2024-03-26T04:44:00.696 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T04:44:00.696 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} --owner scheduled_teuthology@teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053 --name teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053/orig.config.yaml 2024-03-26T04:44:00.731 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T04:44:00.818 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053 branch: squid description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7622053' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 1822 sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a sleep_before_teardown: 0 subset: 53/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de targets: smithi080.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJgeH1Q56uVFk/VLjC3/lV252QgRNpcwV2BCZGbDiqXsPobNpYIrINifR5iz5gSVNoaxtwdTgRDg7KoNE3clG2o= smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCbBxpL2wQKtIofp6kPNucKbb+MVQDHk2s0F3nlHmaXReniGPSLyYC5gVqXo8+TVbSHRsXz8FQoddUqGvmYwT9Y= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_21:08:11 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T04:44:00.818 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa; will attempt to use it 2024-03-26T04:44:00.819 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa/tasks 2024-03-26T04:44:00.819 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T04:44:00.820 INFO:teuthology.task.internal:Checking packages... 2024-03-26T04:44:00.845 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5e70124b46c2a671efcaae4690cd9ea72670710a' 2024-03-26T04:44:00.845 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T04:44:00.845 INFO:teuthology.packaging:ref: None 2024-03-26T04:44:00.845 INFO:teuthology.packaging:tag: None 2024-03-26T04:44:00.845 INFO:teuthology.packaging:branch: squid 2024-03-26T04:44:00.845 INFO:teuthology.packaging:sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:44:00.845 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-26T04:44:01.008 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1863.g9f04c91b 2024-03-26T04:44:01.010 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T04:44:01.017 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T04:44:01.017 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T04:44:01.027 INFO:teuthology.task.internal:Saving configuration 2024-03-26T04:44:01.039 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T04:44:01.048 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:44:01.072 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:38:21.003983', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJgeH1Q56uVFk/VLjC3/lV252QgRNpcwV2BCZGbDiqXsPobNpYIrINifR5iz5gSVNoaxtwdTgRDg7KoNE3clG2o='} 2024-03-26T04:44:01.093 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:38:21.002591', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCbBxpL2wQKtIofp6kPNucKbb+MVQDHk2s0F3nlHmaXReniGPSLyYC5gVqXo8+TVbSHRsXz8FQoddUqGvmYwT9Y='} 2024-03-26T04:44:01.093 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T04:44:01.099 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T04:44:01.100 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-26T04:44:01.100 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T04:44:01.174 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe84f088f70>, signals=[15]) 2024-03-26T04:44:01.175 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T04:44:01.183 INFO:teuthology.task.internal:Opening connections... 2024-03-26T04:44:01.183 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi080.front.sepia.ceph.com 2024-03-26T04:44:01.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:44:01.277 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2024-03-26T04:44:01.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:44:01.369 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T04:44:01.377 DEBUG:teuthology.orchestra.run.smithi080:> uname -m 2024-03-26T04:44:01.394 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64 2024-03-26T04:44:01.394 DEBUG:teuthology.orchestra.run.smithi080:> cat /etc/os-release 2024-03-26T04:44:01.450 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="CentOS Stream" 2024-03-26T04:44:01.450 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="9" 2024-03-26T04:44:01.450 INFO:teuthology.orchestra.run.smithi080.stdout:ID="centos" 2024-03-26T04:44:01.450 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE="rhel fedora" 2024-03-26T04:44:01.450 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="9" 2024-03-26T04:44:01.450 INFO:teuthology.orchestra.run.smithi080.stdout:PLATFORM_ID="platform:el9" 2024-03-26T04:44:01.450 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T04:44:01.451 INFO:teuthology.orchestra.run.smithi080.stdout:ANSI_COLOR="0;31" 2024-03-26T04:44:01.451 INFO:teuthology.orchestra.run.smithi080.stdout:LOGO="fedora-logo-icon" 2024-03-26T04:44:01.451 INFO:teuthology.orchestra.run.smithi080.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T04:44:01.451 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://centos.org/" 2024-03-26T04:44:01.451 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T04:44:01.451 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T04:44:01.451 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T04:44:01.452 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2024-03-26T04:44:01.478 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2024-03-26T04:44:01.495 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2024-03-26T04:44:01.496 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2024-03-26T04:44:01.552 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2024-03-26T04:44:01.552 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="9" 2024-03-26T04:44:01.552 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2024-03-26T04:44:01.552 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2024-03-26T04:44:01.552 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="9" 2024-03-26T04:44:01.552 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el9" 2024-03-26T04:44:01.553 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T04:44:01.553 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2024-03-26T04:44:01.553 INFO:teuthology.orchestra.run.smithi132.stdout:LOGO="fedora-logo-icon" 2024-03-26T04:44:01.553 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T04:44:01.553 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2024-03-26T04:44:01.553 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T04:44:01.553 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T04:44:01.553 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T04:44:01.553 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2024-03-26T04:44:01.578 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T04:44:01.587 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T04:44:01.633 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T04:44:01.633 DEBUG:teuthology.orchestra.run.smithi080:> test '!' -e /home/ubuntu/cephtest 2024-03-26T04:44:01.635 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2024-03-26T04:44:01.650 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T04:44:01.659 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T04:44:01.659 DEBUG:teuthology.orchestra.run.smithi080:> test -z $(ls -A /var/lib/ceph) 2024-03-26T04:44:01.692 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2024-03-26T04:44:01.772 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T04:44:01.825 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T04:44:01.843 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T04:44:01.843 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T04:44:01.844 DEBUG:teuthology.orchestra.run.smithi080:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T04:44:01.844 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T04:44:01.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:44:01.859 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2024-03-26T04:44:01.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:44:01.861 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-03-26T04:44:01.916 INFO:teuthology.orchestra.run.smithi080.stdout:5.14.0-361.el9.x86_64 2024-03-26T04:44:01.916 INFO:teuthology.task.kernel:Running kernel on smithi080: 5.14.0-361.el9.x86_64 2024-03-26T04:44:01.916 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum install -y kernel 2024-03-26T04:44:01.918 INFO:teuthology.orchestra.run.smithi132.stdout:5.14.0-361.el9.x86_64 2024-03-26T04:44:01.918 INFO:teuthology.task.kernel:Running kernel on smithi132: 5.14.0-361.el9.x86_64 2024-03-26T04:44:01.918 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum install -y kernel 2024-03-26T04:44:04.076 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T04:44:04.112 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T04:44:05.526 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-26T04:44:05.554 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-26T04:44:06.329 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-26T04:44:07.558 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-03-26T04:44:23.230 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-26T04:44:25.123 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:16 2024-03-26T04:44:27.437 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.1 MB 00:00 2024-03-26T04:44:29.079 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-26T04:44:30.097 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-26T04:44:30.600 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.1 MB 00:02 2024-03-26T04:44:32.407 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-26T04:44:35.073 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-26T04:44:35.075 INFO:teuthology.orchestra.run.smithi080.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 21 MB 00:02 2024-03-26T04:44:37.246 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T04:44:37.246 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T04:44:37.348 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:44:37.356 INFO:teuthology.orchestra.run.smithi132.stdout:Install 4 Packages 2024-03-26T04:44:37.357 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:44:37.357 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 76 M 2024-03-26T04:44:37.357 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 123 M 2024-03-26T04:44:37.357 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:44:37.532 INFO:teuthology.orchestra.run.smithi132.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 1.1 MB/s | 66 kB 00:00 2024-03-26T04:44:38.916 INFO:teuthology.orchestra.run.smithi132.stdout:(2/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 23 MB/s | 33 MB 00:01 2024-03-26T04:44:39.066 INFO:teuthology.orchestra.run.smithi132.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 9.4 MB/s | 15 MB 00:01 2024-03-26T04:44:39.333 INFO:teuthology.orchestra.run.smithi132.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 15 MB/s | 27 MB 00:01 2024-03-26T04:44:39.333 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:44:39.333 INFO:teuthology.orchestra.run.smithi132.stdout:Total 38 MB/s | 76 MB 00:01 2024-03-26T04:44:39.975 INFO:teuthology.orchestra.run.smithi080.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-26T04:44:40.003 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:44:40.354 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:44:40.354 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:44:41.360 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:44:41.360 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:44:42.081 INFO:teuthology.orchestra.run.smithi080.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T04:44:42.081 INFO:teuthology.orchestra.run.smithi080.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T04:44:42.178 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:44:42.186 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:44:42.187 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-03-26T04:44:42.187 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:44:42.187 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:44:42.187 INFO:teuthology.orchestra.run.smithi080.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T04:44:42.187 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-03-26T04:44:42.187 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T04:44:42.187 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T04:44:42.187 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T04:44:42.187 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:44:42.187 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:44:42.187 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:44:42.188 INFO:teuthology.orchestra.run.smithi080.stdout:Install 4 Packages 2024-03-26T04:44:42.188 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:44:42.188 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 76 M 2024-03-26T04:44:42.188 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 123 M 2024-03-26T04:44:42.188 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:44:42.418 INFO:teuthology.orchestra.run.smithi080.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 583 kB/s | 66 kB 00:00 2024-03-26T04:44:42.776 INFO:teuthology.orchestra.run.smithi080.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 32 MB/s | 15 MB 00:00 2024-03-26T04:44:43.202 INFO:teuthology.orchestra.run.smithi080.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 37 MB/s | 33 MB 00:00 2024-03-26T04:44:43.510 INFO:teuthology.orchestra.run.smithi080.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 25 MB/s | 27 MB 00:01 2024-03-26T04:44:43.510 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:44:43.510 INFO:teuthology.orchestra.run.smithi080.stdout:Total 57 MB/s | 76 MB 00:01 2024-03-26T04:44:43.798 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:44:44.180 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:44:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:44:44.542 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:44:44.549 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:44:44.549 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:44:45.570 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:44:45.570 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:44:47.280 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:44:47.414 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:44:48.019 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:44:48.633 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:44:48.713 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:44:51.471 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:44:51.616 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:44:52.152 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:44:52.272 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:44:56.346 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:44:56.382 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:44:57.021 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:45:01.104 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:45:26.373 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:45:26.373 INFO:teuthology.orchestra.run.smithi132.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T04:45:26.373 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:45:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:45:26.805 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:45:26.805 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:45:26.805 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:45:26.805 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:45:27.215 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:45:27.216 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:45:27.216 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:45:27.216 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T04:45:27.216 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T04:45:27.216 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T04:45:27.216 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T04:45:27.216 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:45:27.216 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:45:27.519 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:45:27.621 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:45:27.621 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:45:27.621 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T04:45:27.622 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T04:45:27.622 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T04:45:27.622 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T04:45:27.622 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi132.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T04:45:27.622 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:45:27.701 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:45:27.701 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:45:27.702 DEBUG:teuthology.orchestra.run.smithi132:> sudo rpm -qi grub2-tools 2024-03-26T04:45:27.735 INFO:teuthology.orchestra.run.smithi132.stdout:Name : grub2-tools 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Epoch : 1 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Version : 2.06 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Release : 68.el9 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Architecture: x86_64 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Group : Unspecified 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Size : 8264254 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:License : GPLv3+ 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Packager : builder@centos.org 2024-03-26T04:45:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:Vendor : CentOS 2024-03-26T04:45:27.737 INFO:teuthology.orchestra.run.smithi132.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T04:45:27.737 INFO:teuthology.orchestra.run.smithi132.stdout:Summary : Support tools for GRUB. 2024-03-26T04:45:27.737 INFO:teuthology.orchestra.run.smithi132.stdout:Description : 2024-03-26T04:45:27.737 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:45:27.737 INFO:teuthology.orchestra.run.smithi132.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T04:45:27.737 INFO:teuthology.orchestra.run.smithi132.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T04:45:27.737 INFO:teuthology.orchestra.run.smithi132.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T04:45:27.737 INFO:teuthology.orchestra.run.smithi132.stdout:hardware devices. 2024-03-26T04:45:27.737 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:45:27.737 INFO:teuthology.orchestra.run.smithi132.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T04:45:27.738 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T04:45:27.738 INFO:teuthology.task.kernel:Updating grub on smithi132 to boot 5.14.0-432.el9.x86_64 2024-03-26T04:45:27.738 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T04:45:28.292 INFO:teuthology.orchestra.run.smithi132.stderr:Generating grub configuration file ... 2024-03-26T04:45:29.733 INFO:teuthology.orchestra.run.smithi132.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T04:45:29.775 INFO:teuthology.orchestra.run.smithi132.stderr:done 2024-03-26T04:45:29.777 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2024-03-26T04:45:29.792 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.mYHr3ECF0m 2024-03-26T04:45:29.793 DEBUG:teuthology.orchestra.run.smithi132:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mYHr3ECF0m 2024-03-26T04:45:29.862 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.mYHr3ECF0m 2024-03-26T04:45:30.001 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.mYHr3ECF0m is 6KB 2024-03-26T04:45:30.050 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.mYHr3ECF0m 2024-03-26T04:45:30.064 DEBUG:teuthology.orchestra.run.smithi132:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T04:45:30.133 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T04:45:30.134 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T04:45:30.134 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T04:45:30.134 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T04:45:30.135 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T04:45:30.290 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:45:30.290 INFO:teuthology.orchestra.run.smithi080.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T04:45:30.291 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:45:30.297 DEBUG:teuthology.orchestra.run.smithi132:> sudo shutdown -r now 2024-03-26T04:45:30.335 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:45:30.661 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:45:30.661 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:45:30.661 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:45:30.661 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:45:31.056 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:45:31.057 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:45:31.057 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:45:31.057 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T04:45:31.057 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T04:45:31.057 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T04:45:31.057 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T04:45:31.057 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:45:31.057 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:45:31.383 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:45:31.490 INFO:teuthology.orchestra.run.smithi080.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:45:31.490 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:45:31.490 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T04:45:31.490 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T04:45:31.490 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T04:45:31.490 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T04:45:31.491 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi080.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T04:45:31.491 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:45:31.570 INFO:teuthology.orchestra.run.smithi080.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:45:31.571 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:45:31.571 DEBUG:teuthology.orchestra.run.smithi080:> sudo rpm -qi grub2-tools 2024-03-26T04:45:31.614 INFO:teuthology.orchestra.run.smithi080.stdout:Name : grub2-tools 2024-03-26T04:45:31.614 INFO:teuthology.orchestra.run.smithi080.stdout:Epoch : 1 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:Version : 2.06 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:Release : 68.el9 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:Architecture: x86_64 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:Group : Unspecified 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:Size : 8264254 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:License : GPLv3+ 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:Packager : builder@centos.org 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:Vendor : CentOS 2024-03-26T04:45:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T04:45:31.616 INFO:teuthology.orchestra.run.smithi080.stdout:Summary : Support tools for GRUB. 2024-03-26T04:45:31.616 INFO:teuthology.orchestra.run.smithi080.stdout:Description : 2024-03-26T04:45:31.616 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:45:31.616 INFO:teuthology.orchestra.run.smithi080.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T04:45:31.616 INFO:teuthology.orchestra.run.smithi080.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T04:45:31.616 INFO:teuthology.orchestra.run.smithi080.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T04:45:31.616 INFO:teuthology.orchestra.run.smithi080.stdout:hardware devices. 2024-03-26T04:45:31.616 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:45:31.616 INFO:teuthology.orchestra.run.smithi080.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T04:45:31.617 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T04:45:31.617 INFO:teuthology.task.kernel:Updating grub on smithi080 to boot 5.14.0-432.el9.x86_64 2024-03-26T04:45:31.617 DEBUG:teuthology.orchestra.run.smithi080:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T04:45:32.255 INFO:teuthology.orchestra.run.smithi080.stderr:Generating grub configuration file ... 2024-03-26T04:45:33.630 INFO:teuthology.orchestra.run.smithi080.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T04:45:33.664 INFO:teuthology.orchestra.run.smithi080.stderr:done 2024-03-26T04:45:33.666 DEBUG:teuthology.orchestra.run.smithi080:> mktemp 2024-03-26T04:45:33.686 INFO:teuthology.orchestra.run.smithi080.stdout:/tmp/tmp.5fEtmTPHl1 2024-03-26T04:45:33.686 DEBUG:teuthology.orchestra.run.smithi080:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5fEtmTPHl1 2024-03-26T04:45:33.756 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0666 /tmp/tmp.5fEtmTPHl1 2024-03-26T04:45:33.892 DEBUG:teuthology.orchestra.remote:smithi080:/tmp/tmp.5fEtmTPHl1 is 6KB 2024-03-26T04:45:33.915 DEBUG:teuthology.orchestra.run.smithi080:> rm -fr /tmp/tmp.5fEtmTPHl1 2024-03-26T04:45:33.930 DEBUG:teuthology.orchestra.run.smithi080:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T04:45:33.996 INFO:teuthology.orchestra.run.smithi080.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T04:45:33.996 INFO:teuthology.orchestra.run.smithi080.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T04:45:33.996 INFO:teuthology.orchestra.run.smithi080.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T04:45:33.996 INFO:teuthology.orchestra.run.smithi080.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T04:45:33.997 DEBUG:teuthology.orchestra.run.smithi080:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T04:45:34.138 DEBUG:teuthology.orchestra.run.smithi080:> sudo shutdown -r now 2024-03-26T04:46:00.326 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T04:46:00.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-26T04:46:00.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:46:04.142 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T04:46:04.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-26T04:46:04.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:46:18.654 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.132 2024-03-26T04:46:22.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.80 2024-03-26T04:46:27.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-26T04:46:27.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:46:30.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.132 2024-03-26T04:46:31.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-26T04:46:31.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:46:34.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.80 2024-03-26T04:46:42.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-26T04:46:42.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:46:46.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-26T04:46:46.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:47:14.632 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-03-26T04:47:15.094 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-26T04:47:15.094 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T04:47:15.095 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-03-26T04:47:15.108 INFO:teuthology.orchestra.run.smithi132.stdout:5.14.0-432.el9.x86_64 2024-03-26T04:47:15.108 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T04:47:15.109 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T04:47:15.109 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T04:47:16.109 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T04:47:16.110 DEBUG:teuthology.orchestra.run.smithi132:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T04:47:16.292 INFO:teuthology.orchestra.run.smithi132.stdout:ttyS1 2024-03-26T04:47:16.360 DEBUG:teuthology.parallel:result is None 2024-03-26T04:47:18.435 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-03-26T04:47:18.863 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-26T04:47:18.864 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T04:47:18.864 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2024-03-26T04:47:18.877 INFO:teuthology.orchestra.run.smithi080.stdout:5.14.0-432.el9.x86_64 2024-03-26T04:47:18.878 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T04:47:18.878 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T04:47:18.878 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T04:47:19.878 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T04:47:19.879 DEBUG:teuthology.orchestra.run.smithi080:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T04:47:19.959 INFO:teuthology.orchestra.run.smithi080.stdout:ttyS1 2024-03-26T04:47:20.009 DEBUG:teuthology.parallel:result is None 2024-03-26T04:47:20.010 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T04:47:20.018 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T04:47:20.018 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T04:47:20.020 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T04:47:20.039 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T04:47:20.050 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T04:47:20.057 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T04:47:20.057 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T04:47:20.079 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T04:47:20.115 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T04:47:20.122 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T04:47:20.122 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T04:47:20.147 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T04:47:20.171 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:47:20.184 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:47:20.184 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:47:20.197 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:47:20.199 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T04:47:20.206 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T04:47:20.206 DEBUG:teuthology.orchestra.run.smithi080:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T04:47:20.233 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T04:47:20.269 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T04:47:20.279 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T04:47:20.280 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T04:47:20.304 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T04:47:20.325 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:47:20.414 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:47:20.453 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:47:20.544 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:47:20.589 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:47:20.589 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T04:47:20.660 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:47:20.703 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:47:20.755 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:47:20.838 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:47:20.884 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T04:47:20.884 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T04:47:20.954 DEBUG:teuthology.orchestra.run.smithi080:> sudo service rsyslog restart 2024-03-26T04:47:20.956 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2024-03-26T04:47:21.016 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:47:21.038 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:47:21.558 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T04:47:21.566 INFO:teuthology.task.internal:Starting timer... 2024-03-26T04:47:21.566 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T04:47:21.577 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T04:47:21.584 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T04:47:21.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo service auditd rotate 2024-03-26T04:47:21.688 INFO:teuthology.orchestra.run.smithi080.stdout:Rotating logs: 2024-03-26T04:47:21.693 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2024-03-26T04:47:21.741 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2024-03-26T04:47:21.743 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T04:47:21.743 DEBUG:teuthology.orchestra.run.smithi080:> /usr/sbin/getenforce 2024-03-26T04:47:21.777 INFO:teuthology.orchestra.run.smithi080.stdout:Permissive 2024-03-26T04:47:21.777 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2024-03-26T04:47:21.820 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2024-03-26T04:47:21.820 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi080.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi132.front.sepia.ceph.com': 'permissive'} 2024-03-26T04:47:21.820 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:47:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:47:21.874 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:47:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:47:21.902 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T04:47:21.902 DEBUG:teuthology.orchestra.run.smithi080:> sudo /usr/sbin/setenforce permissive 2024-03-26T04:47:21.946 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2024-03-26T04:47:21.988 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T04:47:22.001 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T04:47:22.017 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T04:47:22.018 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi080.front.sepia.ceph.com,smithi132.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T04:51:36.650 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi080.front.sepia.ceph.com'), Remote(name='ubuntu@smithi132.front.sepia.ceph.com')] 2024-03-26T04:51:36.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-26T04:51:36.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:51:36.744 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-03-26T04:51:36.829 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-26T04:51:36.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-26T04:51:36.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:51:36.918 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-03-26T04:51:36.996 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-26T04:51:36.997 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T04:51:37.007 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T04:51:37.008 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:51:37.008 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:51:37.011 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:51:37.011 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:51:37.041 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:51:37.058 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:51:37.066 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:51:37.083 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:51:37.086 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: ntpd: command not found 2024-03-26T04:51:37.100 INFO:teuthology.orchestra.run.smithi080.stdout:506 Cannot talk to daemon 2024-03-26T04:51:37.111 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2024-03-26T04:51:37.117 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:51:37.124 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2024-03-26T04:51:37.133 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:51:37.141 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:51:37.158 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:51:37.184 INFO:teuthology.orchestra.run.smithi080.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:51:37.187 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:51:37.188 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2024-03-26T04:51:37.188 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:51:37.188 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:51:37.188 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:51:37.188 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:51:37.209 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:51:37.213 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:51:37.213 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-03-26T04:51:37.213 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:51:37.213 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:51:37.213 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:51:37.213 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:51:37.213 INFO:teuthology.run_tasks:Running task install... 2024-03-26T04:51:37.221 DEBUG:teuthology.task.install:project ceph 2024-03-26T04:51:37.221 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'}} 2024-03-26T04:51:37.221 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'} 2024-03-26T04:51:37.221 INFO:teuthology.task.install:Using flavor: default 2024-03-26T04:51:37.227 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-26T04:51:37.227 INFO:teuthology.task.install:extra packages: [] 2024-03-26T04:51:37.228 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'tag': None, 'wait_for_package': False} 2024-03-26T04:51:37.228 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:51:37.230 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'tag': None, 'wait_for_package': False} 2024-03-26T04:51:37.231 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:51:37.375 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/ 2024-03-26T04:51:37.375 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1775.g5e70124b 2024-03-26T04:51:37.382 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/ 2024-03-26T04:51:37.382 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1775.g5e70124b 2024-03-26T04:51:37.486 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T04:51:37.487 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:51:37.487 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T04:51:37.495 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T04:51:37.495 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T04:51:37.495 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T04:51:37.515 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T04:51:37.515 DEBUG:teuthology.orchestra.run.smithi080:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e70124b46c2a671efcaae4690cd9ea72670710a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T04:51:37.525 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T04:51:37.525 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e70124b46c2a671efcaae4690cd9ea72670710a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T04:51:37.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T04:51:37.593 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T04:51:37.665 DEBUG:teuthology.orchestra.run.smithi080:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T04:51:37.672 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T04:51:37.732 INFO:teuthology.orchestra.run.smithi080.stdout:check_obsoletes = 1 2024-03-26T04:51:37.733 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum clean all 2024-03-26T04:51:37.739 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2024-03-26T04:51:37.740 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2024-03-26T04:51:37.982 INFO:teuthology.orchestra.run.smithi080.stdout:74 files removed 2024-03-26T04:51:37.993 INFO:teuthology.orchestra.run.smithi132.stdout:74 files removed 2024-03-26T04:51:38.012 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-radosgw 2024-03-26T04:51:38.020 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw 2024-03-26T04:51:38.798 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 173 kB/s | 83 kB 00:00 2024-03-26T04:51:38.805 INFO:teuthology.orchestra.run.smithi080.stdout:ceph packages for x86_64 168 kB/s | 83 kB 00:00 2024-03-26T04:51:39.123 INFO:teuthology.orchestra.run.smithi080.stdout:ceph noarch packages 37 kB/s | 10 kB 00:00 2024-03-26T04:51:39.140 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 34 kB/s | 10 kB 00:00 2024-03-26T04:51:39.541 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-03-26T04:51:39.548 INFO:teuthology.orchestra.run.smithi080.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-26T04:51:40.009 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-26T04:51:40.017 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-26T04:51:41.401 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-26T04:51:41.410 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-26T04:51:42.418 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-26T04:51:43.342 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.0 MB 00:01 2024-03-26T04:51:44.227 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-26T04:51:45.343 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-26T04:51:49.969 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.1 MB 00:02 2024-03-26T04:51:50.027 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.1 MB 00:01 2024-03-26T04:51:51.478 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-03-26T04:51:52.222 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 33 MB/s | 21 MB 00:00 2024-03-26T04:51:52.494 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-26T04:51:55.664 INFO:teuthology.orchestra.run.smithi080.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 21 MB 00:03 2024-03-26T04:51:57.188 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-26T04:51:59.625 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:51:59.628 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:51:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-26T04:51:59.628 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:51:59.629 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:51:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 9.8 M 2024-03-26T04:51:59.629 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2024-03-26T04:51:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.4 M 2024-03-26T04:51:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T04:51:59.629 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-26T04:51:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options 2024-03-26T04:51:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T04:51:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.3 M 2024-03-26T04:51:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 20 M 2024-03-26T04:51:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 25 k 2024-03-26T04:51:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T04:51:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T04:51:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T04:51:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 744 k 2024-03-26T04:51:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T04:51:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T04:51:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T04:51:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 498 k 2024-03-26T04:51:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T04:51:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.1 M 2024-03-26T04:51:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T04:51:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T04:51:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T04:51:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse 2024-03-26T04:51:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 45 k 2024-03-26T04:51:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 131 k 2024-03-26T04:51:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 162 k 2024-03-26T04:51:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T04:51:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 321 k 2024-03-26T04:51:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 298 k 2024-03-26T04:51:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 99 k 2024-03-26T04:51:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T04:51:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T04:51:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T04:51:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T04:51:59.632 INFO:teuthology.orchestra.run.smithi132.stdout:Installing weak dependencies: 2024-03-26T04:51:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T04:51:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:51:59.632 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:51:59.632 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:51:59.632 INFO:teuthology.orchestra.run.smithi132.stdout:Install 30 Packages 2024-03-26T04:51:59.632 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 2 Packages 2024-03-26T04:51:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:51:59.633 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 58 M 2024-03-26T04:51:59.633 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:52:00.339 INFO:teuthology.orchestra.run.smithi132.stdout:(1/32): ceph-base-19.0.0-1775.g5e70124b.el9.x86 10 MB/s | 5.3 MB 00:00 2024-03-26T04:52:00.389 INFO:teuthology.orchestra.run.smithi132.stdout:(2/32): ceph-selinux-19.0.0-1775.g5e70124b.el9. 504 kB/s | 25 kB 00:00 2024-03-26T04:52:00.465 INFO:teuthology.orchestra.run.smithi132.stdout:(3/32): libcephfs2-19.0.0-1775.g5e70124b.el9.x8 9.7 MB/s | 744 kB 00:00 2024-03-26T04:52:00.523 INFO:teuthology.orchestra.run.smithi132.stdout:(4/32): libradosstriper1-19.0.0-1775.g5e70124b. 8.4 MB/s | 498 kB 00:00 2024-03-26T04:52:00.595 INFO:teuthology.orchestra.run.smithi080.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-26T04:52:00.823 INFO:teuthology.orchestra.run.smithi132.stdout:(5/32): librgw2-19.0.0-1775.g5e70124b.el9.x86_6 17 MB/s | 5.1 MB 00:00 2024-03-26T04:52:00.857 INFO:teuthology.orchestra.run.smithi132.stdout:(6/32): python3-ceph-argparse-19.0.0-1775.g5e70 1.3 MB/s | 45 kB 00:00 2024-03-26T04:52:00.907 INFO:teuthology.orchestra.run.smithi132.stdout:(7/32): python3-ceph-common-19.0.0-1775.g5e7012 2.6 MB/s | 131 kB 00:00 2024-03-26T04:52:00.966 INFO:teuthology.orchestra.run.smithi132.stdout:(8/32): python3-cephfs-19.0.0-1775.g5e70124b.el 2.7 MB/s | 162 kB 00:00 2024-03-26T04:52:01.033 INFO:teuthology.orchestra.run.smithi132.stdout:(9/32): python3-rados-19.0.0-1775.g5e70124b.el9 4.7 MB/s | 321 kB 00:00 2024-03-26T04:52:01.083 INFO:teuthology.orchestra.run.smithi132.stdout:(10/32): python3-rbd-19.0.0-1775.g5e70124b.el9. 5.8 MB/s | 298 kB 00:00 2024-03-26T04:52:01.133 INFO:teuthology.orchestra.run.smithi132.stdout:(11/32): python3-rgw-19.0.0-1775.g5e70124b.el9. 2.0 MB/s | 99 kB 00:00 2024-03-26T04:52:01.250 INFO:teuthology.orchestra.run.smithi132.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 906 kB/s | 106 kB 00:00 2024-03-26T04:52:01.284 INFO:teuthology.orchestra.run.smithi132.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-26T04:52:01.317 INFO:teuthology.orchestra.run.smithi132.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-26T04:52:01.376 INFO:teuthology.orchestra.run.smithi132.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-26T04:52:01.418 INFO:teuthology.orchestra.run.smithi132.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-26T04:52:01.452 INFO:teuthology.orchestra.run.smithi132.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-26T04:52:01.585 INFO:teuthology.orchestra.run.smithi132.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-03-26T04:52:01.627 INFO:teuthology.orchestra.run.smithi132.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 861 kB/s | 36 kB 00:00 2024-03-26T04:52:01.736 INFO:teuthology.orchestra.run.smithi132.stdout:(20/32): ceph-radosgw-19.0.0-1775.g5e70124b.el9 5.1 MB/s | 9.8 MB 00:01 2024-03-26T04:52:01.761 INFO:teuthology.orchestra.run.smithi132.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 354 kB/s | 47 kB 00:00 2024-03-26T04:52:02.195 INFO:teuthology.orchestra.run.smithi132.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 673 kB/s | 309 kB 00:00 2024-03-26T04:52:02.395 INFO:teuthology.orchestra.run.smithi132.stdout:(23/32): ceph-common-19.0.0-1775.g5e70124b.el9. 7.9 MB/s | 20 MB 00:02 2024-03-26T04:52:02.420 INFO:teuthology.orchestra.run.smithi132.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 115 kB/s | 26 kB 00:00 2024-03-26T04:52:02.504 INFO:teuthology.orchestra.run.smithi132.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 808 kB/s | 67 kB 00:00 2024-03-26T04:52:02.672 INFO:teuthology.orchestra.run.smithi132.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 176 kB/s | 49 kB 00:00 2024-03-26T04:52:02.705 INFO:teuthology.orchestra.run.smithi132.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-03-26T04:52:02.872 INFO:teuthology.orchestra.run.smithi132.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-26T04:52:02.939 INFO:teuthology.orchestra.run.smithi132.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 4.4 MB 00:01 2024-03-26T04:52:02.964 INFO:teuthology.orchestra.run.smithi132.stdout:(30/32): re2-20211101-3.el9.x86_64.rpm 656 kB/s | 192 kB 00:00 2024-03-26T04:52:03.049 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:52:03.061 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:52:03.061 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-03-26T04:52:03.061 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:52:03.061 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:52:03.061 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-radosgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 9.8 M 2024-03-26T04:52:03.061 INFO:teuthology.orchestra.run.smithi080.stdout:Upgrading: 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout: librados2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.4 M 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout: librbd1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout: boost-program-options 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-base x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.3 M 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 20 M 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-selinux x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 25 k 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 744 k 2024-03-26T04:52:03.062 INFO:teuthology.orchestra.run.smithi080.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: libradosstriper1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 498 k 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: librgw2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.1 M 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-argparse 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 45 k 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 131 k 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cephfs x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 162 k 2024-03-26T04:52:03.063 INFO:teuthology.orchestra.run.smithi080.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T04:52:03.064 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rados x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 321 k 2024-03-26T04:52:03.064 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 298 k 2024-03-26T04:52:03.064 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 99 k 2024-03-26T04:52:03.064 INFO:teuthology.orchestra.run.smithi080.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T04:52:03.064 INFO:teuthology.orchestra.run.smithi080.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T04:52:03.064 INFO:teuthology.orchestra.run.smithi080.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T04:52:03.064 INFO:teuthology.orchestra.run.smithi080.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T04:52:03.064 INFO:teuthology.orchestra.run.smithi080.stdout:Installing weak dependencies: 2024-03-26T04:52:03.064 INFO:teuthology.orchestra.run.smithi080.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T04:52:03.064 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:03.065 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:52:03.065 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:52:03.065 INFO:teuthology.orchestra.run.smithi080.stdout:Install 30 Packages 2024-03-26T04:52:03.065 INFO:teuthology.orchestra.run.smithi080.stdout:Upgrade 2 Packages 2024-03-26T04:52:03.065 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:03.065 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 58 M 2024-03-26T04:52:03.065 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:52:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:(31/32): librados2-19.0.0-1775.g5e70124b.el9.x8 9.0 MB/s | 3.4 MB 00:00 2024-03-26T04:52:03.331 INFO:teuthology.orchestra.run.smithi132.stdout:(32/32): librbd1-19.0.0-1775.g5e70124b.el9.x86_ 8.0 MB/s | 3.1 MB 00:00 2024-03-26T04:52:03.334 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:52:03.334 INFO:teuthology.orchestra.run.smithi132.stdout:Total 16 MB/s | 58 MB 00:03 2024-03-26T04:52:03.370 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:52:03.406 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:52:03.407 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:52:03.862 INFO:teuthology.orchestra.run.smithi080.stdout:(1/32): ceph-base-19.0.0-1775.g5e70124b.el9.x86 8.9 MB/s | 5.3 MB 00:00 2024-03-26T04:52:03.894 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:52:03.894 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:52:03.921 INFO:teuthology.orchestra.run.smithi080.stdout:(2/32): ceph-selinux-19.0.0-1775.g5e70124b.el9. 431 kB/s | 25 kB 00:00 2024-03-26T04:52:04.021 INFO:teuthology.orchestra.run.smithi080.stdout:(3/32): libcephfs2-19.0.0-1775.g5e70124b.el9.x8 7.3 MB/s | 744 kB 00:00 2024-03-26T04:52:04.097 INFO:teuthology.orchestra.run.smithi080.stdout:(4/32): libradosstriper1-19.0.0-1775.g5e70124b. 6.5 MB/s | 498 kB 00:00 2024-03-26T04:52:04.514 INFO:teuthology.orchestra.run.smithi080.stdout:(5/32): librgw2-19.0.0-1775.g5e70124b.el9.x86_6 12 MB/s | 5.1 MB 00:00 2024-03-26T04:52:04.589 INFO:teuthology.orchestra.run.smithi080.stdout:(6/32): python3-ceph-argparse-19.0.0-1775.g5e70 602 kB/s | 45 kB 00:00 2024-03-26T04:52:04.648 INFO:teuthology.orchestra.run.smithi080.stdout:(7/32): python3-ceph-common-19.0.0-1775.g5e7012 2.2 MB/s | 131 kB 00:00 2024-03-26T04:52:04.698 INFO:teuthology.orchestra.run.smithi080.stdout:(8/32): python3-cephfs-19.0.0-1775.g5e70124b.el 3.2 MB/s | 162 kB 00:00 2024-03-26T04:52:04.765 INFO:teuthology.orchestra.run.smithi080.stdout:(9/32): python3-rados-19.0.0-1775.g5e70124b.el9 4.7 MB/s | 321 kB 00:00 2024-03-26T04:52:04.965 INFO:teuthology.orchestra.run.smithi080.stdout:(10/32): ceph-common-19.0.0-1775.g5e70124b.el9. 12 MB/s | 20 MB 00:01 2024-03-26T04:52:04.990 INFO:teuthology.orchestra.run.smithi080.stdout:(11/32): python3-rbd-19.0.0-1775.g5e70124b.el9. 1.3 MB/s | 298 kB 00:00 2024-03-26T04:52:05.016 INFO:teuthology.orchestra.run.smithi080.stdout:(12/32): python3-rgw-19.0.0-1775.g5e70124b.el9. 1.9 MB/s | 99 kB 00:00 2024-03-26T04:52:05.150 INFO:teuthology.orchestra.run.smithi080.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 666 kB/s | 106 kB 00:00 2024-03-26T04:52:05.175 INFO:teuthology.orchestra.run.smithi080.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-26T04:52:05.192 INFO:teuthology.orchestra.run.smithi080.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-26T04:52:05.251 INFO:teuthology.orchestra.run.smithi080.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-26T04:52:05.276 INFO:teuthology.orchestra.run.smithi080.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-26T04:52:05.293 INFO:teuthology.orchestra.run.smithi080.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 992 kB/s | 41 kB 00:00 2024-03-26T04:52:05.357 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:52:05.357 INFO:teuthology.orchestra.run.smithi080.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T04:52:05.358 INFO:teuthology.orchestra.run.smithi080.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T04:52:05.438 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T04:52:05.469 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T04:52:05.481 INFO:teuthology.orchestra.run.smithi080.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T04:52:05.494 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T04:52:05.504 INFO:teuthology.orchestra.run.smithi080.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T04:52:05.517 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T04:52:05.534 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T04:52:05.548 INFO:teuthology.orchestra.run.smithi080.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:52:05.569 INFO:teuthology.orchestra.run.smithi080.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:52:05.675 INFO:teuthology.orchestra.run.smithi080.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:52:05.699 INFO:teuthology.orchestra.run.smithi080.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:52:05.726 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T04:52:05.760 INFO:teuthology.orchestra.run.smithi080.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 77 kB/s | 36 kB 00:00 2024-03-26T04:52:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T04:52:05.792 INFO:teuthology.orchestra.run.smithi080.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T04:52:05.818 INFO:teuthology.orchestra.run.smithi080.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 407 kB/s | 221 kB 00:00 2024-03-26T04:52:05.825 INFO:teuthology.orchestra.run.smithi080.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T04:52:05.858 INFO:teuthology.orchestra.run.smithi080.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:52:05.891 INFO:teuthology.orchestra.run.smithi080.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:52:05.952 INFO:teuthology.orchestra.run.smithi080.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 247 kB/s | 47 kB 00:00 2024-03-26T04:52:06.085 INFO:teuthology.orchestra.run.smithi080.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-26T04:52:06.125 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T04:52:06.153 INFO:teuthology.orchestra.run.smithi080.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-03-26T04:52:06.213 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:52:06.228 INFO:teuthology.orchestra.run.smithi080.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-03-26T04:52:06.295 INFO:teuthology.orchestra.run.smithi080.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-26T04:52:06.330 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:52:06.419 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 8/34 2024-03-26T04:52:06.445 INFO:teuthology.orchestra.run.smithi080.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 838 kB 00:00 2024-03-26T04:52:06.447 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T04:52:06.510 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T04:52:06.512 INFO:teuthology.orchestra.run.smithi080.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-26T04:52:06.539 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T04:52:06.589 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 11/34 2024-03-26T04:52:06.660 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T04:52:06.683 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T04:52:06.704 INFO:teuthology.orchestra.run.smithi080.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2024-03-26T04:52:06.730 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T04:52:06.737 INFO:teuthology.orchestra.run.smithi080.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-03-26T04:52:06.765 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T04:52:06.885 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T04:52:06.924 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T04:52:06.965 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T04:52:06.996 INFO:teuthology.orchestra.run.smithi080.stdout:(30/32): ceph-radosgw-19.0.0-1775.g5e70124b.el9 2.6 MB/s | 9.8 MB 00:03 2024-03-26T04:52:07.005 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T04:52:07.034 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T04:52:07.068 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T04:52:07.155 INFO:teuthology.orchestra.run.smithi080.stdout:(31/32): librados2-19.0.0-1775.g5e70124b.el9.x8 7.5 MB/s | 3.4 MB 00:00 2024-03-26T04:52:07.380 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T04:52:07.414 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T04:52:07.488 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T04:52:07.519 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 23/34 2024-03-26T04:52:07.560 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T04:52:08.018 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T04:52:08.050 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T04:52:08.104 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T04:52:08.172 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 27/34 2024-03-26T04:52:08.202 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 28/34 2024-03-26T04:52:08.346 INFO:teuthology.orchestra.run.smithi080.stdout:(32/32): librbd1-19.0.0-1775.g5e70124b.el9.x86_ 2.0 MB/s | 3.1 MB 00:01 2024-03-26T04:52:08.351 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:52:08.351 INFO:teuthology.orchestra.run.smithi080.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-26T04:52:08.382 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:52:08.418 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:52:08.418 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:52:08.909 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:52:08.909 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:52:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:52:10.442 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:52:10.465 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T04:52:10.482 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:52:10.502 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T04:52:10.527 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T04:52:10.550 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T04:52:10.566 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T04:52:10.744 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T04:52:10.806 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T04:52:11.028 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:52:11.060 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T04:52:11.123 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T04:52:11.153 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T04:52:11.153 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T04:52:11.153 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T04:52:11.154 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:11.179 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrading : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:52:11.186 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:52:11.263 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:52:11.352 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 8/34 2024-03-26T04:52:11.372 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T04:52:11.418 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T04:52:11.456 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T04:52:11.510 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 11/34 2024-03-26T04:52:11.602 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T04:52:11.633 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T04:52:11.697 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T04:52:11.732 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T04:52:11.854 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T04:52:11.891 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T04:52:11.932 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T04:52:11.965 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T04:52:11.993 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T04:52:12.026 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T04:52:12.337 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T04:52:12.365 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrading : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T04:52:12.447 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T04:52:12.478 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 23/34 2024-03-26T04:52:12.524 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T04:52:12.978 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T04:52:13.001 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T04:52:13.055 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T04:52:13.123 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 27/34 2024-03-26T04:52:13.154 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 28/34 2024-03-26T04:52:15.358 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:52:15.409 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:52:15.957 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:52:16.155 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T04:52:16.248 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T04:52:16.248 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T04:52:16.248 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T04:52:16.248 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:16.289 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:52:20.763 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:52:20.763 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2024-03-26T04:52:20.763 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2024-03-26T04:52:20.763 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2024-03-26T04:52:20.763 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2024-03-26T04:52:20.763 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2024-03-26T04:52:20.763 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /root 2024-03-26T04:52:20.763 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2024-03-26T04:52:20.763 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:21.675 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T04:52:21.700 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T04:52:21.700 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:52:21.700 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T04:52:21.700 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T04:52:21.700 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T04:52:21.700 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:21.702 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T04:52:21.786 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T04:52:21.786 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:52:22.876 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:52:22.876 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 1/34 2024-03-26T04:52:22.876 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2/34 2024-03-26T04:52:22.876 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 3/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 4/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 5/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 6/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 8/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 9/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 10/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 11/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 13/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T04:52:22.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T04:52:22.878 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T04:52:22.879 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:52:22.879 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T04:52:22.879 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 33/34 2024-03-26T04:52:23.945 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:52:23.945 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:23.945 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2024-03-26T04:52:23.945 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.946 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.946 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:52:23.946 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T04:52:23.946 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.946 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.946 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.946 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.946 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T04:52:23.946 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T04:52:23.946 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T04:52:23.946 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.946 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T04:52:23.947 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T04:52:23.947 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T04:52:23.947 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.947 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T04:52:23.947 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.947 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T04:52:23.947 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T04:52:23.947 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T04:52:23.947 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.947 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.947 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.947 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T04:52:23.948 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.948 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.948 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:23.948 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T04:52:23.948 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T04:52:23.948 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T04:52:23.948 INFO:teuthology.orchestra.run.smithi132.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T04:52:23.948 INFO:teuthology.orchestra.run.smithi132.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T04:52:23.948 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:23.948 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:52:24.292 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-test 2024-03-26T04:52:24.853 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:27 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:52:25.002 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:52:25.004 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:52:25.004 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-26T04:52:25.004 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 47 M 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 163 k 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:25.005 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 48 M 2024-03-26T04:52:25.006 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 205 M 2024-03-26T04:52:25.006 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:52:25.277 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-26T04:52:25.319 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-26T04:52:25.353 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-26T04:52:25.403 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): libcephsqlite-19.0.0-1775.g5e70124b.el9. 585 kB/s | 163 kB 00:00 2024-03-26T04:52:25.553 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 935 kB/s | 187 kB 00:00 2024-03-26T04:52:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:52:25.852 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /sys 2024-03-26T04:52:25.853 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /proc 2024-03-26T04:52:25.853 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /mnt 2024-03-26T04:52:25.853 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /var/tmp 2024-03-26T04:52:25.853 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /home 2024-03-26T04:52:25.853 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /root 2024-03-26T04:52:25.853 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /tmp 2024-03-26T04:52:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:26.761 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T04:52:26.785 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T04:52:26.785 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:52:26.786 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T04:52:26.786 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T04:52:26.786 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T04:52:26.786 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:26.787 INFO:teuthology.orchestra.run.smithi080.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T04:52:26.864 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T04:52:26.864 INFO:teuthology.orchestra.run.smithi080.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:52:27.445 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): ceph-test-19.0.0-1775.g5e70124b.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-26T04:52:27.448 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:52:27.448 INFO:teuthology.orchestra.run.smithi132.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-26T04:52:27.463 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:52:27.484 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:52:27.484 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:52:27.868 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:52:27.868 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:52:27.988 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:52:27.988 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 1/34 2024-03-26T04:52:27.988 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2/34 2024-03-26T04:52:27.988 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 3/34 2024-03-26T04:52:27.988 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 4/34 2024-03-26T04:52:27.988 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 5/34 2024-03-26T04:52:27.988 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 6/34 2024-03-26T04:52:27.988 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 8/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 9/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 10/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 11/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 13/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T04:52:27.989 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T04:52:27.990 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T04:52:27.990 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T04:52:27.990 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T04:52:27.990 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T04:52:27.990 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T04:52:27.990 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T04:52:27.990 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T04:52:27.990 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T04:52:27.990 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T04:52:27.991 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:52:27.991 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T04:52:27.991 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 33/34 2024-03-26T04:52:28.496 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:52:28.570 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T04:52:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T04:52:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T04:52:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T04:52:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T04:52:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:52:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:29.005 INFO:teuthology.orchestra.run.smithi080.stdout:Upgraded: 2024-03-26T04:52:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.005 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:52:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T04:52:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T04:52:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:29.007 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:52:29.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-test 2024-03-26T04:52:29.906 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:00:29 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:52:30.055 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:52:30.055 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:52:30.055 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-03-26T04:52:30.055 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:52:30.055 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:52:30.056 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-test x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 47 M 2024-03-26T04:52:30.056 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-03-26T04:52:30.056 INFO:teuthology.orchestra.run.smithi080.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T04:52:30.056 INFO:teuthology.orchestra.run.smithi080.stdout: libcephsqlite x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 163 k 2024-03-26T04:52:30.056 INFO:teuthology.orchestra.run.smithi080.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T04:52:30.056 INFO:teuthology.orchestra.run.smithi080.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T04:52:30.056 INFO:teuthology.orchestra.run.smithi080.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T04:52:30.056 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:30.056 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:52:30.056 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:52:30.056 INFO:teuthology.orchestra.run.smithi080.stdout:Install 6 Packages 2024-03-26T04:52:30.056 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:30.057 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 48 M 2024-03-26T04:52:30.057 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 205 M 2024-03-26T04:52:30.057 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:52:30.251 INFO:teuthology.orchestra.run.smithi080.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-26T04:52:30.293 INFO:teuthology.orchestra.run.smithi080.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-26T04:52:30.327 INFO:teuthology.orchestra.run.smithi080.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-26T04:52:30.485 INFO:teuthology.orchestra.run.smithi080.stdout:(4/6): libcephsqlite-19.0.0-1775.g5e70124b.el9. 417 kB/s | 163 kB 00:00 2024-03-26T04:52:30.835 INFO:teuthology.orchestra.run.smithi080.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 368 kB/s | 187 kB 00:00 2024-03-26T04:52:32.435 INFO:teuthology.orchestra.run.smithi080.stdout:(6/6): ceph-test-19.0.0-1775.g5e70124b.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-26T04:52:32.438 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:52:32.439 INFO:teuthology.orchestra.run.smithi080.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-26T04:52:32.453 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:52:32.473 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:52:32.474 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:52:32.853 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:52:32.854 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:52:33.182 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T04:52:33.256 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T04:52:33.483 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:52:33.570 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T04:52:33.611 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T04:52:33.643 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T04:52:33.686 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T04:52:33.722 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T04:52:35.673 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T04:52:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 1/6 2024-03-26T04:52:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2/6 2024-03-26T04:52:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T04:52:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T04:52:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T04:52:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T04:52:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:36.086 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:52:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: jq-1.6-16.el9.x86_64 2024-03-26T04:52:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T04:52:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T04:52:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T04:52:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:36.086 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:52:36.384 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph 2024-03-26T04:52:36.971 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:39 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:52:37.128 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:52:37.130 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================= 2024-03-26T04:52:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-26T04:52:37.130 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================= 2024-03-26T04:52:37.130 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 6.6 k 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 2.3 M 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 1.5 M 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 248 k 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 4.7 M 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 17 M 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T04:52:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T04:52:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================= 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout:Install 36 Packages 2024-03-26T04:52:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:37.134 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 30 M 2024-03-26T04:52:37.134 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 107 M 2024-03-26T04:52:37.134 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:52:37.631 INFO:teuthology.orchestra.run.smithi132.stdout:(1/36): ceph-19.0.0-1775.g5e70124b.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-03-26T04:52:37.831 INFO:teuthology.orchestra.run.smithi132.stdout:(2/36): ceph-mgr-19.0.0-1775.g5e70124b.el9.x86_ 4.0 MB/s | 1.5 MB 00:00 2024-03-26T04:52:38.108 INFO:teuthology.orchestra.run.smithi132.stdout:(3/36): ceph-mds-19.0.0-1775.g5e70124b.el9.x86_ 5.3 MB/s | 2.3 MB 00:00 2024-03-26T04:52:38.108 INFO:teuthology.orchestra.run.smithi132.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1775.g5e70 2.9 MB/s | 248 kB 00:00 2024-03-26T04:52:38.124 INFO:teuthology.orchestra.run.smithi132.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-26T04:52:38.166 INFO:teuthology.orchestra.run.smithi132.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-26T04:52:38.191 INFO:teuthology.orchestra.run.smithi132.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-26T04:52:38.193 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T04:52:38.216 INFO:teuthology.orchestra.run.smithi132.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-26T04:52:38.242 INFO:teuthology.orchestra.run.smithi132.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-26T04:52:38.270 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T04:52:38.284 INFO:teuthology.orchestra.run.smithi132.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-26T04:52:38.351 INFO:teuthology.orchestra.run.smithi132.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-26T04:52:38.418 INFO:teuthology.orchestra.run.smithi132.stdout:(12/36): ceph-mon-19.0.0-1775.g5e70124b.el9.x86 6.0 MB/s | 4.7 MB 00:00 2024-03-26T04:52:38.443 INFO:teuthology.orchestra.run.smithi132.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-26T04:52:38.468 INFO:teuthology.orchestra.run.smithi132.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-26T04:52:38.493 INFO:teuthology.orchestra.run.smithi132.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-26T04:52:38.627 INFO:teuthology.orchestra.run.smithi132.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 788 kB/s | 125 kB 00:00 2024-03-26T04:52:38.811 INFO:teuthology.orchestra.run.smithi132.stdout:(17/36): ceph-osd-19.0.0-1775.g5e70124b.el9.x86 17 MB/s | 17 MB 00:00 2024-03-26T04:52:38.853 INFO:teuthology.orchestra.run.smithi132.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 62 kB/s | 22 kB 00:00 2024-03-26T04:52:39.145 INFO:teuthology.orchestra.run.smithi132.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 140 kB/s | 43 kB 00:00 2024-03-26T04:52:39.212 INFO:teuthology.orchestra.run.smithi132.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 259 kB/s | 151 kB 00:00 2024-03-26T04:52:39.279 INFO:teuthology.orchestra.run.smithi132.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 403 kB/s | 172 kB 00:00 2024-03-26T04:52:39.304 INFO:teuthology.orchestra.run.smithi132.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-26T04:52:39.379 INFO:teuthology.orchestra.run.smithi132.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-03-26T04:52:39.404 INFO:teuthology.orchestra.run.smithi132.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 230 kB/s | 23 kB 00:00 2024-03-26T04:52:39.438 INFO:teuthology.orchestra.run.smithi132.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-03-26T04:52:39.488 INFO:teuthology.orchestra.run.smithi132.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 178 kB/s | 19 kB 00:00 2024-03-26T04:52:39.530 INFO:teuthology.orchestra.run.smithi132.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 157 kB/s | 20 kB 00:00 2024-03-26T04:52:39.556 INFO:teuthology.orchestra.run.smithi132.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 395 kB/s | 46 kB 00:00 2024-03-26T04:52:39.622 INFO:teuthology.orchestra.run.smithi132.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 589 kB/s | 79 kB 00:00 2024-03-26T04:52:39.664 INFO:teuthology.orchestra.run.smithi132.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-26T04:52:39.731 INFO:teuthology.orchestra.run.smithi132.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-26T04:52:39.748 INFO:teuthology.orchestra.run.smithi132.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 716 kB/s | 90 kB 00:00 2024-03-26T04:52:39.807 INFO:teuthology.orchestra.run.smithi132.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 251 kB/s | 36 kB 00:00 2024-03-26T04:52:39.907 INFO:teuthology.orchestra.run.smithi132.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-03-26T04:52:39.941 INFO:teuthology.orchestra.run.smithi132.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 150 kB/s | 20 kB 00:00 2024-03-26T04:52:40.074 INFO:teuthology.orchestra.run.smithi132.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.3 MB/s | 427 kB 00:00 2024-03-26T04:52:40.079 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:52:40.079 INFO:teuthology.orchestra.run.smithi132.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-26T04:52:40.104 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:52:40.146 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:52:40.146 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:52:40.548 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:52:40.548 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:52:40.721 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T04:52:40.721 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 1/6 2024-03-26T04:52:40.722 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2/6 2024-03-26T04:52:40.722 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T04:52:40.722 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T04:52:40.722 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T04:52:41.148 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T04:52:41.148 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:41.148 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:52:41.148 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:41.148 INFO:teuthology.orchestra.run.smithi080.stdout: jq-1.6-16.el9.x86_64 2024-03-26T04:52:41.148 INFO:teuthology.orchestra.run.smithi080.stdout: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:41.148 INFO:teuthology.orchestra.run.smithi080.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T04:52:41.148 INFO:teuthology.orchestra.run.smithi080.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T04:52:41.148 INFO:teuthology.orchestra.run.smithi080.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T04:52:41.148 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:41.149 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:52:41.472 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph 2024-03-26T04:52:41.868 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:52:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T04:52:41.976 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T04:52:42.012 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T04:52:42.054 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:00:42 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:52:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T04:52:42.120 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T04:52:42.152 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T04:52:42.208 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:52:42.210 INFO:teuthology.orchestra.run.smithi080.stdout:========================================================================================= 2024-03-26T04:52:42.210 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-03-26T04:52:42.210 INFO:teuthology.orchestra.run.smithi080.stdout:========================================================================================= 2024-03-26T04:52:42.210 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:52:42.210 INFO:teuthology.orchestra.run.smithi080.stdout: ceph x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 6.6 k 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mds x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 2.3 M 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 1.5 M 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 248 k 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mon x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 4.7 M 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-osd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 17 M 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T04:52:42.211 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T04:52:42.212 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout:========================================================================================= 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout:Install 36 Packages 2024-03-26T04:52:42.213 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:42.214 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 30 M 2024-03-26T04:52:42.214 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 107 M 2024-03-26T04:52:42.214 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:52:42.251 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T04:52:42.331 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T04:52:42.371 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T04:52:42.444 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T04:52:42.497 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T04:52:42.535 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T04:52:42.602 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T04:52:42.660 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T04:52:42.870 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T04:52:42.923 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T04:52:42.961 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T04:52:42.997 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T04:52:43.012 INFO:teuthology.orchestra.run.smithi080.stdout:(1/36): ceph-19.0.0-1775.g5e70124b.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-26T04:52:43.027 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T04:52:43.102 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T04:52:43.138 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T04:52:43.146 INFO:teuthology.orchestra.run.smithi080.stdout:(2/36): ceph-mds-19.0.0-1775.g5e70124b.el9.x86_ 6.2 MB/s | 2.3 MB 00:00 2024-03-26T04:52:43.171 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T04:52:43.211 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T04:52:43.246 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T04:52:43.272 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T04:52:43.362 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T04:52:43.404 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T04:52:43.476 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T04:52:43.610 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T04:52:43.621 INFO:teuthology.orchestra.run.smithi080.stdout:(3/36): ceph-mgr-19.0.0-1775.g5e70124b.el9.x86_ 1.8 MB/s | 1.5 MB 00:00 2024-03-26T04:52:43.735 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T04:52:43.788 INFO:teuthology.orchestra.run.smithi080.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1775.g5e70 1.5 MB/s | 248 kB 00:00 2024-03-26T04:52:43.895 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 31/36 2024-03-26T04:52:43.926 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T04:52:43.950 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T04:52:43.951 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:52:43.951 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T04:52:43.951 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T04:52:43.951 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T04:52:43.951 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:44.005 INFO:teuthology.orchestra.run.smithi080.stdout:(5/36): ceph-osd-19.0.0-1775.g5e70124b.el9.x86_ 20 MB/s | 17 MB 00:00 2024-03-26T04:52:44.072 INFO:teuthology.orchestra.run.smithi080.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 144 kB 00:00 2024-03-26T04:52:44.098 INFO:teuthology.orchestra.run.smithi080.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 605 kB/s | 187 kB 00:00 2024-03-26T04:52:44.122 INFO:teuthology.orchestra.run.smithi080.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 630 kB/s | 32 kB 00:00 2024-03-26T04:52:44.148 INFO:teuthology.orchestra.run.smithi080.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-26T04:52:44.173 INFO:teuthology.orchestra.run.smithi080.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-03-26T04:52:44.198 INFO:teuthology.orchestra.run.smithi080.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-26T04:52:44.249 INFO:teuthology.orchestra.run.smithi080.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-26T04:52:44.274 INFO:teuthology.orchestra.run.smithi080.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-26T04:52:44.299 INFO:teuthology.orchestra.run.smithi080.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-26T04:52:44.333 INFO:teuthology.orchestra.run.smithi080.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-26T04:52:44.416 INFO:teuthology.orchestra.run.smithi080.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-26T04:52:44.458 INFO:teuthology.orchestra.run.smithi080.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 787 kB/s | 125 kB 00:00 2024-03-26T04:52:44.592 INFO:teuthology.orchestra.run.smithi080.stdout:(18/36): ceph-mon-19.0.0-1775.g5e70124b.el9.x86 3.0 MB/s | 4.7 MB 00:01 2024-03-26T04:52:44.625 INFO:teuthology.orchestra.run.smithi080.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 260 kB/s | 43 kB 00:00 2024-03-26T04:52:44.692 INFO:teuthology.orchestra.run.smithi080.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 548 kB/s | 151 kB 00:00 2024-03-26T04:52:44.751 INFO:teuthology.orchestra.run.smithi080.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-26T04:52:44.801 INFO:teuthology.orchestra.run.smithi080.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 821 kB/s | 172 kB 00:00 2024-03-26T04:52:44.827 INFO:teuthology.orchestra.run.smithi080.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 234 kB/s | 18 kB 00:00 2024-03-26T04:52:44.852 INFO:teuthology.orchestra.run.smithi080.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-03-26T04:52:44.877 INFO:teuthology.orchestra.run.smithi080.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-26T04:52:44.902 INFO:teuthology.orchestra.run.smithi080.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-03-26T04:52:44.928 INFO:teuthology.orchestra.run.smithi080.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-03-26T04:52:44.978 INFO:teuthology.orchestra.run.smithi080.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-03-26T04:52:45.003 INFO:teuthology.orchestra.run.smithi080.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 784 kB/s | 79 kB 00:00 2024-03-26T04:52:45.053 INFO:teuthology.orchestra.run.smithi080.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-03-26T04:52:45.079 INFO:teuthology.orchestra.run.smithi080.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-03-26T04:52:45.104 INFO:teuthology.orchestra.run.smithi080.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 895 kB/s | 90 kB 00:00 2024-03-26T04:52:45.129 INFO:teuthology.orchestra.run.smithi080.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-03-26T04:52:45.205 INFO:teuthology.orchestra.run.smithi080.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-26T04:52:45.230 INFO:teuthology.orchestra.run.smithi080.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-26T04:52:45.272 INFO:teuthology.orchestra.run.smithi080.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.5 MB/s | 427 kB 00:00 2024-03-26T04:52:45.274 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:52:45.274 INFO:teuthology.orchestra.run.smithi080.stdout:Total 9.8 MB/s | 30 MB 00:03 2024-03-26T04:52:45.297 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:52:45.337 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:52:45.338 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:52:45.376 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T04:52:45.402 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T04:52:45.402 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:52:45.402 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T04:52:45.403 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T04:52:45.403 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T04:52:45.403 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:45.757 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:52:45.757 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:52:45.844 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T04:52:45.867 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T04:52:45.867 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:52:45.867 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T04:52:45.867 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T04:52:45.867 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T04:52:45.867 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:46.086 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T04:52:46.109 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T04:52:46.109 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:52:46.109 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T04:52:46.109 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T04:52:46.109 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T04:52:46.110 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:46.175 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T04:52:47.082 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:52:47.149 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T04:52:47.182 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T04:52:47.210 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T04:52:47.280 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T04:52:47.316 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T04:52:47.349 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T04:52:47.442 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T04:52:47.520 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T04:52:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T04:52:47.641 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T04:52:47.695 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T04:52:47.733 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T04:52:47.781 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T04:52:47.837 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 1/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 3/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 4/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 5/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 6/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T04:52:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T04:52:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T04:52:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T04:52:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T04:52:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T04:52:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T04:52:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T04:52:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T04:52:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T04:52:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T04:52:48.034 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T04:52:48.087 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T04:52:48.131 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T04:52:48.169 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T04:52:48.199 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T04:52:48.276 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T04:52:48.309 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T04:52:48.342 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T04:52:48.383 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T04:52:48.417 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T04:52:48.461 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T04:52:48.550 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T04:52:48.591 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T04:52:48.663 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T04:52:48.801 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T04:52:48.805 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T04:52:48.805 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:48.805 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:52:48.805 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:48.805 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:48.805 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:48.806 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:52:48.806 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:48.806 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:48.806 INFO:teuthology.orchestra.run.smithi132.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T04:52:48.806 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T04:52:48.806 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T04:52:48.806 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T04:52:48.806 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T04:52:48.806 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T04:52:48.806 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T04:52:48.806 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T04:52:48.807 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T04:52:48.808 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T04:52:48.808 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T04:52:48.808 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T04:52:48.808 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T04:52:48.808 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T04:52:48.808 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T04:52:48.808 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T04:52:48.808 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T04:52:48.808 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:48.808 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:52:48.918 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T04:52:49.076 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 31/36 2024-03-26T04:52:49.113 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T04:52:49.134 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-base 2024-03-26T04:52:49.138 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T04:52:49.138 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:52:49.138 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T04:52:49.138 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T04:52:49.138 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T04:52:49.138 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:49.714 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:52 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:52:49.811 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:52:49.863 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:52:49.864 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-26T04:52:49.864 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:52:49.925 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install cephadm 2024-03-26T04:52:50.486 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:53 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:52:50.572 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T04:52:50.598 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T04:52:50.599 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:52:50.599 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T04:52:50.599 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T04:52:50.599 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T04:52:50.599 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:50.634 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:52:50.635 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:52:50.635 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-26T04:52:50.635 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:52:50.636 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:52:50.636 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 748 k 2024-03-26T04:52:50.636 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:50.636 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:52:50.636 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:52:50.636 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-26T04:52:50.636 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:50.636 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 748 k 2024-03-26T04:52:50.636 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 754 k 2024-03-26T04:52:50.636 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:52:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T04:52:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T04:52:51.062 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:52:51.062 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T04:52:51.062 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T04:52:51.063 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T04:52:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:51.077 INFO:teuthology.orchestra.run.smithi132.stdout:cephadm-19.0.0-1775.g5e70124b.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-26T04:52:51.078 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:52:51.078 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-26T04:52:51.078 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:52:51.083 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:52:51.083 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:52:51.089 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:52:51.089 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:52:51.163 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:52:51.290 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T04:52:51.313 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T04:52:51.313 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:52:51.313 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T04:52:51.313 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T04:52:51.313 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T04:52:51.313 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:51.379 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T04:52:51.588 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:52:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:52:52.029 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:52:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:52:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:52.316 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:52:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:52:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:52.317 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:52:52.527 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T04:52:53.096 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:56 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:52:53.149 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T04:52:53.149 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 1/36 2024-03-26T04:52:53.149 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 3/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 4/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 5/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 6/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T04:52:53.150 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T04:52:53.151 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T04:52:53.152 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T04:52:53.152 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T04:52:53.152 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T04:52:53.152 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T04:52:53.152 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T04:52:53.249 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:52:53.250 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:52:53.250 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-26T04:52:53.250 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:52:53.250 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:52:53.251 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 145 k 2024-03-26T04:52:53.251 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:53.251 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:52:53.251 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:52:53.251 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-26T04:52:53.251 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:53.251 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 145 k 2024-03-26T04:52:53.251 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 436 k 2024-03-26T04:52:53.251 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:52:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-immutable-object-cache-19.0.0-1775.g5e7012 440 kB/s | 145 kB 00:00 2024-03-26T04:52:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:52:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:Total 438 kB/s | 145 kB 00:00 2024-03-26T04:52:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:52:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:52:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:52:53.642 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:52:53.642 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:52:53.777 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:52:53.854 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:52:53.879 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:52:53.879 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:52:53.879 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T04:52:53.879 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:54.142 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T04:52:54.142 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:54.142 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:52:54.142 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:54.142 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:54.142 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:54.142 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:52:54.142 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:54.142 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:54.143 INFO:teuthology.orchestra.run.smithi080.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T04:52:54.143 INFO:teuthology.orchestra.run.smithi080.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T04:52:54.145 INFO:teuthology.orchestra.run.smithi080.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T04:52:54.145 INFO:teuthology.orchestra.run.smithi080.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T04:52:54.145 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T04:52:54.145 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T04:52:54.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:54.147 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:52:54.446 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-base 2024-03-26T04:52:54.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:52:54.867 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:54.867 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:52:54.867 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:54.867 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:54.867 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:52:54.999 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:00:55 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:52:55.096 INFO:teuthology.orchestra.run.smithi080.stdout:Package ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:52:55.124 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr 2024-03-26T04:52:55.148 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:52:55.149 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-03-26T04:52:55.149 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:52:55.213 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install cephadm 2024-03-26T04:52:55.697 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:58 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:52:55.762 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:00:55 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:52:55.795 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:52:55.847 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:52:55.847 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-26T04:52:55.848 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:52:55.908 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:52:55.909 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:52:55.909 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-03-26T04:52:55.909 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:52:55.910 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:52:55.910 INFO:teuthology.orchestra.run.smithi080.stdout: cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 748 k 2024-03-26T04:52:55.910 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:55.910 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:52:55.910 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:52:55.910 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-03-26T04:52:55.910 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:55.910 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 748 k 2024-03-26T04:52:55.910 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 754 k 2024-03-26T04:52:55.910 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:52:55.919 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T04:52:56.239 INFO:teuthology.orchestra.run.smithi080.stdout:cephadm-19.0.0-1775.g5e70124b.el9.noarch.rpm 2.2 MB/s | 748 kB 00:00 2024-03-26T04:52:56.240 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:52:56.240 INFO:teuthology.orchestra.run.smithi080.stdout:Total 2.2 MB/s | 748 kB 00:00 2024-03-26T04:52:56.240 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:52:56.245 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:52:56.245 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:52:56.250 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:52:56.251 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:52:56.327 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:52:56.477 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:59 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:52:56.628 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:52:56.629 INFO:teuthology.orchestra.run.smithi132.stdout:====================================================================================== 2024-03-26T04:52:56.630 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-26T04:52:56.630 INFO:teuthology.orchestra.run.smithi132.stdout:====================================================================================== 2024-03-26T04:52:56.630 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:52:56.630 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 1.8 M 2024-03-26T04:52:56.630 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-26T04:52:56.630 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T04:52:56.630 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 31 k 2024-03-26T04:52:56.630 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 15 k 2024-03-26T04:52:56.630 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T04:52:56.630 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T04:52:56.630 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T04:52:56.630 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T04:52:56.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T04:52:56.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T04:52:56.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T04:52:56.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T04:52:56.631 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:56.631 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:52:56.631 INFO:teuthology.orchestra.run.smithi132.stdout:====================================================================================== 2024-03-26T04:52:56.631 INFO:teuthology.orchestra.run.smithi132.stdout:Install 12 Packages 2024-03-26T04:52:56.631 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:52:56.632 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 6.9 M 2024-03-26T04:52:56.632 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 26 M 2024-03-26T04:52:56.632 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:52:56.738 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:52:56.826 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:52:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:52:57.324 INFO:teuthology.orchestra.run.smithi132.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1775.g5e7 52 kB/s | 15 kB 00:00 2024-03-26T04:52:57.349 INFO:teuthology.orchestra.run.smithi132.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1775.g5e 99 kB/s | 31 kB 00:00 2024-03-26T04:52:57.453 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:52:57.453 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:57.454 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:52:57.454 INFO:teuthology.orchestra.run.smithi080.stdout: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:52:57.454 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:57.454 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:52:57.641 INFO:teuthology.orchestra.run.smithi132.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 816 kB/s | 238 kB 00:00 2024-03-26T04:52:57.686 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T04:52:57.733 INFO:teuthology.orchestra.run.smithi132.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-03-26T04:52:58.033 INFO:teuthology.orchestra.run.smithi132.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1775.g5e70124 1.8 MB/s | 1.8 MB 00:01 2024-03-26T04:52:58.125 INFO:teuthology.orchestra.run.smithi132.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.7 MB/s | 863 kB 00:00 2024-03-26T04:52:58.150 INFO:teuthology.orchestra.run.smithi132.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-03-26T04:52:58.184 INFO:teuthology.orchestra.run.smithi132.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-26T04:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 527 kB/s | 31 kB 00:00 2024-03-26T04:52:58.251 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:00:58 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:52:58.310 INFO:teuthology.orchestra.run.smithi132.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-26T04:52:58.360 INFO:teuthology.orchestra.run.smithi132.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 690 kB/s | 144 kB 00:00 2024-03-26T04:52:58.405 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:52:58.405 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:52:58.406 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2024-03-26T04:52:58.406 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:52:58.406 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:52:58.406 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 145 k 2024-03-26T04:52:58.406 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:58.406 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:52:58.406 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:52:58.406 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-03-26T04:52:58.406 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:58.407 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 145 k 2024-03-26T04:52:58.407 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 436 k 2024-03-26T04:52:58.407 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:52:58.477 INFO:teuthology.orchestra.run.smithi132.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.8 MB/s | 2.0 MB 00:00 2024-03-26T04:52:58.480 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:52:58.481 INFO:teuthology.orchestra.run.smithi132.stdout:Total 3.7 MB/s | 6.9 MB 00:01 2024-03-26T04:52:58.499 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:52:58.518 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:52:58.519 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:52:58.628 INFO:teuthology.orchestra.run.smithi080.stdout:ceph-immutable-object-cache-19.0.0-1775.g5e7012 657 kB/s | 145 kB 00:00 2024-03-26T04:52:58.629 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:52:58.629 INFO:teuthology.orchestra.run.smithi080.stdout:Total 652 kB/s | 145 kB 00:00 2024-03-26T04:52:58.629 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:52:58.637 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:52:58.637 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:52:58.689 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:52:58.689 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:52:58.692 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:52:58.693 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:52:58.823 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:52:58.879 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:52:58.904 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:52:58.904 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:52:58.904 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T04:52:58.904 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:59.204 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:52:59.294 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T04:52:59.337 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T04:52:59.385 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T04:52:59.423 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T04:52:59.532 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T04:52:59.656 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T04:52:59.750 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T04:52:59.805 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T04:52:59.831 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T04:52:59.851 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:52:59.852 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:59.852 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:52:59.852 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:52:59.852 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:52:59.852 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:52:59.863 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 10/12 2024-03-26T04:53:00.106 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 11/12 2024-03-26T04:53:00.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr 2024-03-26T04:53:00.138 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T04:53:00.681 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:00 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:00.778 INFO:teuthology.orchestra.run.smithi080.stdout:Package ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:00.830 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:00.831 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-03-26T04:53:00.831 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:00.900 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T04:53:00.930 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T04:53:00.930 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 1/12 2024-03-26T04:53:00.930 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 2/12 2024-03-26T04:53:00.930 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 3/12 2024-03-26T04:53:00.930 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T04:53:00.930 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T04:53:00.930 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T04:53:00.930 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T04:53:00.930 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T04:53:00.930 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T04:53:00.930 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T04:53:00.930 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T04:53:01.274 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T04:53:01.274 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:01.274 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:53:01.274 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:01.275 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:01.452 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:01 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:01.541 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T04:53:01.601 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:01.602 INFO:teuthology.orchestra.run.smithi080.stdout:====================================================================================== 2024-03-26T04:53:01.602 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-03-26T04:53:01.602 INFO:teuthology.orchestra.run.smithi080.stdout:====================================================================================== 2024-03-26T04:53:01.602 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:53:01.602 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 1.8 M 2024-03-26T04:53:01.602 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-03-26T04:53:01.602 INFO:teuthology.orchestra.run.smithi080.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 31 k 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 15 k 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout:====================================================================================== 2024-03-26T04:53:01.603 INFO:teuthology.orchestra.run.smithi080.stdout:Install 12 Packages 2024-03-26T04:53:01.604 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:01.604 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 6.9 M 2024-03-26T04:53:01.604 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 26 M 2024-03-26T04:53:01.604 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:53:02.101 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:05 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:02.124 INFO:teuthology.orchestra.run.smithi080.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1775.g5e 120 kB/s | 31 kB 00:00 2024-03-26T04:53:02.149 INFO:teuthology.orchestra.run.smithi080.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1775.g5e7 53 kB/s | 15 kB 00:00 2024-03-26T04:53:02.251 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:02.252 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-03-26T04:53:02.252 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-26T04:53:02.252 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-03-26T04:53:02.252 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:53:02.252 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 7.4 M 2024-03-26T04:53:02.252 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-26T04:53:02.252 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:53:02.253 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-03-26T04:53:02.254 INFO:teuthology.orchestra.run.smithi132.stdout:Install 12 Packages 2024-03-26T04:53:02.254 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:02.254 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 37 M 2024-03-26T04:53:02.254 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 209 M 2024-03-26T04:53:02.254 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:53:02.324 INFO:teuthology.orchestra.run.smithi080.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-26T04:53:02.358 INFO:teuthology.orchestra.run.smithi080.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-26T04:53:02.584 INFO:teuthology.orchestra.run.smithi132.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 149 kB/s | 32 kB 00:00 2024-03-26T04:53:02.659 INFO:teuthology.orchestra.run.smithi132.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 228 kB/s | 17 kB 00:00 2024-03-26T04:53:02.691 INFO:teuthology.orchestra.run.smithi080.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1775.g5e70124 2.1 MB/s | 1.8 MB 00:00 2024-03-26T04:53:02.718 INFO:teuthology.orchestra.run.smithi132.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 591 kB/s | 34 kB 00:00 2024-03-26T04:53:02.801 INFO:teuthology.orchestra.run.smithi132.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.0 MB/s | 3.0 MB 00:00 2024-03-26T04:53:02.816 INFO:teuthology.orchestra.run.smithi080.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.7 MB/s | 863 kB 00:00 2024-03-26T04:53:03.008 INFO:teuthology.orchestra.run.smithi080.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 61 kB/s | 19 kB 00:00 2024-03-26T04:53:03.034 INFO:teuthology.orchestra.run.smithi080.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 811 kB/s | 548 kB 00:00 2024-03-26T04:53:03.035 INFO:teuthology.orchestra.run.smithi132.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-03-26T04:53:03.110 INFO:teuthology.orchestra.run.smithi132.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-26T04:53:03.126 INFO:teuthology.orchestra.run.smithi080.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 335 kB/s | 31 kB 00:00 2024-03-26T04:53:03.135 INFO:teuthology.orchestra.run.smithi132.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-26T04:53:03.259 INFO:teuthology.orchestra.run.smithi080.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-03-26T04:53:03.301 INFO:teuthology.orchestra.run.smithi080.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 494 kB/s | 144 kB 00:00 2024-03-26T04:53:03.569 INFO:teuthology.orchestra.run.smithi132.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-26T04:53:03.670 INFO:teuthology.orchestra.run.smithi132.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-17 5.7 MB/s | 7.4 MB 00:01 2024-03-26T04:53:03.703 INFO:teuthology.orchestra.run.smithi132.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-26T04:53:03.820 INFO:teuthology.orchestra.run.smithi132.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-26T04:53:03.887 INFO:teuthology.orchestra.run.smithi132.stdout:(12/12): libquadmath-11.4.1-3.el9.x86_64.rpm 616 kB/s | 195 kB 00:00 2024-03-26T04:53:03.888 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:03.889 INFO:teuthology.orchestra.run.smithi132.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-26T04:53:03.930 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:53:03.954 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:53:03.954 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:53:04.309 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:53:04.309 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:53:04.551 INFO:teuthology.orchestra.run.smithi080.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 1.2 MB/s | 2.0 MB 00:01 2024-03-26T04:53:04.554 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:04.555 INFO:teuthology.orchestra.run.smithi080.stdout:Total 2.3 MB/s | 6.9 MB 00:02 2024-03-26T04:53:04.575 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:53:04.596 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:53:04.596 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:53:04.784 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:53:04.784 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:53:05.001 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:53:05.076 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T04:53:05.108 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T04:53:05.201 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T04:53:05.244 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T04:53:05.309 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:53:05.393 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T04:53:05.421 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T04:53:05.445 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T04:53:05.460 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T04:53:05.491 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T04:53:05.522 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T04:53:05.536 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T04:53:05.633 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T04:53:05.764 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T04:53:05.849 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T04:53:05.893 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T04:53:05.922 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T04:53:05.953 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 10/12 2024-03-26T04:53:06.190 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T04:53:06.194 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 11/12 2024-03-26T04:53:06.228 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T04:53:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T04:53:07.027 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T04:53:07.028 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 1/12 2024-03-26T04:53:07.028 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 2/12 2024-03-26T04:53:07.028 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 3/12 2024-03-26T04:53:07.028 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T04:53:07.028 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T04:53:07.028 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T04:53:07.028 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T04:53:07.028 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T04:53:07.028 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T04:53:07.028 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T04:53:07.028 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T04:53:07.375 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T04:53:07.375 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:07.375 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:53:07.375 INFO:teuthology.orchestra.run.smithi080.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:07.376 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:07.545 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T04:53:07.639 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T04:53:08.192 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:08 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:08.342 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:08.343 INFO:teuthology.orchestra.run.smithi080.stdout:============================================================================================ 2024-03-26T04:53:08.344 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-03-26T04:53:08.344 INFO:teuthology.orchestra.run.smithi080.stdout:============================================================================================ 2024-03-26T04:53:08.344 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:53:08.344 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 7.4 M 2024-03-26T04:53:08.344 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-03-26T04:53:08.344 INFO:teuthology.orchestra.run.smithi080.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T04:53:08.344 INFO:teuthology.orchestra.run.smithi080.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T04:53:08.344 INFO:teuthology.orchestra.run.smithi080.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T04:53:08.344 INFO:teuthology.orchestra.run.smithi080.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T04:53:08.344 INFO:teuthology.orchestra.run.smithi080.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T04:53:08.344 INFO:teuthology.orchestra.run.smithi080.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T04:53:08.344 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T04:53:08.345 INFO:teuthology.orchestra.run.smithi080.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T04:53:08.345 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T04:53:08.345 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T04:53:08.345 INFO:teuthology.orchestra.run.smithi080.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T04:53:08.345 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:08.345 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:53:08.345 INFO:teuthology.orchestra.run.smithi080.stdout:============================================================================================ 2024-03-26T04:53:08.345 INFO:teuthology.orchestra.run.smithi080.stdout:Install 12 Packages 2024-03-26T04:53:08.345 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:08.346 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 37 M 2024-03-26T04:53:08.346 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 209 M 2024-03-26T04:53:08.346 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:53:08.579 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T04:53:08.783 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T04:53:08.783 INFO:teuthology.orchestra.run.smithi080.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 205 kB/s | 32 kB 00:00 2024-03-26T04:53:08.783 INFO:teuthology.orchestra.run.smithi080.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 424 kB/s | 17 kB 00:00 2024-03-26T04:53:08.783 INFO:teuthology.orchestra.run.smithi080.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 696 kB/s | 34 kB 00:00 2024-03-26T04:53:08.783 INFO:teuthology.orchestra.run.smithi080.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.8 MB/s | 3.0 MB 00:00 2024-03-26T04:53:09.075 INFO:teuthology.orchestra.run.smithi080.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-03-26T04:53:09.167 INFO:teuthology.orchestra.run.smithi080.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 4.8 MB 00:00 2024-03-26T04:53:09.192 INFO:teuthology.orchestra.run.smithi080.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.8 MB/s | 217 kB 00:00 2024-03-26T04:53:09.801 INFO:teuthology.orchestra.run.smithi080.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-03-26T04:53:09.826 INFO:teuthology.orchestra.run.smithi080.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-26T04:53:09.893 INFO:teuthology.orchestra.run.smithi080.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.6 MB/s | 245 kB 00:00 2024-03-26T04:53:10.068 INFO:teuthology.orchestra.run.smithi080.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 732 kB/s | 195 kB 00:00 2024-03-26T04:53:11.231 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T04:53:11.231 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 1/12 2024-03-26T04:53:11.231 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T04:53:11.231 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T04:53:11.231 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T04:53:11.232 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T04:53:11.232 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T04:53:11.232 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T04:53:11.232 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T04:53:11.232 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T04:53:11.232 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T04:53:11.232 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T04:53:11.670 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T04:53:11.670 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:11.670 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:53:11.670 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:11.670 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T04:53:11.670 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T04:53:11.670 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T04:53:11.670 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T04:53:11.671 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T04:53:11.671 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T04:53:11.671 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T04:53:11.671 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T04:53:11.671 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T04:53:11.671 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T04:53:11.671 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T04:53:11.671 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:11.671 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:12.033 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-rook 2024-03-26T04:53:12.604 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:15 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:12.755 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:12.756 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-26T04:53:12.756 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-26T04:53:12.756 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-26T04:53:12.756 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 49 k 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T04:53:12.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T04:53:12.758 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T04:53:12.758 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:12.758 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:53:12.758 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-26T04:53:12.758 INFO:teuthology.orchestra.run.smithi132.stdout:Install 13 Packages 2024-03-26T04:53:12.758 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:12.758 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 2.0 M 2024-03-26T04:53:12.759 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 26 M 2024-03-26T04:53:12.759 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:53:13.203 INFO:teuthology.orchestra.run.smithi132.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 124 kB/s | 24 kB 00:00 2024-03-26T04:53:13.228 INFO:teuthology.orchestra.run.smithi132.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 82 kB/s | 18 kB 00:00 2024-03-26T04:53:13.396 INFO:teuthology.orchestra.run.smithi132.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 913 kB/s | 174 kB 00:00 2024-03-26T04:53:13.421 INFO:teuthology.orchestra.run.smithi132.stdout:(4/13): ceph-mgr-rook-19.0.0-1775.g5e70124b.el9 119 kB/s | 49 kB 00:00 2024-03-26T04:53:13.446 INFO:teuthology.orchestra.run.smithi132.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 608 kB/s | 132 kB 00:00 2024-03-26T04:53:13.488 INFO:teuthology.orchestra.run.smithi132.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.2 MB/s | 211 kB 00:00 2024-03-26T04:53:13.513 INFO:teuthology.orchestra.run.smithi132.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 463 kB/s | 43 kB 00:00 2024-03-26T04:53:13.914 INFO:teuthology.orchestra.run.smithi132.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 69 kB/s | 32 kB 00:00 2024-03-26T04:53:13.947 INFO:teuthology.orchestra.run.smithi132.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 33 kB/s | 15 kB 00:00 2024-03-26T04:53:14.156 INFO:teuthology.orchestra.run.smithi132.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 284 kB/s | 59 kB 00:00 2024-03-26T04:53:14.239 INFO:teuthology.orchestra.run.smithi132.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 279 kB/s | 203 kB 00:00 2024-03-26T04:53:14.348 INFO:teuthology.orchestra.run.smithi132.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 467 kB/s | 90 kB 00:00 2024-03-26T04:53:15.339 INFO:teuthology.orchestra.run.smithi132.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 733 kB/s | 1.0 MB 00:01 2024-03-26T04:53:15.345 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:15.345 INFO:teuthology.orchestra.run.smithi132.stdout:Total 810 kB/s | 2.0 MB 00:02 2024-03-26T04:53:15.346 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:53:15.361 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:53:15.361 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:53:15.458 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:53:15.459 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:53:15.991 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:53:16.161 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T04:53:16.208 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T04:53:16.271 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T04:53:16.300 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T04:53:16.341 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T04:53:16.435 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T04:53:16.543 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T04:53:16.593 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T04:53:16.642 INFO:teuthology.orchestra.run.smithi080.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 923 kB/s | 7.4 MB 00:08 2024-03-26T04:53:16.644 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:16.645 INFO:teuthology.orchestra.run.smithi080.stdout:Total 4.5 MB/s | 37 MB 00:08 2024-03-26T04:53:16.687 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:53:16.712 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:53:16.713 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:53:17.068 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:53:17.068 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:53:17.312 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T04:53:17.366 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T04:53:17.399 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T04:53:17.447 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T04:53:17.469 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T04:53:17.766 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:53:17.839 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T04:53:17.871 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T04:53:17.962 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T04:53:17.999 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T04:53:18.175 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T04:53:18.206 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T04:53:18.250 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T04:53:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 1/13 2024-03-26T04:53:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T04:53:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T04:53:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T04:53:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T04:53:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T04:53:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T04:53:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T04:53:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T04:53:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T04:53:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T04:53:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T04:53:18.282 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T04:53:18.601 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T04:53:18.601 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:18.601 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:53:18.601 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:18.601 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T04:53:18.601 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T04:53:18.601 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T04:53:18.601 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T04:53:18.602 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T04:53:18.602 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T04:53:18.602 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T04:53:18.602 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T04:53:18.602 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T04:53:18.602 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T04:53:18.602 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T04:53:18.602 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T04:53:18.602 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:18.602 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:18.863 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T04:53:18.916 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T04:53:19.034 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T04:53:19.442 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:22 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:19.592 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:19.592 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-26T04:53:19.592 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-26T04:53:19.592 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-26T04:53:19.593 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:53:19.593 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 138 k 2024-03-26T04:53:19.593 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-26T04:53:19.593 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T04:53:19.593 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T04:53:19.593 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T04:53:19.593 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T04:53:19.593 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T04:53:19.593 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:19.593 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:53:19.593 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-26T04:53:19.594 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2024-03-26T04:53:19.594 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:19.594 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 6.8 M 2024-03-26T04:53:19.594 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 33 M 2024-03-26T04:53:19.594 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:53:20.253 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T04:53:20.425 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-26T04:53:20.450 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 714 kB/s | 248 kB 00:00 2024-03-26T04:53:20.542 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1775.g5e70124b.e 314 kB/s | 138 kB 00:00 2024-03-26T04:53:20.859 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 141 kB/s | 58 kB 00:00 2024-03-26T04:53:20.951 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 125 kB/s | 51 kB 00:00 2024-03-26T04:53:21.467 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T04:53:21.497 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T04:53:21.718 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 424 kB/s | 548 kB 00:01 2024-03-26T04:53:21.719 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:21.719 INFO:teuthology.orchestra.run.smithi132.stdout:Total 3.2 MB/s | 6.8 MB 00:02 2024-03-26T04:53:21.734 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:53:21.745 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:53:21.745 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:53:21.837 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:53:21.838 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:53:22.135 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:53:22.298 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T04:53:22.352 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T04:53:22.867 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T04:53:22.946 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T04:53:23.008 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T04:53:23.031 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T04:53:23.805 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T04:53:23.805 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/6 2024-03-26T04:53:23.805 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T04:53:23.805 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T04:53:23.805 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T04:53:23.805 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T04:53:24.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T04:53:24.103 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:24.103 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:53:24.103 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:24.103 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T04:53:24.103 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T04:53:24.104 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T04:53:24.104 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T04:53:24.104 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T04:53:24.104 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:24.104 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:24.134 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T04:53:24.134 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 1/12 2024-03-26T04:53:24.134 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T04:53:24.134 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T04:53:24.134 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T04:53:24.134 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T04:53:24.134 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T04:53:24.135 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T04:53:24.135 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T04:53:24.135 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T04:53:24.135 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T04:53:24.135 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T04:53:24.358 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-fuse 2024-03-26T04:53:24.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T04:53:24.575 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:24.575 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:53:24.575 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:24.575 INFO:teuthology.orchestra.run.smithi080.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T04:53:24.575 INFO:teuthology.orchestra.run.smithi080.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T04:53:24.576 INFO:teuthology.orchestra.run.smithi080.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T04:53:24.576 INFO:teuthology.orchestra.run.smithi080.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T04:53:24.576 INFO:teuthology.orchestra.run.smithi080.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T04:53:24.576 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T04:53:24.576 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T04:53:24.576 INFO:teuthology.orchestra.run.smithi080.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T04:53:24.576 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T04:53:24.576 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T04:53:24.576 INFO:teuthology.orchestra.run.smithi080.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T04:53:24.576 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:24.576 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:24.907 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-rook 2024-03-26T04:53:24.918 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:27 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:25.068 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:25.069 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:25.069 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-26T04:53:25.069 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:25.069 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:53:25.069 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 888 k 2024-03-26T04:53:25.069 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-26T04:53:25.070 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T04:53:25.070 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:25.070 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:53:25.070 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:25.070 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2024-03-26T04:53:25.070 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 967 k 2024-03-26T04:53:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 2.8 M 2024-03-26T04:53:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:53:25.205 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 585 kB/s | 79 kB 00:00 2024-03-26T04:53:25.467 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:25 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:25.555 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): ceph-fuse-19.0.0-1775.g5e70124b.el9.x86_ 1.8 MB/s | 888 kB 00:00 2024-03-26T04:53:25.556 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:25.557 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.9 MB/s | 967 kB 00:00 2024-03-26T04:53:25.557 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:53:25.577 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:53:25.577 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:53:25.616 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:25.617 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================== 2024-03-26T04:53:25.618 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-03-26T04:53:25.618 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================== 2024-03-26T04:53:25.618 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:53:25.618 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-rook noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 49 k 2024-03-26T04:53:25.618 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-03-26T04:53:25.618 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T04:53:25.618 INFO:teuthology.orchestra.run.smithi080.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T04:53:25.618 INFO:teuthology.orchestra.run.smithi080.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T04:53:25.618 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T04:53:25.618 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T04:53:25.618 INFO:teuthology.orchestra.run.smithi080.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T04:53:25.619 INFO:teuthology.orchestra.run.smithi080.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T04:53:25.619 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T04:53:25.619 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T04:53:25.619 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T04:53:25.619 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T04:53:25.619 INFO:teuthology.orchestra.run.smithi080.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T04:53:25.619 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:25.619 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:53:25.619 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================== 2024-03-26T04:53:25.619 INFO:teuthology.orchestra.run.smithi080.stdout:Install 13 Packages 2024-03-26T04:53:25.619 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:25.620 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 2.0 M 2024-03-26T04:53:25.620 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 26 M 2024-03-26T04:53:25.620 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:53:25.651 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:53:25.651 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:53:25.831 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:53:25.982 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T04:53:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T04:53:26.131 INFO:teuthology.orchestra.run.smithi080.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 198 kB/s | 24 kB 00:00 2024-03-26T04:53:26.156 INFO:teuthology.orchestra.run.smithi080.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2024-03-26T04:53:26.206 INFO:teuthology.orchestra.run.smithi080.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-26T04:53:26.232 INFO:teuthology.orchestra.run.smithi080.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-26T04:53:26.257 INFO:teuthology.orchestra.run.smithi080.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-26T04:53:26.282 INFO:teuthology.orchestra.run.smithi080.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-26T04:53:26.308 INFO:teuthology.orchestra.run.smithi080.stdout:(7/13): ceph-mgr-rook-19.0.0-1775.g5e70124b.el9 164 kB/s | 49 kB 00:00 2024-03-26T04:53:26.574 INFO:teuthology.orchestra.run.smithi080.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-03-26T04:53:26.608 INFO:teuthology.orchestra.run.smithi080.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 92 kB/s | 32 kB 00:00 2024-03-26T04:53:26.754 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T04:53:26.754 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/2 2024-03-26T04:53:26.850 INFO:teuthology.orchestra.run.smithi080.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 244 kB/s | 59 kB 00:00 2024-03-26T04:53:26.883 INFO:teuthology.orchestra.run.smithi080.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 352 kB/s | 203 kB 00:00 2024-03-26T04:53:27.042 INFO:teuthology.orchestra.run.smithi080.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 468 kB/s | 90 kB 00:00 2024-03-26T04:53:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T04:53:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:27.065 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:53:27.066 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T04:53:27.066 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:27.066 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:27.201 INFO:teuthology.orchestra.run.smithi080.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.6 MB/s | 1.0 MB 00:00 2024-03-26T04:53:27.203 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:27.203 INFO:teuthology.orchestra.run.smithi080.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2024-03-26T04:53:27.203 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:53:27.217 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:53:27.218 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:53:27.312 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:53:27.313 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:53:27.361 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-volume 2024-03-26T04:53:27.821 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:53:27.956 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:27.986 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T04:53:28.051 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T04:53:28.109 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:28.109 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-26T04:53:28.109 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:28.110 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:53:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 264 k 2024-03-26T04:53:28.110 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-26T04:53:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T04:53:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:28.110 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:53:28.110 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:28.110 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2024-03-26T04:53:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:28.111 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 332 k 2024-03-26T04:53:28.111 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 1.5 M 2024-03-26T04:53:28.111 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:53:28.137 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T04:53:28.177 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T04:53:28.218 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T04:53:28.235 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 555 kB/s | 69 kB 00:00 2024-03-26T04:53:28.311 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T04:53:28.368 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): ceph-volume-19.0.0-1775.g5e70124b.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-26T04:53:28.370 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:28.370 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-26T04:53:28.370 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:53:28.378 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:53:28.379 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:53:28.416 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:53:28.417 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:53:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T04:53:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T04:53:28.569 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:53:28.753 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T04:53:28.836 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T04:53:28.852 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T04:53:28.852 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:53:28.852 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T04:53:28.852 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:29.179 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T04:53:29.234 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T04:53:29.269 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T04:53:29.315 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T04:53:29.337 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T04:53:29.669 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 1/2 2024-03-26T04:53:30.000 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T04:53:30.000 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:30.000 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:53:30.000 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:30.000 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T04:53:30.000 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:30.000 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:30.127 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T04:53:30.127 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 1/13 2024-03-26T04:53:30.127 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T04:53:30.127 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T04:53:30.127 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T04:53:30.127 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T04:53:30.127 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T04:53:30.127 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T04:53:30.128 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T04:53:30.128 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T04:53:30.128 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T04:53:30.128 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T04:53:30.128 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T04:53:30.334 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados-devel 2024-03-26T04:53:30.478 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T04:53:30.478 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:30.478 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:53:30.478 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:30.478 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T04:53:30.478 INFO:teuthology.orchestra.run.smithi080.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T04:53:30.478 INFO:teuthology.orchestra.run.smithi080.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T04:53:30.478 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T04:53:30.478 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T04:53:30.479 INFO:teuthology.orchestra.run.smithi080.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T04:53:30.479 INFO:teuthology.orchestra.run.smithi080.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T04:53:30.479 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T04:53:30.479 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T04:53:30.479 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T04:53:30.479 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T04:53:30.479 INFO:teuthology.orchestra.run.smithi080.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T04:53:30.479 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:30.479 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:30.730 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T04:53:30.900 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:31.049 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:31.050 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:31.050 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-26T04:53:31.050 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:31.050 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:53:31.050 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 127 k 2024-03-26T04:53:31.050 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:31.050 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:53:31.050 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:31.050 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-26T04:53:31.050 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:31.051 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 127 k 2024-03-26T04:53:31.051 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 456 k 2024-03-26T04:53:31.051 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:53:31.281 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:31 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:31.383 INFO:teuthology.orchestra.run.smithi132.stdout:librados-devel-19.0.0-1775.g5e70124b.el9.x86_64 383 kB/s | 127 kB 00:00 2024-03-26T04:53:31.383 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:31.384 INFO:teuthology.orchestra.run.smithi132.stdout:Total 381 kB/s | 127 kB 00:00 2024-03-26T04:53:31.384 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:53:31.389 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:53:31.389 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:53:31.430 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:31.430 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================== 2024-03-26T04:53:31.430 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-03-26T04:53:31.430 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================== 2024-03-26T04:53:31.430 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:53:31.431 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 138 k 2024-03-26T04:53:31.431 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-03-26T04:53:31.431 INFO:teuthology.orchestra.run.smithi080.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T04:53:31.431 INFO:teuthology.orchestra.run.smithi080.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T04:53:31.431 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T04:53:31.431 INFO:teuthology.orchestra.run.smithi080.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T04:53:31.431 INFO:teuthology.orchestra.run.smithi080.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T04:53:31.431 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:31.431 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:53:31.431 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================== 2024-03-26T04:53:31.431 INFO:teuthology.orchestra.run.smithi080.stdout:Install 6 Packages 2024-03-26T04:53:31.431 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:31.432 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 6.8 M 2024-03-26T04:53:31.432 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 33 M 2024-03-26T04:53:31.432 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:53:31.444 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:53:31.444 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:53:31.587 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:53:31.726 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:31.960 INFO:teuthology.orchestra.run.smithi080.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-26T04:53:31.985 INFO:teuthology.orchestra.run.smithi080.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 776 kB/s | 248 kB 00:00 2024-03-26T04:53:32.077 INFO:teuthology.orchestra.run.smithi080.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1775.g5e70124b.e 335 kB/s | 138 kB 00:00 2024-03-26T04:53:32.158 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:32.302 INFO:teuthology.orchestra.run.smithi080.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2024-03-26T04:53:32.344 INFO:teuthology.orchestra.run.smithi080.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 191 kB/s | 51 kB 00:00 2024-03-26T04:53:32.435 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:32.435 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:32.435 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:53:32.435 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:53:32.435 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:32.435 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:32.552 INFO:teuthology.orchestra.run.smithi080.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 926 kB/s | 548 kB 00:00 2024-03-26T04:53:32.553 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:32.554 INFO:teuthology.orchestra.run.smithi080.stdout:Total 6.1 MB/s | 6.8 MB 00:01 2024-03-26T04:53:32.567 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:53:32.577 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:53:32.577 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:53:32.664 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:53:32.666 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:53:32.686 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs2 2024-03-26T04:53:32.973 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:53:33.143 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T04:53:33.195 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T04:53:33.266 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:36 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:33.365 INFO:teuthology.orchestra.run.smithi132.stdout:Package libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:33.418 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:33.419 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-26T04:53:33.419 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:33.488 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs-devel 2024-03-26T04:53:33.713 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T04:53:33.792 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T04:53:33.854 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T04:53:33.876 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T04:53:34.049 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:37 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:34.200 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:34.201 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:34.201 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-26T04:53:34.201 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:34.201 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:53:34.201 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 31 k 2024-03-26T04:53:34.201 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:34.201 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:53:34.201 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:34.201 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-26T04:53:34.201 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:34.201 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 31 k 2024-03-26T04:53:34.201 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 140 k 2024-03-26T04:53:34.202 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:53:34.453 INFO:teuthology.orchestra.run.smithi132.stdout:libcephfs-devel-19.0.0-1775.g5e70124b.el9.x86_6 124 kB/s | 31 kB 00:00 2024-03-26T04:53:34.453 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:34.454 INFO:teuthology.orchestra.run.smithi132.stdout:Total 124 kB/s | 31 kB 00:00 2024-03-26T04:53:34.454 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:53:34.457 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:53:34.457 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:53:34.493 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:53:34.494 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:53:34.626 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:53:34.662 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T04:53:34.662 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/6 2024-03-26T04:53:34.662 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T04:53:34.662 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T04:53:34.662 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T04:53:34.662 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T04:53:34.734 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:34.961 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T04:53:34.961 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:34.961 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:53:34.961 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:34.961 INFO:teuthology.orchestra.run.smithi080.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T04:53:34.961 INFO:teuthology.orchestra.run.smithi080.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T04:53:34.961 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T04:53:34.961 INFO:teuthology.orchestra.run.smithi080.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T04:53:34.961 INFO:teuthology.orchestra.run.smithi080.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T04:53:34.961 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:34.961 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:35.135 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:35.213 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-fuse 2024-03-26T04:53:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:35.410 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:53:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:53:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:35.410 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:35.589 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados2 2024-03-26T04:53:35.774 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:35 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:35.922 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:35.923 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:35.923 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-03-26T04:53:35.923 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:35.923 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:53:35.923 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 888 k 2024-03-26T04:53:35.923 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-03-26T04:53:35.923 INFO:teuthology.orchestra.run.smithi080.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T04:53:35.923 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:35.924 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:53:35.924 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:35.924 INFO:teuthology.orchestra.run.smithi080.stdout:Install 2 Packages 2024-03-26T04:53:35.924 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:35.924 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 967 k 2024-03-26T04:53:35.924 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 2.8 M 2024-03-26T04:53:35.924 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:53:36.055 INFO:teuthology.orchestra.run.smithi080.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 600 kB/s | 79 kB 00:00 2024-03-26T04:53:36.148 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:36.244 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:36.295 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:36.297 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-26T04:53:36.297 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:36.342 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librbd1 2024-03-26T04:53:36.906 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:37.004 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:37.056 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:37.057 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-26T04:53:37.057 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:37.111 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rados 2024-03-26T04:53:37.264 INFO:teuthology.orchestra.run.smithi080.stdout:(2/2): ceph-fuse-19.0.0-1775.g5e70124b.el9.x86_ 663 kB/s | 888 kB 00:01 2024-03-26T04:53:37.265 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:37.266 INFO:teuthology.orchestra.run.smithi080.stdout:Total 720 kB/s | 967 kB 00:01 2024-03-26T04:53:37.266 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:53:37.289 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:53:37.289 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:53:37.368 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:53:37.368 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:53:37.550 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:53:37.672 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:40 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:37.707 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T04:53:37.768 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:37.773 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T04:53:37.819 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:37.820 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-26T04:53:37.820 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:37.867 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rgw 2024-03-26T04:53:38.428 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:41 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:38.483 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T04:53:38.483 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/2 2024-03-26T04:53:38.524 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:38.575 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:38.576 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-26T04:53:38.576 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:38.626 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-cephfs 2024-03-26T04:53:38.794 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T04:53:38.794 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:38.794 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:53:38.794 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T04:53:38.794 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:38.794 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:39.079 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-volume 2024-03-26T04:53:39.198 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:39.301 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:39.433 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:39.433 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-26T04:53:39.433 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:39.433 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rbd 2024-03-26T04:53:39.654 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:39.804 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:39.804 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:39.805 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-03-26T04:53:39.805 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:39.805 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:53:39.805 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-volume noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 264 k 2024-03-26T04:53:39.805 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-03-26T04:53:39.805 INFO:teuthology.orchestra.run.smithi080.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T04:53:39.805 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:39.805 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:53:39.805 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:39.805 INFO:teuthology.orchestra.run.smithi080.stdout:Install 2 Packages 2024-03-26T04:53:39.805 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:39.806 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 332 k 2024-03-26T04:53:39.806 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 1.5 M 2024-03-26T04:53:39.806 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:53:39.918 INFO:teuthology.orchestra.run.smithi080.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 614 kB/s | 69 kB 00:00 2024-03-26T04:53:40.026 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:40.127 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:40.180 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:40.182 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-26T04:53:40.182 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:40.240 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-fuse 2024-03-26T04:53:40.469 INFO:teuthology.orchestra.run.smithi080.stdout:(2/2): ceph-volume-19.0.0-1775.g5e70124b.el9.no 398 kB/s | 264 kB 00:00 2024-03-26T04:53:40.470 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:40.470 INFO:teuthology.orchestra.run.smithi080.stdout:Total 500 kB/s | 332 kB 00:00 2024-03-26T04:53:40.470 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:53:40.479 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:53:40.479 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:53:40.522 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:53:40.523 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:53:40.669 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:53:40.814 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:40.849 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T04:53:40.887 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T04:53:40.903 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T04:53:40.903 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:53:40.903 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T04:53:40.903 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:40.961 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:40.962 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:40.962 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-03-26T04:53:40.962 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:40.962 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:53:40.962 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 85 k 2024-03-26T04:53:40.962 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:40.962 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:53:40.963 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:40.963 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-26T04:53:40.963 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:40.963 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 85 k 2024-03-26T04:53:40.963 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 231 k 2024-03-26T04:53:40.963 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:53:41.331 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-fuse-19.0.0-1775.g5e70124b.el9.x86_64.rpm 232 kB/s | 85 kB 00:00 2024-03-26T04:53:41.332 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:41.332 INFO:teuthology.orchestra.run.smithi132.stdout:Total 231 kB/s | 85 kB 00:00 2024-03-26T04:53:41.332 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:53:41.339 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:53:41.339 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:53:41.394 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:53:41.395 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:53:41.533 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:53:41.638 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:41.735 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 1/2 2024-03-26T04:53:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:42.069 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T04:53:42.069 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:42.069 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:53:42.069 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:53:42.069 INFO:teuthology.orchestra.run.smithi080.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T04:53:42.069 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:42.069 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:42.324 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:42.324 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:42.324 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:53:42.324 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:53:42.324 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:42.325 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:42.398 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install librados-devel 2024-03-26T04:53:42.569 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-mirror 2024-03-26T04:53:42.970 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:43.119 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:43.120 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:43.120 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2024-03-26T04:53:43.120 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:43.120 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:53:43.120 INFO:teuthology.orchestra.run.smithi080.stdout: librados-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 127 k 2024-03-26T04:53:43.120 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:43.120 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:53:43.120 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:43.120 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-03-26T04:53:43.120 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:43.120 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 127 k 2024-03-26T04:53:43.120 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 456 k 2024-03-26T04:53:43.121 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:53:43.148 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:46 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:43.301 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:43.302 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:43.302 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-26T04:53:43.302 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:43.302 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:53:43.302 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T04:53:43.302 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:43.302 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:53:43.302 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:43.302 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-26T04:53:43.302 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:43.303 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 3.1 M 2024-03-26T04:53:43.303 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 13 M 2024-03-26T04:53:43.303 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:53:43.491 INFO:teuthology.orchestra.run.smithi080.stdout:librados-devel-19.0.0-1775.g5e70124b.el9.x86_64 343 kB/s | 127 kB 00:00 2024-03-26T04:53:43.491 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:43.491 INFO:teuthology.orchestra.run.smithi080.stdout:Total 341 kB/s | 127 kB 00:00 2024-03-26T04:53:43.492 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:53:43.497 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:53:43.497 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:53:43.553 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:53:43.553 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:53:43.698 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:53:43.806 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:44.231 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:44.293 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-mirror-19.0.0-1775.g5e70124b.el9.x86_64.rpm 3.1 MB/s | 3.1 MB 00:00 2024-03-26T04:53:44.294 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:44.294 INFO:teuthology.orchestra.run.smithi132.stdout:Total 3.1 MB/s | 3.1 MB 00:00 2024-03-26T04:53:44.295 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:53:44.304 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:53:44.304 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:53:44.377 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:53:44.378 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:53:44.517 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:44.517 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:44.517 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:53:44.517 INFO:teuthology.orchestra.run.smithi080.stdout: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:53:44.517 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:44.517 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:44.767 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install libcephfs2 2024-03-26T04:53:44.799 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:53:44.852 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:44.878 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:44.878 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:53:44.878 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T04:53:44.878 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T04:53:44.878 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T04:53:44.878 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:45.335 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:45.434 INFO:teuthology.orchestra.run.smithi080.stdout:Package libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:45.487 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:45.488 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-03-26T04:53:45.488 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:45.559 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install libcephfs-devel 2024-03-26T04:53:46.042 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:46.042 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:46.042 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:53:46.042 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:53:46.042 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:46.043 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:46.114 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:46 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:46.263 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:46.265 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:46.265 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2024-03-26T04:53:46.265 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:46.265 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:53:46.266 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 31 k 2024-03-26T04:53:46.266 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:46.266 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:53:46.266 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:46.266 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-03-26T04:53:46.266 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:46.267 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 31 k 2024-03-26T04:53:46.267 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 140 k 2024-03-26T04:53:46.267 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:53:46.298 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-nbd 2024-03-26T04:53:46.518 INFO:teuthology.orchestra.run.smithi080.stdout:libcephfs-devel-19.0.0-1775.g5e70124b.el9.x86_6 123 kB/s | 31 kB 00:00 2024-03-26T04:53:46.519 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:46.519 INFO:teuthology.orchestra.run.smithi080.stdout:Total 123 kB/s | 31 kB 00:00 2024-03-26T04:53:46.519 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:53:46.522 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:53:46.523 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:53:46.546 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:53:46.546 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:53:46.693 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:53:46.791 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:46.873 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:49 ago on Tue 26 Mar 2024 04:51:57 AM UTC. 2024-03-26T04:53:47.023 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-26T04:53:47.024 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:47.024 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-03-26T04:53:47.024 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:47.024 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-26T04:53:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 171 k 2024-03-26T04:53:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:47.025 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-26T04:53:47.025 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-26T04:53:47.025 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-26T04:53:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:47.025 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 171 k 2024-03-26T04:53:47.025 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 494 k 2024-03-26T04:53:47.025 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-26T04:53:47.185 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:47.255 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-nbd-19.0.0-1775.g5e70124b.el9.x86_64.rpm 742 kB/s | 171 kB 00:00 2024-03-26T04:53:47.256 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:47.256 INFO:teuthology.orchestra.run.smithi132.stdout:Total 737 kB/s | 171 kB 00:00 2024-03-26T04:53:47.256 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-26T04:53:47.264 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-26T04:53:47.264 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-26T04:53:47.320 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-26T04:53:47.321 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-26T04:53:47.447 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:47.448 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:47.448 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:53:47.448 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:53:47.448 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:47.448 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:47.468 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-26T04:53:47.572 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:47.596 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install librados2 2024-03-26T04:53:48.017 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:48.154 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:48 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:48.250 INFO:teuthology.orchestra.run.smithi080.stdout:Package librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:48.299 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-26T04:53:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:53:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:53:48.299 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-26T04:53:48.303 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:48.304 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-03-26T04:53:48.304 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:48.351 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install librbd1 2024-03-26T04:53:48.518 DEBUG:teuthology.parallel:result is None 2024-03-26T04:53:48.912 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:48 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:49.010 INFO:teuthology.orchestra.run.smithi080.stdout:Package librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:49.061 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:49.062 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-03-26T04:53:49.062 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:49.117 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-rados 2024-03-26T04:53:49.680 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:49 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:49.776 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:49.828 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:49.829 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-03-26T04:53:49.829 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:49.872 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-rgw 2024-03-26T04:53:50.436 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:50 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:50.532 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:50.583 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:50.584 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-03-26T04:53:50.584 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:50.636 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-cephfs 2024-03-26T04:53:51.193 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:51 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:51.295 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:51.349 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:51.351 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-03-26T04:53:51.351 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:51.407 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-rbd 2024-03-26T04:53:51.991 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:51 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:52.088 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:53:52.141 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:52.142 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-03-26T04:53:52.142 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:52.191 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-fuse 2024-03-26T04:53:52.765 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:52 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:52.911 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 85 k 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 85 k 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 231 k 2024-03-26T04:53:52.912 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:53:53.113 INFO:teuthology.orchestra.run.smithi080.stdout:rbd-fuse-19.0.0-1775.g5e70124b.el9.x86_64.rpm 426 kB/s | 85 kB 00:00 2024-03-26T04:53:53.113 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:53.113 INFO:teuthology.orchestra.run.smithi080.stdout:Total 423 kB/s | 85 kB 00:00 2024-03-26T04:53:53.114 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:53:53.120 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:53:53.121 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:53:53.175 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:53:53.176 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:53:53.314 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:53:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:53.862 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:54.165 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:54.165 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:54.165 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:53:54.165 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:53:54.165 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:54.165 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:54.430 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-mirror 2024-03-26T04:53:54.992 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:54 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:55.145 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:55.145 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:55.145 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2024-03-26T04:53:55.145 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:55.145 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:53:55.146 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-mirror x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T04:53:55.146 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:55.146 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:53:55.146 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:55.146 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-03-26T04:53:55.146 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:55.146 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 3.1 M 2024-03-26T04:53:55.146 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 13 M 2024-03-26T04:53:55.147 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:53:55.774 INFO:teuthology.orchestra.run.smithi080.stdout:rbd-mirror-19.0.0-1775.g5e70124b.el9.x86_64.rpm 4.9 MB/s | 3.1 MB 00:00 2024-03-26T04:53:55.775 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:55.775 INFO:teuthology.orchestra.run.smithi080.stdout:Total 4.9 MB/s | 3.1 MB 00:00 2024-03-26T04:53:55.775 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:53:55.784 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:53:55.784 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:53:55.857 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:53:55.858 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:53:56.273 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:53:56.342 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:56.367 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:56.367 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:53:56.367 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T04:53:56.367 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T04:53:56.367 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T04:53:56.367 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:57.490 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:57.490 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:57.490 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:53:57.490 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:53:57.490 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:57.490 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:53:57.726 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-nbd 2024-03-26T04:53:58.303 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:58 ago on Tue 26 Mar 2024 04:52:00 AM UTC. 2024-03-26T04:53:58.451 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-26T04:53:58.452 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:58.452 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-03-26T04:53:58.452 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:58.452 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-26T04:53:58.453 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-nbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 171 k 2024-03-26T04:53:58.453 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:58.453 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-26T04:53:58.453 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-26T04:53:58.453 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-03-26T04:53:58.453 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:58.453 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 171 k 2024-03-26T04:53:58.453 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 494 k 2024-03-26T04:53:58.453 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-26T04:53:58.819 INFO:teuthology.orchestra.run.smithi080.stdout:rbd-nbd-19.0.0-1775.g5e70124b.el9.x86_64.rpm 467 kB/s | 171 kB 00:00 2024-03-26T04:53:58.820 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:53:58.820 INFO:teuthology.orchestra.run.smithi080.stdout:Total 465 kB/s | 171 kB 00:00 2024-03-26T04:53:58.820 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-26T04:53:58.828 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-26T04:53:58.828 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-26T04:53:58.885 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-26T04:53:58.886 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-26T04:53:59.036 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-26T04:53:59.151 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:59.570 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:59.888 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:53:59.888 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:59.888 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-26T04:53:59.888 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:53:59.888 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:53:59.888 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-26T04:54:00.113 DEBUG:teuthology.parallel:result is None 2024-03-26T04:54:00.114 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:54:00.382 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T04:54:00.402 INFO:teuthology.orchestra.run.smithi080.stdout:19.0.0-1775.g5e70124b.el9 2024-03-26T04:54:00.403 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1775.g5e70124b.el9 2024-03-26T04:54:00.403 INFO:teuthology.task.install:The correct ceph version 19.0.0-1775.g5e70124b is installed. 2024-03-26T04:54:00.405 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:54:00.680 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T04:54:00.704 INFO:teuthology.orchestra.run.smithi132.stdout:19.0.0-1775.g5e70124b.el9 2024-03-26T04:54:00.705 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1775.g5e70124b.el9 2024-03-26T04:54:00.705 INFO:teuthology.task.install:The correct ceph version 19.0.0-1775.g5e70124b is installed. 2024-03-26T04:54:00.707 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-26T04:54:00.707 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:54:00.707 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T04:54:00.734 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T04:54:00.734 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T04:54:00.777 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-26T04:54:00.777 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:54:00.777 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T04:54:00.804 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T04:54:00.871 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T04:54:00.871 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T04:54:00.903 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T04:54:00.973 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-26T04:54:00.973 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:54:00.973 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T04:54:01.000 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T04:54:01.066 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T04:54:01.066 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T04:54:01.096 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T04:54:01.166 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-26T04:54:01.167 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:54:01.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T04:54:01.193 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T04:54:01.259 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T04:54:01.259 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T04:54:01.289 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T04:54:01.358 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T04:54:01.456 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'cephadm_mode': 'root'} 2024-03-26T04:54:01.456 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:54:01.456 INFO:tasks.cephadm:Cluster fsid is dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:54:01.456 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T04:54:01.457 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.80', 'mon.c': '[v2:172.21.15.80:3301,v1:172.21.15.80:6790]', 'mon.b': '172.21.15.132'} 2024-03-26T04:54:01.457 INFO:tasks.cephadm:First mon is mon.a on smithi080 2024-03-26T04:54:01.457 INFO:tasks.cephadm:First mgr is y 2024-03-26T04:54:01.457 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T04:54:01.457 DEBUG:teuthology.orchestra.run.smithi080:> sudo hostname $(hostname -s) 2024-03-26T04:54:01.484 DEBUG:teuthology.orchestra.run.smithi132:> sudo hostname $(hostname -s) 2024-03-26T04:54:01.513 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-26T04:54:01.513 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:54:01.765 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54690/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1775-g5e70124b', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1775.g5e70124b'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-25 18:23:16.641218', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-26T04:54:01.913 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 5e70124b46c2a671efcaae4690cd9ea72670710a from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:54:01.915 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm 2024-03-26T04:54:01.915 DEBUG:teuthology.orchestra.run.smithi080:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T04:54:02.435 INFO:teuthology.orchestra.run.smithi080.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 04:54 /home/ubuntu/cephtest/cephadm 2024-03-26T04:54:02.437 DEBUG:teuthology.orchestra.run.smithi132:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T04:54:02.912 INFO:teuthology.orchestra.run.smithi132.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 04:54 /home/ubuntu/cephtest/cephadm 2024-03-26T04:54:02.912 DEBUG:teuthology.orchestra.run.smithi080:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T04:54:02.929 DEBUG:teuthology.orchestra.run.smithi132:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T04:54:02.959 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a on all hosts... 2024-03-26T04:54:02.960 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T04:54:02.972 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T04:54:03.823 INFO:teuthology.orchestra.run.smithi132.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:54:03.829 INFO:teuthology.orchestra.run.smithi080.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:54:29.606 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T04:54:29.607 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:54:29.607 INFO:teuthology.orchestra.run.smithi132.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T04:54:29.607 INFO:teuthology.orchestra.run.smithi132.stdout: "repo_digests": [ 2024-03-26T04:54:29.607 INFO:teuthology.orchestra.run.smithi132.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T04:54:29.607 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T04:54:29.607 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T04:54:42.206 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T04:54:42.206 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:54:42.206 INFO:teuthology.orchestra.run.smithi080.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T04:54:42.206 INFO:teuthology.orchestra.run.smithi080.stdout: "repo_digests": [ 2024-03-26T04:54:42.206 INFO:teuthology.orchestra.run.smithi080.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T04:54:42.206 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T04:54:42.206 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T04:54:42.231 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /etc/ceph 2024-03-26T04:54:42.284 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph 2024-03-26T04:54:42.320 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 777 /etc/ceph 2024-03-26T04:54:42.347 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /etc/ceph 2024-03-26T04:54:42.390 INFO:tasks.cephadm:Writing seed config... 2024-03-26T04:54:42.391 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T04:54:42.391 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T04:54:42.391 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-26T04:54:42.391 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-26T04:54:42.391 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-26T04:54:42.391 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-26T04:54:42.392 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-26T04:54:42.393 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T04:54:42.393 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-26T04:54:42.393 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:54:42.393 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T04:54:42.408 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = dd09e9cc-eb2c-11ee-95d0-87774f69a715 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-26T04:54:42.408 DEBUG:teuthology.orchestra.run.smithi080:mon.a> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T04:54:42.450 DEBUG:teuthology.orchestra.run.smithi080:mgr.y> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T04:54:42.493 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T04:54:42.493 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a -v bootstrap --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.80 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:54:42.682 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:54:42.682 INFO:teuthology.orchestra.run.smithi080.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a', '-v', 'bootstrap', '--fsid', 'dd09e9cc-eb2c-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.80', '--skip-admin-label'] 2024-03-26T04:54:42.707 INFO:teuthology.orchestra.run.smithi080.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:54:42.708 INFO:teuthology.orchestra.run.smithi080.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T04:54:42.708 INFO:teuthology.orchestra.run.smithi080.stdout:Verifying podman|docker is present... 2024-03-26T04:54:42.731 INFO:teuthology.orchestra.run.smithi080.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:54:42.731 INFO:teuthology.orchestra.run.smithi080.stdout:Verifying lvm2 is present... 2024-03-26T04:54:42.731 INFO:teuthology.orchestra.run.smithi080.stdout:Verifying time synchronization is in place... 2024-03-26T04:54:42.738 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T04:54:42.738 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T04:54:42.744 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T04:54:42.744 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout inactive 2024-03-26T04:54:42.751 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout enabled 2024-03-26T04:54:42.757 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout active 2024-03-26T04:54:42.757 INFO:teuthology.orchestra.run.smithi080.stdout:Unit chronyd.service is enabled and running 2024-03-26T04:54:42.757 INFO:teuthology.orchestra.run.smithi080.stdout:Repeating the final host check... 2024-03-26T04:54:42.779 INFO:teuthology.orchestra.run.smithi080.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:54:42.780 INFO:teuthology.orchestra.run.smithi080.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T04:54:42.780 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl is present 2024-03-26T04:54:42.780 INFO:teuthology.orchestra.run.smithi080.stdout:lvcreate is present 2024-03-26T04:54:42.786 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T04:54:42.786 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T04:54:42.792 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T04:54:42.792 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout inactive 2024-03-26T04:54:42.798 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout enabled 2024-03-26T04:54:42.805 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout active 2024-03-26T04:54:42.805 INFO:teuthology.orchestra.run.smithi080.stdout:Unit chronyd.service is enabled and running 2024-03-26T04:54:42.805 INFO:teuthology.orchestra.run.smithi080.stdout:Host looks OK 2024-03-26T04:54:42.805 INFO:teuthology.orchestra.run.smithi080.stdout:Cluster fsid: dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:54:42.805 INFO:teuthology.orchestra.run.smithi080.stdout:Acquiring lock 139812557504464 on /run/cephadm/dd09e9cc-eb2c-11ee-95d0-87774f69a715.lock 2024-03-26T04:54:42.805 INFO:teuthology.orchestra.run.smithi080.stdout:Lock 139812557504464 acquired on /run/cephadm/dd09e9cc-eb2c-11ee-95d0-87774f69a715.lock 2024-03-26T04:54:42.805 INFO:teuthology.orchestra.run.smithi080.stdout:Verifying IP 172.21.15.80 port 3300 ... 2024-03-26T04:54:42.806 INFO:teuthology.orchestra.run.smithi080.stdout:Verifying IP 172.21.15.80 port 6789 ... 2024-03-26T04:54:42.806 INFO:teuthology.orchestra.run.smithi080.stdout:Base mon IP(s) is [172.21.15.80:3300, 172.21.15.80:6789], mon addrv is [v2:172.21.15.80:3300,v1:172.21.15.80:6789] 2024-03-26T04:54:42.809 INFO:teuthology.orchestra.run.smithi080.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.80 metric 100 2024-03-26T04:54:42.809 INFO:teuthology.orchestra.run.smithi080.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.80 metric 100 2024-03-26T04:54:42.811 INFO:teuthology.orchestra.run.smithi080.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T04:54:42.811 INFO:teuthology.orchestra.run.smithi080.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-26T04:54:42.811 INFO:teuthology.orchestra.run.smithi080.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-03-26T04:54:42.814 INFO:teuthology.orchestra.run.smithi080.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T04:54:42.814 INFO:teuthology.orchestra.run.smithi080.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T04:54:42.814 INFO:teuthology.orchestra.run.smithi080.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T04:54:42.814 INFO:teuthology.orchestra.run.smithi080.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-26T04:54:42.814 INFO:teuthology.orchestra.run.smithi080.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cba9/64 scope link 2024-03-26T04:54:42.814 INFO:teuthology.orchestra.run.smithi080.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T04:54:42.815 INFO:teuthology.orchestra.run.smithi080.stdout:Mon IP `172.21.15.80` is in CIDR network `172.21.0.0/20` 2024-03-26T04:54:42.815 INFO:teuthology.orchestra.run.smithi080.stdout:Mon IP `172.21.15.80` is in CIDR network `172.21.0.0/20` 2024-03-26T04:54:42.815 INFO:teuthology.orchestra.run.smithi080.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T04:54:42.815 INFO:teuthology.orchestra.run.smithi080.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T04:54:42.816 INFO:teuthology.orchestra.run.smithi080.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:54:43.283 INFO:teuthology.orchestra.run.smithi080.stdout:/bin/podman: stdout 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T04:54:43.283 INFO:teuthology.orchestra.run.smithi080.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:54:43.283 INFO:teuthology.orchestra.run.smithi080.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T04:54:43.283 INFO:teuthology.orchestra.run.smithi080.stdout:/bin/podman: stderr Copying blob sha256:45ddbf13772d0d6535ecfdcaa58ecfc496e5849512dfbf602c5218332a216bcc 2024-03-26T04:54:43.283 INFO:teuthology.orchestra.run.smithi080.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T04:54:43.284 INFO:teuthology.orchestra.run.smithi080.stdout:/bin/podman: stderr Copying config sha256:5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T04:54:43.284 INFO:teuthology.orchestra.run.smithi080.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T04:54:44.214 INFO:teuthology.orchestra.run.smithi080.stdout:ceph: stdout ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T04:54:44.215 INFO:teuthology.orchestra.run.smithi080.stdout:Ceph version: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T04:54:44.215 INFO:teuthology.orchestra.run.smithi080.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T04:54:45.037 INFO:teuthology.orchestra.run.smithi080.stdout:stat: stdout 167 167 2024-03-26T04:54:45.037 INFO:teuthology.orchestra.run.smithi080.stdout:Creating initial keys... 2024-03-26T04:54:46.077 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph-authtool: stdout AQAVVQJmCN3xFRAAin3F++p5KklRxleN6EQmHg== 2024-03-26T04:54:46.951 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph-authtool: stdout AQAWVQJmPjDoFhAAkMMUyXNlNvTMZTRLJaVXgg== 2024-03-26T04:54:47.840 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph-authtool: stdout AQAXVQJmzfFaEBAAzJwovAiLa/V2a21CdCatfA== 2024-03-26T04:54:47.841 INFO:teuthology.orchestra.run.smithi080.stdout:Creating initial monmap... 2024-03-26T04:54:48.747 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T04:54:48.747 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T04:54:48.747 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:54:48.747 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T04:54:48.747 INFO:teuthology.orchestra.run.smithi080.stdout:monmaptool for a [v2:172.21.15.80:3300,v1:172.21.15.80:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T04:54:48.748 INFO:teuthology.orchestra.run.smithi080.stdout:setting min_mon_release = quincy 2024-03-26T04:54:48.748 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/monmaptool: set fsid to dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:54:48.748 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T04:54:48.748 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:54:48.748 INFO:teuthology.orchestra.run.smithi080.stdout:Creating mon... 2024-03-26T04:54:49.645 INFO:teuthology.orchestra.run.smithi080.stdout:create mon.a on 2024-03-26T04:54:49.861 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-26T04:54:49.992 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T04:54:50.151 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715.target. 2024-03-26T04:54:50.151 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715.target. 2024-03-26T04:54:50.391 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.a 2024-03-26T04:54:50.391 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stderr Failed to reset failed state of unit ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T04:54:50.568 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715.target.wants/ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@.service. 2024-03-26T04:54:51.119 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:54:51.119 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout disabled 2024-03-26T04:54:51.126 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:54:51.126 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout inactive 2024-03-26T04:54:51.127 INFO:teuthology.orchestra.run.smithi080.stdout:firewalld.service is not enabled 2024-03-26T04:54:51.127 INFO:teuthology.orchestra.run.smithi080.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T04:54:51.127 INFO:teuthology.orchestra.run.smithi080.stdout:Waiting for mon to start... 2024-03-26T04:54:51.127 INFO:teuthology.orchestra.run.smithi080.stdout:Waiting for mon... 2024-03-26T04:54:52.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:52 smithi080 ceph-mon[29315]: from='client.? 172.21.15.80:0/2388160777' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T04:54:52.215 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T04:54:52.217 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout id: dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:54:52.217 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T04:54:52.217 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 2024-03-26T04:54:52.217 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout services: 2024-03-26T04:54:52.217 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.558612s) 2024-03-26T04:54:52.217 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T04:54:52.217 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T04:54:52.217 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 2024-03-26T04:54:52.217 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout data: 2024-03-26T04:54:52.217 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T04:54:52.218 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T04:54:52.218 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T04:54:52.218 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T04:54:52.218 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 2024-03-26T04:54:52.218 INFO:teuthology.orchestra.run.smithi080.stdout:mon is available 2024-03-26T04:54:52.218 INFO:teuthology.orchestra.run.smithi080.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T04:54:53.278 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:53 smithi080 ceph-mon[29315]: from='client.? 172.21.15.80:0/1573372170' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T04:54:53.278 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:53 smithi080 ceph-mon[29315]: from='client.? 172.21.15.80:0/1573372170' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T04:54:53.282 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 2024-03-26T04:54:53.282 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T04:54:53.282 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout fsid = dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:54:53.282 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.80:3300,v1:172.21.15.80:6789] 2024-03-26T04:54:53.282 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T04:54:53.282 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T04:54:53.282 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T04:54:53.282 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T04:54:53.282 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 2024-03-26T04:54:53.283 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T04:54:53.283 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T04:54:53.283 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 2024-03-26T04:54:53.283 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T04:54:53.283 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T04:54:53.283 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T04:54:53.283 INFO:teuthology.orchestra.run.smithi080.stdout:Generating new minimal ceph.conf... 2024-03-26T04:54:54.313 INFO:teuthology.orchestra.run.smithi080.stdout:Restarting the monitor... 2024-03-26T04:54:54.327 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:54 smithi080 ceph-mon[29315]: from='client.? 172.21.15.80:0/3328299979' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:54:54.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:54 smithi080 systemd[1]: Stopping Ceph mon.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T04:54:54.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a[29311]: 2024-03-26T04:54:54.577+0000 7f896c29b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:54:54.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a[29311]: 2024-03-26T04:54:54.577+0000 7f896c29b700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T04:54:54.997 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:54 smithi080 podman[29561]: 2024-03-26 04:54:54.751233524 +0000 UTC m=+0.279152916 container died 7b670010817983505e7a2a5bc30f7563175f694c8c2c89608f10c97478f6d29f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T04:54:55.339 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 podman[29561]: 2024-03-26 04:54:55.093894362 +0000 UTC m=+0.621813745 container cleanup 7b670010817983505e7a2a5bc30f7563175f694c8c2c89608f10c97478f6d29f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-26T04:54:55.339 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 bash[29561]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a 2024-03-26T04:54:55.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 podman[29576]: 2024-03-26 04:54:55.344782199 +0000 UTC m=+0.594582727 container remove 7b670010817983505e7a2a5bc30f7563175f694c8c2c89608f10c97478f6d29f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T04:54:55.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T04:54:55.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 systemd[1]: Stopped Ceph mon.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T04:54:55.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 systemd[1]: Starting Ceph mon.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T04:54:55.910 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 podman[29672]: 2024-03-26 04:54:55.66234487 +0000 UTC m=+0.109258224 container create d7b1d029f188e01e68fd186f3f6072fc520eb5b4a9bd58256503a363d33db90b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:54:55.910 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 podman[29672]: 2024-03-26 04:54:55.570518115 +0000 UTC m=+0.017431484 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:54:55.910 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 podman[29672]: 2024-03-26 04:54:55.889877799 +0000 UTC m=+0.336791153 container init d7b1d029f188e01e68fd186f3f6072fc520eb5b4a9bd58256503a363d33db90b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T04:54:55.910 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 podman[29672]: 2024-03-26 04:54:55.89270632 +0000 UTC m=+0.339619684 container start d7b1d029f188e01e68fd186f3f6072fc520eb5b4a9bd58256503a363d33db90b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-5e70124, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T04:54:55.972 INFO:teuthology.orchestra.run.smithi080.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T04:54:56.205 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:54:56.205 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T04:54:56.205 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: pidfile_write: ignore empty --pid-file 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: load: jerasure load: lrc 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Git sha 0 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: DB SUMMARY 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: DB Session ID: R3RNMS3YG935XKR0LKR3 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 90819 ; 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:54:56.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.env: 0x56122ca2dae0 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.info_log: 0x56122e643920 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.statistics: (nil) 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.use_fsync: 0 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:54:56.207 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:54:56.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:54:56.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:54:56.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:54:56.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:54:56.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:54:56.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:54:56.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:54:56.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.db_log_dir: 2024-03-26T04:54:56.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.wal_dir: 2024-03-26T04:54:56.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:54:56.209 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.write_buffer_manager: 0x56122e6161e0 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:54:56.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:54:56.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:54:56.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:54:56.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.unordered_write: 0 2024-03-26T04:54:56.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:54:56.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:54:56.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:54:56.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:54:56.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.row_cache: None 2024-03-26T04:54:56.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.wal_filter: None 2024-03-26T04:54:56.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:54:56.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:54:56.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:54:56.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:54:56.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.wal_compression: 0 2024-03-26T04:54:56.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:54:56.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:54:56.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:54:56.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:54:56.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:54:56.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:54:56.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:54:56.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:54:56.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:54:56.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_open_files: -1 2024-03-26T04:54:56.213 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Compression algorithms supported: 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: kZSTD supported: 0 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:54:56.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.merge_operator: 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_filter: None 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56122e643a20) 2024-03-26T04:54:56.215 INFO:journalctl@ceph.mon.a.smithi080.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: index_type: 0 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: data_block_index_type: 0 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: index_shortening: 1 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: checksum: 4 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: no_block_cache: 0 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_cache: 0x56122e647090 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_cache_options: 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: capacity : 536870912 2024-03-26T04:54:56.216 INFO:journalctl@ceph.mon.a.smithi080.stdout: num_shard_bits : 4 2024-03-26T04:54:56.217 INFO:journalctl@ceph.mon.a.smithi080.stdout: strict_capacity_limit : 0 2024-03-26T04:54:56.217 INFO:journalctl@ceph.mon.a.smithi080.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:54:56.217 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_cache_compressed: (nil) 2024-03-26T04:54:56.217 INFO:journalctl@ceph.mon.a.smithi080.stdout: persistent_cache: (nil) 2024-03-26T04:54:56.217 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_size: 4096 2024-03-26T04:54:56.217 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_size_deviation: 10 2024-03-26T04:54:56.217 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_restart_interval: 16 2024-03-26T04:54:56.217 INFO:journalctl@ceph.mon.a.smithi080.stdout: index_block_restart_interval: 1 2024-03-26T04:54:56.217 INFO:journalctl@ceph.mon.a.smithi080.stdout: metadata_block_size: 4096 2024-03-26T04:54:56.217 INFO:journalctl@ceph.mon.a.smithi080.stdout: partition_filters: 0 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout: use_delta_encoding: 1 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout: filter_policy: bloomfilter 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout: whole_key_filtering: 1 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout: verify_compression: 0 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout: format_version: 5 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout: enable_index_compression: 1 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_align: 0 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout: max_auto_readahead_size: 262144 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout: prepopulate_block_cache: 0 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:54:56.218 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compression: NoCompression 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.num_levels: 7 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:54:56.219 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:54:56.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:54:56.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:54:56.222 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:54:56.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.ttl: 2592000 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T04:54:56.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e37e2973-47f5-40fc-9d37-c6b7f0a32cae 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428895921421, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428895922360, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 86814, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 278, "table_properties": {"data_size": 84896, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11580, "raw_average_key_size": 48, "raw_value_size": 78517, "raw_average_value_size": 331, "num_data_blocks": 11, "num_entries": 237, "num_filter_entries": 237, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711428895, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e37e2973-47f5-40fc-9d37-c6b7f0a32cae", "db_session_id": "R3RNMS3YG935XKR0LKR3", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428895922446, "job": 1, "event": "recovery_finished"} 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56122e738000 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: DB pointer 0x56122e722000 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** DB Stats ** 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:54:56.225 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: L0 2/0 86.61 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: Sum 2/0 86.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:54:56.226 INFO:journalctl@ceph.mon.a.smithi080.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 132.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative compaction: 0.00 GB write, 28.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval compaction: 0.00 GB write, 28.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: Block cache BinnedLRUCache@0x56122e647090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T04:54:56.227 INFO:journalctl@ceph.mon.a.smithi080.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: starting mon.a rank 0 at public addrs [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] at bind addrs [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: mon.a@-1(???) e1 preinit fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: mon.a@-1(???).mds e1 new map 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: mon.a@-1(???).mds e1 print_map 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout: e1 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout: legacy client fscid: -1 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout: No filesystems configured 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:54:56.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:54:56.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:54:56.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:54:56.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T04:54:56.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 bash[29672]: d7b1d029f188e01e68fd186f3f6072fc520eb5b4a9bd58256503a363d33db90b 2024-03-26T04:54:56.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 systemd[1]: Started Ceph mon.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T04:54:56.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T04:54:56.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: monmap e1: 1 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:54:56.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: fsmap 2024-03-26T04:54:56.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T04:54:56.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:55 smithi080 ceph-mon[29691]: mgrmap e1: no daemons active 2024-03-26T04:54:57.078 INFO:teuthology.orchestra.run.smithi080.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T04:54:57.100 INFO:teuthology.orchestra.run.smithi080.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:54:57.100 INFO:teuthology.orchestra.run.smithi080.stdout:Creating mgr... 2024-03-26T04:54:57.100 INFO:teuthology.orchestra.run.smithi080.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T04:54:57.101 INFO:teuthology.orchestra.run.smithi080.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T04:54:57.354 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mgr.y 2024-03-26T04:54:57.355 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stderr Failed to reset failed state of unit ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T04:54:57.503 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715.target.wants/ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@.service. 2024-03-26T04:54:57.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:57 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3881723844' entity='client.admin' 2024-03-26T04:54:58.035 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:54:58.035 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout disabled 2024-03-26T04:54:58.042 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:54:58.042 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout inactive 2024-03-26T04:54:58.043 INFO:teuthology.orchestra.run.smithi080.stdout:firewalld.service is not enabled 2024-03-26T04:54:58.043 INFO:teuthology.orchestra.run.smithi080.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T04:54:58.051 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:54:58.051 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout disabled 2024-03-26T04:54:58.058 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:54:58.058 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout inactive 2024-03-26T04:54:58.058 INFO:teuthology.orchestra.run.smithi080.stdout:firewalld.service is not enabled 2024-03-26T04:54:58.058 INFO:teuthology.orchestra.run.smithi080.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T04:54:58.059 INFO:teuthology.orchestra.run.smithi080.stdout:Waiting for mgr to start... 2024-03-26T04:54:58.059 INFO:teuthology.orchestra.run.smithi080.stdout:Waiting for mgr... 2024-03-26T04:54:58.120 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:54:58 smithi080 systemd[1]: Started Ceph mgr.y for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T04:54:58.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:54:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4134018009' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:54:58.871 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:54:58 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:54:58.457+0000 7f28a1a00200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:54:58.871 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:54:58 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:54:58.554+0000 7f28a1a00200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:54:59.201 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 2024-03-26T04:54:59.201 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:54:59.201 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "fsid": "dd09e9cc-eb2c-11ee-95d0-87774f69a715", 2024-03-26T04:54:59.201 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:54:59.201 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:54:59.201 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:54:59.201 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:54:59.201 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:54:59.201 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:54:59.201 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:54:59.202 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:54:59.203 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:54:59.203 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:54:59.203 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:54:59.203 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:54:59.203 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:54:59.203 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:54:59.206 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:54:59.206 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:54:59.206 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:54:59.207 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:54:51.107415+0000", 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:54:59.208 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:54:59.209 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:54:59.209 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:54:59.209 INFO:teuthology.orchestra.run.smithi080.stdout:mgr not available, waiting (1/15)... 2024-03-26T04:54:59.530 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:54:59 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:54:59.208+0000 7f28a1a00200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:54:59.530 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:54:59 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:54:59.301+0000 7f28a1a00200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:54:59.530 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:54:59 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:54:59.440+0000 7f28a1a00200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:54:59.870 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:54:59 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:54:59.534+0000 7f28a1a00200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:54:59.870 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:54:59 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:54:59.618+0000 7f28a1a00200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:55:00.163 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:54:59 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:54:59.878+0000 7f28a1a00200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:55:00.163 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:54:59 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:54:59.987+0000 7f28a1a00200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:55:00.163 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:00 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:00.074+0000 7f28a1a00200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:55:00.163 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:00 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:00.167+0000 7f28a1a00200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:55:00.620 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:00 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:00.341+0000 7f28a1a00200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:55:01.557 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:01 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:01.310+0000 7f28a1a00200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:55:01.557 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:01 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:01.444+0000 7f28a1a00200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:55:01.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:01 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3022194133' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:55:01.870 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:01 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:01.800+0000 7f28a1a00200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:55:02.266 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:01 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:01.890+0000 7f28a1a00200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:55:02.266 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:02 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:02.086+0000 7f28a1a00200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:55:02.266 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:02 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:02.271+0000 7f28a1a00200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:55:02.817 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:02 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:02.555+0000 7f28a1a00200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:55:02.817 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:02 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:02.822+0000 7f28a1a00200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:55:03.098 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:03.059+0000 7f28a1a00200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:55:04.370 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:03.878+0000 7f28a1a00200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:55:04.762 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:04.455+0000 7f28a1a00200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:55:04.762 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:04.553+0000 7f28a1a00200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:55:04.762 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:04.766+0000 7f28a1a00200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:55:05.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: Activating manager daemon y 2024-03-26T04:55:05.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: mgrmap e2: y(active, starting, since 0.00298265s) 2024-03-26T04:55:05.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: from='mgr.14100 172.21.15.80:0/2427381951' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:55:05.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: from='mgr.14100 172.21.15.80:0/2427381951' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:55:05.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: from='mgr.14100 172.21.15.80:0/2427381951' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:55:05.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: from='mgr.14100 172.21.15.80:0/2427381951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:55:05.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: from='mgr.14100 172.21.15.80:0/2427381951' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:55:05.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: Manager daemon y is now available 2024-03-26T04:55:05.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: from='mgr.14100 172.21.15.80:0/2427381951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:55:05.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: from='mgr.14100 172.21.15.80:0/2427381951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:55:05.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: from='mgr.14100 172.21.15.80:0/2427381951' entity='mgr.y' 2024-03-26T04:55:05.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: from='mgr.14100 172.21.15.80:0/2427381951' entity='mgr.y' 2024-03-26T04:55:05.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:04 smithi080 ceph-mon[29691]: from='mgr.14100 172.21.15.80:0/2427381951' entity='mgr.y' 2024-03-26T04:55:05.865 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 2024-03-26T04:55:05.865 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:55:05.865 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "fsid": "dd09e9cc-eb2c-11ee-95d0-87774f69a715", 2024-03-26T04:55:05.865 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:55:05.865 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:55:05.865 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:55:05.865 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:05.866 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:55:05.867 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:55:05.867 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:55:05.867 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:55:05.867 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:55:05.867 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:55:05.867 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:55:05.867 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:55:05.870 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:05.870 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:55:05.870 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:55:05.870 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:55:05.870 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:55:05.870 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:55:05.871 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:54:51.107415+0000", 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:55:05.872 INFO:teuthology.orchestra.run.smithi080.stdout:mgr not available, waiting (2/15)... 2024-03-26T04:55:07.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:06 smithi080 ceph-mon[29691]: mgrmap e3: y(active, since 1.00681s) 2024-03-26T04:55:09.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:08 smithi080 ceph-mon[29691]: mgrmap e4: y(active, since 3s) 2024-03-26T04:55:12.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:12 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2468207458' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:55:13.013 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 2024-03-26T04:55:13.013 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:55:13.013 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "fsid": "dd09e9cc-eb2c-11ee-95d0-87774f69a715", 2024-03-26T04:55:13.013 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:55:13.013 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:55:13.013 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:55:13.013 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:55:13.013 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:13.013 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:55:13.013 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:55:13.014 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:55:13.015 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:55:13.016 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:55:13.017 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:55:13.017 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:13.017 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:55:13.017 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:55:13.017 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:54:51.107415+0000", 2024-03-26T04:55:13.017 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:55:13.017 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:55:13.017 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:55:13.017 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:55:13.017 INFO:teuthology.orchestra.run.smithi080.stdout:mgr is available 2024-03-26T04:55:14.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:13 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1283827453' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T04:55:14.182 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 2024-03-26T04:55:14.182 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T04:55:14.182 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout fsid = dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:55:14.183 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T04:55:14.183 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T04:55:14.183 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T04:55:14.183 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T04:55:14.183 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 2024-03-26T04:55:14.183 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T04:55:14.183 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T04:55:14.183 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 2024-03-26T04:55:14.183 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T04:55:14.183 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T04:55:14.183 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T04:55:14.183 INFO:teuthology.orchestra.run.smithi080.stdout:Enabling cephadm module... 2024-03-26T04:55:15.202 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:14 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3132470457' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T04:55:15.202 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ignoring --setuser ceph since I am not root 2024-03-26T04:55:15.202 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ignoring --setgroup ceph since I am not root 2024-03-26T04:55:15.709 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:15 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:15.462+0000 7f0bfcd49200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:55:15.709 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:15 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:15.564+0000 7f0bfcd49200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:55:16.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:15 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3132470457' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T04:55:16.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:15 smithi080 ceph-mon[29691]: mgrmap e5: y(active, since 10s) 2024-03-26T04:55:16.462 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:16 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:16.229+0000 7f0bfcd49200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:55:16.463 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:16 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:16.321+0000 7f0bfcd49200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:55:16.463 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:16 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:16.467+0000 7f0bfcd49200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:55:16.670 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:55:16.671 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T04:55:16.671 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:55:16.671 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T04:55:16.671 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T04:55:16.671 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:55:16.671 INFO:teuthology.orchestra.run.smithi080.stdout:Waiting for the mgr to restart... 2024-03-26T04:55:16.671 INFO:teuthology.orchestra.run.smithi080.stdout:Waiting for mgr epoch 5... 2024-03-26T04:55:16.776 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:16 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:16.563+0000 7f0bfcd49200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:55:16.776 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:16 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:16.648+0000 7f0bfcd49200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:55:17.029 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:16 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2299937754' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T04:55:17.029 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:16 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:16.922+0000 7f0bfcd49200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:55:17.370 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:17 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:17.034+0000 7f0bfcd49200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:55:17.370 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:17 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:17.124+0000 7f0bfcd49200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:55:17.370 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:17 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:17.219+0000 7f0bfcd49200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:55:17.870 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:17 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:17.394+0000 7f0bfcd49200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:55:18.620 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:18 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:18.342+0000 7f0bfcd49200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:55:18.620 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:18 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:18.470+0000 7f0bfcd49200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:55:19.077 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:18 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:18.799+0000 7f0bfcd49200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:55:19.077 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:18 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:18.888+0000 7f0bfcd49200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:55:19.078 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:19 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:19.082+0000 7f0bfcd49200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:55:19.370 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:19 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:19.267+0000 7f0bfcd49200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:55:19.802 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:19 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:19.548+0000 7f0bfcd49200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:55:19.802 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:19 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:19.806+0000 7f0bfcd49200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:55:20.120 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:20.038+0000 7f0bfcd49200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:55:21.120 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:20.827+0000 7f0bfcd49200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:55:21.693 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:21.389+0000 7f0bfcd49200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:55:21.693 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:21.485+0000 7f0bfcd49200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:55:21.693 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:21.698+0000 7f0bfcd49200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:55:22.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: Active manager daemon y restarted 2024-03-26T04:55:22.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: Activating manager daemon y 2024-03-26T04:55:22.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T04:55:22.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: mgrmap e6: y(active, starting, since 0.00320955s) 2024-03-26T04:55:22.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:55:22.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:55:22.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:55:22.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:55:22.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:55:22.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: Manager daemon y is now available 2024-03-26T04:55:22.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:22.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:22.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:55:22.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:21 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:55:23.081 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:22 smithi080 ceph-mon[29691]: Found migration_current of "None". Setting to last migration. 2024-03-26T04:55:23.082 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:22 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:55:23.082 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:22 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:55:23.082 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:22 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:55:23.082 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:22 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:23.082 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:22 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:23.082 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:22 smithi080 ceph-mon[29691]: mgrmap e7: y(active, since 1.00576s) 2024-03-26T04:55:23.327 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:55:23.329 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T04:55:23.329 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T04:55:23.329 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:55:23.329 INFO:teuthology.orchestra.run.smithi080.stdout:mgr epoch 5 is available 2024-03-26T04:55:23.329 INFO:teuthology.orchestra.run.smithi080.stdout:Setting orchestrator backend to cephadm... 2024-03-26T04:55:23.989 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:23 smithi080 ceph-mon[29691]: [26/Mar/2024:04:55:22] ENGINE Bus STARTING 2024-03-26T04:55:23.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:23 smithi080 ceph-mon[29691]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T04:55:23.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:23 smithi080 ceph-mon[29691]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T04:55:23.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:23 smithi080 ceph-mon[29691]: [26/Mar/2024:04:55:22] ENGINE Serving on https://172.21.15.80:7150 2024-03-26T04:55:23.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:23 smithi080 ceph-mon[29691]: [26/Mar/2024:04:55:22] ENGINE Serving on http://172.21.15.80:8765 2024-03-26T04:55:23.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:23 smithi080 ceph-mon[29691]: [26/Mar/2024:04:55:22] ENGINE Bus STARTED 2024-03-26T04:55:23.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:23 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:55:24.901 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:24 smithi080 ceph-mon[29691]: mgrmap e8: y(active, since 2s) 2024-03-26T04:55:24.901 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:24 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:24.902 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:24 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:55:25.708 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T04:55:25.708 INFO:teuthology.orchestra.run.smithi080.stdout:Generating ssh key... 2024-03-26T04:55:26.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:25 smithi080 ceph-mon[29691]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:55:27.036 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-mon[29691]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:55:27.036 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:27.036 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:27.037 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: Generating public/private rsa key pair. 2024-03-26T04:55:27.037 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: Your identification has been saved in /tmp/tmpfrt6s_vv/key. 2024-03-26T04:55:27.037 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: Your public key has been saved in /tmp/tmpfrt6s_vv/key.pub. 2024-03-26T04:55:27.037 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: The key fingerprint is: 2024-03-26T04:55:27.037 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: SHA256:wc3qaVn/WcEutEaUXIhpfFT1mnQKti9nRBaQdEfhMW0 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:55:27.037 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: The key's randomart image is: 2024-03-26T04:55:27.037 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: +---[RSA 3072]----+ 2024-03-26T04:55:27.037 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: | ..*+=BB| 2024-03-26T04:55:27.037 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: | . o =o++oE| 2024-03-26T04:55:27.037 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: | o + +++oo| 2024-03-26T04:55:27.037 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: | o ..*.= | 2024-03-26T04:55:27.038 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: | S . .o=o | 2024-03-26T04:55:27.038 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: | . + .ooo .| 2024-03-26T04:55:27.038 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: | = o++..| 2024-03-26T04:55:27.038 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: | . .=.o | 2024-03-26T04:55:27.038 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: | o | 2024-03-26T04:55:27.038 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:26 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: +----[SHA256]-----+ 2024-03-26T04:55:27.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:27 smithi080 ceph-mon[29691]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:55:27.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:27 smithi080 ceph-mon[29691]: Generating ssh key... 2024-03-26T04:55:28.396 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNOVmm50kkDdJIEUJ4qz57Jjyh8xAvp7zXyEjqAn3WB32kUE7Kyc6kdI/4BOQzgVlbYOcCUQSduwSL8TgJA43lHhCFfI5zcNNy9Fpot1L4FJWvrih7mBh3iYR4cWas3CtjGVTPz9u3bqfxrxLLcHGWXSaejPuyfIyXjq4t/OY6vwBYrAF4Ki+oyQYZNxBO0QUTjMhR5CQlMOrH45DKFouLEVlyuLtrFpZcD/PJhvw6ZcdeT0073EvhN6x+vvxwOEv+TfflpGpXgC0BOr6p23/1l8mPV16ausxTS+CHl8frENIZO1T8/2zLihx+Ci2CLHovOQITtjliaOcQzhWq5m2FpLPWowvZayGQNmeGyOROCzfPLS8kCTN+Zt+JtkmWpxWNeIesuOkg1/VV4ubyratlA/DL4fvGavaS1IWfjHPrZwn/A+fa7T97ilVcy/hyTL04zBvnLFhgQoy/3LXNEcAPfNhJwYQV3jbUCyIChRXhvNNgKFDmRpz+jS+RocUwXI0= ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:55:28.397 INFO:teuthology.orchestra.run.smithi080.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:55:28.397 INFO:teuthology.orchestra.run.smithi080.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T04:55:28.397 INFO:teuthology.orchestra.run.smithi080.stdout:Adding host smithi080... 2024-03-26T04:55:29.026 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:28 smithi080 ceph-mon[29691]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:55:30.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:30 smithi080 ceph-mon[29691]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi080", "addr": "172.21.15.80", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:55:30.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:30 smithi080 ceph-mon[29691]: Deploying cephadm binary to smithi080 2024-03-26T04:55:33.855 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout Added host 'smithi080' with addr '172.21.15.80' 2024-03-26T04:55:33.856 INFO:teuthology.orchestra.run.smithi080.stdout:Deploying unmanaged mon service... 2024-03-26T04:55:34.460 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:34 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:34.461 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:34 smithi080 ceph-mon[29691]: Added host smithi080 2024-03-26T04:55:34.461 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:34 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:55:35.811 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:35 smithi080 ceph-mon[29691]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:55:35.811 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:35 smithi080 ceph-mon[29691]: Saving service mon spec with placement count:5 2024-03-26T04:55:35.811 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:35 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:36.114 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T04:55:36.114 INFO:teuthology.orchestra.run.smithi080.stdout:Deploying unmanaged mgr service... 2024-03-26T04:55:36.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:36 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:36.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:36 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:37.833 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T04:55:37.948 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:37 smithi080 ceph-mon[29691]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:55:37.948 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:37 smithi080 ceph-mon[29691]: Saving service mgr spec with placement count:2 2024-03-26T04:55:37.948 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:37 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:37.948 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:37 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:39.737 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:39 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2114973050' entity='client.admin' 2024-03-26T04:55:39.738 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:39 smithi080 ceph-mon[29691]: from='mgr.14118 172.21.15.80:0/723038550' entity='mgr.y' 2024-03-26T04:55:40.408 INFO:teuthology.orchestra.run.smithi080.stdout:Enabling the dashboard module... 2024-03-26T04:55:40.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:40 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3427390716' entity='client.admin' 2024-03-26T04:55:41.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:41 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/450553513' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T04:55:41.870 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:41 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ignoring --setuser ceph since I am not root 2024-03-26T04:55:41.870 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:41 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ignoring --setgroup ceph since I am not root 2024-03-26T04:55:42.370 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:42 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:42.299+0000 7f0bed13b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:55:42.695 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:42 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:42.402+0000 7f0bed13b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:55:42.999 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:42 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/450553513' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T04:55:42.999 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:42 smithi080 ceph-mon[29691]: mgrmap e9: y(active, since 20s) 2024-03-26T04:55:43.292 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:43 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:43.064+0000 7f0bed13b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:55:43.293 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:43 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:43.156+0000 7f0bed13b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:55:43.293 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:43 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:43.297+0000 7f0bed13b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:55:43.574 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:43 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:43.393+0000 7f0bed13b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:55:43.574 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:43 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:43.477+0000 7f0bed13b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:55:43.576 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:55:43.577 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T04:55:43.577 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:55:43.577 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T04:55:43.577 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T04:55:43.577 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:55:43.577 INFO:teuthology.orchestra.run.smithi080.stdout:Waiting for the mgr to restart... 2024-03-26T04:55:43.578 INFO:teuthology.orchestra.run.smithi080.stdout:Waiting for mgr epoch 9... 2024-03-26T04:55:43.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:43 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1238435146' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T04:55:43.836 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:43 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:43.750+0000 7f0bed13b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:55:44.120 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:43 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:43.863+0000 7f0bed13b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:55:44.120 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:43 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:43.951+0000 7f0bed13b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:55:44.120 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:44.052+0000 7f0bed13b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:55:44.620 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:44.227+0000 7f0bed13b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:55:45.620 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:45.172+0000 7f0bed13b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:55:45.620 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:45.300+0000 7f0bed13b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:55:45.907 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:45.628+0000 7f0bed13b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:55:45.907 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:45.716+0000 7f0bed13b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:55:45.907 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:45.912+0000 7f0bed13b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:55:46.370 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:46 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:46.096+0000 7f0bed13b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:55:46.628 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:46 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:46.379+0000 7f0bed13b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:55:46.628 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:46 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:46.633+0000 7f0bed13b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:55:47.120 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:46 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:46.867+0000 7f0bed13b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:55:48.120 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:47 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:47.657+0000 7f0bed13b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:55:48.523 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:48.216+0000 7f0bed13b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:55:48.523 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:48.313+0000 7f0bed13b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:55:48.523 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T04:55:48.528+0000 7f0bed13b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:55:48.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-mon[29691]: Active manager daemon y restarted 2024-03-26T04:55:48.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-mon[29691]: Activating manager daemon y 2024-03-26T04:55:48.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-mon[29691]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T04:55:48.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-mon[29691]: mgrmap e10: y(active, starting, since 0.00301941s) 2024-03-26T04:55:48.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:55:48.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:55:48.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:55:48.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:55:48.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:55:48.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-mon[29691]: Manager daemon y is now available 2024-03-26T04:55:48.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:55:48.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:55:49.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:49 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:55:49.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:49 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:55:49.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:49 smithi080 ceph-mon[29691]: mgrmap e11: y(active, since 1.00633s) 2024-03-26T04:55:50.472 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:55:50.473 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T04:55:50.473 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T04:55:50.473 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:55:50.473 INFO:teuthology.orchestra.run.smithi080.stdout:mgr epoch 9 is available 2024-03-26T04:55:50.473 INFO:teuthology.orchestra.run.smithi080.stdout:Generating a dashboard self-signed certificate... 2024-03-26T04:55:50.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:50 smithi080 ceph-mon[29691]: [26/Mar/2024:04:55:49] ENGINE Bus STARTING 2024-03-26T04:55:50.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:50 smithi080 ceph-mon[29691]: [26/Mar/2024:04:55:49] ENGINE Serving on http://172.21.15.80:8765 2024-03-26T04:55:50.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:50 smithi080 ceph-mon[29691]: [26/Mar/2024:04:55:49] ENGINE Serving on https://172.21.15.80:7150 2024-03-26T04:55:50.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:50 smithi080 ceph-mon[29691]: [26/Mar/2024:04:55:49] ENGINE Bus STARTED 2024-03-26T04:55:50.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:50 smithi080 ceph-mon[29691]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T04:55:50.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:50 smithi080 ceph-mon[29691]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T04:55:50.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:50 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:55:51.886 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T04:55:51.887 INFO:teuthology.orchestra.run.smithi080.stdout:Creating initial admin user... 2024-03-26T04:55:52.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:51 smithi080 ceph-mon[29691]: mgrmap e12: y(active, since 2s) 2024-03-26T04:55:52.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:51 smithi080 ceph-mon[29691]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:55:52.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:51 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:55:52.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:51 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:55:52.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:51 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:55:53.455 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Y4jueO3NCMwUL.wxd85MEuribTWWlhGUztNKy1S3RqZMxjhsIKw9q", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711428952, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T04:55:53.455 INFO:teuthology.orchestra.run.smithi080.stdout:Fetching dashboard port number... 2024-03-26T04:55:53.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:53 smithi080 ceph-mon[29691]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:55:53.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:53 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:55:54.973 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T04:55:54.982 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:55:54.982 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout disabled 2024-03-26T04:55:54.988 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:55:54.989 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stdout inactive 2024-03-26T04:55:54.989 INFO:teuthology.orchestra.run.smithi080.stdout:firewalld.service is not enabled 2024-03-26T04:55:54.989 INFO:teuthology.orchestra.run.smithi080.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T04:55:54.991 INFO:teuthology.orchestra.run.smithi080.stdout:Ceph Dashboard is now available at: 2024-03-26T04:55:54.991 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:55:54.991 INFO:teuthology.orchestra.run.smithi080.stdout: URL: https://smithi080.front.sepia.ceph.com:8443/ 2024-03-26T04:55:54.991 INFO:teuthology.orchestra.run.smithi080.stdout: User: admin 2024-03-26T04:55:54.991 INFO:teuthology.orchestra.run.smithi080.stdout: Password: 468kbtbv4u 2024-03-26T04:55:54.991 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:55:54.991 INFO:teuthology.orchestra.run.smithi080.stdout:Saving cluster configuration to /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config directory 2024-03-26T04:55:54.991 INFO:teuthology.orchestra.run.smithi080.stdout:Enabling autotune for osd_memory_target 2024-03-26T04:55:55.096 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:54 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1328044679' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T04:55:56.099 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:55 smithi080 ceph-mon[29691]: mgrmap e13: y(active, since 6s) 2024-03-26T04:55:57.470 INFO:teuthology.orchestra.run.smithi080.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T04:55:57.470 INFO:teuthology.orchestra.run.smithi080.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout: ceph telemetry on 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout:For more information see: 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:55:57.471 INFO:teuthology.orchestra.run.smithi080.stdout:Bootstrap complete. 2024-03-26T04:55:57.514 INFO:tasks.cephadm:Fetching config... 2024-03-26T04:55:57.514 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:55:57.514 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T04:55:57.529 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T04:55:57.530 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:55:57.530 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T04:55:57.584 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T04:55:57.585 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:55:57.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T04:55:57.651 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T04:55:57.651 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:55:57.651 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T04:55:57.706 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T04:55:57.706 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNOVmm50kkDdJIEUJ4qz57Jjyh8xAvp7zXyEjqAn3WB32kUE7Kyc6kdI/4BOQzgVlbYOcCUQSduwSL8TgJA43lHhCFfI5zcNNy9Fpot1L4FJWvrih7mBh3iYR4cWas3CtjGVTPz9u3bqfxrxLLcHGWXSaejPuyfIyXjq4t/OY6vwBYrAF4Ki+oyQYZNxBO0QUTjMhR5CQlMOrH45DKFouLEVlyuLtrFpZcD/PJhvw6ZcdeT0073EvhN6x+vvxwOEv+TfflpGpXgC0BOr6p23/1l8mPV16ausxTS+CHl8frENIZO1T8/2zLihx+Ci2CLHovOQITtjliaOcQzhWq5m2FpLPWowvZayGQNmeGyOROCzfPLS8kCTN+Zt+JtkmWpxWNeIesuOkg1/VV4ubyratlA/DL4fvGavaS1IWfjHPrZwn/A+fa7T97ilVcy/hyTL04zBvnLFhgQoy/3LXNEcAPfNhJwYQV3jbUCyIChRXhvNNgKFDmRpz+jS+RocUwXI0= ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:55:57.785 INFO:teuthology.orchestra.run.smithi080.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNOVmm50kkDdJIEUJ4qz57Jjyh8xAvp7zXyEjqAn3WB32kUE7Kyc6kdI/4BOQzgVlbYOcCUQSduwSL8TgJA43lHhCFfI5zcNNy9Fpot1L4FJWvrih7mBh3iYR4cWas3CtjGVTPz9u3bqfxrxLLcHGWXSaejPuyfIyXjq4t/OY6vwBYrAF4Ki+oyQYZNxBO0QUTjMhR5CQlMOrH45DKFouLEVlyuLtrFpZcD/PJhvw6ZcdeT0073EvhN6x+vvxwOEv+TfflpGpXgC0BOr6p23/1l8mPV16ausxTS+CHl8frENIZO1T8/2zLihx+Ci2CLHovOQITtjliaOcQzhWq5m2FpLPWowvZayGQNmeGyOROCzfPLS8kCTN+Zt+JtkmWpxWNeIesuOkg1/VV4ubyratlA/DL4fvGavaS1IWfjHPrZwn/A+fa7T97ilVcy/hyTL04zBvnLFhgQoy/3LXNEcAPfNhJwYQV3jbUCyIChRXhvNNgKFDmRpz+jS+RocUwXI0= ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:55:57.812 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNOVmm50kkDdJIEUJ4qz57Jjyh8xAvp7zXyEjqAn3WB32kUE7Kyc6kdI/4BOQzgVlbYOcCUQSduwSL8TgJA43lHhCFfI5zcNNy9Fpot1L4FJWvrih7mBh3iYR4cWas3CtjGVTPz9u3bqfxrxLLcHGWXSaejPuyfIyXjq4t/OY6vwBYrAF4Ki+oyQYZNxBO0QUTjMhR5CQlMOrH45DKFouLEVlyuLtrFpZcD/PJhvw6ZcdeT0073EvhN6x+vvxwOEv+TfflpGpXgC0BOr6p23/1l8mPV16ausxTS+CHl8frENIZO1T8/2zLihx+Ci2CLHovOQITtjliaOcQzhWq5m2FpLPWowvZayGQNmeGyOROCzfPLS8kCTN+Zt+JtkmWpxWNeIesuOkg1/VV4ubyratlA/DL4fvGavaS1IWfjHPrZwn/A+fa7T97ilVcy/hyTL04zBvnLFhgQoy/3LXNEcAPfNhJwYQV3jbUCyIChRXhvNNgKFDmRpz+jS+RocUwXI0= ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:55:57.858 INFO:teuthology.orchestra.run.smithi132.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNOVmm50kkDdJIEUJ4qz57Jjyh8xAvp7zXyEjqAn3WB32kUE7Kyc6kdI/4BOQzgVlbYOcCUQSduwSL8TgJA43lHhCFfI5zcNNy9Fpot1L4FJWvrih7mBh3iYR4cWas3CtjGVTPz9u3bqfxrxLLcHGWXSaejPuyfIyXjq4t/OY6vwBYrAF4Ki+oyQYZNxBO0QUTjMhR5CQlMOrH45DKFouLEVlyuLtrFpZcD/PJhvw6ZcdeT0073EvhN6x+vvxwOEv+TfflpGpXgC0BOr6p23/1l8mPV16ausxTS+CHl8frENIZO1T8/2zLihx+Ci2CLHovOQITtjliaOcQzhWq5m2FpLPWowvZayGQNmeGyOROCzfPLS8kCTN+Zt+JtkmWpxWNeIesuOkg1/VV4ubyratlA/DL4fvGavaS1IWfjHPrZwn/A+fa7T97ilVcy/hyTL04zBvnLFhgQoy/3LXNEcAPfNhJwYQV3jbUCyIChRXhvNNgKFDmRpz+jS+RocUwXI0= ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:55:57.873 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T04:55:58.118 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:55:58.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:55:57 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2858103420' entity='client.admin' 2024-03-26T04:56:01.690 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T04:56:01.690 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T04:56:02.110 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:56:02.232 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:01 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4247612643' entity='client.admin' 2024-03-26T04:56:03.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:03 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:03.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:03 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:03.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:03 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi080", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:03.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:03 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:03.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:03 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:03.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:03 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:04.507 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi132 2024-03-26T04:56:04.507 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T04:56:04.508 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/etc/ceph/ceph.conf 2024-03-26T04:56:04.526 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T04:56:04.526 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:56:04.584 INFO:tasks.cephadm:Adding host smithi132 to orchestrator... 2024-03-26T04:56:04.584 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch host add smithi132 2024-03-26T04:56:04.823 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:56:04.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:04 smithi080 ceph-mon[29691]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:56:04.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:04 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:04.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:04 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:04.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:04 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:04.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:04 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:04.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:04 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:04.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:04 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:04.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:04 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:05.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:05 smithi080 ceph-mon[29691]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-26T04:56:05.895 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:05 smithi080 ceph-mon[29691]: Updating smithi080:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:56:05.895 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:05 smithi080 ceph-mon[29691]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:56:05.895 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:05 smithi080 ceph-mon[29691]: Updating smithi080:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:56:07.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:06 smithi080 ceph-mon[29691]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:56:08.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:07 smithi080 ceph-mon[29691]: Deploying cephadm binary to smithi132 2024-03-26T04:56:09.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:08 smithi080 ceph-mon[29691]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:11.010 INFO:teuthology.orchestra.run.smithi080.stdout:Added host 'smithi132' with addr '172.21.15.132' 2024-03-26T04:56:11.624 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T04:56:11.856 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:56:11.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:11 smithi080 ceph-mon[29691]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:11.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:11.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:11.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:12.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:12 smithi080 ceph-mon[29691]: Added host smithi132 2024-03-26T04:56:12.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:13.634 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T04:56:13.634 INFO:teuthology.orchestra.run.smithi080.stdout:[{"addr": "172.21.15.80", "hostname": "smithi080", "labels": [], "status": ""}, {"addr": "172.21.15.132", "hostname": "smithi132", "labels": [], "status": ""}] 2024-03-26T04:56:13.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:13 smithi080 ceph-mon[29691]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:13.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:13 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:14.138 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T04:56:14.139 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T04:56:14.371 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:56:14.789 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:14 smithi080 ceph-mon[29691]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:56:15.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:15 smithi080 ceph-mon[29691]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:16.610 INFO:teuthology.orchestra.run.smithi080.stderr:adjusted tunables profile to default 2024-03-26T04:56:16.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:16 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4265823479' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:56:17.270 INFO:tasks.cephadm:Adding mon.a on smithi080 2024-03-26T04:56:17.270 INFO:tasks.cephadm:Adding mon.c on smithi080 2024-03-26T04:56:17.270 INFO:tasks.cephadm:Adding mon.b on smithi132 2024-03-26T04:56:17.270 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi080:172.21.15.80=a;smithi080:[v2:172.21.15.80:3301,v1:172.21.15.80:6790]=c;smithi132:172.21.15.132=b' 2024-03-26T04:56:17.503 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:56:17.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:17.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4265823479' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:56:17.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:56:17.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:17.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:17.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:17.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:17.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:17.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:17.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:17.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: Updating smithi132:/etc/ceph/ceph.conf 2024-03-26T04:56:17.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:17.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:17.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:18.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:18 smithi080 ceph-mon[29691]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:56:18.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:18 smithi080 ceph-mon[29691]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:56:18.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:18 smithi080 ceph-mon[29691]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:56:19.231 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled mon update... 2024-03-26T04:56:19.801 DEBUG:teuthology.orchestra.run.smithi080:mon.c> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T04:56:19.804 DEBUG:teuthology.orchestra.run.smithi132:mon.b> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T04:56:19.806 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:56:19.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:56:19.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:19 smithi080 ceph-mon[29691]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:19.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:19.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:19.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:19.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:19.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:19.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:19.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:20.892 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:56:21.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:20 smithi080 ceph-mon[29691]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi080:172.21.15.80=a;smithi080:[v2:172.21.15.80:3301,v1:172.21.15.80:6790]=c;smithi132:172.21.15.132=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:56:21.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:20 smithi080 ceph-mon[29691]: Saving service mon spec with placement smithi080:172.21.15.80=a;smithi080:[v2:172.21.15.80:3301,v1:172.21.15.80:6790]=c;smithi132:172.21.15.132=b;count:3 2024-03-26T04:56:21.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:20 smithi080 ceph-mon[29691]: Deploying daemon mon.b on smithi132 2024-03-26T04:56:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:21 smithi080 ceph-mon[29691]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:23.673 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:56:23.673 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"dd09e9cc-eb2c-11ee-95d0-87774f69a715","modified":"2024-03-26T04:54:48.169242Z","created":"2024-03-26T04:54:48.169242Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T04:56:23.675 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-03-26T04:56:23.983 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:23 smithi132 systemd[1]: Starting Ceph mon.b for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T04:56:24.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:23 smithi080 ceph-mon[29691]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:24.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 podman[33481]: 2024-03-26 04:56:24.4810642 +0000 UTC m=+0.093636539 container create 7747b853140c851ae358488216623b90ae62dcd5e46bc5429f501eaccf9b0600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-26T04:56:24.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 podman[33481]: 2024-03-26 04:56:24.405086279 +0000 UTC m=+0.017658617 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:56:25.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 podman[33481]: 2024-03-26 04:56:24.691954836 +0000 UTC m=+0.304527185 container init 7747b853140c851ae358488216623b90ae62dcd5e46bc5429f501eaccf9b0600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124) 2024-03-26T04:56:25.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 podman[33481]: 2024-03-26 04:56:24.695277844 +0000 UTC m=+0.307850183 container start 7747b853140c851ae358488216623b90ae62dcd5e46bc5429f501eaccf9b0600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T04:56:25.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: pidfile_write: ignore empty --pid-file 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: load: jerasure load: lrc 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Git sha 0 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: DB SUMMARY 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: DB Session ID: 3KUJHBQB2YT8K5D952EQ 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-03-26T04:56:25.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:56:25.048 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:56:25.048 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:56:25.048 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:56:25.048 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:56:25.048 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:56:25.048 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.env: 0x56203fe55ae0 2024-03-26T04:56:25.048 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:56:25.048 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.info_log: 0x562041c07920 2024-03-26T04:56:25.048 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:56:25.049 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.statistics: (nil) 2024-03-26T04:56:25.049 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.use_fsync: 0 2024-03-26T04:56:25.049 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:56:25.049 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:56:25.049 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:56:25.049 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:56:25.049 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:56:25.050 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:56:25.050 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:56:25.050 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:56:25.050 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:56:25.050 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:56:25.050 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:56:25.050 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.db_log_dir: 2024-03-26T04:56:25.050 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.wal_dir: 2024-03-26T04:56:25.050 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:56:25.050 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:56:25.050 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:56:25.050 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:56:25.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:56:25.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:56:25.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:56:25.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:56:25.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.write_buffer_manager: 0x562041bda1e0 2024-03-26T04:56:25.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:56:25.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:56:25.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:56:25.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:56:25.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:56:25.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:56:25.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.unordered_write: 0 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.row_cache: None 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.wal_filter: None 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.wal_compression: 0 2024-03-26T04:56:25.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:56:25.053 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_open_files: -1 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:56:25.054 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Compression algorithms supported: 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: kZSTD supported: 0 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:56:25.055 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.merge_operator: 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_filter: None 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562041c07a20) 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout: index_type: 0 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout: data_block_index_type: 0 2024-03-26T04:56:25.056 INFO:journalctl@ceph.mon.b.smithi132.stdout: index_shortening: 1 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: checksum: 4 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: no_block_cache: 0 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_cache: 0x562041c0b090 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_cache_options: 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: capacity : 536870912 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: num_shard_bits : 4 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: strict_capacity_limit : 0 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_cache_compressed: (nil) 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: persistent_cache: (nil) 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_size: 4096 2024-03-26T04:56:25.057 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_size_deviation: 10 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_restart_interval: 16 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: index_block_restart_interval: 1 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: metadata_block_size: 4096 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: partition_filters: 0 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: use_delta_encoding: 1 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: filter_policy: bloomfilter 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: whole_key_filtering: 1 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: verify_compression: 0 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: format_version: 5 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: enable_index_compression: 1 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_align: 0 2024-03-26T04:56:25.058 INFO:journalctl@ceph.mon.b.smithi132.stdout: max_auto_readahead_size: 262144 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout: prepopulate_block_cache: 0 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compression: NoCompression 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.num_levels: 7 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:56:25.059 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:56:25.060 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:56:25.061 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:56:25.062 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:56:25.062 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:56:25.062 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:56:25.062 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:56:25.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:56:25.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.ttl: 2592000 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:56:25.065 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 048feab8-bf87-4024-b1da-7e8fecb62c2a 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428984725060, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428984725469, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711428984, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "048feab8-bf87-4024-b1da-7e8fecb62c2a", "db_session_id": "3KUJHBQB2YT8K5D952EQ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428984725548, "job": 1, "event": "recovery_finished"} 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562041cfc000 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: DB pointer 0x562041ce6000 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout: ** DB Stats ** 2024-03-26T04:56:25.066 INFO:journalctl@ceph.mon.b.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: ** Compaction Stats [default] ** 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:56:25.067 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: ** Compaction Stats [default] ** 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:56:25.068 INFO:journalctl@ceph.mon.b.smithi132.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout: Block cache BinnedLRUCache@0x562041c0b090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: using public_addr v2:172.21.15.132:0/0 -> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: starting mon.b rank -1 at public addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] at bind addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(???) e0 preinit fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).mds e1 new map 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout: e1 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:56:25.069 INFO:journalctl@ceph.mon.b.smithi132.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout: legacy client fscid: -1 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout: No filesystems configured 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: Added host smithi132 2024-03-26T04:56:25.070 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4265823479' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4265823479' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:25.071 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:25.072 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: Updating smithi132:/etc/ceph/ceph.conf 2024-03-26T04:56:25.072 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:25.072 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:25.072 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:25.072 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:56:25.072 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:56:25.072 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:56:25.072 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:25.072 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:25.072 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:25.072 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:25.072 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:25.073 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:25.073 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:25.073 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:25.073 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi080:172.21.15.80=a;smithi080:[v2:172.21.15.80:3301,v1:172.21.15.80:6790]=c;smithi132:172.21.15.132=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:56:25.073 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: Saving service mon spec with placement smithi080:172.21.15.80=a;smithi080:[v2:172.21.15.80:3301,v1:172.21.15.80:6790]=c;smithi132:172.21.15.132=b;count:3 2024-03-26T04:56:25.073 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: Deploying daemon mon.b on smithi132 2024-03-26T04:56:25.073 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:25.073 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:25.073 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/1490283853' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:56:25.073 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 ceph-mon[33500]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T04:56:25.073 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 bash[33481]: 7747b853140c851ae358488216623b90ae62dcd5e46bc5429f501eaccf9b0600 2024-03-26T04:56:25.073 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:24 smithi132 systemd[1]: Started Ceph mon.b for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T04:56:25.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:24 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/1490283853' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:56:25.301 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:56:25.301 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:56:25.537 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:56:30.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:30.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: mon.a calling monitor election 2024-03-26T04:56:30.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:56:30.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:30.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:30.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: mon.b calling monitor election 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: monmap e2: 2 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: fsmap 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: mgrmap e13: y(active, since 41s) 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: overall HEALTH_OK 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:30.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:30.048 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:30.048 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:30.048 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:30.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:30.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: mon.a calling monitor election 2024-03-26T04:56:30.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:56:30.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:30.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:30.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:30.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: mon.b calling monitor election 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: monmap e2: 2 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: fsmap 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: mgrmap e13: y(active, since 41s) 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: overall HEALTH_OK 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:30.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:30.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:30.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:30.462 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:56:30.462 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":2,"fsid":"dd09e9cc-eb2c-11ee-95d0-87774f69a715","modified":"2024-03-26T04:56:24.749140Z","created":"2024-03-26T04:54:48.169242Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T04:56:30.465 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 2 2024-03-26T04:56:31.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:30 smithi132 ceph-mon[33500]: Deploying daemon mon.c on smithi080 2024-03-26T04:56:31.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:30 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/3014842213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:56:31.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:30 smithi132 ceph-mon[33500]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:31.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:30 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:31.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:30 smithi080 ceph-mon[29691]: Deploying daemon mon.c on smithi080 2024-03-26T04:56:31.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:30 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/3014842213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:56:31.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:30 smithi080 ceph-mon[29691]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:31.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:30 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:32.084 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:56:32.085 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:56:32.318 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:56:33.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:33 smithi080 systemd[1]: Started Ceph mon.c for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T04:56:38.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:56:38.555 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: mon.a calling monitor election 2024-03-26T04:56:38.555 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: mon.b calling monitor election 2024-03-26T04:56:38.555 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:38.555 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.555 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: mon.c calling monitor election 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: monmap e3: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: fsmap 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: mgrmap e13: y(active, since 49s) 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: overall HEALTH_OK 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:38.556 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:38 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:38.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:56:38.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: mon.a calling monitor election 2024-03-26T04:56:38.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: mon.b calling monitor election 2024-03-26T04:56:38.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:38.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:38.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: mon.c calling monitor election 2024-03-26T04:56:38.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:38.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T04:56:38.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: monmap e3: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:56:38.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: fsmap 2024-03-26T04:56:38.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:56:38.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: mgrmap e13: y(active, since 49s) 2024-03-26T04:56:38.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: overall HEALTH_OK 2024-03-26T04:56:38.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:38.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:38.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:56:38.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: mon.a calling monitor election 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: mon.b calling monitor election 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: mon.c calling monitor election 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T04:56:38.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: monmap e3: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:56:38.623 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: fsmap 2024-03-26T04:56:38.623 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:56:38.623 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: mgrmap e13: y(active, since 49s) 2024-03-26T04:56:38.623 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: overall HEALTH_OK 2024-03-26T04:56:38.623 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:38.623 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:38 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:39.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:39.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:39.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:39.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:39 smithi132 ceph-mon[33500]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:39.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:39.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:39.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:39.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:39.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:39.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:39 smithi080 ceph-mon[29691]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:39.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:39.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:39.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:39.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:39.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:39.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:39 smithi080 ceph-mon[40404]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:39.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:39.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:56:39.938 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-26T04:56:39.938 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":3,"fsid":"dd09e9cc-eb2c-11ee-95d0-87774f69a715","modified":"2024-03-26T04:56:33.199614Z","created":"2024-03-26T04:54:48.169242Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3301","nonce":0},{"type":"v1","addr":"172.21.15.80:6790","nonce":0}]},"addr":"172.21.15.80:6790/0","public_addr":"172.21.15.80:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-26T04:56:39.940 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 3 2024-03-26T04:56:40.632 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T04:56:40.632 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T04:56:40.867 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:56:40.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:40.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:40.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/4280442414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:56:40.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:40.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:56:40.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:40.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/4280442414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:40.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:40.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:56:40.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:41.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:41.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:41.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/4280442414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:56:41.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:41.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:41.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:56:41.047 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:41.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[40404]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-26T04:56:41.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[40404]: Updating smithi132:/etc/ceph/ceph.conf 2024-03-26T04:56:41.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[40404]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:56:41.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[40404]: Updating smithi080:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:56:41.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[40404]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T04:56:41.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[40404]: Reconfiguring daemon mon.a on smithi080 2024-03-26T04:56:41.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[40404]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:41.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[29691]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-26T04:56:41.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[29691]: Updating smithi132:/etc/ceph/ceph.conf 2024-03-26T04:56:41.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[29691]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:56:41.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[29691]: Updating smithi080:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:56:41.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[29691]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T04:56:41.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[29691]: Reconfiguring daemon mon.a on smithi080 2024-03-26T04:56:41.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:41 smithi080 ceph-mon[29691]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:42.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:41 smithi132 ceph-mon[33500]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-26T04:56:42.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:41 smithi132 ceph-mon[33500]: Updating smithi132:/etc/ceph/ceph.conf 2024-03-26T04:56:42.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:41 smithi132 ceph-mon[33500]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:56:42.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:41 smithi132 ceph-mon[33500]: Updating smithi080:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:56:42.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:41 smithi132 ceph-mon[33500]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T04:56:42.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:41 smithi132 ceph-mon[33500]: Reconfiguring daemon mon.a on smithi080 2024-03-26T04:56:42.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:41 smithi132 ceph-mon[33500]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:43.639 INFO:teuthology.orchestra.run.smithi080.stdout:# minimal ceph.conf for dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:56:43.639 INFO:teuthology.orchestra.run.smithi080.stdout:[global] 2024-03-26T04:56:43.639 INFO:teuthology.orchestra.run.smithi080.stdout: fsid = dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T04:56:43.639 INFO:teuthology.orchestra.run.smithi080.stdout: mon_host = [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] [v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0] 2024-03-26T04:56:43.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:43 smithi080 ceph-mon[40404]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:43.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:43 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:43.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:43 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:43.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:43 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:43.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:43 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:56:43.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:43 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:43.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:43 smithi080 ceph-mon[29691]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:43.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:43 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:43.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:43 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:43.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:43 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:43.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:43 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:56:43.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:43 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:44.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:43 smithi132 ceph-mon[33500]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:44.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:43 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:44.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:43 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:44.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:43 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:44.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:43 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:56:44.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:43 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:44.200 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T04:56:44.200 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:56:44.200 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:56:44.229 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:56:44.229 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:56:44.296 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T04:56:44.296 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:56:44.331 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T04:56:44.331 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:56:44.402 INFO:tasks.cephadm:Adding mgr.y on smithi080 2024-03-26T04:56:44.402 INFO:tasks.cephadm:Adding mgr.x on smithi132 2024-03-26T04:56:44.402 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch apply mgr '2;smithi080=y;smithi132=x' 2024-03-26T04:56:44.676 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:56:44.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:44 smithi080 ceph-mon[29691]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:56:44.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:44 smithi080 ceph-mon[29691]: Reconfiguring daemon mon.c on smithi080 2024-03-26T04:56:44.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:44 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2756239303' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:44.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:44 smithi080 ceph-mon[40404]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:56:44.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:44 smithi080 ceph-mon[40404]: Reconfiguring daemon mon.c on smithi080 2024-03-26T04:56:44.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:44 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2756239303' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:44.933 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:44 smithi132 ceph-mon[33500]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:56:44.933 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:44 smithi132 ceph-mon[33500]: Reconfiguring daemon mon.c on smithi080 2024-03-26T04:56:44.933 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:44 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2756239303' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:45.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:45 smithi080 ceph-mon[40404]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:45.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:45 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:45.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:45 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:45.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:45 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:45.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:45 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:56:45.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:45 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:45.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:45 smithi080 ceph-mon[29691]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:45.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:45 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:45.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:45 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:45.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:45 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:45.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:45 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:56:45.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:45 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:45.920 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:45 smithi132 ceph-mon[33500]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:45.920 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:45 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:45.920 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:45 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:45.920 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:45 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:56:45.920 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:45 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:56:45.920 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:45 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:46.379 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled mgr update... 2024-03-26T04:56:46.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:46 smithi132 ceph-mon[33500]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:56:46.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:46 smithi132 ceph-mon[33500]: Reconfiguring daemon mon.b on smithi132 2024-03-26T04:56:46.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:46 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:46.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:46 smithi080 ceph-mon[29691]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:56:46.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:46 smithi080 ceph-mon[29691]: Reconfiguring daemon mon.b on smithi132 2024-03-26T04:56:46.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:46 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:46.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:46 smithi080 ceph-mon[40404]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:56:46.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:46 smithi080 ceph-mon[40404]: Reconfiguring daemon mon.b on smithi132 2024-03-26T04:56:46.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:46 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:47.314 DEBUG:teuthology.orchestra.run.smithi132:mgr.x> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T04:56:47.317 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T04:56:47.317 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T04:56:47.317 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:56:47.336 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:56:47.336 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_1 2024-03-26T04:56:47.394 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:56:47.394 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:56:47.394 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-26T04:56:47.394 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:56:47.394 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:56:47.394 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-26 04:56:32.615534104 +0000 2024-03-26T04:56:47.394 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-26 04:51:13.504360411 +0000 2024-03-26T04:56:47.394 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-26 04:51:13.504360411 +0000 2024-03-26T04:56:47.394 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-26 04:51:13.504360411 +0000 2024-03-26T04:56:47.395 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:56:47.465 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-26T04:56:47.466 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-26T04:56:47.466 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000228876 s, 2.2 MB/s 2024-03-26T04:56:47.467 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:56:47.526 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_2 2024-03-26T04:56:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:56:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:56:47.584 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T04:56:47.584 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:56:47.584 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:56:47.584 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-26 04:56:32.615534104 +0000 2024-03-26T04:56:47.584 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-26 04:51:13.857352230 +0000 2024-03-26T04:56:47.584 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-26 04:51:13.857352230 +0000 2024-03-26T04:56:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-26 04:51:13.857352230 +0000 2024-03-26T04:56:47.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:56:47.655 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-26T04:56:47.655 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-26T04:56:47.655 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000246661 s, 2.1 MB/s 2024-03-26T04:56:47.657 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:56:47.715 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_3 2024-03-26T04:56:47.771 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:56:47.771 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:56:47.771 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-26T04:56:47.771 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:56:47.771 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:56:47.772 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-26 04:56:32.615534104 +0000 2024-03-26T04:56:47.772 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-26 04:51:14.207344118 +0000 2024-03-26T04:56:47.772 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-26 04:51:14.207344118 +0000 2024-03-26T04:56:47.772 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-26 04:51:14.207344118 +0000 2024-03-26T04:56:47.772 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:56:47.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:47 smithi080 ceph-mon[29691]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi080=y;smithi132=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:56:47.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:47 smithi080 ceph-mon[29691]: Saving service mgr spec with placement smithi080=y;smithi132=x;count:2 2024-03-26T04:56:47.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:47 smithi080 ceph-mon[29691]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:47.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:47 smithi080 ceph-mon[40404]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi080=y;smithi132=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:56:47.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:47 smithi080 ceph-mon[40404]: Saving service mgr spec with placement smithi080=y;smithi132=x;count:2 2024-03-26T04:56:47.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:47 smithi080 ceph-mon[40404]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:47.838 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-26T04:56:47.838 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-26T04:56:47.838 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000265794 s, 1.9 MB/s 2024-03-26T04:56:47.839 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:56:47.896 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_4 2024-03-26T04:56:47.951 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:56:47.951 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:56:47.951 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-26T04:56:47.951 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:56:47.951 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:56:47.952 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-26 04:56:32.616534082 +0000 2024-03-26T04:56:47.952 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-26 04:51:14.539336423 +0000 2024-03-26T04:56:47.952 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-26 04:51:14.539336423 +0000 2024-03-26T04:56:47.952 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-26 04:51:14.539336423 +0000 2024-03-26T04:56:47.952 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:56:48.016 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-26T04:56:48.016 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-26T04:56:48.017 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000197656 s, 2.6 MB/s 2024-03-26T04:56:48.018 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:56:48.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:47 smithi132 ceph-mon[33500]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi080=y;smithi132=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:56:48.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:47 smithi132 ceph-mon[33500]: Saving service mgr spec with placement smithi080=y;smithi132=x;count:2 2024-03-26T04:56:48.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:47 smithi132 ceph-mon[33500]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:48.073 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T04:56:48.073 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:56:48.089 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:56:48.090 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_1 2024-03-26T04:56:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:56:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:56:48.145 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-26T04:56:48.145 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:56:48.145 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:56:48.145 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-26 04:56:23.730885997 +0000 2024-03-26T04:56:48.146 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-26 04:51:13.510109689 +0000 2024-03-26T04:56:48.146 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-26 04:51:13.510109689 +0000 2024-03-26T04:56:48.146 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-26 04:51:13.510109689 +0000 2024-03-26T04:56:48.146 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:56:48.211 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-26T04:56:48.211 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-26T04:56:48.211 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000196418 s, 2.6 MB/s 2024-03-26T04:56:48.213 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:56:48.269 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_2 2024-03-26T04:56:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:56:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:56:48.324 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-26T04:56:48.325 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:56:48.325 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:56:48.325 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-26 04:56:23.731885970 +0000 2024-03-26T04:56:48.325 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-26 04:51:13.862100445 +0000 2024-03-26T04:56:48.325 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-26 04:51:13.862100445 +0000 2024-03-26T04:56:48.325 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-26 04:51:13.862100445 +0000 2024-03-26T04:56:48.325 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:56:48.389 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-26T04:56:48.389 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-26T04:56:48.390 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000200547 s, 2.6 MB/s 2024-03-26T04:56:48.391 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:56:48.446 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_3 2024-03-26T04:56:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:56:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:56:48.501 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-26T04:56:48.501 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:56:48.501 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:56:48.501 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-26 04:56:23.731885970 +0000 2024-03-26T04:56:48.501 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-26 04:51:14.198091622 +0000 2024-03-26T04:56:48.501 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-26 04:51:14.198091622 +0000 2024-03-26T04:56:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-26 04:51:14.198091622 +0000 2024-03-26T04:56:48.502 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:56:48.566 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-26T04:56:48.566 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-26T04:56:48.566 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000189113 s, 2.7 MB/s 2024-03-26T04:56:48.568 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:56:48.623 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_4 2024-03-26T04:56:48.677 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:56:48.678 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:56:48.678 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-26T04:56:48.678 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:56:48.678 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:56:48.678 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-26 04:56:23.732885945 +0000 2024-03-26T04:56:48.678 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-26 04:51:14.536082745 +0000 2024-03-26T04:56:48.678 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-26 04:51:14.536082745 +0000 2024-03-26T04:56:48.678 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-26 04:51:14.536082745 +0000 2024-03-26T04:56:48.679 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:56:48.742 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:48.742 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:48.742 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:48.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:48.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:48.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:48.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:56:48.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:56:48.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:56:48.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:48.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:48 smithi132 ceph-mon[33500]: Deploying daemon mgr.x on smithi132 2024-03-26T04:56:48.744 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-26T04:56:48.744 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-26T04:56:48.744 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.00020327 s, 2.5 MB/s 2024-03-26T04:56:48.746 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:56:48.801 INFO:tasks.cephadm:Deploying osd.0 on smithi080 with /dev/vg_nvme/lv_4... 2024-03-26T04:56:48.802 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T04:56:49.035 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:56:49.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:49.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:49.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:49.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:49.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:49.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:49.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:56:49.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:56:49.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:56:49.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:49.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[29691]: Deploying daemon mgr.x on smithi132 2024-03-26T04:56:49.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:49.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:49.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:49.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:49.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:49.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:49.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:56:49.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:56:49.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:56:49.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:49.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:48 smithi080 ceph-mon[40404]: Deploying daemon mgr.x on smithi132 2024-03-26T04:56:49.934 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:49 smithi132 ceph-mon[33500]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:50.060 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:49 smithi080 ceph-mon[29691]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:50.060 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:49 smithi080 ceph-mon[40404]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:50.203 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:50 smithi132 ceph-mgr[35371]: -- 172.21.15.132:0/3253926480 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562a1a89e1a0 con 0x562a1a874800 2024-03-26T04:56:50.545 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:50 smithi132 bash[35352]: 09998ae5d2ac79cc4768befc5ed43a49f9958cfa05e4480dd2daa6224a9c3e3f 2024-03-26T04:56:50.545 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:50 smithi132 systemd[1]: Started Ceph mgr.x for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T04:56:50.894 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T04:56:50.894 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T04:56:50.895 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-03-26T04:56:50.895 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-03-26T04:56:50.895 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011377 s, 922 MB/s 2024-03-26T04:56:50.895 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-03-26T04:56:51.045 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:50 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:50.705+0000 7fa91bbfd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:56:51.045 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:50 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:50.801+0000 7fa91bbfd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:56:51.438 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:51 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:51.438 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:51 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:51.438 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:51 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:51.438 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:51 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:51.438 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:51 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:51.438 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:51 smithi132 ceph-mon[33500]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:51.509 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:51 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:51.509 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:51 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:51.509 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:51 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:51.509 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:51 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:51.510 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:51 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:51.510 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:51 smithi080 ceph-mon[29691]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:51.510 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:51 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:51.510 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:51 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:51.510 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:51 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:51.510 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:51 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:51.510 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:51 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:51.510 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:51 smithi080 ceph-mon[40404]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:51.536 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi080:vg_nvme/lv_4 2024-03-26T04:56:51.700 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:51 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:51.488+0000 7fa91bbfd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:56:51.700 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:51 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:51.583+0000 7fa91bbfd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:56:51.774 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:56:52.045 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:51 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:51.737+0000 7fa91bbfd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:56:52.045 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:51 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:51.839+0000 7fa91bbfd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:56:52.045 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:51 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:51.927+0000 7fa91bbfd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:56:52.488 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:52.200+0000 7fa91bbfd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:56:52.488 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:52.311+0000 7fa91bbfd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:56:52.488 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:52.398+0000 7fa91bbfd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:56:52.488 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:52.493+0000 7fa91bbfd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:56:52.795 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:52.672+0000 7fa91bbfd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:56:52.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:52.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:52.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:52.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:52.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:52.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:56:52.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:56:52.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:52 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:52.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:52.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:56:52.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:56:52.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:52 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:53.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:53 smithi080 ceph-mon[29691]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:56:53.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:53 smithi080 ceph-mon[29691]: Reconfiguring daemon mgr.y on smithi080 2024-03-26T04:56:53.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:53 smithi080 ceph-mon[29691]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:53.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:53 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:53.807 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:53 smithi080 ceph-mon[40404]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:56:53.807 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:53 smithi080 ceph-mon[40404]: Reconfiguring daemon mgr.y on smithi080 2024-03-26T04:56:53.807 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:53 smithi080 ceph-mon[40404]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:53.807 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:53 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:54.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:53 smithi132 ceph-mon[33500]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:56:54.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:53 smithi132 ceph-mon[33500]: Reconfiguring daemon mgr.y on smithi080 2024-03-26T04:56:54.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:53 smithi132 ceph-mon[33500]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:54.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:53 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:54.046 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:53 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:53.639+0000 7fa91bbfd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:56:54.046 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:53 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:53.770+0000 7fa91bbfd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:56:54.390 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:54.106+0000 7fa91bbfd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:56:54.390 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:54.197+0000 7fa91bbfd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:56:54.390 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:54.395+0000 7fa91bbfd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:56:54.795 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:54.582+0000 7fa91bbfd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:56:55.121 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:54.865+0000 7fa91bbfd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:56:55.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:55.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:55.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:55.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:55.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:55.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:55.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-mon[33500]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:56:55.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:56:55.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-mon[33500]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:55.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:56:55.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:54 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:55.145 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:55.145 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:55.145 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:55.145 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:55.145 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:55.145 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:55.145 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[29691]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:56:55.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:56:55.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[29691]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:55.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:56:55.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:55.146 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:55.146 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:55.146 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:55.146 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:55.146 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:55.146 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:56:55.146 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[40404]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:56:55.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:56:55.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[40404]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:55.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:56:55.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:54 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:55.545 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:55.126+0000 7fa91bbfd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:56:55.545 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:55.360+0000 7fa91bbfd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:56:56.545 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:56.159+0000 7fa91bbfd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:56:57.039 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:56.730+0000 7fa91bbfd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:56:57.039 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:56.828+0000 7fa91bbfd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:56:57.039 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 04:56:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T04:56:57.044+0000 7fa91bbfd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:56:57.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[29691]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:57.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[29691]: Standby manager daemon x started 2024-03-26T04:56:57.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[29691]: from='mgr.? 172.21.15.132:0/3738502388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:56:57.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[29691]: from='mgr.? 172.21.15.132:0/3738502388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:56:57.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[29691]: from='mgr.? 172.21.15.132:0/3738502388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:56:57.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[29691]: from='mgr.? 172.21.15.132:0/3738502388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:56:57.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3649435208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366"}]: dispatch 2024-03-26T04:56:57.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[29691]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366"}]: dispatch 2024-03-26T04:56:57.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[29691]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366"}]': finished 2024-03-26T04:56:57.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[29691]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:56:57.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:56:57.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[40404]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:57.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[40404]: Standby manager daemon x started 2024-03-26T04:56:57.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[40404]: from='mgr.? 172.21.15.132:0/3738502388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:56:57.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[40404]: from='mgr.? 172.21.15.132:0/3738502388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:56:57.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[40404]: from='mgr.? 172.21.15.132:0/3738502388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:56:57.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[40404]: from='mgr.? 172.21.15.132:0/3738502388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:56:57.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3649435208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366"}]: dispatch 2024-03-26T04:56:57.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[40404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366"}]: dispatch 2024-03-26T04:56:57.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[40404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366"}]': finished 2024-03-26T04:56:57.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[40404]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:56:57.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:57 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:56:58.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:57 smithi132 ceph-mon[33500]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:58.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:57 smithi132 ceph-mon[33500]: Standby manager daemon x started 2024-03-26T04:56:58.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:57 smithi132 ceph-mon[33500]: from='mgr.? 172.21.15.132:0/3738502388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:56:58.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:57 smithi132 ceph-mon[33500]: from='mgr.? 172.21.15.132:0/3738502388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:56:58.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:57 smithi132 ceph-mon[33500]: from='mgr.? 172.21.15.132:0/3738502388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:56:58.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:57 smithi132 ceph-mon[33500]: from='mgr.? 172.21.15.132:0/3738502388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:56:58.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:57 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3649435208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366"}]: dispatch 2024-03-26T04:56:58.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:57 smithi132 ceph-mon[33500]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366"}]: dispatch 2024-03-26T04:56:58.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:57 smithi132 ceph-mon[33500]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366"}]': finished 2024-03-26T04:56:58.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:57 smithi132 ceph-mon[33500]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:56:58.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:57 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:56:58.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:58 smithi080 ceph-mon[29691]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-26T04:56:58.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:58 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:56:58.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1364792021' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:56:58.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:58 smithi080 ceph-mon[40404]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-26T04:56:58.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:58 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:56:58.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1364792021' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:56:59.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:58 smithi132 ceph-mon[33500]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-26T04:56:59.056 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:58 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:56:59.056 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1364792021' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:56:59.994 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:56:59 smithi080 ceph-mon[29691]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:56:59.994 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:56:59 smithi080 ceph-mon[40404]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:00.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:56:59 smithi132 ceph-mon[33500]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:01.625 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:01 smithi080 ceph-mon[29691]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:02.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:01 smithi132 ceph-mon[33500]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:02.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:01 smithi080 ceph-mon[40404]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:03.779 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:03 smithi080 ceph-mon[29691]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:03.779 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:03 smithi080 ceph-mon[40404]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:04.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:03 smithi132 ceph-mon[33500]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:04.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:04 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:57:04.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:04 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:04.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:04 smithi080 ceph-mon[29691]: Deploying daemon osd.0 on smithi080 2024-03-26T04:57:04.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:04 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:57:04.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:04 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:04.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:04 smithi080 ceph-mon[40404]: Deploying daemon osd.0 on smithi080 2024-03-26T04:57:05.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:04 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:57:05.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:04 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:05.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:04 smithi132 ceph-mon[33500]: Deploying daemon osd.0 on smithi080 2024-03-26T04:57:05.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:05 smithi080 ceph-mon[29691]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:05.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:05 smithi080 ceph-mon[40404]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:06.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:05 smithi132 ceph-mon[33500]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:08.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:07 smithi132 ceph-mon[33500]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:08.092 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:07 smithi080 ceph-mon[29691]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:08.092 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:07 smithi080 ceph-mon[40404]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:09.731 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:09 smithi080 ceph-mon[29691]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:09.731 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:09 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:09.731 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:09 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:09.731 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:09 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:09.731 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:09 smithi080 ceph-mon[40404]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:09.731 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:09 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:09.731 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:09 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:09.731 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:09 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:10.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:09 smithi132 ceph-mon[33500]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:10.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:09 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:10.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:09 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:10.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:09 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:11.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:11 smithi080 ceph-mon[29691]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:11.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:11 smithi080 ceph-mon[40404]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:11 smithi132 ceph-mon[33500]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:12.547 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 0 on host 'smithi080' 2024-03-26T04:57:12.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:12.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:12.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:12.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:12.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:12.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[29691]: from='osd.0 [v2:172.21.15.80:6802/1840686622,v1:172.21.15.80:6803/1840686622]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:57:12.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[29691]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:57:12.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:12.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:12.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:12.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:12.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:12.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:12.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:12.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:12.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[40404]: from='osd.0 [v2:172.21.15.80:6802/1840686622,v1:172.21.15.80:6803/1840686622]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:57:12.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[40404]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:57:12.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:12.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:12.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:12 smithi132 ceph-mon[33500]: from='osd.0 [v2:172.21.15.80:6802/1840686622,v1:172.21.15.80:6803/1840686622]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:57:13.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:12 smithi132 ceph-mon[33500]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:57:13.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:13.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:13.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:13.253 DEBUG:teuthology.orchestra.run.smithi080:osd.0> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T04:57:13.255 INFO:tasks.cephadm:Deploying osd.1 on smithi080 with /dev/vg_nvme/lv_3... 2024-03-26T04:57:13.255 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T04:57:13.814 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-mon[29691]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:13.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-mon[29691]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:57:13.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-mon[29691]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:57:13.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:13.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-mon[29691]: from='osd.0 [v2:172.21.15.80:6802/1840686622,v1:172.21.15.80:6803/1840686622]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:57:13.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-mon[29691]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:57:13.815 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-mon[40404]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:13.815 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-mon[40404]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:57:13.815 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-mon[40404]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:57:13.816 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:13.816 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-mon[40404]: from='osd.0 [v2:172.21.15.80:6802/1840686622,v1:172.21.15.80:6803/1840686622]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:57:13.816 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-mon[40404]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:57:13.816 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0[44401]: 2024-03-26T04:57:13.693+0000 7f10843ff700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T04:57:13.816 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 04:57:13 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0[44401]: 2024-03-26T04:57:13.695+0000 7f10801f2700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:57:14.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:13 smithi132 ceph-mon[33500]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:14.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:13 smithi132 ceph-mon[33500]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:57:14.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:13 smithi132 ceph-mon[33500]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:57:14.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:13 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:14.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:13 smithi132 ceph-mon[33500]: from='osd.0 [v2:172.21.15.80:6802/1840686622,v1:172.21.15.80:6803/1840686622]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:57:14.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:13 smithi132 ceph-mon[33500]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:57:14.453 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:57:15.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:14 smithi132 ceph-mon[33500]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-26T04:57:15.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:14 smithi132 ceph-mon[33500]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:57:15.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:14 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:15.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:14 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:15.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:14 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:15.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:14 smithi080 ceph-mon[29691]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-26T04:57:15.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:14 smithi080 ceph-mon[29691]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:57:15.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:14 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:15.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:14 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:15.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:14 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:14 smithi080 ceph-mon[40404]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-26T04:57:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:14 smithi080 ceph-mon[40404]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:57:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:14 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:14 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:14 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:15.760 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:15 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:57:15.761 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:15 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:57:15.761 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:15 smithi080 ceph-mon[40404]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:15.761 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:15 smithi080 ceph-mon[40404]: osd.0 [v2:172.21.15.80:6802/1840686622,v1:172.21.15.80:6803/1840686622] boot 2024-03-26T04:57:15.761 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:15 smithi080 ceph-mon[40404]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:57:15.761 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:15 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:15.761 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:15 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:57:15.761 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:15 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:57:15.761 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:15 smithi080 ceph-mon[29691]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:15.761 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:15 smithi080 ceph-mon[29691]: osd.0 [v2:172.21.15.80:6802/1840686622,v1:172.21.15.80:6803/1840686622] boot 2024-03-26T04:57:15.761 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:15 smithi080 ceph-mon[29691]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:57:15.762 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:15 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:16.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:15 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:57:16.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:15 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:57:16.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:15 smithi132 ceph-mon[33500]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:57:16.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:15 smithi132 ceph-mon[33500]: osd.0 [v2:172.21.15.80:6802/1840686622,v1:172.21.15.80:6803/1840686622] boot 2024-03-26T04:57:16.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:15 smithi132 ceph-mon[33500]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:57:16.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:15 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:57:16.153 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T04:57:16.153 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T04:57:16.153 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-03-26T04:57:16.153 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-03-26T04:57:16.153 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135337 s, 775 MB/s 2024-03-26T04:57:16.154 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-03-26T04:57:16.805 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi080:vg_nvme/lv_3 2024-03-26T04:57:17.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:16 smithi132 ceph-mon[33500]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:57:17.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:16 smithi080 ceph-mon[40404]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:57:17.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:16 smithi080 ceph-mon[29691]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:57:17.587 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:57:17.792 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[40404]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:17.792 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:17.792 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:17.792 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:57:17.792 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:17.793 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:17.793 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:17.793 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:17.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[29691]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:17.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:17.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:17.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:57:17.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:17.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:17.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:17.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:18.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:17 smithi132 ceph-mon[33500]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:18.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:17 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:18.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:17 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:18.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:17 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:57:18.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:17 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:18.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:17 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:18.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:17 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:18.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:17 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:18.748 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:18 smithi080 ceph-mon[29691]: Detected new or changed devices on smithi080 2024-03-26T04:57:18.748 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:18 smithi080 ceph-mon[29691]: Adjusting osd_memory_target on smithi080 to 16064M 2024-03-26T04:57:18.749 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:18 smithi080 ceph-mon[40404]: Detected new or changed devices on smithi080 2024-03-26T04:57:18.749 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:18 smithi080 ceph-mon[40404]: Adjusting osd_memory_target on smithi080 to 16064M 2024-03-26T04:57:19.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:18 smithi132 ceph-mon[33500]: Detected new or changed devices on smithi080 2024-03-26T04:57:19.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:18 smithi132 ceph-mon[33500]: Adjusting osd_memory_target on smithi080 to 16064M 2024-03-26T04:57:20.002 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:19 smithi080 ceph-mon[29691]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:20.002 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:57:20.002 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:57:20.002 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:20.002 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:19 smithi080 ceph-mon[40404]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:20.002 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:19 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:57:20.002 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:19 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:57:20.002 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:19 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:20.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:19 smithi132 ceph-mon[33500]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:20.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:19 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:57:20.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:19 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:57:20.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:19 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:20.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:20 smithi080 ceph-mon[29691]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:57:20.991 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:20 smithi080 ceph-mon[40404]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:57:21.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:20 smithi132 ceph-mon[33500]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:57:22.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:21 smithi132 ceph-mon[33500]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:21 smithi080 ceph-mon[29691]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:22.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:21 smithi080 ceph-mon[40404]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:23.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:22 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3602613572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23"}]: dispatch 2024-03-26T04:57:23.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:22 smithi132 ceph-mon[33500]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23"}]: dispatch 2024-03-26T04:57:23.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:22 smithi132 ceph-mon[33500]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23"}]': finished 2024-03-26T04:57:23.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:22 smithi132 ceph-mon[33500]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:57:23.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:22 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:23.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:22 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3506031047' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:57:23.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:22 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3602613572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23"}]: dispatch 2024-03-26T04:57:23.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:22 smithi080 ceph-mon[29691]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23"}]: dispatch 2024-03-26T04:57:23.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:22 smithi080 ceph-mon[29691]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23"}]': finished 2024-03-26T04:57:23.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:22 smithi080 ceph-mon[29691]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:57:23.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:22 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:23.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:22 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3506031047' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:57:23.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:22 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3602613572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23"}]: dispatch 2024-03-26T04:57:23.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:22 smithi080 ceph-mon[40404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23"}]: dispatch 2024-03-26T04:57:23.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:22 smithi080 ceph-mon[40404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23"}]': finished 2024-03-26T04:57:23.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:22 smithi080 ceph-mon[40404]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:57:23.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:22 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:23.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:22 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3506031047' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:57:24.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:23 smithi132 ceph-mon[33500]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:24.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:23 smithi080 ceph-mon[29691]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:24.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:23 smithi080 ceph-mon[40404]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:26.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:25 smithi080 ceph-mon[29691]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:26.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:25 smithi080 ceph-mon[40404]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:26.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:25 smithi132 ceph-mon[33500]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:27.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:27 smithi080 ceph-mon[40404]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:27.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:27 smithi080 ceph-mon[29691]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:28.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:27 smithi132 ceph-mon[33500]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:29.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:28 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:57:29.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:28 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:29.097 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:28 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:57:29.097 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:28 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:29.097 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:28 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:57:29.097 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:28 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:30.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:29 smithi132 ceph-mon[33500]: Deploying daemon osd.1 on smithi080 2024-03-26T04:57:30.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:29 smithi132 ceph-mon[33500]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:30.103 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:29 smithi080 ceph-mon[29691]: Deploying daemon osd.1 on smithi080 2024-03-26T04:57:30.103 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:29 smithi080 ceph-mon[29691]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:30.104 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:29 smithi080 ceph-mon[40404]: Deploying daemon osd.1 on smithi080 2024-03-26T04:57:30.104 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:29 smithi080 ceph-mon[40404]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:31.845 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:31 smithi080 ceph-mon[29691]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:31.845 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:31 smithi080 ceph-mon[40404]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:32.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:31 smithi132 ceph-mon[33500]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:34.105 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:33 smithi080 ceph-mon[29691]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:34.105 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:33 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:34.105 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:33 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:34.105 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:33 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:34.105 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:33 smithi080 ceph-mon[40404]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:34.105 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:33 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:34.105 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:33 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:34.105 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:33 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:34.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:33 smithi132 ceph-mon[33500]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:34.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:33 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:34.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:33 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:34.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:33 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:36.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:35 smithi080 ceph-mon[29691]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:36.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:35 smithi080 ceph-mon[40404]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:36.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:35 smithi132 ceph-mon[33500]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:36.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:36 smithi080 ceph-mon[29691]: from='osd.1 [v2:172.21.15.80:6810/1726833996,v1:172.21.15.80:6811/1726833996]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:57:36.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:36 smithi080 ceph-mon[29691]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:57:36.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:36 smithi080 ceph-mon[40404]: from='osd.1 [v2:172.21.15.80:6810/1726833996,v1:172.21.15.80:6811/1726833996]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:57:36.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:36 smithi080 ceph-mon[40404]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:57:36.961 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 1 on host 'smithi080' 2024-03-26T04:57:37.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:36 smithi132 ceph-mon[33500]: from='osd.1 [v2:172.21.15.80:6810/1726833996,v1:172.21.15.80:6811/1726833996]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:57:37.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:36 smithi132 ceph-mon[33500]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:57:37.665 DEBUG:teuthology.orchestra.run.smithi080:osd.1> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T04:57:37.668 INFO:tasks.cephadm:Deploying osd.2 on smithi080 with /dev/vg_nvme/lv_2... 2024-03-26T04:57:37.668 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T04:57:37.892 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[29691]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:37.892 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[29691]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:57:37.892 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[29691]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:57:37.892 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:37.892 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[29691]: from='osd.1 [v2:172.21.15.80:6810/1726833996,v1:172.21.15.80:6811/1726833996]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:57:37.892 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[29691]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:57:37.893 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:37.893 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:37.893 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:37.893 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:37.893 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[40404]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:37.894 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[40404]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:57:37.894 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[40404]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:57:37.894 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:37.894 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[40404]: from='osd.1 [v2:172.21.15.80:6810/1726833996,v1:172.21.15.80:6811/1726833996]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:57:37.894 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[40404]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:57:37.894 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:37.894 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:37.894 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:37.894 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:37 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:38.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:37 smithi132 ceph-mon[33500]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:38.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:37 smithi132 ceph-mon[33500]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:57:38.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:37 smithi132 ceph-mon[33500]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:57:38.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:37 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:38.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:37 smithi132 ceph-mon[33500]: from='osd.1 [v2:172.21.15.80:6810/1726833996,v1:172.21.15.80:6811/1726833996]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:57:38.296 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:37 smithi132 ceph-mon[33500]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:57:38.296 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:37 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:38.296 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:37 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:38.296 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:37 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:38.296 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:37 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:38.619 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1[50406]: 2024-03-26T04:57:38.259+0000 7f3d19c0f700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T04:57:38.620 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1[50406]: 2024-03-26T04:57:38.262+0000 7f3d119fa700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:57:38.879 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:57:39.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-mon[29691]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-26T04:57:39.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-mon[29691]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:57:39.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:39.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:39.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-mon[29691]: from='osd.1 ' entity='osd.1' 2024-03-26T04:57:39.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-mon[29691]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:39.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-mon[40404]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-26T04:57:39.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-mon[40404]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:57:39.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:39.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:39.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-mon[40404]: from='osd.1 ' entity='osd.1' 2024-03-26T04:57:39.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:38 smithi080 ceph-mon[40404]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:39.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:38 smithi132 ceph-mon[33500]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-26T04:57:39.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:38 smithi132 ceph-mon[33500]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:57:39.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:38 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:39.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:38 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:39.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:38 smithi132 ceph-mon[33500]: from='osd.1 ' entity='osd.1' 2024-03-26T04:57:39.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:38 smithi132 ceph-mon[33500]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:57:40.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:39 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:57:40.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:39 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:57:40.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:40.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:39 smithi080 ceph-mon[29691]: osd.1 [v2:172.21.15.80:6810/1726833996,v1:172.21.15.80:6811/1726833996] boot 2024-03-26T04:57:40.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:39 smithi080 ceph-mon[29691]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:57:40.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:40.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:39 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:57:40.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:39 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:57:40.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:40.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:39 smithi080 ceph-mon[40404]: osd.1 [v2:172.21.15.80:6810/1726833996,v1:172.21.15.80:6811/1726833996] boot 2024-03-26T04:57:40.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:39 smithi080 ceph-mon[40404]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:40.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:39 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:57:40.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:39 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:57:40.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:40.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:39 smithi132 ceph-mon[33500]: osd.1 [v2:172.21.15.80:6810/1726833996,v1:172.21.15.80:6811/1726833996] boot 2024-03-26T04:57:40.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:39 smithi132 ceph-mon[33500]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:57:40.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:57:41.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:57:41.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:41.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:57:41.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:57:41.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:41.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:41.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:41.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:41.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:41.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:41.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:57:41.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:41.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:57:41.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:57:41.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:41.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:41.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:41.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:41.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:41.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:41.752 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T04:57:41.752 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T04:57:41.753 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-03-26T04:57:41.753 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-03-26T04:57:41.753 INFO:teuthology.orchestra.run.smithi080.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112547 s, 932 MB/s 2024-03-26T04:57:41.753 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-03-26T04:57:42.319 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:42 smithi080 ceph-mon[29691]: Detected new or changed devices on smithi080 2024-03-26T04:57:42.319 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:42 smithi080 ceph-mon[29691]: Adjusting osd_memory_target on smithi080 to 8032M 2024-03-26T04:57:42.319 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:42 smithi080 ceph-mon[40404]: Detected new or changed devices on smithi080 2024-03-26T04:57:42.319 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:42 smithi080 ceph-mon[40404]: Adjusting osd_memory_target on smithi080 to 8032M 2024-03-26T04:57:42.347 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi080:vg_nvme/lv_2 2024-03-26T04:57:42.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:42 smithi132 ceph-mon[33500]: Detected new or changed devices on smithi080 2024-03-26T04:57:42.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:42 smithi132 ceph-mon[33500]: Adjusting osd_memory_target on smithi080 to 8032M 2024-03-26T04:57:42.583 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:57:43.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:43 smithi132 ceph-mon[33500]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:43.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:43 smithi080 ceph-mon[29691]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:43.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:43 smithi080 ceph-mon[40404]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:44.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:44 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:57:44.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:44 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:57:44.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:44 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:44.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:44 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:57:44.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:44 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:57:44.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:44 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:44.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:44 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:57:44.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:44 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:57:44.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:44 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:45.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:45 smithi080 ceph-mon[29691]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:57:45.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:45 smithi080 ceph-mon[29691]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:45.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:45 smithi080 ceph-mon[40404]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:57:45.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:45 smithi080 ceph-mon[40404]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:45.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:45 smithi132 ceph-mon[33500]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:57:45.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:45 smithi132 ceph-mon[33500]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:47.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:46 smithi132 ceph-mon[33500]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:47.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:46 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3910120122' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "536bec9b-9636-4e67-b452-ed1a55c22ab6"}]: dispatch 2024-03-26T04:57:47.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:46 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3910120122' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "536bec9b-9636-4e67-b452-ed1a55c22ab6"}]': finished 2024-03-26T04:57:47.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:46 smithi132 ceph-mon[33500]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:57:47.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:46 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:57:47.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:46 smithi080 ceph-mon[29691]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:47.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:46 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3910120122' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "536bec9b-9636-4e67-b452-ed1a55c22ab6"}]: dispatch 2024-03-26T04:57:47.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:46 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3910120122' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "536bec9b-9636-4e67-b452-ed1a55c22ab6"}]': finished 2024-03-26T04:57:47.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:46 smithi080 ceph-mon[29691]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:57:47.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:46 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:57:47.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:46 smithi080 ceph-mon[40404]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:47.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:46 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3910120122' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "536bec9b-9636-4e67-b452-ed1a55c22ab6"}]: dispatch 2024-03-26T04:57:47.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:46 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3910120122' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "536bec9b-9636-4e67-b452-ed1a55c22ab6"}]': finished 2024-03-26T04:57:47.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:46 smithi080 ceph-mon[40404]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:57:47.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:46 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:57:48.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:47 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/259873105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:57:48.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:47 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/259873105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:57:48.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:47 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/259873105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:57:49.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:48 smithi132 ceph-mon[33500]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:49.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:48 smithi080 ceph-mon[29691]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:49.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:48 smithi080 ceph-mon[40404]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:51.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:51 smithi080 ceph-mon[29691]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:51.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:51 smithi080 ceph-mon[40404]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:52.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:51 smithi132 ceph-mon[33500]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:53.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:53 smithi080 ceph-mon[40404]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:53.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:53 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:57:53.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:53 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:53.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:53 smithi080 ceph-mon[29691]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:53.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:53 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:57:53.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:53 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:54.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:53 smithi132 ceph-mon[33500]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:54.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:53 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:57:54.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:53 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:54.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:54 smithi080 ceph-mon[29691]: Deploying daemon osd.2 on smithi080 2024-03-26T04:57:54.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:54 smithi080 ceph-mon[40404]: Deploying daemon osd.2 on smithi080 2024-03-26T04:57:55.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:54 smithi132 ceph-mon[33500]: Deploying daemon osd.2 on smithi080 2024-03-26T04:57:56.038 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:55 smithi080 ceph-mon[29691]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:56.038 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:55 smithi080 ceph-mon[40404]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:56.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:55 smithi132 ceph-mon[33500]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:57.696 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:57 smithi080 ceph-mon[29691]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:57.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:57 smithi080 ceph-mon[40404]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:58.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:57 smithi132 ceph-mon[33500]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:59.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:58 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:59.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:58 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:59.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:58 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:59.059 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:58 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:59.059 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:58 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:59.059 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:58 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:59.059 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:58 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:59.060 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:58 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:59.060 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:58 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:57:59.965 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:57:59 smithi080 ceph-mon[29691]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:57:59.965 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:57:59 smithi080 ceph-mon[40404]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:58:00.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:57:59 smithi132 ceph-mon[33500]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:58:01.486 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 2 on host 'smithi080' 2024-03-26T04:58:01.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:01 smithi080 ceph-mon[29691]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:58:01.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:01 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:01.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:01 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:01.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:01 smithi080 ceph-mon[29691]: from='osd.2 [v2:172.21.15.80:6818/128713757,v1:172.21.15.80:6819/128713757]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:58:01.712 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:01 smithi080 ceph-mon[29691]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:58:01.712 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:01 smithi080 ceph-mon[40404]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:58:01.712 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:01 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:01.712 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:01 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:01.712 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:01 smithi080 ceph-mon[40404]: from='osd.2 [v2:172.21.15.80:6818/128713757,v1:172.21.15.80:6819/128713757]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:58:01.712 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:01 smithi080 ceph-mon[40404]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:58:02.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:01 smithi132 ceph-mon[33500]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:58:02.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:01 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:02.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:01 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:02.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:01 smithi132 ceph-mon[33500]: from='osd.2 [v2:172.21.15.80:6818/128713757,v1:172.21.15.80:6819/128713757]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:58:02.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:01 smithi132 ceph-mon[33500]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:58:02.431 DEBUG:teuthology.orchestra.run.smithi080:osd.2> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T04:58:02.433 INFO:tasks.cephadm:Deploying osd.3 on smithi080 with /dev/vg_nvme/lv_1... 2024-03-26T04:58:02.433 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T04:58:02.729 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:58:03.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:03 smithi132 ceph-mon[33500]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:58:03.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:03 smithi132 ceph-mon[33500]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:58:03.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:03 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:03.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:03 smithi132 ceph-mon[33500]: from='osd.2 [v2:172.21.15.80:6818/128713757,v1:172.21.15.80:6819/128713757]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:58:03.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:03 smithi132 ceph-mon[33500]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:58:03.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:03 smithi132 ceph-mon[33500]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:58:03.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:03 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:03.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:03 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:03.829 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[40404]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:58:03.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[40404]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:58:03.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:03.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[40404]: from='osd.2 [v2:172.21.15.80:6818/128713757,v1:172.21.15.80:6819/128713757]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:58:03.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[40404]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:58:03.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[40404]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:58:03.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:03.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:03.830 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[29691]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:58:03.830 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[29691]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:58:03.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:03.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[29691]: from='osd.2 [v2:172.21.15.80:6818/128713757,v1:172.21.15.80:6819/128713757]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:58:03.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[29691]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:58:03.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[29691]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:58:03.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:03.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:04.115 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2[56429]: 2024-03-26T04:58:03.911+0000 7fd548584700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T04:58:04.115 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 04:58:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2[56429]: 2024-03-26T04:58:03.913+0000 7fd54036f700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:58:04.502 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T04:58:04.503 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T04:58:04.503 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-03-26T04:58:04.503 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-03-26T04:58:04.503 INFO:teuthology.orchestra.run.smithi080.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125104 s, 838 MB/s 2024-03-26T04:58:04.503 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 2024-03-26T04:58:04.503 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-03-26T04:58:04.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:04 smithi132 ceph-mon[33500]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-26T04:58:04.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:04 smithi132 ceph-mon[33500]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:58:04.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:04 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:04.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:04 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:04.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:04 smithi132 ceph-mon[33500]: from='osd.2 ' entity='osd.2' 2024-03-26T04:58:04.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:04 smithi080 ceph-mon[29691]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-26T04:58:04.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:04 smithi080 ceph-mon[29691]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:58:04.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:04 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:04.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:04 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:04.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:04 smithi080 ceph-mon[29691]: from='osd.2 ' entity='osd.2' 2024-03-26T04:58:04.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:04 smithi080 ceph-mon[40404]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-26T04:58:04.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:04 smithi080 ceph-mon[40404]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:58:04.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:04 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:04.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:04 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:04.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:04 smithi080 ceph-mon[40404]: from='osd.2 ' entity='osd.2' 2024-03-26T04:58:05.277 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi080:vg_nvme/lv_1 2024-03-26T04:58:05.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:58:05.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:58:05.515 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:05.515 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[29691]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:58:05.515 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[29691]: osd.2 [v2:172.21.15.80:6818/128713757,v1:172.21.15.80:6819/128713757] boot 2024-03-26T04:58:05.515 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[29691]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:58:05.515 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:05.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:58:05.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:58:05.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:05.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[40404]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:58:05.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[40404]: osd.2 [v2:172.21.15.80:6818/128713757,v1:172.21.15.80:6819/128713757] boot 2024-03-26T04:58:05.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[40404]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:58:05.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:05 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:05.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:05 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:58:05.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:05 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:58:05.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:05 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:05.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:05 smithi132 ceph-mon[33500]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:58:05.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:05 smithi132 ceph-mon[33500]: osd.2 [v2:172.21.15.80:6818/128713757,v1:172.21.15.80:6819/128713757] boot 2024-03-26T04:58:05.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:05 smithi132 ceph-mon[33500]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:58:05.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:05 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:58:06.696 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:58:07.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:07 smithi132 ceph-mon[33500]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:58:07.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:07 smithi132 ceph-mon[33500]: pgmap v78: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:07.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:07 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:07.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:07 smithi080 ceph-mon[29691]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:58:07.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:07 smithi080 ceph-mon[29691]: pgmap v78: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:07.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:07 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:07.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:07 smithi080 ceph-mon[40404]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:58:07.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:07 smithi080 ceph-mon[40404]: pgmap v78: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:07.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:07 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:08.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:58:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:08.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:08.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:08.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:58:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:08.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:08.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:08.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:08.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:08.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:08.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:08.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:58:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:08.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:08.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:08.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:08.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:08.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:08.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:09.694 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[29691]: Detected new or changed devices on smithi080 2024-03-26T04:58:09.694 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[29691]: Adjusting osd_memory_target on smithi080 to 5354M 2024-03-26T04:58:09.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:09.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[29691]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:58:09.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[29691]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:09.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:58:09.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:58:09.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:09.695 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[40404]: Detected new or changed devices on smithi080 2024-03-26T04:58:09.695 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[40404]: Adjusting osd_memory_target on smithi080 to 5354M 2024-03-26T04:58:09.695 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:09.695 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[40404]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:58:09.695 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[40404]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:09.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:58:09.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:58:09.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:09 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:09.696 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 04:58:09 smithi080 sudo[60691]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:58:09.696 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 04:58:09 smithi080 sudo[60691]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:58:09.696 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 04:58:09 smithi080 sudo[60691]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:58:09.696 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 04:58:09 smithi080 sudo[60691]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:58:09.696 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 04:58:09 smithi080 sudo[60750]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:58:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:09 smithi132 ceph-mon[33500]: Detected new or changed devices on smithi080 2024-03-26T04:58:09.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:09 smithi132 ceph-mon[33500]: Adjusting osd_memory_target on smithi080 to 5354M 2024-03-26T04:58:09.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:09 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:09.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:09 smithi132 ceph-mon[33500]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:58:09.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:09 smithi132 ceph-mon[33500]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:09.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:09 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:58:09.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:09 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:58:09.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:09 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:09.951 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 04:58:09 smithi080 sudo[60750]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:58:09.951 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 04:58:09 smithi080 sudo[60750]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:58:09.951 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 04:58:09 smithi080 sudo[60750]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:58:10.223 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 04:58:09 smithi080 sudo[60760]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:58:10.223 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 04:58:09 smithi080 sudo[60760]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:58:10.223 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 04:58:09 smithi080 sudo[60760]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:58:10.223 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 04:58:09 smithi080 sudo[60760]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:58:10.474 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60774]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:58:10.474 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60774]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:58:10.474 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60774]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:58:10.474 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60774]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:58:10.475 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60780]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:58:10.475 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60780]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:58:10.475 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60780]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:58:10.734 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60780]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:58:10.735 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:10 smithi080 ceph-mon[29691]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:58:10.735 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:10 smithi080 ceph-mon[29691]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T04:58:10.735 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:10 smithi080 ceph-mon[29691]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T04:58:10.735 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:10 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:10.735 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:10 smithi080 ceph-mon[40404]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:58:10.736 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:10 smithi080 ceph-mon[40404]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T04:58:10.736 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:10 smithi080 ceph-mon[40404]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T04:58:10.736 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:10 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:10.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:10 smithi132 ceph-mon[33500]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:58:10.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:10 smithi132 ceph-mon[33500]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T04:58:10.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:10 smithi132 ceph-mon[33500]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T04:58:10.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:10 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:10.997 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60807]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:58:10.997 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60807]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:58:10.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60807]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:58:10.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 sudo[60807]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:58:10.998 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60804]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:58:10.998 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60804]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:58:10.998 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60804]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:58:10.998 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 04:58:10 smithi080 sudo[60804]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:58:11.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 sudo[60818]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:58:11.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 sudo[60818]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:58:11.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 sudo[60818]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:58:11.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 sudo[60818]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:58:11.747 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 sudo[36086]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:58:11.748 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 sudo[36086]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:58:11.748 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 sudo[36086]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:58:11.748 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 sudo[36086]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:58:12.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[40404]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:12.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[40404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:58:12.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[40404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:58:12.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:58:12.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:58:12.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:58:12.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:58:12.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:58:12.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:58:12.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[29691]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:12.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[29691]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:58:12.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[29691]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:58:12.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:58:12.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:58:12.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:58:12.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:58:12.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:58:12.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 sudo[36089]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 sudo[36089]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 sudo[36089]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 sudo[36089]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 ceph-mon[33500]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 ceph-mon[33500]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 ceph-mon[33500]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:58:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:58:12.267 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 sudo[60865]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:58:12.267 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 sudo[60865]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:58:12.267 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 sudo[60865]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:58:12.267 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 sudo[60865]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:58:12.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 sudo[60868]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:58:12.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 sudo[60868]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:58:12.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 sudo[60868]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:58:12.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 sudo[60868]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:58:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:12 smithi132 ceph-mon[33500]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:58:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:12 smithi132 ceph-mon[33500]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:58:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:12 smithi132 ceph-mon[33500]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:58:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:58:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:58:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:58:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:12 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3306375834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "640cfdb6-b9c5-446d-8312-504ae12c7818"}]: dispatch 2024-03-26T04:58:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:12 smithi132 ceph-mon[33500]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:58:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:12 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3306375834' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "640cfdb6-b9c5-446d-8312-504ae12c7818"}]': finished 2024-03-26T04:58:13.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:12 smithi132 ceph-mon[33500]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T04:58:13.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[40404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:58:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[40404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:58:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[40404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:58:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:58:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:58:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:58:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3306375834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "640cfdb6-b9c5-446d-8312-504ae12c7818"}]: dispatch 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[40404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3306375834' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "640cfdb6-b9c5-446d-8312-504ae12c7818"}]': finished 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[40404]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[29691]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[29691]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[29691]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3306375834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "640cfdb6-b9c5-446d-8312-504ae12c7818"}]: dispatch 2024-03-26T04:58:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[29691]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:58:13.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3306375834' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "640cfdb6-b9c5-446d-8312-504ae12c7818"}]': finished 2024-03-26T04:58:13.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[29691]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T04:58:13.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:14.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:13 smithi132 ceph-mon[33500]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:14.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:13 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2881716601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:58:14.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:13 smithi080 ceph-mon[40404]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:14.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:13 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2881716601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:58:14.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:13 smithi080 ceph-mon[29691]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:14.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:13 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2881716601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:58:16.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:15 smithi132 ceph-mon[33500]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:16.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:15 smithi080 ceph-mon[29691]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:16.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:15 smithi080 ceph-mon[40404]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:17.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:17 smithi080 ceph-mon[40404]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-26T04:58:17.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:17 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:58:17.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:17 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:58:17.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:17 smithi080 ceph-mon[40404]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T04:58:17.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:17 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:17.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:17 smithi080 ceph-mon[29691]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-26T04:58:17.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:17 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:58:17.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:17 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:58:17.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:17 smithi080 ceph-mon[29691]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T04:58:17.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:18.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:17 smithi132 ceph-mon[33500]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-26T04:58:18.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:17 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:58:18.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:17 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:58:18.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:17 smithi132 ceph-mon[33500]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T04:58:18.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:17 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:19.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:18 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:58:19.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:18 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:19.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:18 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:58:19.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:18 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:18 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:58:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:18 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:19.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:19 smithi080 ceph-mon[29691]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-26T04:58:19.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:19 smithi080 ceph-mon[29691]: Deploying daemon osd.3 on smithi080 2024-03-26T04:58:19.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:19 smithi080 ceph-mon[40404]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-26T04:58:19.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:19 smithi080 ceph-mon[40404]: Deploying daemon osd.3 on smithi080 2024-03-26T04:58:20.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:19 smithi132 ceph-mon[33500]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-26T04:58:20.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:19 smithi132 ceph-mon[33500]: Deploying daemon osd.3 on smithi080 2024-03-26T04:58:22.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:21 smithi132 ceph-mon[33500]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 95 KiB/s wr, 6 op/s 2024-03-26T04:58:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:21 smithi080 ceph-mon[29691]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 95 KiB/s wr, 6 op/s 2024-03-26T04:58:22.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:21 smithi080 ceph-mon[40404]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 95 KiB/s wr, 6 op/s 2024-03-26T04:58:24.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:23 smithi080 ceph-mon[29691]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-26T04:58:24.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:23 smithi080 ceph-mon[40404]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-26T04:58:24.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:23 smithi132 ceph-mon[33500]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-26T04:58:25.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:25.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:25.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:24 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:25.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:24 smithi132 ceph-mon[33500]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-26T04:58:25.298 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:24 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:25.298 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:24 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:25.298 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:24 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:25.298 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:24 smithi080 ceph-mon[40404]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-26T04:58:25.298 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:24 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:25.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:24 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:25.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:24 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:25.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:24 smithi080 ceph-mon[29691]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-26T04:58:27.591 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 3 on host 'smithi080' 2024-03-26T04:58:28.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[40404]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:28.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:28.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:28.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[40404]: from='osd.3 [v2:172.21.15.80:6826/418771441,v1:172.21.15.80:6827/418771441]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:58:28.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[40404]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:58:28.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:28.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[29691]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:28.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:28.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:28.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[29691]: from='osd.3 [v2:172.21.15.80:6826/418771441,v1:172.21.15.80:6827/418771441]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:58:28.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[29691]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:58:28.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:27 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:28.235 DEBUG:teuthology.orchestra.run.smithi080:osd.3> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T04:58:28.237 INFO:tasks.cephadm:Deploying osd.4 on smithi132 with /dev/vg_nvme/lv_4... 2024-03-26T04:58:28.238 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T04:58:28.268 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:27 smithi132 ceph-mon[33500]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:28.268 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:27 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.268 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:27 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.268 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:27 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:28.268 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:27 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:28.268 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:27 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.268 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:27 smithi132 ceph-mon[33500]: from='osd.3 [v2:172.21.15.80:6826/418771441,v1:172.21.15.80:6827/418771441]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:58:28.269 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:27 smithi132 ceph-mon[33500]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:58:28.269 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:27 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.269 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:27 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:28.269 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:27 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:28.481 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:58:28.909 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:28 smithi132 ceph-mon[33500]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:58:28.909 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:28 smithi132 ceph-mon[33500]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T04:58:28.909 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:28 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:28.909 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:28 smithi132 ceph-mon[33500]: from='osd.3 [v2:172.21.15.80:6826/418771441,v1:172.21.15.80:6827/418771441]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:58:28.909 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:28 smithi132 ceph-mon[33500]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:58:28.909 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:28 smithi132 ceph-mon[33500]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:29.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:28 smithi080 ceph-mon[29691]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:58:29.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:28 smithi080 ceph-mon[29691]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T04:58:29.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:28 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:29.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:28 smithi080 ceph-mon[29691]: from='osd.3 [v2:172.21.15.80:6826/418771441,v1:172.21.15.80:6827/418771441]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:58:29.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:28 smithi080 ceph-mon[29691]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:58:29.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:28 smithi080 ceph-mon[29691]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:29.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:28 smithi080 ceph-mon[40404]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:58:29.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:28 smithi080 ceph-mon[40404]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T04:58:29.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:28 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:29.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:28 smithi080 ceph-mon[40404]: from='osd.3 [v2:172.21.15.80:6826/418771441,v1:172.21.15.80:6827/418771441]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:58:29.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:28 smithi080 ceph-mon[40404]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-26T04:58:29.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:28 smithi080 ceph-mon[40404]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:29.571 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 04:58:29 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3[62541]: 2024-03-26T04:58:29.262+0000 7fc5d2e79700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T04:58:29.571 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 04:58:29 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3[62541]: 2024-03-26T04:58:29.265+0000 7fc5ca463700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:58:29.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:29 smithi080 ceph-mon[40404]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-26T04:58:29.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:29 smithi080 ceph-mon[40404]: osdmap e26: 4 total, 3 up, 4 in 2024-03-26T04:58:29.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:29 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:29.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:29 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:29.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:29 smithi080 ceph-mon[40404]: from='osd.3 ' entity='osd.3' 2024-03-26T04:58:29.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:29 smithi080 ceph-mon[29691]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-26T04:58:29.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:29 smithi080 ceph-mon[29691]: osdmap e26: 4 total, 3 up, 4 in 2024-03-26T04:58:29.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:29.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:29.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:29 smithi080 ceph-mon[29691]: from='osd.3 ' entity='osd.3' 2024-03-26T04:58:30.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:29 smithi132 ceph-mon[33500]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-26T04:58:30.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:29 smithi132 ceph-mon[33500]: osdmap e26: 4 total, 3 up, 4 in 2024-03-26T04:58:30.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:30.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:30.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:29 smithi132 ceph-mon[33500]: from='osd.3 ' entity='osd.3' 2024-03-26T04:58:30.367 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T04:58:30.367 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T04:58:30.367 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-03-26T04:58:30.367 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-03-26T04:58:30.367 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113096 s, 927 MB/s 2024-03-26T04:58:30.367 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-03-26T04:58:30.970 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:30 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:58:30.970 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:30 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:58:30.970 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:30 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:30.970 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:30 smithi132 ceph-mon[33500]: osd.3 [v2:172.21.15.80:6826/418771441,v1:172.21.15.80:6827/418771441] boot 2024-03-26T04:58:30.970 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:30 smithi132 ceph-mon[33500]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T04:58:30.970 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:30 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:30.970 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:30 smithi132 ceph-mon[33500]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:30.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:58:30.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:58:30.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:30.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[29691]: osd.3 [v2:172.21.15.80:6826/418771441,v1:172.21.15.80:6827/418771441] boot 2024-03-26T04:58:30.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[29691]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T04:58:30.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:30.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[29691]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:30.971 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:58:30.972 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:58:30.972 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:30.972 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[40404]: osd.3 [v2:172.21.15.80:6826/418771441,v1:172.21.15.80:6827/418771441] boot 2024-03-26T04:58:30.972 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[40404]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T04:58:30.972 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:58:30.972 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:30 smithi080 ceph-mon[40404]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:58:30.998 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi132:vg_nvme/lv_4 2024-03-26T04:58:31.293 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:58:32.200 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:32 smithi132 ceph-mon[33500]: Detected new or changed devices on smithi080 2024-03-26T04:58:32.200 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:32.201 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:32.201 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:32.201 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:32.201 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:32.201 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:32.201 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:32 smithi132 ceph-mon[33500]: Adjusting osd_memory_target on smithi080 to 4016M 2024-03-26T04:58:32.201 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:32.201 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:32.201 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:32.201 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:32.359 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[29691]: Detected new or changed devices on smithi080 2024-03-26T04:58:32.359 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:32.359 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:32.359 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:32.359 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:32.359 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:32.359 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:32.359 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[29691]: Adjusting osd_memory_target on smithi080 to 4016M 2024-03-26T04:58:32.359 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:32.359 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[40404]: Detected new or changed devices on smithi080 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[40404]: Adjusting osd_memory_target on smithi080 to 4016M 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:32.360 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:32.361 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:33.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:33 smithi080 ceph-mon[29691]: osdmap e28: 4 total, 4 up, 4 in 2024-03-26T04:58:33.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:33 smithi080 ceph-mon[29691]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:58:33.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:33 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:58:33.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:33 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:58:33.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:33 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:33.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:33 smithi080 ceph-mon[40404]: osdmap e28: 4 total, 4 up, 4 in 2024-03-26T04:58:33.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:33 smithi080 ceph-mon[40404]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:58:33.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:33 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:58:33.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:33 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:58:33.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:33 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:33.513 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:33 smithi132 ceph-mon[33500]: osdmap e28: 4 total, 4 up, 4 in 2024-03-26T04:58:33.513 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:33 smithi132 ceph-mon[33500]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:58:33.513 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:33 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:58:33.513 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:33 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:58:33.514 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:33 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:34.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:34 smithi132 ceph-mon[33500]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:58:34.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:34 smithi080 ceph-mon[29691]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:58:34.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:34 smithi080 ceph-mon[40404]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:58:35.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:35 smithi080 ceph-mon[40404]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:58:35.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:35 smithi080 ceph-mon[29691]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:58:35.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:35 smithi132 ceph-mon[33500]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:58:36.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:36 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/966352602' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639"}]: dispatch 2024-03-26T04:58:36.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:36 smithi080 ceph-mon[29691]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639"}]: dispatch 2024-03-26T04:58:36.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:36 smithi080 ceph-mon[29691]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639"}]': finished 2024-03-26T04:58:36.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:36 smithi080 ceph-mon[29691]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T04:58:36.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:36 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:36.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:36 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/1783879000' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:58:36.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:36 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/966352602' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639"}]: dispatch 2024-03-26T04:58:36.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:36 smithi080 ceph-mon[40404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639"}]: dispatch 2024-03-26T04:58:36.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:36 smithi080 ceph-mon[40404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639"}]': finished 2024-03-26T04:58:36.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:36 smithi080 ceph-mon[40404]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T04:58:36.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:36 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:36.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:36 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/1783879000' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:58:36.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:36 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/966352602' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639"}]: dispatch 2024-03-26T04:58:36.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:36 smithi132 ceph-mon[33500]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639"}]: dispatch 2024-03-26T04:58:36.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:36 smithi132 ceph-mon[33500]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639"}]': finished 2024-03-26T04:58:36.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:36 smithi132 ceph-mon[33500]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T04:58:36.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:36 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:36.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:36 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/1783879000' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:58:37.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:37 smithi080 ceph-mon[29691]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:37.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:37 smithi080 ceph-mon[40404]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:37.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:37 smithi132 ceph-mon[33500]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:39.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:39 smithi132 ceph-mon[33500]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:40.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:39 smithi080 ceph-mon[40404]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:40.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:39 smithi080 ceph-mon[29691]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:41.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:41 smithi132 ceph-mon[33500]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:42.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:41 smithi080 ceph-mon[29691]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:42.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:41 smithi080 ceph-mon[40404]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:42.693 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:42 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:58:42.694 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:42 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:42.694 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:42 smithi132 ceph-mon[33500]: Deploying daemon osd.4 on smithi132 2024-03-26T04:58:43.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:42 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:58:43.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:42 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:43.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:42 smithi080 ceph-mon[40404]: Deploying daemon osd.4 on smithi132 2024-03-26T04:58:43.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:42 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:58:43.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:42 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:43.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:42 smithi080 ceph-mon[29691]: Deploying daemon osd.4 on smithi132 2024-03-26T04:58:43.973 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:43 smithi132 ceph-mon[33500]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:44.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:43 smithi080 ceph-mon[29691]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:44.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:43 smithi080 ceph-mon[40404]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:45.949 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:45 smithi132 ceph-mon[33500]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:45.949 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:45 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:58:45.949 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:45 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:58:46.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:45 smithi080 ceph-mon[40404]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:46.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:45 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:58:46.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:45 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:58:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:45 smithi080 ceph-mon[29691]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:45 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:58:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:45 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:58:46.857 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:46 smithi132 ceph-mon[33500]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T04:58:46.858 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:46 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:47.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:46 smithi080 ceph-mon[29691]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T04:58:47.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:46 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:47.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:46 smithi080 ceph-mon[40404]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T04:58:47.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:46 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:47.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:47 smithi132 ceph-mon[33500]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:47.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:47 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:47.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:47 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:47.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:47 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:48.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:47 smithi080 ceph-mon[29691]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:48.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:47 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:48.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:47 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:48.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:47 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:48.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:47 smithi080 ceph-mon[40404]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:48.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:47 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:48.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:47 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:48.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:47 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:49.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:49 smithi132 ceph-mon[33500]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:49 smithi080 ceph-mon[29691]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:50.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:49 smithi080 ceph-mon[40404]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:50.644 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 4 on host 'smithi132' 2024-03-26T04:58:51.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:51.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:51.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[29691]: from='osd.4 [v2:172.21.15.132:6800/1441071423,v1:172.21.15.132:6801/1441071423]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:58:51.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[29691]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:58:51.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[29691]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:51.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:51.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:51.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:51.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:51.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[40404]: from='osd.4 [v2:172.21.15.132:6800/1441071423,v1:172.21.15.132:6801/1441071423]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:58:51.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[40404]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:58:51.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[40404]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:51.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:51.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:50 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:50 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:50 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:50 smithi132 ceph-mon[33500]: from='osd.4 [v2:172.21.15.132:6800/1441071423,v1:172.21.15.132:6801/1441071423]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:58:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:50 smithi132 ceph-mon[33500]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:58:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:50 smithi132 ceph-mon[33500]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:50 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:50 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:51.326 DEBUG:teuthology.orchestra.run.smithi132:osd.4> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T04:58:51.328 INFO:tasks.cephadm:Deploying osd.5 on smithi132 with /dev/vg_nvme/lv_3... 2024-03-26T04:58:51.329 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T04:58:52.045 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:58:52.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:51 smithi080 ceph-mon[29691]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:58:52.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:51 smithi080 ceph-mon[29691]: osdmap e31: 5 total, 4 up, 5 in 2024-03-26T04:58:52.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:51 smithi080 ceph-mon[29691]: from='osd.4 [v2:172.21.15.132:6800/1441071423,v1:172.21.15.132:6801/1441071423]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:58:52.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:51 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:52.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:51 smithi080 ceph-mon[29691]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:58:52.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:51 smithi080 ceph-mon[40404]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:58:52.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:51 smithi080 ceph-mon[40404]: osdmap e31: 5 total, 4 up, 5 in 2024-03-26T04:58:52.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:51 smithi080 ceph-mon[40404]: from='osd.4 [v2:172.21.15.132:6800/1441071423,v1:172.21.15.132:6801/1441071423]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:58:52.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:51 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:52.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:51 smithi080 ceph-mon[40404]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:58:52.167 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:51 smithi132 ceph-mon[33500]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:58:52.167 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:51 smithi132 ceph-mon[33500]: osdmap e31: 5 total, 4 up, 5 in 2024-03-26T04:58:52.167 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:51 smithi132 ceph-mon[33500]: from='osd.4 [v2:172.21.15.132:6800/1441071423,v1:172.21.15.132:6801/1441071423]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:58:52.167 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:51 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:52.167 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:51 smithi132 ceph-mon[33500]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:58:52.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 04:58:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[38419]: 2024-03-26T04:58:52.310+0000 7f259f13b700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T04:58:52.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 04:58:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[38419]: 2024-03-26T04:58:52.313+0000 7f2596f26700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:58:53.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:52 smithi080 ceph-mon[29691]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-26T04:58:53.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:52 smithi080 ceph-mon[29691]: osdmap e32: 5 total, 4 up, 5 in 2024-03-26T04:58:53.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:52 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:53.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:52 smithi080 ceph-mon[29691]: from='osd.4 ' entity='osd.4' 2024-03-26T04:58:53.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:52 smithi080 ceph-mon[29691]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:53.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:52 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:52 smithi080 ceph-mon[40404]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-26T04:58:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:52 smithi080 ceph-mon[40404]: osdmap e32: 5 total, 4 up, 5 in 2024-03-26T04:58:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:52 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:52 smithi080 ceph-mon[40404]: from='osd.4 ' entity='osd.4' 2024-03-26T04:58:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:52 smithi080 ceph-mon[40404]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:52 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:53.180 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:52 smithi132 ceph-mon[33500]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-26T04:58:53.180 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:52 smithi132 ceph-mon[33500]: osdmap e32: 5 total, 4 up, 5 in 2024-03-26T04:58:53.181 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:52 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:53.181 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:52 smithi132 ceph-mon[33500]: from='osd.4 ' entity='osd.4' 2024-03-26T04:58:53.181 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:52 smithi132 ceph-mon[33500]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:58:53.181 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:52 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:54.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:53 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:58:54.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:53 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:58:54.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:53 smithi132 ceph-mon[33500]: osd.4 [v2:172.21.15.132:6800/1441071423,v1:172.21.15.132:6801/1441071423] boot 2024-03-26T04:58:54.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:53 smithi132 ceph-mon[33500]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T04:58:54.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:53 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:54.330 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T04:58:54.330 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T04:58:54.330 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-03-26T04:58:54.330 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-03-26T04:58:54.330 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0136258 s, 770 MB/s 2024-03-26T04:58:54.330 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-03-26T04:58:54.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:53 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:58:54.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:53 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:58:54.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:53 smithi080 ceph-mon[29691]: osd.4 [v2:172.21.15.132:6800/1441071423,v1:172.21.15.132:6801/1441071423] boot 2024-03-26T04:58:54.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:53 smithi080 ceph-mon[29691]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T04:58:54.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:53 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:54.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:53 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:58:54.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:53 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:58:54.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:53 smithi080 ceph-mon[40404]: osd.4 [v2:172.21.15.132:6800/1441071423,v1:172.21.15.132:6801/1441071423] boot 2024-03-26T04:58:54.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:53 smithi080 ceph-mon[40404]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T04:58:54.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:53 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:58:54.993 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi132:vg_nvme/lv_3 2024-03-26T04:58:55.280 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:54 smithi132 ceph-mon[33500]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T04:58:55.280 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:54 smithi132 ceph-mon[33500]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:58:55.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:54 smithi080 ceph-mon[29691]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T04:58:55.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:54 smithi080 ceph-mon[29691]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:58:55.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:54 smithi080 ceph-mon[40404]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T04:58:55.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:54 smithi080 ceph-mon[40404]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:58:55.608 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:58:56.135 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: Detected new or changed devices on smithi132 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: Adjusting osd_memory_target on smithi132 to 17088M 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:56.136 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T04:58:56.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: Detected new or changed devices on smithi132 2024-03-26T04:58:56.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:56.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: Adjusting osd_memory_target on smithi132 to 17088M 2024-03-26T04:58:56.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:56.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:56.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:56.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:56.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:56.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T04:58:56.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: Detected new or changed devices on smithi132 2024-03-26T04:58:56.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:58:56.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: Adjusting osd_memory_target on smithi132 to 17088M 2024-03-26T04:58:56.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:56.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:56.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:56.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:56.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:56.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:56.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:58:57.034 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:56 smithi132 ceph-mon[33500]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T04:58:57.034 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:56 smithi132 ceph-mon[33500]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:58:57.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:56 smithi080 ceph-mon[29691]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T04:58:57.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:56 smithi080 ceph-mon[29691]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:58:57.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:56 smithi080 ceph-mon[40404]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T04:58:57.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:56 smithi080 ceph-mon[40404]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:58:58.166 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:57 smithi132 ceph-mon[33500]: osdmap e37: 5 total, 5 up, 5 in 2024-03-26T04:58:58.166 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:57 smithi132 ceph-mon[33500]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:58:58.166 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:57 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:58:58.166 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:57 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:58:58.166 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:57 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:58.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:57 smithi080 ceph-mon[29691]: osdmap e37: 5 total, 5 up, 5 in 2024-03-26T04:58:58.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:57 smithi080 ceph-mon[29691]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:58:58.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:57 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:58:58.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:57 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:58:58.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:57 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:58.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:57 smithi080 ceph-mon[40404]: osdmap e37: 5 total, 5 up, 5 in 2024-03-26T04:58:58.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:57 smithi080 ceph-mon[40404]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:58:58.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:57 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:58:58.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:57 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:58:58.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:57 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:59.187 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:58:58 smithi132 ceph-mon[33500]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:58:59.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:58:58 smithi080 ceph-mon[29691]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:58:59.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:58:58 smithi080 ceph-mon[40404]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:00.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:00 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/14257703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc"}]: dispatch 2024-03-26T04:59:00.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:00 smithi080 ceph-mon[29691]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc"}]: dispatch 2024-03-26T04:59:00.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:00 smithi080 ceph-mon[29691]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc"}]': finished 2024-03-26T04:59:00.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:00 smithi080 ceph-mon[29691]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T04:59:00.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:00 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:00.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:00 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/14257703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc"}]: dispatch 2024-03-26T04:59:00.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:00 smithi080 ceph-mon[40404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc"}]: dispatch 2024-03-26T04:59:00.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:00 smithi080 ceph-mon[40404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc"}]': finished 2024-03-26T04:59:00.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:00 smithi080 ceph-mon[40404]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T04:59:00.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:00 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:00.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:00 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/14257703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc"}]: dispatch 2024-03-26T04:59:00.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:00 smithi132 ceph-mon[33500]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc"}]: dispatch 2024-03-26T04:59:00.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:00 smithi132 ceph-mon[33500]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc"}]': finished 2024-03-26T04:59:00.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:00 smithi132 ceph-mon[33500]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T04:59:00.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:00 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:01.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:01 smithi080 ceph-mon[29691]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-26T04:59:01.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:01 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/3966421757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:59:01.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:01 smithi080 ceph-mon[40404]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-26T04:59:01.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:01 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/3966421757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:59:01.446 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:01 smithi132 ceph-mon[33500]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-26T04:59:01.446 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:01 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/3966421757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:59:04.046 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:03 smithi132 ceph-mon[33500]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T04:59:04.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:03 smithi080 ceph-mon[40404]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T04:59:04.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:03 smithi080 ceph-mon[29691]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T04:59:06.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:05 smithi132 ceph-mon[33500]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T04:59:06.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:05 smithi080 ceph-mon[29691]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T04:59:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:05 smithi080 ceph-mon[40404]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T04:59:07.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:07 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:59:07.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:07 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:07.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:07 smithi132 ceph-mon[33500]: Deploying daemon osd.5 on smithi132 2024-03-26T04:59:07.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:07 smithi132 ceph-mon[33500]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-26T04:59:07.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:07 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:59:07.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:07 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:07.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:07 smithi080 ceph-mon[29691]: Deploying daemon osd.5 on smithi132 2024-03-26T04:59:07.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:07 smithi080 ceph-mon[29691]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-26T04:59:07.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:07 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:59:07.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:07 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:07.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:07 smithi080 ceph-mon[40404]: Deploying daemon osd.5 on smithi132 2024-03-26T04:59:07.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:07 smithi080 ceph-mon[40404]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-26T04:59:09.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:09 smithi132 ceph-mon[33500]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T04:59:10.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:09 smithi080 ceph-mon[29691]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T04:59:10.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:09 smithi080 ceph-mon[40404]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T04:59:11.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:11 smithi132 ceph-mon[33500]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:11.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:11 smithi080 ceph-mon[29691]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:11.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:11 smithi080 ceph-mon[40404]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:13.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:12 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:13.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:13.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:13.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:12 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:13.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:12 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:14.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:13 smithi132 ceph-mon[33500]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:14.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:13 smithi080 ceph-mon[29691]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:14.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:13 smithi080 ceph-mon[40404]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:15.482 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 5 on host 'smithi132' 2024-03-26T04:59:15.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:15 smithi132 ceph-mon[33500]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:15.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:15 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:15 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:15 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:15.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:15 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:15.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:15 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:15 smithi132 ceph-mon[33500]: from='osd.5 [v2:172.21.15.132:6808/794504625,v1:172.21.15.132:6809/794504625]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:59:15.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:15 smithi132 ceph-mon[33500]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:59:15.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:15 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:15 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:15 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[29691]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[29691]: from='osd.5 [v2:172.21.15.132:6808/794504625,v1:172.21.15.132:6809/794504625]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:59:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[29691]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:59:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:15.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[40404]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:15.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:15.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:15.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[40404]: from='osd.5 [v2:172.21.15.132:6808/794504625,v1:172.21.15.132:6809/794504625]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:59:15.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[40404]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:59:15.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:15.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:15 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:16.111 DEBUG:teuthology.orchestra.run.smithi132:osd.5> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T04:59:16.113 INFO:tasks.cephadm:Deploying osd.6 on smithi132 with /dev/vg_nvme/lv_2... 2024-03-26T04:59:16.113 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T04:59:16.967 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:16 smithi132 ceph-mon[33500]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:59:16.968 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:16 smithi132 ceph-mon[33500]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T04:59:16.968 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:16 smithi132 ceph-mon[33500]: from='osd.5 [v2:172.21.15.132:6808/794504625,v1:172.21.15.132:6809/794504625]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:59:16.968 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:16 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:16.968 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:16 smithi132 ceph-mon[33500]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:59:16.968 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 04:59:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T04:59:16.730+0000 7f63cd216700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T04:59:16.968 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 04:59:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T04:59:16.735+0000 7f63c9009700 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:59:17.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:16 smithi080 ceph-mon[40404]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:59:17.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:16 smithi080 ceph-mon[40404]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T04:59:17.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:16 smithi080 ceph-mon[40404]: from='osd.5 [v2:172.21.15.132:6808/794504625,v1:172.21.15.132:6809/794504625]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:59:17.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:16 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:17.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:16 smithi080 ceph-mon[40404]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:59:17.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:16 smithi080 ceph-mon[29691]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:59:17.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:16 smithi080 ceph-mon[29691]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T04:59:17.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:16 smithi080 ceph-mon[29691]: from='osd.5 [v2:172.21.15.132:6808/794504625,v1:172.21.15.132:6809/794504625]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:59:17.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:16 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:17.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:16 smithi080 ceph-mon[29691]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:59:17.718 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:59:17.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:17 smithi132 ceph-mon[33500]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:17.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:17 smithi132 ceph-mon[33500]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-26T04:59:17.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:17 smithi132 ceph-mon[33500]: osdmap e40: 6 total, 5 up, 6 in 2024-03-26T04:59:17.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:17 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:17.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:17 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:18.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:17 smithi080 ceph-mon[29691]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:18.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:17 smithi080 ceph-mon[29691]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-26T04:59:18.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:17 smithi080 ceph-mon[29691]: osdmap e40: 6 total, 5 up, 6 in 2024-03-26T04:59:18.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:18.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:17 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:18.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:17 smithi080 ceph-mon[40404]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:59:18.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:17 smithi080 ceph-mon[40404]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-26T04:59:18.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:17 smithi080 ceph-mon[40404]: osdmap e40: 6 total, 5 up, 6 in 2024-03-26T04:59:18.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:17 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:18.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:17 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:19.031 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:18 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:59:19.031 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:18 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:59:19.031 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:18 smithi132 ceph-mon[33500]: osd.5 [v2:172.21.15.132:6808/794504625,v1:172.21.15.132:6809/794504625] boot 2024-03-26T04:59:19.031 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:18 smithi132 ceph-mon[33500]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T04:59:19.031 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:18 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:18 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:59:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:18 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:59:19.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:18 smithi080 ceph-mon[29691]: osd.5 [v2:172.21.15.132:6808/794504625,v1:172.21.15.132:6809/794504625] boot 2024-03-26T04:59:19.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:18 smithi080 ceph-mon[29691]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T04:59:19.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:18 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:18 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:59:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:18 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:59:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:18 smithi080 ceph-mon[40404]: osd.5 [v2:172.21.15.132:6808/794504625,v1:172.21.15.132:6809/794504625] boot 2024-03-26T04:59:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:18 smithi080 ceph-mon[40404]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T04:59:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:18 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:59:19.769 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:19 smithi132 ceph-mon[33500]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:19.769 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:19 smithi132 ceph-mon[33500]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T04:59:19.769 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:19 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:19.769 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:19 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:19.769 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:19 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:19.769 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:19 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:19.769 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:19 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:19.769 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:19 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:19.769 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:19 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:19.769 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:19 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:20.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[29691]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:20.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[29691]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T04:59:20.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:20.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:20.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:20.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:20.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:20.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:20.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:20.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:20.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[40404]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:20.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[40404]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T04:59:20.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:20.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:20.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:20.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:20.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:20.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:20.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:20.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:19 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:20.344 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T04:59:20.344 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T04:59:20.344 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-03-26T04:59:20.344 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-03-26T04:59:20.344 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129272 s, 811 MB/s 2024-03-26T04:59:20.344 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-03-26T04:59:20.982 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:20 smithi132 ceph-mon[33500]: Detected new or changed devices on smithi132 2024-03-26T04:59:20.982 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:20 smithi132 ceph-mon[33500]: Adjusting osd_memory_target on smithi132 to 8544M 2024-03-26T04:59:20.982 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:20 smithi132 ceph-mon[33500]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T04:59:21.010 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi132:vg_nvme/lv_2 2024-03-26T04:59:21.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:20 smithi080 ceph-mon[29691]: Detected new or changed devices on smithi132 2024-03-26T04:59:21.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:20 smithi080 ceph-mon[29691]: Adjusting osd_memory_target on smithi132 to 8544M 2024-03-26T04:59:21.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:20 smithi080 ceph-mon[29691]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T04:59:21.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:20 smithi080 ceph-mon[40404]: Detected new or changed devices on smithi132 2024-03-26T04:59:21.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:20 smithi080 ceph-mon[40404]: Adjusting osd_memory_target on smithi132 to 8544M 2024-03-26T04:59:21.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:20 smithi080 ceph-mon[40404]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T04:59:21.247 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:59:22.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:21 smithi132 ceph-mon[33500]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:22.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:21 smithi132 ceph-mon[33500]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T04:59:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:21 smithi080 ceph-mon[29691]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:21 smithi080 ceph-mon[29691]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T04:59:22.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:21 smithi080 ceph-mon[40404]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:22.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:21 smithi080 ceph-mon[40404]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T04:59:23.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:22 smithi132 ceph-mon[33500]: osdmap e45: 6 total, 6 up, 6 in 2024-03-26T04:59:23.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:22 smithi080 ceph-mon[29691]: osdmap e45: 6 total, 6 up, 6 in 2024-03-26T04:59:23.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:22 smithi080 ceph-mon[40404]: osdmap e45: 6 total, 6 up, 6 in 2024-03-26T04:59:23.929 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:23 smithi132 ceph-mon[33500]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:23.929 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:23 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:59:23.930 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:23 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:59:23.930 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:23 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:24.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:23 smithi080 ceph-mon[29691]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:24.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:23 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:59:24.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:23 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:59:24.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:23 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:24.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:23 smithi080 ceph-mon[40404]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:24.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:23 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:59:24.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:23 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:59:24.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:23 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:25.002 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:24 smithi132 ceph-mon[33500]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:59:25.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:24 smithi080 ceph-mon[29691]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:59:25.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:24 smithi080 ceph-mon[40404]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:59:26.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:25 smithi132 ceph-mon[33500]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:26.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:25 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/3778751396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5"}]: dispatch 2024-03-26T04:59:26.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:25 smithi132 ceph-mon[33500]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5"}]: dispatch 2024-03-26T04:59:26.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:25 smithi132 ceph-mon[33500]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5"}]': finished 2024-03-26T04:59:26.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:25 smithi132 ceph-mon[33500]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T04:59:26.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:25 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:26.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:25 smithi080 ceph-mon[40404]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:26.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:25 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/3778751396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5"}]: dispatch 2024-03-26T04:59:26.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:25 smithi080 ceph-mon[40404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5"}]: dispatch 2024-03-26T04:59:26.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:25 smithi080 ceph-mon[40404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5"}]': finished 2024-03-26T04:59:26.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:25 smithi080 ceph-mon[40404]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T04:59:26.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:25 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:26.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:25 smithi080 ceph-mon[29691]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:26.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:25 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/3778751396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5"}]: dispatch 2024-03-26T04:59:26.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:25 smithi080 ceph-mon[29691]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5"}]: dispatch 2024-03-26T04:59:26.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:25 smithi080 ceph-mon[29691]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5"}]': finished 2024-03-26T04:59:26.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:25 smithi080 ceph-mon[29691]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T04:59:26.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:25 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:27.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:26 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/1229045884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:59:27.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:26 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/1229045884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:59:27.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:26 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/1229045884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:59:28.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:27 smithi132 ceph-mon[33500]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:28.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:27 smithi080 ceph-mon[29691]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:28.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:27 smithi080 ceph-mon[40404]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:30.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:29 smithi132 ceph-mon[33500]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:30.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:29 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:59:30.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:29 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:59:30.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:29 smithi132 ceph-mon[33500]: osdmap e47: 7 total, 6 up, 7 in 2024-03-26T04:59:30.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:29 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:30.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:29 smithi080 ceph-mon[29691]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:30.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:29 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:59:30.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:29 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:59:30.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:29 smithi080 ceph-mon[29691]: osdmap e47: 7 total, 6 up, 7 in 2024-03-26T04:59:30.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:29 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:30.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:29 smithi080 ceph-mon[40404]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:30.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:29 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:59:30.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:29 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:59:30.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:29 smithi080 ceph-mon[40404]: osdmap e47: 7 total, 6 up, 7 in 2024-03-26T04:59:30.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:29 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:32.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:31 smithi132 ceph-mon[33500]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:32.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:31 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:59:32.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:31 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:59:32.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:31 smithi080 ceph-mon[29691]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:32.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:31 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:59:32.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:31 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:59:32.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:31 smithi080 ceph-mon[40404]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:32.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:31 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:59:32.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:31 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:59:33.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:32 smithi132 ceph-mon[33500]: osdmap e48: 7 total, 6 up, 7 in 2024-03-26T04:59:33.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:33.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:59:33.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:32 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:33.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:32 smithi080 ceph-mon[29691]: osdmap e48: 7 total, 6 up, 7 in 2024-03-26T04:59:33.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:33.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:59:33.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:32 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:33.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:32 smithi080 ceph-mon[40404]: osdmap e48: 7 total, 6 up, 7 in 2024-03-26T04:59:33.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:33.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:59:33.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:32 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:34.012 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:33 smithi132 ceph-mon[33500]: Deploying daemon osd.6 on smithi132 2024-03-26T04:59:34.012 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:33 smithi132 ceph-mon[33500]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:34.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:33 smithi080 ceph-mon[29691]: Deploying daemon osd.6 on smithi132 2024-03-26T04:59:34.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:33 smithi080 ceph-mon[29691]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:34.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:33 smithi080 ceph-mon[40404]: Deploying daemon osd.6 on smithi132 2024-03-26T04:59:34.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:33 smithi080 ceph-mon[40404]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:35.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:34 smithi080 ceph-mon[29691]: 1.0 scrub starts 2024-03-26T04:59:35.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:34 smithi080 ceph-mon[29691]: 1.0 scrub ok 2024-03-26T04:59:35.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:34 smithi080 ceph-mon[40404]: 1.0 scrub starts 2024-03-26T04:59:35.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:34 smithi080 ceph-mon[40404]: 1.0 scrub ok 2024-03-26T04:59:35.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:34 smithi132 ceph-mon[33500]: 1.0 scrub starts 2024-03-26T04:59:35.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:34 smithi132 ceph-mon[33500]: 1.0 scrub ok 2024-03-26T04:59:35.951 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:35 smithi132 ceph-mon[33500]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:36.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:35 smithi080 ceph-mon[40404]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:36.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:35 smithi080 ceph-mon[29691]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:38.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:37 smithi080 ceph-mon[29691]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:38.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:37 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:38.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:37 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:38.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:37 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:38.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:37 smithi080 ceph-mon[40404]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:38.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:37 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:38.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:37 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:38.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:37 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:38.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:37 smithi132 ceph-mon[33500]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:38.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:37 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:38.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:37 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:38.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:37 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:40.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:39 smithi132 ceph-mon[33500]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:40.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:39 smithi080 ceph-mon[29691]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:40.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:39 smithi080 ceph-mon[40404]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:40.966 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 6 on host 'smithi132' 2024-03-26T04:59:41.605 DEBUG:teuthology.orchestra.run.smithi132:osd.6> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T04:59:41.607 INFO:tasks.cephadm:Deploying osd.7 on smithi132 with /dev/vg_nvme/lv_1... 2024-03-26T04:59:41.607 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T04:59:41.891 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:59:42.017 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:41 smithi132 ceph-mon[33500]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:42.017 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:41 smithi132 ceph-mon[33500]: from='osd.6 [v2:172.21.15.132:6816/2209078972,v1:172.21.15.132:6817/2209078972]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:59:42.017 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:41 smithi132 ceph-mon[33500]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:59:42.017 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:42.017 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:42.017 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:42.017 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:41 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:42.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[40404]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:42.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[40404]: from='osd.6 [v2:172.21.15.132:6816/2209078972,v1:172.21.15.132:6817/2209078972]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:59:42.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[40404]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:59:42.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:42.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:42.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:42.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:42.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[29691]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:42.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[29691]: from='osd.6 [v2:172.21.15.132:6816/2209078972,v1:172.21.15.132:6817/2209078972]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:59:42.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[29691]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:59:42.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:42.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:42.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:42.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:41 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:42.868 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:42 smithi132 ceph-mon[33500]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:59:42.869 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:42 smithi132 ceph-mon[33500]: osdmap e49: 7 total, 6 up, 7 in 2024-03-26T04:59:42.869 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:42 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:42.869 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:42 smithi132 ceph-mon[33500]: from='osd.6 [v2:172.21.15.132:6816/2209078972,v1:172.21.15.132:6817/2209078972]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:59:42.869 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:42 smithi132 ceph-mon[33500]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:59:43.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:42 smithi080 ceph-mon[29691]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:59:43.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:42 smithi080 ceph-mon[29691]: osdmap e49: 7 total, 6 up, 7 in 2024-03-26T04:59:43.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:42 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:43.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:42 smithi080 ceph-mon[29691]: from='osd.6 [v2:172.21.15.132:6816/2209078972,v1:172.21.15.132:6817/2209078972]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:59:43.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:42 smithi080 ceph-mon[29691]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:59:43.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:42 smithi080 ceph-mon[40404]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:59:43.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:42 smithi080 ceph-mon[40404]: osdmap e49: 7 total, 6 up, 7 in 2024-03-26T04:59:43.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:42 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:43.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:42 smithi080 ceph-mon[40404]: from='osd.6 [v2:172.21.15.132:6816/2209078972,v1:172.21.15.132:6817/2209078972]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:59:43.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:42 smithi080 ceph-mon[40404]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T04:59:43.529 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 04:59:43 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T04:59:43.268+0000 7f278f1e0700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T04:59:43.530 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 04:59:43 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T04:59:43.277+0000 7f2786fcb700 -1 osd.6 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:59:43.712 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T04:59:43.713 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T04:59:43.713 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-03-26T04:59:43.713 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-03-26T04:59:43.713 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124922 s, 839 MB/s 2024-03-26T04:59:43.713 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-03-26T04:59:44.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:43 smithi132 ceph-mon[33500]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:44.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:43 smithi132 ceph-mon[33500]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-26T04:59:44.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:43 smithi132 ceph-mon[33500]: osdmap e50: 7 total, 6 up, 7 in 2024-03-26T04:59:44.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:43 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:44.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:43 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:44.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:43 smithi132 ceph-mon[33500]: from='osd.6 ' entity='osd.6' 2024-03-26T04:59:44.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:43 smithi080 ceph-mon[29691]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:44.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:43 smithi080 ceph-mon[29691]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-26T04:59:44.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:43 smithi080 ceph-mon[29691]: osdmap e50: 7 total, 6 up, 7 in 2024-03-26T04:59:44.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:43 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:44.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:43 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:44.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:43 smithi080 ceph-mon[29691]: from='osd.6 ' entity='osd.6' 2024-03-26T04:59:44.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:43 smithi080 ceph-mon[40404]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:59:44.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:43 smithi080 ceph-mon[40404]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-26T04:59:44.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:43 smithi080 ceph-mon[40404]: osdmap e50: 7 total, 6 up, 7 in 2024-03-26T04:59:44.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:43 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:44.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:43 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:44.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:43 smithi080 ceph-mon[40404]: from='osd.6 ' entity='osd.6' 2024-03-26T04:59:44.668 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi132:vg_nvme/lv_1 2024-03-26T04:59:45.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:44 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:59:45.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:44 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:59:45.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:44 smithi080 ceph-mon[29691]: osd.6 [v2:172.21.15.132:6816/2209078972,v1:172.21.15.132:6817/2209078972] boot 2024-03-26T04:59:45.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:44 smithi080 ceph-mon[29691]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T04:59:45.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:44 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:45.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:44 smithi080 ceph-mon[29691]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:59:45.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:44 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:59:45.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:44 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:59:45.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:44 smithi080 ceph-mon[40404]: osd.6 [v2:172.21.15.132:6816/2209078972,v1:172.21.15.132:6817/2209078972] boot 2024-03-26T04:59:45.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:44 smithi080 ceph-mon[40404]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T04:59:45.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:44 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:45.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:44 smithi080 ceph-mon[40404]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:59:45.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:44 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:59:45.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:44 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:59:45.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:44 smithi132 ceph-mon[33500]: osd.6 [v2:172.21.15.132:6816/2209078972,v1:172.21.15.132:6817/2209078972] boot 2024-03-26T04:59:45.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:44 smithi132 ceph-mon[33500]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T04:59:45.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:44 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:59:45.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:44 smithi132 ceph-mon[33500]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:59:45.242 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:59:46.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:45 smithi132 ceph-mon[33500]: osdmap e52: 7 total, 7 up, 7 in 2024-03-26T04:59:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:45 smithi080 ceph-mon[29691]: osdmap e52: 7 total, 7 up, 7 in 2024-03-26T04:59:46.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:45 smithi080 ceph-mon[40404]: osdmap e52: 7 total, 7 up, 7 in 2024-03-26T04:59:47.007 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:46 smithi132 ceph-mon[33500]: osdmap e53: 7 total, 7 up, 7 in 2024-03-26T04:59:47.007 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:46 smithi132 ceph-mon[33500]: pgmap v162: 1 pgs: 1 remapped+peering; 577 KiB data, 305 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:59:47.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:46 smithi080 ceph-mon[29691]: osdmap e53: 7 total, 7 up, 7 in 2024-03-26T04:59:47.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:46 smithi080 ceph-mon[29691]: pgmap v162: 1 pgs: 1 remapped+peering; 577 KiB data, 305 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:59:47.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:46 smithi080 ceph-mon[40404]: osdmap e53: 7 total, 7 up, 7 in 2024-03-26T04:59:47.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:46 smithi080 ceph-mon[40404]: pgmap v162: 1 pgs: 1 remapped+peering; 577 KiB data, 305 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:59:48.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: osdmap e54: 7 total, 7 up, 7 in 2024-03-26T04:59:48.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: Detected new or changed devices on smithi132 2024-03-26T04:59:48.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:48.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:48.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:48.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: Adjusting osd_memory_target on smithi132 to 5696M 2024-03-26T04:59:48.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:48.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:48.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:48.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:48.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:48.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:59:48.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:59:48.546 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: osdmap e54: 7 total, 7 up, 7 in 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: Detected new or changed devices on smithi132 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: Adjusting osd_memory_target on smithi132 to 5696M 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:48.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:48.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:48.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:59:48.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:59:48.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:48.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: osdmap e54: 7 total, 7 up, 7 in 2024-03-26T04:59:48.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: Detected new or changed devices on smithi132 2024-03-26T04:59:48.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:48.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: Adjusting osd_memory_target on smithi132 to 5696M 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:59:48.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:49.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:49 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:59:49.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:49 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:59:49.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:49 smithi132 ceph-mon[33500]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:59:49.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:49 smithi132 ceph-mon[33500]: osdmap e55: 7 total, 7 up, 7 in 2024-03-26T04:59:49.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:49 smithi132 ceph-mon[33500]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 305 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:59:49.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:49 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:59:49.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:49 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:59:49.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:49 smithi080 ceph-mon[29691]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:59:49.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:49 smithi080 ceph-mon[29691]: osdmap e55: 7 total, 7 up, 7 in 2024-03-26T04:59:49.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:49 smithi080 ceph-mon[29691]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 305 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:59:49.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:49 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:59:49.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:49 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:59:49.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:49 smithi080 ceph-mon[40404]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:59:49.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:49 smithi080 ceph-mon[40404]: osdmap e55: 7 total, 7 up, 7 in 2024-03-26T04:59:49.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:49 smithi080 ceph-mon[40404]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 305 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:59:52.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:51 smithi132 ceph-mon[33500]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 199 MiB used, 626 GiB / 626 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-26T04:59:52.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:51 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/2067666313' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d"}]: dispatch 2024-03-26T04:59:52.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:51 smithi132 ceph-mon[33500]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d"}]: dispatch 2024-03-26T04:59:52.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:51 smithi132 ceph-mon[33500]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d"}]': finished 2024-03-26T04:59:52.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:51 smithi132 ceph-mon[33500]: osdmap e56: 8 total, 7 up, 8 in 2024-03-26T04:59:52.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:51 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:59:52.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:51 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/3520485246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:59:52.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[40404]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 199 MiB used, 626 GiB / 626 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-26T04:59:52.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/2067666313' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d"}]: dispatch 2024-03-26T04:59:52.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[40404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d"}]: dispatch 2024-03-26T04:59:52.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[40404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d"}]': finished 2024-03-26T04:59:52.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[40404]: osdmap e56: 8 total, 7 up, 8 in 2024-03-26T04:59:52.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:59:52.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/3520485246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:59:52.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[29691]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 199 MiB used, 626 GiB / 626 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-26T04:59:52.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/2067666313' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d"}]: dispatch 2024-03-26T04:59:52.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[29691]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d"}]: dispatch 2024-03-26T04:59:52.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[29691]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d"}]': finished 2024-03-26T04:59:52.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[29691]: osdmap e56: 8 total, 7 up, 8 in 2024-03-26T04:59:52.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:59:52.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:51 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/3520485246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:59:53.928 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:53 smithi132 ceph-mon[33500]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 199 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:59:54.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:53 smithi080 ceph-mon[29691]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 199 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:59:54.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:53 smithi080 ceph-mon[40404]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 199 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:59:55.916 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:55 smithi132 ceph-mon[33500]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-26T04:59:56.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:55 smithi080 ceph-mon[29691]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-26T04:59:56.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:55 smithi080 ceph-mon[40404]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-26T04:59:56.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:56 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T04:59:56.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:56 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T04:59:56.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:56 smithi132 ceph-mon[33500]: osdmap e57: 8 total, 7 up, 8 in 2024-03-26T04:59:56.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:56 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:59:57.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:56 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T04:59:57.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:56 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T04:59:57.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:56 smithi080 ceph-mon[29691]: osdmap e57: 8 total, 7 up, 8 in 2024-03-26T04:59:57.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:56 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:59:57.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:56 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T04:59:57.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:56 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T04:59:57.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:56 smithi080 ceph-mon[40404]: osdmap e57: 8 total, 7 up, 8 in 2024-03-26T04:59:57.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:56 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:59:57.955 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:57 smithi132 ceph-mon[33500]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T04:59:57.955 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:57 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:59:57.955 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:57 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:58.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:57 smithi080 ceph-mon[29691]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T04:59:58.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:57 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:59:58.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:57 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:58.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:57 smithi080 ceph-mon[40404]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T04:59:58.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:57 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:59:58.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:57 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:58.909 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:58 smithi132 ceph-mon[33500]: Deploying daemon osd.7 on smithi132 2024-03-26T04:59:59.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:58 smithi080 ceph-mon[29691]: Deploying daemon osd.7 on smithi132 2024-03-26T04:59:59.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:58 smithi080 ceph-mon[40404]: Deploying daemon osd.7 on smithi132 2024-03-26T05:00:00.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:59 smithi132 ceph-mon[33500]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:00.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:59 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:00:00.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 04:59:59 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:00:00.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:59 smithi080 ceph-mon[29691]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:00.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:59 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:00:00.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 04:59:59 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:00:00.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:59 smithi080 ceph-mon[40404]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:00.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:59 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:00:00.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 04:59:59 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:00:00.934 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:00 smithi132 ceph-mon[33500]: osdmap e58: 8 total, 7 up, 8 in 2024-03-26T05:00:00.934 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:00 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:00.934 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:00 smithi132 ceph-mon[33500]: overall HEALTH_OK 2024-03-26T05:00:01.020 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:00 smithi080 ceph-mon[29691]: osdmap e58: 8 total, 7 up, 8 in 2024-03-26T05:00:01.020 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:00 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:01.020 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:00 smithi080 ceph-mon[29691]: overall HEALTH_OK 2024-03-26T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:00 smithi080 ceph-mon[40404]: osdmap e58: 8 total, 7 up, 8 in 2024-03-26T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:00 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:00 smithi080 ceph-mon[40404]: overall HEALTH_OK 2024-03-26T05:00:02.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:01 smithi132 ceph-mon[33500]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:02.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:01 smithi080 ceph-mon[29691]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:02.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:01 smithi080 ceph-mon[40404]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:04.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:03 smithi132 ceph-mon[33500]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:04.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:03 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:04.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:03 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:04.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:03 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:04.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:03 smithi080 ceph-mon[29691]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:04.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:03 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:04.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:03 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:04.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:03 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:04.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:03 smithi080 ceph-mon[40404]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:04.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:03 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:04.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:03 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:04.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:03 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:06.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:05 smithi132 ceph-mon[33500]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:05 smithi080 ceph-mon[29691]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:05 smithi080 ceph-mon[40404]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:06.562 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 7 on host 'smithi132' 2024-03-26T05:00:07.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:06 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:07.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:06 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:07.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:06 smithi132 ceph-mon[33500]: from='osd.7 [v2:172.21.15.132:6824/3655388282,v1:172.21.15.132:6825/3655388282]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:00:07.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:06 smithi132 ceph-mon[33500]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:00:07.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:06 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:07.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:06 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:07.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:06 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:07.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:06 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:07.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:06 smithi080 ceph-mon[29691]: from='osd.7 [v2:172.21.15.132:6824/3655388282,v1:172.21.15.132:6825/3655388282]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:00:07.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:06 smithi080 ceph-mon[29691]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:00:07.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:06 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:07.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:06 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:07.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:06 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:07.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:06 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:07.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:06 smithi080 ceph-mon[40404]: from='osd.7 [v2:172.21.15.132:6824/3655388282,v1:172.21.15.132:6825/3655388282]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:00:07.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:06 smithi080 ceph-mon[40404]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:00:07.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:06 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:07.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:06 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:07.425 DEBUG:teuthology.orchestra.run.smithi132:osd.7> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T05:00:07.427 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-26T05:00:07.428 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T05:00:07.686 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:00:08.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:07 smithi132 ceph-mon[33500]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:08.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:07 smithi132 ceph-mon[33500]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T05:00:08.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:07 smithi132 ceph-mon[33500]: osdmap e59: 8 total, 7 up, 8 in 2024-03-26T05:00:08.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:07 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:08.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:07 smithi132 ceph-mon[33500]: from='osd.7 [v2:172.21.15.132:6824/3655388282,v1:172.21.15.132:6825/3655388282]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:00:08.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:07 smithi132 ceph-mon[33500]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:00:08.045 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:00:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[55659]: 2024-03-26T05:00:07.807+0000 7ff444548700 -1 osd.7 0 waiting for initial osdmap 2024-03-26T05:00:08.045 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:00:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[55659]: 2024-03-26T05:00:07.813+0000 7ff44033b700 -1 osd.7 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T05:00:08.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:07 smithi080 ceph-mon[29691]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:08.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:07 smithi080 ceph-mon[29691]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T05:00:08.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:07 smithi080 ceph-mon[29691]: osdmap e59: 8 total, 7 up, 8 in 2024-03-26T05:00:08.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:07 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:08.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:07 smithi080 ceph-mon[29691]: from='osd.7 [v2:172.21.15.132:6824/3655388282,v1:172.21.15.132:6825/3655388282]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:00:08.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:07 smithi080 ceph-mon[29691]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:00:08.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:07 smithi080 ceph-mon[40404]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:08.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:07 smithi080 ceph-mon[40404]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T05:00:08.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:07 smithi080 ceph-mon[40404]: osdmap e59: 8 total, 7 up, 8 in 2024-03-26T05:00:08.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:07 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:08.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:07 smithi080 ceph-mon[40404]: from='osd.7 [v2:172.21.15.132:6824/3655388282,v1:172.21.15.132:6825/3655388282]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:00:08.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:07 smithi080 ceph-mon[40404]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:00:09.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:08 smithi080 ceph-mon[29691]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-26T05:00:09.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:08 smithi080 ceph-mon[29691]: osdmap e60: 8 total, 7 up, 8 in 2024-03-26T05:00:09.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:09.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:08 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:09.111 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:08 smithi080 ceph-mon[40404]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-26T05:00:09.111 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:08 smithi080 ceph-mon[40404]: osdmap e60: 8 total, 7 up, 8 in 2024-03-26T05:00:09.111 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:09.111 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:08 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:08 smithi132 ceph-mon[33500]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-26T05:00:09.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:08 smithi132 ceph-mon[33500]: osdmap e60: 8 total, 7 up, 8 in 2024-03-26T05:00:09.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:09.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:08 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:09.530 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:00:09.997 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:09 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:00:09.998 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:09 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:00:09.998 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:09 smithi132 ceph-mon[33500]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:09.998 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:09 smithi132 ceph-mon[33500]: osd.7 [v2:172.21.15.132:6824/3655388282,v1:172.21.15.132:6825/3655388282] boot 2024-03-26T05:00:09.998 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:09 smithi132 ceph-mon[33500]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T05:00:09.998 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:09 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:09.998 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:09 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/688015151' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T05:00:09.998 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:09 smithi132 ceph-mon[33500]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T05:00:10.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:00:10.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[29691]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[29691]: osd.7 [v2:172.21.15.132:6824/3655388282,v1:172.21.15.132:6825/3655388282] boot 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[29691]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/688015151' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[29691]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[40404]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[40404]: osd.7 [v2:172.21.15.132:6824/3655388282,v1:172.21.15.132:6825/3655388282] boot 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[40404]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:00:10.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/688015151' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T05:00:10.070 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:09 smithi080 ceph-mon[40404]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T05:00:10.095 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":61,"num_osds":8,"num_up_osds":8,"osd_up_since":1711429208,"num_in_osds":8,"osd_in_since":1711429190,"num_remapped_pgs":1} 2024-03-26T05:00:10.096 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T05:00:10.342 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:00:11.468 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: Detected new or changed devices on smithi132 2024-03-26T05:00:11.469 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.469 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.469 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:00:11.469 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:00:11.469 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:00:11.469 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:00:11.469 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: Adjusting osd_memory_target on smithi132 to 4272M 2024-03-26T05:00:11.469 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.469 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:11.469 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:11.469 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.470 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:11.470 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:11.470 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:11.470 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.470 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:11.470 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[40404]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T05:00:11.470 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: Detected new or changed devices on smithi132 2024-03-26T05:00:11.470 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: Adjusting osd_memory_target on smithi132 to 4272M 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:11.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:11.472 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.472 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:11.472 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:11 smithi080 ceph-mon[29691]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T05:00:11.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: Detected new or changed devices on smithi132 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: Adjusting osd_memory_target on smithi132 to 4272M 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:11.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:11.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:11.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:11.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:11.796 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:11 smithi132 ceph-mon[33500]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T05:00:12.171 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:00:12.172 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":63,"fsid":"dd09e9cc-eb2c-11ee-95d0-87774f69a715","created":"2024-03-26T04:54:51.106954+0000","modified":"2024-03-26T05:00:10.804637+0000","last_up_change":"2024-03-26T05:00:08.801055+0000","last_in_change":"2024-03-26T04:59:50.693588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:58:06.876819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"35a52544-0b0b-47d6-8d6a-571c48ceb366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6803","nonce":1840686622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6805","nonce":1840686622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6809","nonce":1840686622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6807","nonce":1840686622}]},"public_addr":"172.21.15.80:6803/1840686622","cluster_addr":"172.21.15.80:6805/1840686622","heartbeat_back_addr":"172.21.15.80:6809/1840686622","heartbeat_front_addr":"172.21.15.80:6807/1840686622","state":["exists","up"]},{"osd":1,"uuid":"efa5ea33-4f11-49b7-a3f3-f60d5329db23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6811","nonce":1726833996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6813","nonce":1726833996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6817","nonce":1726833996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6815","nonce":1726833996}]},"public_addr":"172.21.15.80:6811/1726833996","cluster_addr":"172.21.15.80:6813/1726833996","heartbeat_back_addr":"172.21.15.80:6817/1726833996","heartbeat_front_addr":"172.21.15.80:6815/1726833996","state":["exists","up"]},{"osd":2,"uuid":"536bec9b-9636-4e67-b452-ed1a55c22ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6819","nonce":128713757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6821","nonce":128713757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6825","nonce":128713757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6823","nonce":128713757}]},"public_addr":"172.21.15.80:6819/128713757","cluster_addr":"172.21.15.80:6821/128713757","heartbeat_back_addr":"172.21.15.80:6825/128713757","heartbeat_front_addr":"172.21.15.80:6823/128713757","state":["exists","up"]},{"osd":3,"uuid":"640cfdb6-b9c5-446d-8312-504ae12c7818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6827","nonce":418771441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6829","nonce":418771441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6833","nonce":418771441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6831","nonce":418771441}]},"public_addr":"172.21.15.80:6827/418771441","cluster_addr":"172.21.15.80:6829/418771441","heartbeat_back_addr":"172.21.15.80:6833/418771441","heartbeat_front_addr":"172.21.15.80:6831/418771441","state":["exists","up"]},{"osd":4,"uuid":"0cc7822a-ed73-4c79-a30d-1e151ddd6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1441071423},{"type":"v1","addr":"172.21.15.132:6801","nonce":1441071423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1441071423},{"type":"v1","addr":"172.21.15.132:6803","nonce":1441071423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1441071423},{"type":"v1","addr":"172.21.15.132:6807","nonce":1441071423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1441071423},{"type":"v1","addr":"172.21.15.132:6805","nonce":1441071423}]},"public_addr":"172.21.15.132:6801/1441071423","cluster_addr":"172.21.15.132:6803/1441071423","heartbeat_back_addr":"172.21.15.132:6807/1441071423","heartbeat_front_addr":"172.21.15.132:6805/1441071423","state":["exists","up"]},{"osd":5,"uuid":"acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6809","nonce":794504625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6811","nonce":794504625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6815","nonce":794504625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6813","nonce":794504625}]},"public_addr":"172.21.15.132:6809/794504625","cluster_addr":"172.21.15.132:6811/794504625","heartbeat_back_addr":"172.21.15.132:6815/794504625","heartbeat_front_addr":"172.21.15.132:6813/794504625","state":["exists","up"]},{"osd":6,"uuid":"2970bbab-11b2-4d1b-b57e-037bdac089b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6817","nonce":2209078972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6819","nonce":2209078972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6823","nonce":2209078972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6821","nonce":2209078972}]},"public_addr":"172.21.15.132:6817/2209078972","cluster_addr":"172.21.15.132:6819/2209078972","heartbeat_back_addr":"172.21.15.132:6823/2209078972","heartbeat_front_addr":"172.21.15.132:6821/2209078972","state":["exists","up"]},{"osd":7,"uuid":"12f4b466-28f9-469b-8fd0-65b91c6dc22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3655388282},{"type":"v1","addr":"172.21.15.132:6825","nonce":3655388282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3655388282},{"type":"v1","addr":"172.21.15.132:6827","nonce":3655388282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3655388282},{"type":"v1","addr":"172.21.15.132:6831","nonce":3655388282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3655388282},{"type":"v1","addr":"172.21.15.132:6829","nonce":3655388282}]},"public_addr":"172.21.15.132:6825/3655388282","cluster_addr":"172.21.15.132:6827/3655388282","heartbeat_back_addr":"172.21.15.132:6831/3655388282","heartbeat_front_addr":"172.21.15.132:6829/3655388282","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:28.618939+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:47.656666+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:30.837272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:55.511378+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:59.232560+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:15.860500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:41.861229+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:00:06.871179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.0","osds":[0,6]}],"primary_temp":[],"blocklist":{"172.21.15.80:0/1451012560":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/571905941":"2024-03-27T04:55:48.542948+0000","172.21.15.80:6801/2770889532":"2024-03-27T04:55:48.542948+0000","172.21.15.80:6800/2466284312":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/1993138634":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/1986135300":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/490056571":"2024-03-27T04:55:21.711776+0000","172.21.15.80:6801/2466284312":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/3796611170":"2024-03-27T04:55:21.711776+0000","172.21.15.80:6800/2770889532":"2024-03-27T04:55:48.542948+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T05:00:12.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:12 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3632995682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:00:12.843 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:12 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3632995682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:00:12.843 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:12 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3632995682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:00:12.873 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T04:58:06.876819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T05:00:12.874 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T05:00:13.110 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:00:13.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:13 smithi132 ceph-mon[33500]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T05:00:13.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:13 smithi132 ceph-mon[33500]: pgmap v186: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:13 smithi080 ceph-mon[40404]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T05:00:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:13 smithi080 ceph-mon[40404]: pgmap v186: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:13.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:13 smithi080 ceph-mon[29691]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T05:00:13.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:13 smithi080 ceph-mon[29691]: pgmap v186: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:14.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:14 smithi132 ceph-mon[33500]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T05:00:14.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:14 smithi080 ceph-mon[29691]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T05:00:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:14 smithi080 ceph-mon[40404]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T05:00:14.976 INFO:teuthology.orchestra.run.smithi080.stdout:pg_num: 1 2024-03-26T05:00:15.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:15 smithi080 ceph-mon[29691]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-26T05:00:15.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:15 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3272729319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T05:00:15.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:15 smithi080 ceph-mon[40404]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-26T05:00:15.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:15 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3272729319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T05:00:15.672 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi080 2024-03-26T05:00:15.673 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi080=foo.a' 2024-03-26T05:00:15.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:15 smithi132 ceph-mon[33500]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-26T05:00:15.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:15 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3272729319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T05:00:15.923 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:00:17.734 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled rgw.foo.a update... 2024-03-26T05:00:18.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:17 smithi132 ceph-mon[33500]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T05:00:18.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:17 smithi080 ceph-mon[29691]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T05:00:18.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:17 smithi080 ceph-mon[40404]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T05:00:18.367 DEBUG:teuthology.orchestra.run.smithi080:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T05:00:18.369 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi132 2024-03-26T05:00:18.369 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-26T05:00:18.608 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:00:19.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:18 smithi132 ceph-mon[33500]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi080=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:19.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:18 smithi132 ceph-mon[33500]: Saving service rgw.foo.a spec with placement smithi080=foo.a;count:1 2024-03-26T05:00:19.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:18 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:19.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:18 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:19.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:18 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:19.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:18 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:19.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:18 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:19.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:18 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T05:00:19.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:18 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T05:00:19.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:18 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:19.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:18 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:19.074 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[29691]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi080=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:19.074 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[29691]: Saving service rgw.foo.a spec with placement smithi080=foo.a;count:1 2024-03-26T05:00:19.074 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:19.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:19.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:19.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:19.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:19.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T05:00:19.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T05:00:19.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:19.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:19.075 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[40404]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi080=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:19.075 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[40404]: Saving service rgw.foo.a spec with placement smithi080=foo.a;count:1 2024-03-26T05:00:19.075 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:19.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:19.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:19.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:19.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:19.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T05:00:19.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T05:00:19.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:19.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:18 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:20.031 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:19 smithi132 ceph-mon[33500]: Deploying daemon rgw.foo.a on smithi080 2024-03-26T05:00:20.031 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:19 smithi132 ceph-mon[33500]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-26T05:00:20.046 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:19 smithi080 ceph-mon[29691]: Deploying daemon rgw.foo.a on smithi080 2024-03-26T05:00:20.046 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:19 smithi080 ceph-mon[29691]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-26T05:00:20.046 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:19 smithi080 ceph-mon[40404]: Deploying daemon rgw.foo.a on smithi080 2024-03-26T05:00:20.046 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:19 smithi080 ceph-mon[40404]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-26T05:00:20.735 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:00:20 smithi080 podman[67216]: 2024-03-26 05:00:20.630950045 +0000 UTC m=+0.117576658 container create e30d26b8491614153ed9bc8a081f5bf47eb914d62ed44f73f16469adbda3f81d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T05:00:20.746 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'datapool' created 2024-03-26T05:00:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:20 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/3869832293' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T05:00:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:20 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T05:00:21.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:20 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/3869832293' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T05:00:21.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:20 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T05:00:21.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:20 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/3869832293' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T05:00:21.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:20 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T05:00:21.119 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:00:20 smithi080 podman[67216]: 2024-03-26 05:00:20.875659928 +0000 UTC m=+0.362286540 container init e30d26b8491614153ed9bc8a081f5bf47eb914d62ed44f73f16469adbda3f81d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_CLEAN=True) 2024-03-26T05:00:21.119 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:00:20 smithi080 podman[67216]: 2024-03-26 05:00:20.878346109 +0000 UTC m=+0.364972722 container start e30d26b8491614153ed9bc8a081f5bf47eb914d62ed44f73f16469adbda3f81d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T05:00:21.119 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:00:20 smithi080 bash[67216]: e30d26b8491614153ed9bc8a081f5bf47eb914d62ed44f73f16469adbda3f81d 2024-03-26T05:00:21.119 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:00:20 smithi080 systemd[1]: Started Ceph rgw.foo.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:00:21.371 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- rbd pool init datapool 2024-03-26T05:00:21.605 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:00:22.039 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:21 smithi132 ceph-mon[33500]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-26T05:00:22.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:21 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T05:00:22.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:21 smithi132 ceph-mon[33500]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T05:00:22.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:21 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:21 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:21 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:21 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:21 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:21 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:22.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[29691]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-26T05:00:22.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T05:00:22.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[29691]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T05:00:22.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.058 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:22.058 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[40404]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-26T05:00:22.058 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T05:00:22.058 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[40404]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T05:00:22.058 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.058 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.058 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.058 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.058 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:22.058 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:21 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:22.976 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:22 smithi132 ceph-mon[33500]: Saving service rgw.foo.a spec with placement smithi080=foo.a;count:1 2024-03-26T05:00:22.976 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:22 smithi132 ceph-mon[33500]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T05:00:22.977 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:22 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2244010176' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T05:00:23.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:22 smithi080 ceph-mon[29691]: Saving service rgw.foo.a spec with placement smithi080=foo.a;count:1 2024-03-26T05:00:23.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:22 smithi080 ceph-mon[29691]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T05:00:23.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:22 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2244010176' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T05:00:23.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:22 smithi080 ceph-mon[40404]: Saving service rgw.foo.a spec with placement smithi080=foo.a;count:1 2024-03-26T05:00:23.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:22 smithi080 ceph-mon[40404]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T05:00:23.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:22 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2244010176' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T05:00:24.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:23 smithi132 ceph-mon[33500]: pgmap v194: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:24.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:23 smithi132 ceph-mon[33500]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:24.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:23 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2244010176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T05:00:24.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:23 smithi132 ceph-mon[33500]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T05:00:24.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:23 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:24.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:23 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:24.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:23 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:24.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:23 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:24.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:23 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/1371180842' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T05:00:24.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:23 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T05:00:24.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:23 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:24.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[29691]: pgmap v194: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:24.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[29691]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:24.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2244010176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T05:00:24.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[29691]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T05:00:24.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:24.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:24.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:24.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:24.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/1371180842' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T05:00:24.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T05:00:24.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:24.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[40404]: pgmap v194: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:24.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[40404]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:24.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2244010176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T05:00:24.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[40404]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T05:00:24.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:24.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:24.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:24.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:24.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/1371180842' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T05:00:24.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T05:00:24.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:23 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:25.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:24 smithi132 ceph-mon[33500]: Checking dashboard <-> RGW credentials 2024-03-26T05:00:25.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:24 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T05:00:25.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:24 smithi132 ceph-mon[33500]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T05:00:25.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1545910408' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:00:25.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2191591708' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:00:25.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:24 smithi132 ceph-mon[33500]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:00:25.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:24 smithi132 ceph-mon[33500]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:00:25.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:24 smithi132 ceph-mon[33500]: pgmap v197: 68 pgs: 5 creating+peering, 15 active+clean, 48 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-26T05:00:25.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:24 smithi132 ceph-mon[33500]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T05:00:25.296 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:24 smithi132 ceph-mon[33500]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T05:00:25.296 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:24 smithi132 ceph-mon[33500]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T05:00:25.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[29691]: Checking dashboard <-> RGW credentials 2024-03-26T05:00:25.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T05:00:25.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[29691]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T05:00:25.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1545910408' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:00:25.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2191591708' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[29691]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[29691]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[29691]: pgmap v197: 68 pgs: 5 creating+peering, 15 active+clean, 48 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[29691]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[29691]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[29691]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[40404]: Checking dashboard <-> RGW credentials 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[40404]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1545910408' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2191591708' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[40404]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:00:25.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[40404]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:00:25.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[40404]: pgmap v197: 68 pgs: 5 creating+peering, 15 active+clean, 48 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-26T05:00:25.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[40404]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T05:00:25.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[40404]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T05:00:25.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:24 smithi080 ceph-mon[40404]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T05:00:26.651 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.132 --placement '1;smithi132=iscsi.a' 2024-03-26T05:00:26.890 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:00:27.261 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:26 smithi132 ceph-mon[33500]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T05:00:27.262 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:26 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2191591708' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:00:27.262 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:26 smithi132 ceph-mon[33500]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:00:27.262 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:26 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1545910408' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:00:27.262 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:26 smithi132 ceph-mon[33500]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:00:27.262 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:26 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:00:27.262 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:26 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:00:27.262 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:26 smithi132 ceph-mon[33500]: pgmap v200: 100 pgs: 10 creating+peering, 44 active+clean, 46 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T05:00:27.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[29691]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T05:00:27.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2191591708' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:00:27.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[29691]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:00:27.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1545910408' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:00:27.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[29691]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:00:27.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:00:27.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:00:27.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[29691]: pgmap v200: 100 pgs: 10 creating+peering, 44 active+clean, 46 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T05:00:27.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[40404]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T05:00:27.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2191591708' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:00:27.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[40404]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:00:27.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1545910408' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:00:27.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[40404]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:00:27.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:00:27.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:00:27.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:26 smithi080 ceph-mon[40404]: pgmap v200: 100 pgs: 10 creating+peering, 44 active+clean, 46 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T05:00:28.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:27 smithi132 ceph-mon[33500]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T05:00:28.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:27 smithi132 ceph-mon[33500]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T05:00:28.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:27 smithi132 ceph-mon[33500]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T05:00:28.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:27 smithi080 ceph-mon[40404]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T05:00:28.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:27 smithi080 ceph-mon[40404]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T05:00:28.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:27 smithi080 ceph-mon[40404]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T05:00:28.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:27 smithi080 ceph-mon[29691]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T05:00:28.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:27 smithi080 ceph-mon[29691]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T05:00:28.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:27 smithi080 ceph-mon[29691]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T05:00:28.628 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled iscsi.datapool update... 2024-03-26T05:00:29.232 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:28 smithi132 ceph-mon[33500]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T05:00:29.232 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:28 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1545910408' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:00:29.232 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:28 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2191591708' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:00:29.232 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:28 smithi132 ceph-mon[33500]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:00:29.232 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:28 smithi132 ceph-mon[33500]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:00:29.232 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:28 smithi132 ceph-mon[33500]: pgmap v203: 132 pgs: 5 creating+peering, 56 active+clean, 71 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2024-03-26T05:00:29.232 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:28 smithi132 ceph-mon[33500]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.132", "placement": "1;smithi132=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:29.232 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:28 smithi132 ceph-mon[33500]: Saving service iscsi.datapool spec with placement smithi132=iscsi.a;count:1 2024-03-26T05:00:29.232 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:28 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:29.260 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-26T05:00:29.260 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T05:00:29.260 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T05:00:29.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[29691]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T05:00:29.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1545910408' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:00:29.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2191591708' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:00:29.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[29691]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:00:29.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[29691]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:00:29.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[29691]: pgmap v203: 132 pgs: 5 creating+peering, 56 active+clean, 71 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2024-03-26T05:00:29.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[29691]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.132", "placement": "1;smithi132=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:29.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[29691]: Saving service iscsi.datapool spec with placement smithi132=iscsi.a;count:1 2024-03-26T05:00:29.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:29.290 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[40404]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T05:00:29.291 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1545910408' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:00:29.291 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2191591708' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:00:29.291 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[40404]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:00:29.291 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[40404]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:00:29.291 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[40404]: pgmap v203: 132 pgs: 5 creating+peering, 56 active+clean, 71 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2024-03-26T05:00:29.291 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[40404]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.132", "placement": "1;smithi132=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:29.291 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[40404]: Saving service iscsi.datapool spec with placement smithi132=iscsi.a;count:1 2024-03-26T05:00:29.291 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:28 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:29.294 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T05:00:29.295 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T05:00:29.325 DEBUG:teuthology.orchestra.run.smithi132:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@iscsi.iscsi.a.service 2024-03-26T05:00:29.367 INFO:tasks.cephadm:Adding prometheus.a on smithi132 2024-03-26T05:00:29.368 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi132=a' 2024-03-26T05:00:29.645 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:00:30.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:29 smithi132 ceph-mon[33500]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T05:00:30.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:29 smithi132 ceph-mon[33500]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T05:00:30.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:29 smithi132 ceph-mon[33500]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T05:00:30.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:29 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1545910408' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:00:30.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:29 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2191591708' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:00:30.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:29 smithi132 ceph-mon[33500]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:00:30.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:29 smithi132 ceph-mon[33500]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:00:30.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[29691]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T05:00:30.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[29691]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T05:00:30.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[29691]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T05:00:30.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1545910408' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:00:30.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2191591708' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:00:30.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[29691]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:00:30.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[29691]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:00:30.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[40404]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T05:00:30.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[40404]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T05:00:30.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[40404]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T05:00:30.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1545910408' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:00:30.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2191591708' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:00:30.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[40404]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:00:30.371 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:29 smithi080 ceph-mon[40404]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:00:31.219 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:30 smithi132 ceph-mon[33500]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T05:00:31.220 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:30 smithi132 ceph-mon[33500]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T05:00:31.220 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:30 smithi132 ceph-mon[33500]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T05:00:31.220 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:30 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:31.220 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:30 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:31.220 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:30 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:31.220 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:30 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T05:00:31.220 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:30 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T05:00:31.220 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:30 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:31.220 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:30 smithi132 ceph-mon[33500]: Deploying daemon iscsi.iscsi.a on smithi132 2024-03-26T05:00:31.220 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:30 smithi132 ceph-mon[33500]: pgmap v206: 132 pgs: 11 creating+peering, 98 active+clean, 23 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-03-26T05:00:31.270 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[40404]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T05:00:31.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[40404]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T05:00:31.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[40404]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T05:00:31.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:31.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:31.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:31.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T05:00:31.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T05:00:31.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:31.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[40404]: Deploying daemon iscsi.iscsi.a on smithi132 2024-03-26T05:00:31.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[40404]: pgmap v206: 132 pgs: 11 creating+peering, 98 active+clean, 23 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-03-26T05:00:31.271 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[29691]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T05:00:31.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[29691]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T05:00:31.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[29691]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T05:00:31.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:31.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:31.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:31.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T05:00:31.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T05:00:31.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:31.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[29691]: Deploying daemon iscsi.iscsi.a on smithi132 2024-03-26T05:00:31.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:30 smithi080 ceph-mon[29691]: pgmap v206: 132 pgs: 11 creating+peering, 98 active+clean, 23 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-03-26T05:00:31.454 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled prometheus update... 2024-03-26T05:00:32.287 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:31 smithi080 ceph-mon[40404]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:00:32.287 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:31 smithi080 ceph-mon[40404]: Cluster is now healthy 2024-03-26T05:00:32.287 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:31 smithi080 ceph-mon[40404]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T05:00:32.287 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:31 smithi080 ceph-mon[40404]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi132=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:32.287 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:31 smithi080 ceph-mon[40404]: Saving service prometheus spec with placement smithi132=a;count:1 2024-03-26T05:00:32.287 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:31 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:32.287 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:31 smithi080 ceph-mon[29691]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:00:32.287 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:31 smithi080 ceph-mon[29691]: Cluster is now healthy 2024-03-26T05:00:32.287 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:31 smithi080 ceph-mon[29691]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T05:00:32.287 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:31 smithi080 ceph-mon[29691]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi132=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:32.288 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:31 smithi080 ceph-mon[29691]: Saving service prometheus spec with placement smithi132=a;count:1 2024-03-26T05:00:32.288 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:31 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:32.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:31 smithi132 ceph-mon[33500]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:00:32.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:31 smithi132 ceph-mon[33500]: Cluster is now healthy 2024-03-26T05:00:32.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:31 smithi132 ceph-mon[33500]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T05:00:32.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:31 smithi132 ceph-mon[33500]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi132=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:32.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:31 smithi132 ceph-mon[33500]: Saving service prometheus spec with placement smithi132=a;count:1 2024-03-26T05:00:32.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:31 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:33.191 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:32 smithi132 ceph-mon[33500]: pgmap v208: 132 pgs: 11 creating+peering, 117 active+clean, 4 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 3.0 KiB/s wr, 146 op/s 2024-03-26T05:00:33.218 DEBUG:teuthology.orchestra.run.smithi132:prometheus.a> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T05:00:33.220 INFO:tasks.cephadm:Adding node-exporter.a on smithi080 2024-03-26T05:00:33.220 INFO:tasks.cephadm:Adding node-exporter.b on smithi132 2024-03-26T05:00:33.220 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '2;smithi080=a;smithi132=b' 2024-03-26T05:00:33.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:32 smithi080 ceph-mon[29691]: pgmap v208: 132 pgs: 11 creating+peering, 117 active+clean, 4 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 3.0 KiB/s wr, 146 op/s 2024-03-26T05:00:33.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:32 smithi080 ceph-mon[40404]: pgmap v208: 132 pgs: 11 creating+peering, 117 active+clean, 4 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 3.0 KiB/s wr, 146 op/s 2024-03-26T05:00:33.983 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:00:34.108 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:33 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:00:34.108 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:33 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:00:34.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:33 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:00:34.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:33 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:00:34.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:33 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:00:34.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:33 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:00:35.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:34 smithi132 ceph-mon[33500]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T05:00:35.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:34 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:35.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:34 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:35.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:34 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:35.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:34 smithi132 ceph-mon[33500]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T05:00:35.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:34 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:35.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:34 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:35.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:34 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/188009381' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T05:00:35.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:34 smithi132 ceph-mon[33500]: pgmap v210: 132 pgs: 132 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 4.8 KiB/s wr, 238 op/s 2024-03-26T05:00:35.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[29691]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T05:00:35.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:35.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:35.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:35.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[29691]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T05:00:35.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:35.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:35.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/188009381' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T05:00:35.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[29691]: pgmap v210: 132 pgs: 132 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 4.8 KiB/s wr, 238 op/s 2024-03-26T05:00:35.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[40404]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T05:00:35.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:35.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:35.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:35.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[40404]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T05:00:35.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:35.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:35.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/188009381' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T05:00:35.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:34 smithi080 ceph-mon[40404]: pgmap v210: 132 pgs: 132 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 4.8 KiB/s wr, 238 op/s 2024-03-26T05:00:36.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:36 smithi132 ceph-mon[33500]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T05:00:36.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:36 smithi080 ceph-mon[40404]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T05:00:36.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:36 smithi080 ceph-mon[29691]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T05:00:37.029 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled node-exporter update... 2024-03-26T05:00:37.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:37 smithi132 ceph-mon[33500]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 5.1 KiB/s wr, 240 op/s 2024-03-26T05:00:37.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:37 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:37.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:37 smithi080 ceph-mon[29691]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 5.1 KiB/s wr, 240 op/s 2024-03-26T05:00:37.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:37 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:37.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:37 smithi080 ceph-mon[40404]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 5.1 KiB/s wr, 240 op/s 2024-03-26T05:00:37.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:37 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:38.025 DEBUG:teuthology.orchestra.run.smithi080:node-exporter.a> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T05:00:38.028 DEBUG:teuthology.orchestra.run.smithi132:node-exporter.b> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T05:00:38.030 INFO:tasks.cephadm:Adding alertmanager.a on smithi080 2024-03-26T05:00:38.030 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi080=a' 2024-03-26T05:00:38.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:38 smithi132 ceph-mon[33500]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi080=a;smithi132=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:38.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:38 smithi132 ceph-mon[33500]: Saving service node-exporter spec with placement smithi080=a;smithi132=b;count:2 2024-03-26T05:00:38.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:38 smithi080 ceph-mon[40404]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi080=a;smithi132=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:38.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:38 smithi080 ceph-mon[40404]: Saving service node-exporter spec with placement smithi080=a;smithi132=b;count:2 2024-03-26T05:00:38.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:38 smithi080 ceph-mon[29691]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi080=a;smithi132=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:38.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:38 smithi080 ceph-mon[29691]: Saving service node-exporter spec with placement smithi080=a;smithi132=b;count:2 2024-03-26T05:00:38.413 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:00:39.688 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:39.688 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:39.688 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:39.688 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:39.688 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:39.688 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:39 smithi132 ceph-mon[33500]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 4.1 KiB/s wr, 194 op/s 2024-03-26T05:00:39.688 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:39 smithi132 ceph-mon[33500]: Deploying daemon prometheus.a on smithi132 2024-03-26T05:00:39.688 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:39 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:39.689 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:39 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:00:39.689 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:39 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:00:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[40404]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 4.1 KiB/s wr, 194 op/s 2024-03-26T05:00:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[40404]: Deploying daemon prometheus.a on smithi132 2024-03-26T05:00:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:00:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:00:39.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:39.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:39.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:39.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:39.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:39.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[29691]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 4.1 KiB/s wr, 194 op/s 2024-03-26T05:00:39.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[29691]: Deploying daemon prometheus.a on smithi132 2024-03-26T05:00:39.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:39.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:00:39.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:39 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:00:40.216 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled alertmanager update... 2024-03-26T05:00:40.937 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:40 smithi132 ceph-mon[33500]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T05:00:40.937 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:40 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:40.963 DEBUG:teuthology.orchestra.run.smithi080:alertmanager.a> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T05:00:40.966 INFO:tasks.cephadm:Adding grafana.a on smithi132 2024-03-26T05:00:40.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi132=a' 2024-03-26T05:00:40.994 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:40 smithi080 ceph-mon[40404]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T05:00:40.994 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:40 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:40.994 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:40 smithi080 ceph-mon[29691]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T05:00:40.994 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:40 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:00:42.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:41 smithi132 ceph-mon[33500]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi080=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:42.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:41 smithi132 ceph-mon[33500]: Saving service alertmanager spec with placement smithi080=a;count:1 2024-03-26T05:00:42.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:41 smithi132 ceph-mon[33500]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 2.9 KiB/s wr, 117 op/s 2024-03-26T05:00:42.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:41 smithi132 ceph-mon[33500]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T05:00:42.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:41 smithi080 ceph-mon[29691]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi080=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:42.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:41 smithi080 ceph-mon[29691]: Saving service alertmanager spec with placement smithi080=a;count:1 2024-03-26T05:00:42.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:41 smithi080 ceph-mon[29691]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 2.9 KiB/s wr, 117 op/s 2024-03-26T05:00:42.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:41 smithi080 ceph-mon[29691]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T05:00:42.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:41 smithi080 ceph-mon[40404]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi080=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:42.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:41 smithi080 ceph-mon[40404]: Saving service alertmanager spec with placement smithi080=a;count:1 2024-03-26T05:00:42.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:41 smithi080 ceph-mon[40404]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 2.9 KiB/s wr, 117 op/s 2024-03-26T05:00:42.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:41 smithi080 ceph-mon[40404]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T05:00:43.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:43 smithi132 ceph-mon[33500]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1.1 KiB/s wr, 33 op/s 2024-03-26T05:00:43.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:43 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:43.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:43 smithi080 ceph-mon[40404]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1.1 KiB/s wr, 33 op/s 2024-03-26T05:00:43.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:43 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:43.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:43 smithi080 ceph-mon[29691]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1.1 KiB/s wr, 33 op/s 2024-03-26T05:00:43.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:43 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:44.002 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled grafana update... 2024-03-26T05:00:44.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:00:44 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:00:45.030 DEBUG:teuthology.orchestra.run.smithi132:grafana.a> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T05:00:45.032 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T05:00:45.033 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T05:00:45.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:45 smithi132 ceph-mon[33500]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi132=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:45.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:45 smithi132 ceph-mon[33500]: Saving service grafana spec with placement smithi132=a;count:1 2024-03-26T05:00:45.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:45 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:45.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:45 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:45.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:45 smithi132 ceph-mon[33500]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-26T05:00:45.295 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:00:45.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:45 smithi080 ceph-mon[29691]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi132=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:45.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:45 smithi080 ceph-mon[29691]: Saving service grafana spec with placement smithi132=a;count:1 2024-03-26T05:00:45.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:45 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:45.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:45 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:45.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:45 smithi080 ceph-mon[29691]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-26T05:00:45.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:45 smithi080 ceph-mon[40404]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi132=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:00:45.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:45 smithi080 ceph-mon[40404]: Saving service grafana spec with placement smithi132=a;count:1 2024-03-26T05:00:45.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:45 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:45.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:45 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:45.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:45 smithi080 ceph-mon[40404]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-26T05:00:47.140 INFO:teuthology.orchestra.run.smithi080.stdout:[client.0] 2024-03-26T05:00:47.140 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQB/VgJmgR27CBAASxvnqdXDQRSkwbGifhy57A== 2024-03-26T05:00:47.390 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:47 smithi080 ceph-mon[29691]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T05:00:47.390 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:47 smithi080 ceph-mon[29691]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-03-26T05:00:47.390 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:47 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2920030703' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:00:47.391 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:47 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2920030703' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T05:00:47.391 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:47 smithi080 ceph-mon[40404]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T05:00:47.391 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:47 smithi080 ceph-mon[40404]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-03-26T05:00:47.391 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:47 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2920030703' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:00:47.391 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:47 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2920030703' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T05:00:47.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:47 smithi132 ceph-mon[33500]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T05:00:47.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:47 smithi132 ceph-mon[33500]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-03-26T05:00:47.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:47 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2920030703' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:00:47.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:47 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2920030703' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T05:00:47.794 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-26T05:00:47.794 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T05:00:47.794 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T05:00:47.837 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T05:00:48.428 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:00:48.929 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-26T05:00:48.929 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-26T05:00:48.929 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:48 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-26T05:00:49.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-26T05:00:49.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-26T05:00:49.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:48 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-26T05:00:49.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-26T05:00:49.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-26T05:00:49.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:48 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-26T05:00:49.921 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:49 smithi132 ceph-mon[33500]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:00:49.921 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:49 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-26T05:00:49.921 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:49 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-26T05:00:49.921 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:49 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-26T05:00:49.921 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:49 smithi132 ceph-mon[33500]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T05:00:49.921 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:49 smithi132 ceph-mon[33500]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T05:00:50.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:49 smithi080 ceph-mon[40404]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:00:50.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:49 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-26T05:00:50.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:49 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-26T05:00:50.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:49 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-26T05:00:50.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:49 smithi080 ceph-mon[40404]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T05:00:50.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:49 smithi080 ceph-mon[40404]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T05:00:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:49 smithi080 ceph-mon[29691]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:00:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:49 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-26T05:00:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:49 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-26T05:00:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:49 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-26T05:00:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:49 smithi080 ceph-mon[29691]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T05:00:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:49 smithi080 ceph-mon[29691]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T05:00:51.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:51 smithi132 ceph-mon[33500]: pgmap v223: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T05:00:51.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:51 smithi132 ceph-mon[33500]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T05:00:52.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:51 smithi080 ceph-mon[29691]: pgmap v223: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T05:00:52.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:51 smithi080 ceph-mon[29691]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T05:00:52.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:51 smithi080 ceph-mon[40404]: pgmap v223: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T05:00:52.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:51 smithi080 ceph-mon[40404]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T05:00:53.336 INFO:teuthology.orchestra.run.smithi132.stdout:[client.1] 2024-03-26T05:00:53.337 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCFVgJmiGBIFBAABixGUnyjvH42okNUHB3y2Q== 2024-03-26T05:00:53.944 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:53 smithi132 ceph-mon[33500]: pgmap v225: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T05:00:53.944 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:53 smithi132 ceph-mon[33500]: from='client.? 172.21.15.132:0/4162734800' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:00:53.944 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:53 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:00:53.945 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:53 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T05:00:53.971 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-26T05:00:53.971 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T05:00:53.971 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T05:00:54.014 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T05:00:54.015 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T05:00:54.015 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T05:00:54.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:53 smithi080 ceph-mon[29691]: pgmap v225: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T05:00:54.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:53 smithi080 ceph-mon[29691]: from='client.? 172.21.15.132:0/4162734800' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:00:54.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:53 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:00:54.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:53 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T05:00:54.043 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:53 smithi080 ceph-mon[40404]: pgmap v225: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T05:00:54.043 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:53 smithi080 ceph-mon[40404]: from='client.? 172.21.15.132:0/4162734800' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:00:54.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:53 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:00:54.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:53 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T05:00:54.268 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:00:54.278 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:00:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:00:55.544 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:00:55 smithi132 systemd[1]: Started Ceph prometheus.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:00:55.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:55 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:55.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:55 smithi080 ceph-mon[40404]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 18 B/s, 2 objects/s recovering 2024-03-26T05:00:55.818 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:55.818 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:55.818 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:55.818 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:55 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T05:00:55.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:55 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:55.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:55 smithi080 ceph-mon[29691]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 18 B/s, 2 objects/s recovering 2024-03-26T05:00:55.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:55.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:55.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:55.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:55 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T05:00:56.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:55 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:56.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:55 smithi132 ceph-mon[33500]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 18 B/s, 2 objects/s recovering 2024-03-26T05:00:56.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:56.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:56.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' 2024-03-26T05:00:56.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:55 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T05:00:56.308 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:00:56.618 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:56 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ignoring --setuser ceph since I am not root 2024-03-26T05:00:56.618 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:56 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ignoring --setgroup ceph since I am not root 2024-03-26T05:00:56.794 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: ignoring --setuser ceph since I am not root 2024-03-26T05:00:56.794 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: ignoring --setgroup ceph since I am not root 2024-03-26T05:00:56.794 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:56 smithi132 ceph-mgr[35371]: -- 172.21.15.132:0/3896914369 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560c39a7c1a0 con 0x560c39a53000 2024-03-26T05:00:56.952 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:56 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:56.864+0000 7f30580c3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T05:00:57.053 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":1418683843},{"type":"v1","addr":"172.21.15.80:6801","nonce":1418683843}]},"active_addr":"172.21.15.80:6801/1418683843","active_change":"2024-03-26T04:55:48.543060+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.80:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":1832498181}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":1235125152}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":1650870847}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":3992539385}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":3147301987}]}]} 2024-03-26T05:00:57.057 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T05:00:57.057 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T05:00:57.057 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T05:00:57.281 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:56 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:56.959+0000 7f30580c3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T05:00:57.281 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:56.853+0000 7fbbb9be2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T05:00:57.281 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:56.947+0000 7fbbb9be2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T05:00:57.343 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:00:57.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:57 smithi132 ceph-mon[33500]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T05:00:57.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:57 smithi132 ceph-mon[33500]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T05:00:57.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:57 smithi132 ceph-mon[33500]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T05:00:57.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:57 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3404677034' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T05:00:57.579 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:57 smithi080 ceph-mon[40404]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T05:00:57.579 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:57 smithi080 ceph-mon[40404]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T05:00:57.579 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:57 smithi080 ceph-mon[40404]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T05:00:57.579 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:57 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3404677034' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T05:00:57.579 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:57 smithi080 ceph-mon[29691]: from='mgr.14150 172.21.15.80:0/619124370' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T05:00:57.579 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:57 smithi080 ceph-mon[29691]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T05:00:57.580 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:57 smithi080 ceph-mon[29691]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T05:00:57.580 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:57 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3404677034' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T05:00:57.868 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:57 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:57.637+0000 7f30580c3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T05:00:57.868 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:57 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:57.735+0000 7f30580c3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T05:00:57.905 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:57.580+0000 7fbbb9be2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T05:00:57.905 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:57.673+0000 7fbbb9be2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T05:00:57.905 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:57.815+0000 7fbbb9be2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T05:00:58.140 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:57 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:57.883+0000 7f30580c3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T05:00:58.140 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:57 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:57.979+0000 7f30580c3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T05:00:58.140 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:58 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:58.063+0000 7f30580c3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T05:00:58.257 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:57.911+0000 7fbbb9be2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T05:00:58.257 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:57.995+0000 7fbbb9be2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T05:00:58.396 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:58 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:00:58.396 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:58 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:00:58.396 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:58 smithi080 ceph-mon[40404]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T05:00:58.396 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:58 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:00:58.396 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:58 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:00:58.396 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:58 smithi080 ceph-mon[29691]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T05:00:58.396 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:58 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:58.332+0000 7f30580c3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T05:00:58.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:58 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:00:58.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:58 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:00:58.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:58 smithi132 ceph-mon[33500]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T05:00:58.544 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:58 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:58.263+0000 7fbbb9be2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T05:00:58.544 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:58 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:58.372+0000 7fbbb9be2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T05:00:58.544 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:58 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:58.459+0000 7fbbb9be2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T05:00:58.689 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:58 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:58.452+0000 7f30580c3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T05:00:58.689 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:58 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:58.541+0000 7f30580c3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T05:00:58.689 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:58 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:58.642+0000 7f30580c3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T05:00:58.961 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:58 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:58.821+0000 7f30580c3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T05:00:59.044 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:58 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:58.554+0000 7fbbb9be2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T05:00:59.044 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:58 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:58.730+0000 7fbbb9be2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T05:00:59.314 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:00:59.314 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":85,"fsid":"dd09e9cc-eb2c-11ee-95d0-87774f69a715","created":"2024-03-26T04:54:51.106954+0000","modified":"2024-03-26T05:00:57.296762+0000","last_up_change":"2024-03-26T05:00:08.801055+0000","last_in_change":"2024-03-26T04:59:50.693588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:58:06.876819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T05:00:20.705417+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T05:00:21.150368+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T05:00:23.638065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T05:00:25.181000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T05:00:27.249998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"35a52544-0b0b-47d6-8d6a-571c48ceb366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6803","nonce":1840686622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6805","nonce":1840686622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6809","nonce":1840686622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6807","nonce":1840686622}]},"public_addr":"172.21.15.80:6803/1840686622","cluster_addr":"172.21.15.80:6805/1840686622","heartbeat_back_addr":"172.21.15.80:6809/1840686622","heartbeat_front_addr":"172.21.15.80:6807/1840686622","state":["exists","up"]},{"osd":1,"uuid":"efa5ea33-4f11-49b7-a3f3-f60d5329db23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6811","nonce":1726833996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6813","nonce":1726833996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6817","nonce":1726833996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6815","nonce":1726833996}]},"public_addr":"172.21.15.80:6811/1726833996","cluster_addr":"172.21.15.80:6813/1726833996","heartbeat_back_addr":"172.21.15.80:6817/1726833996","heartbeat_front_addr":"172.21.15.80:6815/1726833996","state":["exists","up"]},{"osd":2,"uuid":"536bec9b-9636-4e67-b452-ed1a55c22ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6819","nonce":128713757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6821","nonce":128713757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6825","nonce":128713757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6823","nonce":128713757}]},"public_addr":"172.21.15.80:6819/128713757","cluster_addr":"172.21.15.80:6821/128713757","heartbeat_back_addr":"172.21.15.80:6825/128713757","heartbeat_front_addr":"172.21.15.80:6823/128713757","state":["exists","up"]},{"osd":3,"uuid":"640cfdb6-b9c5-446d-8312-504ae12c7818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6827","nonce":418771441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6829","nonce":418771441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6833","nonce":418771441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6831","nonce":418771441}]},"public_addr":"172.21.15.80:6827/418771441","cluster_addr":"172.21.15.80:6829/418771441","heartbeat_back_addr":"172.21.15.80:6833/418771441","heartbeat_front_addr":"172.21.15.80:6831/418771441","state":["exists","up"]},{"osd":4,"uuid":"0cc7822a-ed73-4c79-a30d-1e151ddd6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1441071423},{"type":"v1","addr":"172.21.15.132:6801","nonce":1441071423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1441071423},{"type":"v1","addr":"172.21.15.132:6803","nonce":1441071423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1441071423},{"type":"v1","addr":"172.21.15.132:6807","nonce":1441071423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1441071423},{"type":"v1","addr":"172.21.15.132:6805","nonce":1441071423}]},"public_addr":"172.21.15.132:6801/1441071423","cluster_addr":"172.21.15.132:6803/1441071423","heartbeat_back_addr":"172.21.15.132:6807/1441071423","heartbeat_front_addr":"172.21.15.132:6805/1441071423","state":["exists","up"]},{"osd":5,"uuid":"acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6809","nonce":794504625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6811","nonce":794504625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6815","nonce":794504625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6813","nonce":794504625}]},"public_addr":"172.21.15.132:6809/794504625","cluster_addr":"172.21.15.132:6811/794504625","heartbeat_back_addr":"172.21.15.132:6815/794504625","heartbeat_front_addr":"172.21.15.132:6813/794504625","state":["exists","up"]},{"osd":6,"uuid":"2970bbab-11b2-4d1b-b57e-037bdac089b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6817","nonce":2209078972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6819","nonce":2209078972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6823","nonce":2209078972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6821","nonce":2209078972}]},"public_addr":"172.21.15.132:6817/2209078972","cluster_addr":"172.21.15.132:6819/2209078972","heartbeat_back_addr":"172.21.15.132:6823/2209078972","heartbeat_front_addr":"172.21.15.132:6821/2209078972","state":["exists","up"]},{"osd":7,"uuid":"12f4b466-28f9-469b-8fd0-65b91c6dc22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3655388282},{"type":"v1","addr":"172.21.15.132:6825","nonce":3655388282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3655388282},{"type":"v1","addr":"172.21.15.132:6827","nonce":3655388282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3655388282},{"type":"v1","addr":"172.21.15.132:6831","nonce":3655388282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3655388282},{"type":"v1","addr":"172.21.15.132:6829","nonce":3655388282}]},"public_addr":"172.21.15.132:6825/3655388282","cluster_addr":"172.21.15.132:6827/3655388282","heartbeat_back_addr":"172.21.15.132:6831/3655388282","heartbeat_front_addr":"172.21.15.132:6829/3655388282","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:00:38.681699+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:47.656666+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:00:33.391905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:55.511378+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:59.232560+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:00:26.169057+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:00:57.224185+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:00:06.871179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.80:0/1451012560":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/571905941":"2024-03-27T04:55:48.542948+0000","172.21.15.80:6801/2770889532":"2024-03-27T04:55:48.542948+0000","172.21.15.80:6800/2466284312":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/1993138634":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/1986135300":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/490056571":"2024-03-27T04:55:21.711776+0000","172.21.15.80:6801/2466284312":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/3796611170":"2024-03-27T04:55:21.711776+0000","172.21.15.80:6800/2770889532":"2024-03-27T04:55:48.542948+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T05:00:59.607 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:00:59 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3872275134' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:00:59.607 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:00:59 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3872275134' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:00:59.684 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:00:59 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3872275134' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:00:59.868 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:59 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:59.840+0000 7f30580c3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T05:00:59.912 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T05:00:59.913 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T05:01:00.044 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:59 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:59.691+0000 7fbbb9be2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T05:01:00.044 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:00:59 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:00:59.821+0000 7fbbb9be2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T05:01:00.156 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:00.278 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:00:59 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:00:59.979+0000 7f30580c3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T05:01:00.432 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:00 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:01:00.153+0000 7fbbb9be2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T05:01:00.432 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:00 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:01:00.243+0000 7fbbb9be2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T05:01:00.570 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:00 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:01:00.327+0000 7f30580c3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T05:01:00.571 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:00 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:01:00.421+0000 7f30580c3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T05:01:00.794 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:00 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:01:00.439+0000 7fbbb9be2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:01:00.794 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:00 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:01:00.624+0000 7fbbb9be2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T05:01:00.868 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:00 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:01:00.625+0000 7f30580c3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:01:00.868 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:00 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:01:00.813+0000 7f30580c3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T05:01:01.155 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:00 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:01:00.909+0000 7fbbb9be2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:01:01.368 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:01 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:01:01.110+0000 7f30580c3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:01:01.419 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:01 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:01:01.174+0000 7fbbb9be2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T05:01:01.622 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:01 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:01:01.374+0000 7f30580c3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T05:01:01.622 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:01 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:01:01.629+0000 7f30580c3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T05:01:01.794 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:01 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:01:01.426+0000 7fbbb9be2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T05:01:02.247 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:01:02.247 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":86,"fsid":"dd09e9cc-eb2c-11ee-95d0-87774f69a715","created":"2024-03-26T04:54:51.106954+0000","modified":"2024-03-26T05:01:01.296680+0000","last_up_change":"2024-03-26T05:00:08.801055+0000","last_in_change":"2024-03-26T04:59:50.693588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:58:06.876819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T05:00:20.705417+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T05:00:21.150368+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T05:00:23.638065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T05:00:25.181000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T05:00:27.249998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"35a52544-0b0b-47d6-8d6a-571c48ceb366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6803","nonce":1840686622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6805","nonce":1840686622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6809","nonce":1840686622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6807","nonce":1840686622}]},"public_addr":"172.21.15.80:6803/1840686622","cluster_addr":"172.21.15.80:6805/1840686622","heartbeat_back_addr":"172.21.15.80:6809/1840686622","heartbeat_front_addr":"172.21.15.80:6807/1840686622","state":["exists","up"]},{"osd":1,"uuid":"efa5ea33-4f11-49b7-a3f3-f60d5329db23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6811","nonce":1726833996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6813","nonce":1726833996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6817","nonce":1726833996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6815","nonce":1726833996}]},"public_addr":"172.21.15.80:6811/1726833996","cluster_addr":"172.21.15.80:6813/1726833996","heartbeat_back_addr":"172.21.15.80:6817/1726833996","heartbeat_front_addr":"172.21.15.80:6815/1726833996","state":["exists","up"]},{"osd":2,"uuid":"536bec9b-9636-4e67-b452-ed1a55c22ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6819","nonce":128713757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6821","nonce":128713757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6825","nonce":128713757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6823","nonce":128713757}]},"public_addr":"172.21.15.80:6819/128713757","cluster_addr":"172.21.15.80:6821/128713757","heartbeat_back_addr":"172.21.15.80:6825/128713757","heartbeat_front_addr":"172.21.15.80:6823/128713757","state":["exists","up"]},{"osd":3,"uuid":"640cfdb6-b9c5-446d-8312-504ae12c7818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6827","nonce":418771441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6829","nonce":418771441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6833","nonce":418771441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6831","nonce":418771441}]},"public_addr":"172.21.15.80:6827/418771441","cluster_addr":"172.21.15.80:6829/418771441","heartbeat_back_addr":"172.21.15.80:6833/418771441","heartbeat_front_addr":"172.21.15.80:6831/418771441","state":["exists","up"]},{"osd":4,"uuid":"0cc7822a-ed73-4c79-a30d-1e151ddd6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1441071423},{"type":"v1","addr":"172.21.15.132:6801","nonce":1441071423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1441071423},{"type":"v1","addr":"172.21.15.132:6803","nonce":1441071423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1441071423},{"type":"v1","addr":"172.21.15.132:6807","nonce":1441071423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1441071423},{"type":"v1","addr":"172.21.15.132:6805","nonce":1441071423}]},"public_addr":"172.21.15.132:6801/1441071423","cluster_addr":"172.21.15.132:6803/1441071423","heartbeat_back_addr":"172.21.15.132:6807/1441071423","heartbeat_front_addr":"172.21.15.132:6805/1441071423","state":["exists","up"]},{"osd":5,"uuid":"acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6809","nonce":794504625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6811","nonce":794504625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6815","nonce":794504625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6813","nonce":794504625}]},"public_addr":"172.21.15.132:6809/794504625","cluster_addr":"172.21.15.132:6811/794504625","heartbeat_back_addr":"172.21.15.132:6815/794504625","heartbeat_front_addr":"172.21.15.132:6813/794504625","state":["exists","up"]},{"osd":6,"uuid":"2970bbab-11b2-4d1b-b57e-037bdac089b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6817","nonce":2209078972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6819","nonce":2209078972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6823","nonce":2209078972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6821","nonce":2209078972}]},"public_addr":"172.21.15.132:6817/2209078972","cluster_addr":"172.21.15.132:6819/2209078972","heartbeat_back_addr":"172.21.15.132:6823/2209078972","heartbeat_front_addr":"172.21.15.132:6821/2209078972","state":["exists","up"]},{"osd":7,"uuid":"12f4b466-28f9-469b-8fd0-65b91c6dc22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3655388282},{"type":"v1","addr":"172.21.15.132:6825","nonce":3655388282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3655388282},{"type":"v1","addr":"172.21.15.132:6827","nonce":3655388282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3655388282},{"type":"v1","addr":"172.21.15.132:6831","nonce":3655388282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3655388282},{"type":"v1","addr":"172.21.15.132:6829","nonce":3655388282}]},"public_addr":"172.21.15.132:6825/3655388282","cluster_addr":"172.21.15.132:6827/3655388282","heartbeat_back_addr":"172.21.15.132:6831/3655388282","heartbeat_front_addr":"172.21.15.132:6829/3655388282","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:00:38.681699+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:47.656666+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:00:33.391905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:55.511378+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:59:59.232560+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:00:26.169057+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:00:57.224185+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:00:06.871179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.80:0/1451012560":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/571905941":"2024-03-27T04:55:48.542948+0000","172.21.15.80:6801/2770889532":"2024-03-27T04:55:48.542948+0000","172.21.15.80:6800/2466284312":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/1993138634":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/1986135300":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/490056571":"2024-03-27T04:55:21.711776+0000","172.21.15.80:6801/2466284312":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/3796611170":"2024-03-27T04:55:21.711776+0000","172.21.15.80:6800/2770889532":"2024-03-27T04:55:48.542948+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T05:01:02.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:02 smithi132 ceph-mon[33500]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T05:01:02.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:02 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/601093169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:01:02.544 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:01:02.296+0000 7fbbb9be2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T05:01:02.547 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:02 smithi080 ceph-mon[40404]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T05:01:02.547 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:02 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/601093169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:01:02.548 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:02 smithi080 ceph-mon[29691]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T05:01:02.548 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:02 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/601093169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:01:02.548 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:02 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:01:02.490+0000 7f30580c3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T05:01:03.040 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T05:01:03.041 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T05:01:03.041 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T05:01:03.042 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T05:01:03.042 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T05:01:03.042 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T05:01:03.043 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T05:01:03.043 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T05:01:03.185 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:01:02.876+0000 7fbbb9be2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T05:01:03.185 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:01:02.975+0000 7fbbb9be2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T05:01:03.281 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:01:03.105+0000 7f30580c3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T05:01:03.281 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:01:03.287+0000 7f30580c3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T05:01:03.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-mon[33500]: Standby manager daemon x restarted 2024-03-26T05:01:03.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-mon[33500]: Standby manager daemon x started 2024-03-26T05:01:03.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-mon[33500]: from='mgr.? 172.21.15.132:0/2239913628' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T05:01:03.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-mon[33500]: from='mgr.? 172.21.15.132:0/2239913628' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T05:01:03.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-mon[33500]: from='mgr.? 172.21.15.132:0/2239913628' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T05:01:03.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-mon[33500]: from='mgr.? 172.21.15.132:0/2239913628' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T05:01:03.544 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:01:03.191+0000 7fbbb9be2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T05:01:03.545 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: [26/Mar/2024:05:01:03] ENGINE Bus STARTING 2024-03-26T05:01:03.545 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: CherryPy Checker: 2024-03-26T05:01:03.545 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: The Application mounted at '' has an empty config. 2024-03-26T05:01:03.545 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: 2024-03-26T05:01:03.545 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: [26/Mar/2024:05:01:03] ENGINE Serving on http://:::9283 2024-03-26T05:01:03.545 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:01:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x[35367]: [26/Mar/2024:05:01:03] ENGINE Bus STARTED 2024-03-26T05:01:03.572 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:03.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-mon[29691]: Standby manager daemon x restarted 2024-03-26T05:01:03.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-mon[29691]: Standby manager daemon x started 2024-03-26T05:01:03.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-mon[29691]: from='mgr.? 172.21.15.132:0/2239913628' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T05:01:03.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-mon[29691]: from='mgr.? 172.21.15.132:0/2239913628' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T05:01:03.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-mon[29691]: from='mgr.? 172.21.15.132:0/2239913628' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T05:01:03.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-mon[29691]: from='mgr.? 172.21.15.132:0/2239913628' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T05:01:03.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-mon[40404]: Standby manager daemon x restarted 2024-03-26T05:01:03.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-mon[40404]: Standby manager daemon x started 2024-03-26T05:01:03.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-mon[40404]: from='mgr.? 172.21.15.132:0/2239913628' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T05:01:03.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-mon[40404]: from='mgr.? 172.21.15.132:0/2239913628' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T05:01:03.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-mon[40404]: from='mgr.? 172.21.15.132:0/2239913628' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T05:01:03.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-mon[40404]: from='mgr.? 172.21.15.132:0/2239913628' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T05:01:03.633 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:03.982 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:01:03.675+0000 7f30580c3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T05:01:03.982 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:03] ENGINE Bus STARTING 2024-03-26T05:01:03.982 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: CherryPy Checker: 2024-03-26T05:01:03.982 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: The Application mounted at '' has an empty config. 2024-03-26T05:01:03.982 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:01:03.982 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:03] ENGINE Serving on http://:::9283 2024-03-26T05:01:03.982 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:03 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:03] ENGINE Bus STARTED 2024-03-26T05:01:04.323 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:01:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:01:04.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: Active manager daemon y restarted 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: Activating manager daemon y 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: mgrmap e19: y(active, starting, since 0.00792743s), standbys: x 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:01:04.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:01:04.325 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:01:04.325 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:01:04.325 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:01:04.331 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:04.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:01:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:01:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:01:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T05:01:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T05:01:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T05:01:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: Manager daemon y is now available 2024-03-26T05:01:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:01:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:01:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T05:01:04.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T05:01:04.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:01:04.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:01:04.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T05:01:04.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T05:01:04.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: Active manager daemon y restarted 2024-03-26T05:01:04.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: Activating manager daemon y 2024-03-26T05:01:04.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T05:01:04.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: mgrmap e19: y(active, starting, since 0.00792743s), standbys: x 2024-03-26T05:01:04.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:01:04.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:01:04.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:01:04.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T05:01:04.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T05:01:04.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:01:04.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:01:04.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:01:04.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:01:04.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:01:04.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:01:04.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:01:04.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:01:04.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T05:01:04.621 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T05:01:04.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T05:01:04.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: Manager daemon y is now available 2024-03-26T05:01:04.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:01:04.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:01:04.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:04.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:04.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T05:01:04.622 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T05:01:04.697 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:01:04.697 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:01:04.697 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T05:01:04.697 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T05:01:04.697 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: Active manager daemon y restarted 2024-03-26T05:01:04.697 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: Activating manager daemon y 2024-03-26T05:01:04.697 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T05:01:04.697 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: mgrmap e19: y(active, starting, since 0.00792743s), standbys: x 2024-03-26T05:01:04.697 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T05:01:04.698 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T05:01:04.699 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T05:01:04.699 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: Manager daemon y is now available 2024-03-26T05:01:04.699 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:01:04.699 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:01:04.699 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:04.699 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:04.699 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T05:01:04.699 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T05:01:05.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:01:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:01:05.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:05 smithi080 ceph-mon[40404]: [26/Mar/2024:05:01:04] ENGINE Bus STARTING 2024-03-26T05:01:05.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:05 smithi080 ceph-mon[40404]: [26/Mar/2024:05:01:04] ENGINE Serving on http://172.21.15.80:8765 2024-03-26T05:01:05.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:05 smithi080 ceph-mon[40404]: mgrmap e20: y(active, since 1.01163s), standbys: x 2024-03-26T05:01:05.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:05 smithi080 ceph-mon[40404]: 1.0 scrub starts 2024-03-26T05:01:05.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:05 smithi080 ceph-mon[40404]: 1.0 scrub ok 2024-03-26T05:01:05.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:05 smithi080 ceph-mon[29691]: [26/Mar/2024:05:01:04] ENGINE Bus STARTING 2024-03-26T05:01:05.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:05 smithi080 ceph-mon[29691]: [26/Mar/2024:05:01:04] ENGINE Serving on http://172.21.15.80:8765 2024-03-26T05:01:05.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:05 smithi080 ceph-mon[29691]: mgrmap e20: y(active, since 1.01163s), standbys: x 2024-03-26T05:01:05.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:05 smithi080 ceph-mon[29691]: 1.0 scrub starts 2024-03-26T05:01:05.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:05 smithi080 ceph-mon[29691]: 1.0 scrub ok 2024-03-26T05:01:05.705 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:05.706 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:05.706 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:05.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:05 smithi132 ceph-mon[33500]: [26/Mar/2024:05:01:04] ENGINE Bus STARTING 2024-03-26T05:01:05.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:05 smithi132 ceph-mon[33500]: [26/Mar/2024:05:01:04] ENGINE Serving on http://172.21.15.80:8765 2024-03-26T05:01:05.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:05 smithi132 ceph-mon[33500]: mgrmap e20: y(active, since 1.01163s), standbys: x 2024-03-26T05:01:05.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:05 smithi132 ceph-mon[33500]: 1.0 scrub starts 2024-03-26T05:01:05.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:05 smithi132 ceph-mon[33500]: 1.0 scrub ok 2024-03-26T05:01:06.582 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:06 smithi080 ceph-mon[40404]: [26/Mar/2024:05:01:04] ENGINE Serving on https://172.21.15.80:7150 2024-03-26T05:01:06.582 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:06 smithi080 ceph-mon[40404]: [26/Mar/2024:05:01:04] ENGINE Bus STARTED 2024-03-26T05:01:06.582 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:06 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:06.582 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:06 smithi080 ceph-mon[40404]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:06.582 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:06 smithi080 ceph-mon[29691]: [26/Mar/2024:05:01:04] ENGINE Serving on https://172.21.15.80:7150 2024-03-26T05:01:06.583 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:06 smithi080 ceph-mon[29691]: [26/Mar/2024:05:01:04] ENGINE Bus STARTED 2024-03-26T05:01:06.583 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:06 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:06.583 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:06 smithi080 ceph-mon[29691]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:06.651 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:06 smithi132 ceph-mon[33500]: [26/Mar/2024:05:01:04] ENGINE Serving on https://172.21.15.80:7150 2024-03-26T05:01:06.651 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:06 smithi132 ceph-mon[33500]: [26/Mar/2024:05:01:04] ENGINE Bus STARTED 2024-03-26T05:01:06.652 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:06 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:06.652 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:06 smithi132 ceph-mon[33500]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:07.512 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:07 smithi080 ceph-mon[40404]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:07.512 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:07 smithi080 ceph-mon[40404]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T05:01:07.513 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:07 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:07.513 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:07 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:07.513 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:07 smithi080 ceph-mon[29691]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:07.513 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:07 smithi080 ceph-mon[29691]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T05:01:07.513 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:07 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:07.513 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:07 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:07.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:07 smithi132 ceph-mon[33500]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:07.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:07 smithi132 ceph-mon[33500]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T05:01:07.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:07 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:07.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:07 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:08.100 INFO:teuthology.orchestra.run.smithi080.stdout:55834574892 2024-03-26T05:01:08.101 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T05:01:08.110 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:08.110 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:09.244 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[29691]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:09.244 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:09.244 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:09.244 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:09.244 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:09.244 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:09.244 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:09.245 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[29691]: Adjusting osd_memory_target on smithi132 to 3760M 2024-03-26T05:01:09.245 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:09.245 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[29691]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T05:01:09.245 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[40404]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:09.245 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:09.245 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:09.245 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:09.245 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:09.245 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:09.245 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:09.245 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[40404]: Adjusting osd_memory_target on smithi132 to 3760M 2024-03-26T05:01:09.245 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:09.246 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:08 smithi080 ceph-mon[40404]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T05:01:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:08 smithi132 ceph-mon[33500]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:08 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:08 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:08 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:08 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:08 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:08 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:08 smithi132 ceph-mon[33500]: Adjusting osd_memory_target on smithi132 to 3760M 2024-03-26T05:01:09.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:08 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:09.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:08 smithi132 ceph-mon[33500]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T05:01:11.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:10 smithi080 ceph-mon[29691]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:11.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:10 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:01:11.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:10 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:01:11.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:10 smithi080 ceph-mon[40404]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:11.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:10 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:01:11.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:10 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:01:11.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:10 smithi132 ceph-mon[33500]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:11.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:10 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:01:11.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:10 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:01:12.199 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:11 smithi080 ceph-mon[40404]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T05:01:12.199 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:11 smithi080 ceph-mon[40404]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T05:01:12.200 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:11 smithi080 ceph-mon[29691]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T05:01:12.200 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:11 smithi080 ceph-mon[29691]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T05:01:12.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:11 smithi132 ceph-mon[33500]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T05:01:12.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:11 smithi132 ceph-mon[33500]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T05:01:13.602 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:13 smithi080 ceph-mon[29691]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T05:01:13.602 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:13 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:01:13.602 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:13 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:01:13.603 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:13 smithi080 ceph-mon[40404]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T05:01:13.603 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:13 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:01:13.603 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:13 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:01:13.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:13 smithi132 ceph-mon[33500]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T05:01:13.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:13 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:01:13.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:13 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:01:14.319 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:01:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:01:14.586 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:14 smithi080 ceph-mon[40404]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T05:01:14.587 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:14 smithi080 ceph-mon[29691]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T05:01:14.707 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:14 smithi132 ceph-mon[33500]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T05:01:15.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:01:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:01:15.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:15 smithi080 ceph-mon[29691]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-26T05:01:15.585 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:15 smithi080 ceph-mon[40404]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-26T05:01:15.612 INFO:teuthology.orchestra.run.smithi080.stdout:219043332115 2024-03-26T05:01:15.612 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T05:01:15.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:15 smithi132 ceph-mon[33500]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-26T05:01:16.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:16 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:16.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:16 smithi080 ceph-mon[29691]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T05:01:16.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:16 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:16.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:16 smithi080 ceph-mon[40404]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T05:01:16.678 INFO:teuthology.orchestra.run.smithi080.stdout:261993005070 2024-03-26T05:01:16.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T05:01:16.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:16 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:16.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:16 smithi132 ceph-mon[33500]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T05:01:17.202 INFO:teuthology.orchestra.run.smithi080.stdout:34359738418 2024-03-26T05:01:17.202 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T05:01:17.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:17 smithi080 ceph-mon[29691]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-26T05:01:17.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:17 smithi080 ceph-mon[40404]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-26T05:01:17.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:17 smithi132 ceph-mon[33500]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-26T05:01:18.610 INFO:teuthology.orchestra.run.smithi080.stdout:176093659161 2024-03-26T05:01:18.610 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T05:01:19.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:19 smithi080 ceph-mon[40404]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 799 B/s rd, 0 op/s 2024-03-26T05:01:19.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:19.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:19 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:01:19.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:19 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:01:19.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:19 smithi080 ceph-mon[29691]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 799 B/s rd, 0 op/s 2024-03-26T05:01:19.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:19.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:19 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:01:19.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:19 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:01:19.663 INFO:teuthology.orchestra.run.smithi080.stdout:115964117027 2024-03-26T05:01:19.664 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T05:01:19.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:19 smithi132 ceph-mon[33500]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 799 B/s rd, 0 op/s 2024-03-26T05:01:19.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:19.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:19 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:01:19.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:19 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:01:20.175 INFO:teuthology.orchestra.run.smithi080.stdout:77309411369 2024-03-26T05:01:20.175 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T05:01:20.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:20 smithi080 ceph-mon[40404]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T05:01:20.584 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:20 smithi080 ceph-mon[29691]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T05:01:20.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:20 smithi132 ceph-mon[33500]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T05:01:21.423 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:21.424 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:21.424 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:21.424 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:21.426 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:21.440 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:21.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:21 smithi080 ceph-mon[40404]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T05:01:21.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:21 smithi080 ceph-mon[40404]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T05:01:21.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:21 smithi080 ceph-mon[29691]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T05:01:21.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:21 smithi080 ceph-mon[29691]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T05:01:21.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:21 smithi132 ceph-mon[33500]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T05:01:21.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:21 smithi132 ceph-mon[33500]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T05:01:21.821 INFO:teuthology.orchestra.run.smithi080.stdout:141733920800 2024-03-26T05:01:21.822 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T05:01:23.260 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:23 smithi080 ceph-mon[29691]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:23.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:23 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:23.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:23 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:23.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:23 smithi080 ceph-mon[29691]: 3.f scrub starts 2024-03-26T05:01:23.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:23 smithi080 ceph-mon[29691]: 3.f scrub ok 2024-03-26T05:01:23.261 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:23 smithi080 ceph-mon[40404]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:23.261 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:23 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:23.261 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:23 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:23.261 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:23 smithi080 ceph-mon[40404]: 3.f scrub starts 2024-03-26T05:01:23.261 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:23 smithi080 ceph-mon[40404]: 3.f scrub ok 2024-03-26T05:01:23.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:23 smithi132 ceph-mon[33500]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:23.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:23 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:23.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:23 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:23.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:23 smithi132 ceph-mon[33500]: 3.f scrub starts 2024-03-26T05:01:23.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:23 smithi132 ceph-mon[33500]: 3.f scrub ok 2024-03-26T05:01:24.202 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:01:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:01:24.236 INFO:teuthology.orchestra.run.smithi080.stdout:219043332117 2024-03-26T05:01:25.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:01:24 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:01:25.311 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-03-26T05:01:25.311 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: Adjusting osd_memory_target on smithi080 to 3760M 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: Updating smithi132:/etc/ceph/ceph.conf 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3033042408' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: Updating smithi080:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:01:25.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:01:25.313 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: 4.1c scrub starts 2024-03-26T05:01:25.313 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[29691]: 4.1c scrub ok 2024-03-26T05:01:25.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-03-26T05:01:25.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:25.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:25.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:25.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:25.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:25.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:25.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: Adjusting osd_memory_target on smithi080 to 3760M 2024-03-26T05:01:25.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:25.314 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:01:25.314 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:01:25.314 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-26T05:01:25.314 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: Updating smithi132:/etc/ceph/ceph.conf 2024-03-26T05:01:25.314 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3033042408' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T05:01:25.314 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: Updating smithi080:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:01:25.314 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:01:25.314 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: 4.1c scrub starts 2024-03-26T05:01:25.314 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:25 smithi080 ceph-mon[40404]: 4.1c scrub ok 2024-03-26T05:01:25.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-03-26T05:01:25.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:25.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:25.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:25.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:25.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:25.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:01:25.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: Adjusting osd_memory_target on smithi080 to 3760M 2024-03-26T05:01:25.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:25.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:01:25.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:01:25.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-26T05:01:25.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: Updating smithi132:/etc/ceph/ceph.conf 2024-03-26T05:01:25.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3033042408' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T05:01:25.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: Updating smithi080:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:01:25.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:01:25.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: 4.1c scrub starts 2024-03-26T05:01:25.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:25 smithi132 ceph-mon[33500]: 4.1c scrub ok 2024-03-26T05:01:26.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:01:26.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:01:26.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:26.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:01:26.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: Updating smithi080:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:01:26.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: Deploying daemon alertmanager.a on smithi080 2024-03-26T05:01:26.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: 3.2 scrub starts 2024-03-26T05:01:26.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:26 smithi132 ceph-mon[33500]: 3.2 scrub ok 2024-03-26T05:01:26.588 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:01:26.588 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:01:26.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:26.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:01:26.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: Updating smithi080:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:01:26.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: Deploying daemon alertmanager.a on smithi080 2024-03-26T05:01:26.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: 3.2 scrub starts 2024-03-26T05:01:26.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[40404]: 3.2 scrub ok 2024-03-26T05:01:26.589 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:01:26.590 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:01:26.590 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:26.590 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: Updating smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:01:26.590 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: Updating smithi080:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:01:26.590 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.590 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.590 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.590 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.590 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:26.590 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: Deploying daemon alertmanager.a on smithi080 2024-03-26T05:01:26.590 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: 3.2 scrub starts 2024-03-26T05:01:26.591 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:26 smithi080 ceph-mon[29691]: 3.2 scrub ok 2024-03-26T05:01:27.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:27 smithi132 ceph-mon[33500]: 3.1 scrub starts 2024-03-26T05:01:27.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:27 smithi132 ceph-mon[33500]: 3.1 scrub ok 2024-03-26T05:01:27.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:27 smithi132 ceph-mon[33500]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:27.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:27 smithi132 ceph-mon[33500]: 5.8 scrub starts 2024-03-26T05:01:27.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:27 smithi132 ceph-mon[33500]: 5.8 scrub ok 2024-03-26T05:01:27.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:27 smithi132 ceph-mon[33500]: 3.13 scrub starts 2024-03-26T05:01:27.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:27 smithi132 ceph-mon[33500]: 3.13 scrub ok 2024-03-26T05:01:27.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[29691]: 3.1 scrub starts 2024-03-26T05:01:27.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[29691]: 3.1 scrub ok 2024-03-26T05:01:27.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[29691]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:27.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[29691]: 5.8 scrub starts 2024-03-26T05:01:27.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[29691]: 5.8 scrub ok 2024-03-26T05:01:27.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[29691]: 3.13 scrub starts 2024-03-26T05:01:27.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[29691]: 3.13 scrub ok 2024-03-26T05:01:27.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[40404]: 3.1 scrub starts 2024-03-26T05:01:27.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[40404]: 3.1 scrub ok 2024-03-26T05:01:27.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[40404]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:27.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[40404]: 5.8 scrub starts 2024-03-26T05:01:27.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[40404]: 5.8 scrub ok 2024-03-26T05:01:27.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[40404]: 3.13 scrub starts 2024-03-26T05:01:27.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:27 smithi080 ceph-mon[40404]: 3.13 scrub ok 2024-03-26T05:01:28.432 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:28 smithi080 ceph-mon[40404]: 3.9 scrub starts 2024-03-26T05:01:28.432 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:28 smithi080 ceph-mon[40404]: 3.9 scrub ok 2024-03-26T05:01:28.432 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:28 smithi080 ceph-mon[40404]: 4.17 scrub starts 2024-03-26T05:01:28.432 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:28 smithi080 ceph-mon[40404]: 4.17 scrub ok 2024-03-26T05:01:28.433 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:28 smithi080 ceph-mon[29691]: 3.9 scrub starts 2024-03-26T05:01:28.433 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:28 smithi080 ceph-mon[29691]: 3.9 scrub ok 2024-03-26T05:01:28.433 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:28 smithi080 ceph-mon[29691]: 4.17 scrub starts 2024-03-26T05:01:28.433 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:28 smithi080 ceph-mon[29691]: 4.17 scrub ok 2024-03-26T05:01:28.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:28 smithi132 ceph-mon[33500]: 3.9 scrub starts 2024-03-26T05:01:28.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:28 smithi132 ceph-mon[33500]: 3.9 scrub ok 2024-03-26T05:01:28.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:28 smithi132 ceph-mon[33500]: 4.17 scrub starts 2024-03-26T05:01:28.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:28 smithi132 ceph-mon[33500]: 4.17 scrub ok 2024-03-26T05:01:28.547 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:28.966 INFO:teuthology.orchestra.run.smithi080.stdout:261993005073 2024-03-26T05:01:29.233 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:29 smithi080 ceph-mon[40404]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:29.233 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:29 smithi080 ceph-mon[40404]: 3.a deep-scrub starts 2024-03-26T05:01:29.233 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:29 smithi080 ceph-mon[40404]: 3.a deep-scrub ok 2024-03-26T05:01:29.233 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:29 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1894685909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T05:01:29.233 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:29 smithi080 ceph-mon[40404]: 6.b scrub starts 2024-03-26T05:01:29.233 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:29 smithi080 ceph-mon[40404]: 6.b scrub ok 2024-03-26T05:01:29.233 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:29 smithi080 ceph-mon[29691]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:29.233 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:29 smithi080 ceph-mon[29691]: 3.a deep-scrub starts 2024-03-26T05:01:29.234 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:29 smithi080 ceph-mon[29691]: 3.a deep-scrub ok 2024-03-26T05:01:29.234 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:29 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1894685909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T05:01:29.234 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:29 smithi080 ceph-mon[29691]: 6.b scrub starts 2024-03-26T05:01:29.234 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:29 smithi080 ceph-mon[29691]: 6.b scrub ok 2024-03-26T05:01:29.257 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332115 got 219043332117 for osd.6 2024-03-26T05:01:29.257 DEBUG:teuthology.parallel:result is None 2024-03-26T05:01:29.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:29 smithi132 ceph-mon[33500]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:29.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:29 smithi132 ceph-mon[33500]: 3.a deep-scrub starts 2024-03-26T05:01:29.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:29 smithi132 ceph-mon[33500]: 3.a deep-scrub ok 2024-03-26T05:01:29.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:29 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1894685909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T05:01:29.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:29 smithi132 ceph-mon[33500]: 6.b scrub starts 2024-03-26T05:01:29.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:29 smithi132 ceph-mon[33500]: 6.b scrub ok 2024-03-26T05:01:30.342 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[29691]: 4.f scrub starts 2024-03-26T05:01:30.343 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[29691]: 4.f scrub ok 2024-03-26T05:01:30.343 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[29691]: 4.8 scrub starts 2024-03-26T05:01:30.343 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[29691]: 4.8 scrub ok 2024-03-26T05:01:30.343 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[29691]: 3.6 scrub starts 2024-03-26T05:01:30.343 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[29691]: 3.6 scrub ok 2024-03-26T05:01:30.343 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[29691]: 6.1f scrub starts 2024-03-26T05:01:30.343 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[29691]: 6.1f scrub ok 2024-03-26T05:01:30.343 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[40404]: 4.f scrub starts 2024-03-26T05:01:30.344 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[40404]: 4.f scrub ok 2024-03-26T05:01:30.344 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[40404]: 4.8 scrub starts 2024-03-26T05:01:30.344 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[40404]: 4.8 scrub ok 2024-03-26T05:01:30.344 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[40404]: 3.6 scrub starts 2024-03-26T05:01:30.344 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[40404]: 3.6 scrub ok 2024-03-26T05:01:30.344 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[40404]: 6.1f scrub starts 2024-03-26T05:01:30.344 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:30 smithi080 ceph-mon[40404]: 6.1f scrub ok 2024-03-26T05:01:30.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:30 smithi132 ceph-mon[33500]: 4.f scrub starts 2024-03-26T05:01:30.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:30 smithi132 ceph-mon[33500]: 4.f scrub ok 2024-03-26T05:01:30.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:30 smithi132 ceph-mon[33500]: 4.8 scrub starts 2024-03-26T05:01:30.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:30 smithi132 ceph-mon[33500]: 4.8 scrub ok 2024-03-26T05:01:30.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:30 smithi132 ceph-mon[33500]: 3.6 scrub starts 2024-03-26T05:01:30.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:30 smithi132 ceph-mon[33500]: 3.6 scrub ok 2024-03-26T05:01:30.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:30 smithi132 ceph-mon[33500]: 6.1f scrub starts 2024-03-26T05:01:30.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:30 smithi132 ceph-mon[33500]: 6.1f scrub ok 2024-03-26T05:01:30.655 INFO:teuthology.orchestra.run.smithi080.stdout:55834574897 2024-03-26T05:01:31.434 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:31 smithi080 ceph-mon[40404]: 5.6 scrub starts 2024-03-26T05:01:31.435 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:31 smithi080 ceph-mon[40404]: 5.6 scrub ok 2024-03-26T05:01:31.435 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:31 smithi080 ceph-mon[40404]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:01:31.435 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:31 smithi080 ceph-mon[40404]: 3.16 scrub starts 2024-03-26T05:01:31.435 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:31 smithi080 ceph-mon[40404]: 3.16 scrub ok 2024-03-26T05:01:31.435 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:31 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1386139181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T05:01:31.435 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:31 smithi080 ceph-mon[29691]: 5.6 scrub starts 2024-03-26T05:01:31.435 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:31 smithi080 ceph-mon[29691]: 5.6 scrub ok 2024-03-26T05:01:31.436 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:31 smithi080 ceph-mon[29691]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:01:31.436 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:31 smithi080 ceph-mon[29691]: 3.16 scrub starts 2024-03-26T05:01:31.436 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:31 smithi080 ceph-mon[29691]: 3.16 scrub ok 2024-03-26T05:01:31.436 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:31 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1386139181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T05:01:31.445 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:31 smithi132 ceph-mon[33500]: 5.6 scrub starts 2024-03-26T05:01:31.445 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:31 smithi132 ceph-mon[33500]: 5.6 scrub ok 2024-03-26T05:01:31.445 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:31 smithi132 ceph-mon[33500]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:01:31.445 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:31 smithi132 ceph-mon[33500]: 3.16 scrub starts 2024-03-26T05:01:31.445 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:31 smithi132 ceph-mon[33500]: 3.16 scrub ok 2024-03-26T05:01:31.446 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:31 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1386139181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T05:01:31.802 INFO:teuthology.orchestra.run.smithi080.stdout:176093659164 2024-03-26T05:01:31.930 INFO:teuthology.orchestra.run.smithi080.stdout:34359738422 2024-03-26T05:01:32.457 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:32 smithi132 ceph-mon[33500]: 4.1a scrub starts 2024-03-26T05:01:32.457 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:32 smithi132 ceph-mon[33500]: 4.1a scrub ok 2024-03-26T05:01:32.457 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:32 smithi132 ceph-mon[33500]: 5.12 deep-scrub starts 2024-03-26T05:01:32.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:32 smithi132 ceph-mon[33500]: 5.12 deep-scrub ok 2024-03-26T05:01:32.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:32 smithi132 ceph-mon[33500]: 3.1d scrub starts 2024-03-26T05:01:32.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:32 smithi132 ceph-mon[33500]: 3.1d scrub ok 2024-03-26T05:01:32.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:32 smithi132 ceph-mon[33500]: 6.14 scrub starts 2024-03-26T05:01:32.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:32 smithi132 ceph-mon[33500]: 6.14 scrub ok 2024-03-26T05:01:32.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:32 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4130555333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T05:01:32.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:32 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1637081006' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T05:01:32.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[40404]: 4.1a scrub starts 2024-03-26T05:01:32.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[40404]: 4.1a scrub ok 2024-03-26T05:01:32.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[40404]: 5.12 deep-scrub starts 2024-03-26T05:01:32.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[40404]: 5.12 deep-scrub ok 2024-03-26T05:01:32.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[40404]: 3.1d scrub starts 2024-03-26T05:01:32.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[40404]: 3.1d scrub ok 2024-03-26T05:01:32.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[40404]: 6.14 scrub starts 2024-03-26T05:01:32.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[40404]: 6.14 scrub ok 2024-03-26T05:01:32.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4130555333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T05:01:32.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1637081006' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T05:01:32.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[29691]: 4.1a scrub starts 2024-03-26T05:01:32.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[29691]: 4.1a scrub ok 2024-03-26T05:01:32.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[29691]: 5.12 deep-scrub starts 2024-03-26T05:01:32.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[29691]: 5.12 deep-scrub ok 2024-03-26T05:01:32.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[29691]: 3.1d scrub starts 2024-03-26T05:01:32.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[29691]: 3.1d scrub ok 2024-03-26T05:01:32.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[29691]: 6.14 scrub starts 2024-03-26T05:01:32.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[29691]: 6.14 scrub ok 2024-03-26T05:01:32.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4130555333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T05:01:32.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:32 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1637081006' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T05:01:33.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:33 smithi080 ceph-mon[40404]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-03-26T05:01:33.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:33 smithi080 ceph-mon[40404]: 6.18 scrub starts 2024-03-26T05:01:33.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:33 smithi080 ceph-mon[40404]: 6.18 scrub ok 2024-03-26T05:01:33.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:33 smithi080 ceph-mon[29691]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-03-26T05:01:33.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:33 smithi080 ceph-mon[29691]: 6.18 scrub starts 2024-03-26T05:01:33.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:33 smithi080 ceph-mon[29691]: 6.18 scrub ok 2024-03-26T05:01:33.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:33 smithi132 ceph-mon[33500]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-03-26T05:01:33.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:33 smithi132 ceph-mon[33500]: 6.18 scrub starts 2024-03-26T05:01:33.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:33 smithi132 ceph-mon[33500]: 6.18 scrub ok 2024-03-26T05:01:34.118 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:01:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:01:34.499 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:34 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:34.499 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:34 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:34.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:34 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:35.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:01:34 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:01:35.103 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738422 for osd.0 2024-03-26T05:01:35.103 DEBUG:teuthology.parallel:result is None 2024-03-26T05:01:35.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:35 smithi080 ceph-mon[29691]: 5.b scrub starts 2024-03-26T05:01:35.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:35 smithi080 ceph-mon[29691]: 5.b scrub ok 2024-03-26T05:01:35.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:35 smithi080 ceph-mon[29691]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:35.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:35 smithi080 ceph-mon[40404]: 5.b scrub starts 2024-03-26T05:01:35.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:35 smithi080 ceph-mon[40404]: 5.b scrub ok 2024-03-26T05:01:35.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:35 smithi080 ceph-mon[40404]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:35.425 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574897 for osd.1 2024-03-26T05:01:35.425 DEBUG:teuthology.parallel:result is None 2024-03-26T05:01:35.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:35 smithi132 ceph-mon[33500]: 5.b scrub starts 2024-03-26T05:01:35.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:35 smithi132 ceph-mon[33500]: 5.b scrub ok 2024-03-26T05:01:35.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:35 smithi132 ceph-mon[33500]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:35.703 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659161 got 176093659164 for osd.5 2024-03-26T05:01:35.703 DEBUG:teuthology.parallel:result is None 2024-03-26T05:01:36.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:36.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:36 smithi080 ceph-mon[40404]: 3.1f scrub starts 2024-03-26T05:01:36.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:36 smithi080 ceph-mon[40404]: 3.1f scrub ok 2024-03-26T05:01:36.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:36.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:36 smithi080 ceph-mon[29691]: 3.1f scrub starts 2024-03-26T05:01:36.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:36 smithi080 ceph-mon[29691]: 3.1f scrub ok 2024-03-26T05:01:36.520 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005070 got 261993005073 for osd.7 2024-03-26T05:01:36.520 DEBUG:teuthology.parallel:result is None 2024-03-26T05:01:36.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:36.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:36 smithi132 ceph-mon[33500]: 3.1f scrub starts 2024-03-26T05:01:36.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:36 smithi132 ceph-mon[33500]: 3.1f scrub ok 2024-03-26T05:01:36.977 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:37.303 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[29691]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:37.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[29691]: 3.4 deep-scrub starts 2024-03-26T05:01:37.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[29691]: 3.4 deep-scrub ok 2024-03-26T05:01:37.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[29691]: 2.2 scrub starts 2024-03-26T05:01:37.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[29691]: 2.2 scrub ok 2024-03-26T05:01:37.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[29691]: 2.0 deep-scrub starts 2024-03-26T05:01:37.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[29691]: 2.0 deep-scrub ok 2024-03-26T05:01:37.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[40404]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:37.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[40404]: 3.4 deep-scrub starts 2024-03-26T05:01:37.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[40404]: 3.4 deep-scrub ok 2024-03-26T05:01:37.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[40404]: 2.2 scrub starts 2024-03-26T05:01:37.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[40404]: 2.2 scrub ok 2024-03-26T05:01:37.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[40404]: 2.0 deep-scrub starts 2024-03-26T05:01:37.369 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:37 smithi080 ceph-mon[40404]: 2.0 deep-scrub ok 2024-03-26T05:01:37.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:37 smithi132 ceph-mon[33500]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:37.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:37 smithi132 ceph-mon[33500]: 3.4 deep-scrub starts 2024-03-26T05:01:37.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:37 smithi132 ceph-mon[33500]: 3.4 deep-scrub ok 2024-03-26T05:01:37.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:37 smithi132 ceph-mon[33500]: 2.2 scrub starts 2024-03-26T05:01:37.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:37 smithi132 ceph-mon[33500]: 2.2 scrub ok 2024-03-26T05:01:37.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:37 smithi132 ceph-mon[33500]: 2.0 deep-scrub starts 2024-03-26T05:01:37.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:37 smithi132 ceph-mon[33500]: 2.0 deep-scrub ok 2024-03-26T05:01:38.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[29691]: 6.a deep-scrub starts 2024-03-26T05:01:38.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[29691]: 6.a deep-scrub ok 2024-03-26T05:01:38.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[29691]: 3.1e scrub starts 2024-03-26T05:01:38.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[29691]: 3.1e scrub ok 2024-03-26T05:01:38.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[29691]: 3.10 scrub starts 2024-03-26T05:01:38.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[29691]: 3.10 scrub ok 2024-03-26T05:01:38.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[29691]: 5.9 scrub starts 2024-03-26T05:01:38.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[29691]: 5.9 scrub ok 2024-03-26T05:01:38.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[40404]: 6.a deep-scrub starts 2024-03-26T05:01:38.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[40404]: 6.a deep-scrub ok 2024-03-26T05:01:38.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[40404]: 3.1e scrub starts 2024-03-26T05:01:38.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[40404]: 3.1e scrub ok 2024-03-26T05:01:38.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[40404]: 3.10 scrub starts 2024-03-26T05:01:38.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[40404]: 3.10 scrub ok 2024-03-26T05:01:38.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[40404]: 5.9 scrub starts 2024-03-26T05:01:38.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:38 smithi080 ceph-mon[40404]: 5.9 scrub ok 2024-03-26T05:01:38.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:38 smithi132 ceph-mon[33500]: 6.a deep-scrub starts 2024-03-26T05:01:38.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:38 smithi132 ceph-mon[33500]: 6.a deep-scrub ok 2024-03-26T05:01:38.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:38 smithi132 ceph-mon[33500]: 3.1e scrub starts 2024-03-26T05:01:38.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:38 smithi132 ceph-mon[33500]: 3.1e scrub ok 2024-03-26T05:01:38.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:38 smithi132 ceph-mon[33500]: 3.10 scrub starts 2024-03-26T05:01:38.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:38 smithi132 ceph-mon[33500]: 3.10 scrub ok 2024-03-26T05:01:38.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:38 smithi132 ceph-mon[33500]: 5.9 scrub starts 2024-03-26T05:01:38.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:38 smithi132 ceph-mon[33500]: 5.9 scrub ok 2024-03-26T05:01:39.536 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[29691]: 2.1 scrub starts 2024-03-26T05:01:39.536 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[29691]: 2.1 scrub ok 2024-03-26T05:01:39.536 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[29691]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:39.536 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[29691]: 5.c scrub starts 2024-03-26T05:01:39.536 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[29691]: 5.c scrub ok 2024-03-26T05:01:39.536 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[29691]: 4.16 scrub starts 2024-03-26T05:01:39.536 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[29691]: 4.16 scrub ok 2024-03-26T05:01:39.536 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[29691]: 3.b scrub starts 2024-03-26T05:01:39.537 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[29691]: 3.b scrub ok 2024-03-26T05:01:39.537 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[40404]: 2.1 scrub starts 2024-03-26T05:01:39.537 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[40404]: 2.1 scrub ok 2024-03-26T05:01:39.537 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[40404]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:39.537 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[40404]: 5.c scrub starts 2024-03-26T05:01:39.537 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[40404]: 5.c scrub ok 2024-03-26T05:01:39.537 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[40404]: 4.16 scrub starts 2024-03-26T05:01:39.537 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[40404]: 4.16 scrub ok 2024-03-26T05:01:39.537 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[40404]: 3.b scrub starts 2024-03-26T05:01:39.537 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:39 smithi080 ceph-mon[40404]: 3.b scrub ok 2024-03-26T05:01:39.736 INFO:teuthology.orchestra.run.smithi080.stdout:115964117031 2024-03-26T05:01:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:39 smithi132 ceph-mon[33500]: 2.1 scrub starts 2024-03-26T05:01:39.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:39 smithi132 ceph-mon[33500]: 2.1 scrub ok 2024-03-26T05:01:39.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:39 smithi132 ceph-mon[33500]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:39.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:39 smithi132 ceph-mon[33500]: 5.c scrub starts 2024-03-26T05:01:39.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:39 smithi132 ceph-mon[33500]: 5.c scrub ok 2024-03-26T05:01:39.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:39 smithi132 ceph-mon[33500]: 4.16 scrub starts 2024-03-26T05:01:39.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:39 smithi132 ceph-mon[33500]: 4.16 scrub ok 2024-03-26T05:01:39.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:39 smithi132 ceph-mon[33500]: 3.b scrub starts 2024-03-26T05:01:39.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:39 smithi132 ceph-mon[33500]: 3.b scrub ok 2024-03-26T05:01:40.569 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[29691]: 6.4 scrub starts 2024-03-26T05:01:40.569 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[29691]: 6.4 scrub ok 2024-03-26T05:01:40.569 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[29691]: 3.1c scrub starts 2024-03-26T05:01:40.569 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[29691]: 3.1c scrub ok 2024-03-26T05:01:40.569 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1711046006' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T05:01:40.569 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[29691]: 4.14 scrub starts 2024-03-26T05:01:40.570 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[29691]: 4.14 scrub ok 2024-03-26T05:01:40.570 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[40404]: 6.4 scrub starts 2024-03-26T05:01:40.570 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[40404]: 6.4 scrub ok 2024-03-26T05:01:40.570 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[40404]: 3.1c scrub starts 2024-03-26T05:01:40.570 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[40404]: 3.1c scrub ok 2024-03-26T05:01:40.570 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1711046006' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T05:01:40.570 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[40404]: 4.14 scrub starts 2024-03-26T05:01:40.570 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:40 smithi080 ceph-mon[40404]: 4.14 scrub ok 2024-03-26T05:01:40.595 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117027 got 115964117031 for osd.3 2024-03-26T05:01:40.595 DEBUG:teuthology.parallel:result is None 2024-03-26T05:01:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:40 smithi132 ceph-mon[33500]: 6.4 scrub starts 2024-03-26T05:01:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:40 smithi132 ceph-mon[33500]: 6.4 scrub ok 2024-03-26T05:01:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:40 smithi132 ceph-mon[33500]: 3.1c scrub starts 2024-03-26T05:01:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:40 smithi132 ceph-mon[33500]: 3.1c scrub ok 2024-03-26T05:01:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:40 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1711046006' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T05:01:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:40 smithi132 ceph-mon[33500]: 4.14 scrub starts 2024-03-26T05:01:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:40 smithi132 ceph-mon[33500]: 4.14 scrub ok 2024-03-26T05:01:41.015 INFO:teuthology.orchestra.run.smithi080.stdout:77309411374 2024-03-26T05:01:41.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[40404]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:41.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[40404]: 5.19 scrub starts 2024-03-26T05:01:41.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[40404]: 5.19 scrub ok 2024-03-26T05:01:41.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:01:41.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:01:41.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[40404]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T05:01:41.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[40404]: 4.13 deep-scrub starts 2024-03-26T05:01:41.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[40404]: 4.13 deep-scrub ok 2024-03-26T05:01:41.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3876750832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T05:01:41.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[40404]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T05:01:41.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[29691]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:41.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[29691]: 5.19 scrub starts 2024-03-26T05:01:41.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[29691]: 5.19 scrub ok 2024-03-26T05:01:41.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:01:41.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:01:41.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[29691]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T05:01:41.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[29691]: 4.13 deep-scrub starts 2024-03-26T05:01:41.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[29691]: 4.13 deep-scrub ok 2024-03-26T05:01:41.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3876750832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T05:01:41.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:41 smithi080 ceph-mon[29691]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T05:01:41.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:41 smithi132 ceph-mon[33500]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:41.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:41 smithi132 ceph-mon[33500]: 5.19 scrub starts 2024-03-26T05:01:41.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:41 smithi132 ceph-mon[33500]: 5.19 scrub ok 2024-03-26T05:01:41.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:41 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:01:41.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:41 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:01:41.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:41 smithi132 ceph-mon[33500]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T05:01:41.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:41 smithi132 ceph-mon[33500]: 4.13 deep-scrub starts 2024-03-26T05:01:41.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:41 smithi132 ceph-mon[33500]: 4.13 deep-scrub ok 2024-03-26T05:01:41.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:41 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3876750832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T05:01:41.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:41 smithi132 ceph-mon[33500]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T05:01:42.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[29691]: 4.4 scrub starts 2024-03-26T05:01:42.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[29691]: 4.4 scrub ok 2024-03-26T05:01:42.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[29691]: 5.e scrub starts 2024-03-26T05:01:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[29691]: 5.e scrub ok 2024-03-26T05:01:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[29691]: 4.1e scrub starts 2024-03-26T05:01:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[29691]: 4.1e scrub ok 2024-03-26T05:01:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[29691]: 6.5 scrub starts 2024-03-26T05:01:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[29691]: 6.5 scrub ok 2024-03-26T05:01:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[29691]: 4.0 deep-scrub starts 2024-03-26T05:01:42.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[40404]: 4.4 scrub starts 2024-03-26T05:01:42.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[40404]: 4.4 scrub ok 2024-03-26T05:01:42.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[40404]: 5.e scrub starts 2024-03-26T05:01:42.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[40404]: 5.e scrub ok 2024-03-26T05:01:42.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[40404]: 4.1e scrub starts 2024-03-26T05:01:42.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[40404]: 4.1e scrub ok 2024-03-26T05:01:42.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[40404]: 6.5 scrub starts 2024-03-26T05:01:42.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[40404]: 6.5 scrub ok 2024-03-26T05:01:42.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:42 smithi080 ceph-mon[40404]: 4.0 deep-scrub starts 2024-03-26T05:01:42.691 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411374 for osd.2 2024-03-26T05:01:42.692 DEBUG:teuthology.parallel:result is None 2024-03-26T05:01:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:42 smithi132 ceph-mon[33500]: 4.4 scrub starts 2024-03-26T05:01:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:42 smithi132 ceph-mon[33500]: 4.4 scrub ok 2024-03-26T05:01:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:42 smithi132 ceph-mon[33500]: 5.e scrub starts 2024-03-26T05:01:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:42 smithi132 ceph-mon[33500]: 5.e scrub ok 2024-03-26T05:01:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:42 smithi132 ceph-mon[33500]: 4.1e scrub starts 2024-03-26T05:01:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:42 smithi132 ceph-mon[33500]: 4.1e scrub ok 2024-03-26T05:01:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:42 smithi132 ceph-mon[33500]: 6.5 scrub starts 2024-03-26T05:01:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:42 smithi132 ceph-mon[33500]: 6.5 scrub ok 2024-03-26T05:01:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:42 smithi132 ceph-mon[33500]: 4.0 deep-scrub starts 2024-03-26T05:01:43.344 INFO:teuthology.orchestra.run.smithi080.stdout:141733920804 2024-03-26T05:01:43.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:43 smithi132 ceph-mon[33500]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:43.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:43 smithi132 ceph-mon[33500]: 5.7 scrub starts 2024-03-26T05:01:43.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:43 smithi132 ceph-mon[33500]: 5.7 scrub ok 2024-03-26T05:01:43.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:43 smithi132 ceph-mon[33500]: 4.0 deep-scrub ok 2024-03-26T05:01:43.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:43 smithi080 ceph-mon[29691]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:43.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:43 smithi080 ceph-mon[29691]: 5.7 scrub starts 2024-03-26T05:01:43.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:43 smithi080 ceph-mon[29691]: 5.7 scrub ok 2024-03-26T05:01:43.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:43 smithi080 ceph-mon[29691]: 4.0 deep-scrub ok 2024-03-26T05:01:43.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:43 smithi080 ceph-mon[40404]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:43.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:43 smithi080 ceph-mon[40404]: 5.7 scrub starts 2024-03-26T05:01:43.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:43 smithi080 ceph-mon[40404]: 5.7 scrub ok 2024-03-26T05:01:43.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:43 smithi080 ceph-mon[40404]: 4.0 deep-scrub ok 2024-03-26T05:01:44.036 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:01:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:01:44.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[29691]: 4.2 scrub starts 2024-03-26T05:01:44.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[29691]: 4.2 scrub ok 2024-03-26T05:01:44.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[29691]: 3.18 scrub starts 2024-03-26T05:01:44.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[29691]: 3.18 scrub ok 2024-03-26T05:01:44.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1321700621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T05:01:44.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:01:44.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:01:44.619 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:44 smithi080 systemd[1]: Starting Ceph alertmanager.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:01:44.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[40404]: 4.2 scrub starts 2024-03-26T05:01:44.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[40404]: 4.2 scrub ok 2024-03-26T05:01:44.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[40404]: 3.18 scrub starts 2024-03-26T05:01:44.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[40404]: 3.18 scrub ok 2024-03-26T05:01:44.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1321700621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T05:01:44.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:01:44.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:44 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:01:44.675 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920800 got 141733920804 for osd.4 2024-03-26T05:01:44.675 DEBUG:teuthology.parallel:result is None 2024-03-26T05:01:44.675 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T05:01:44.675 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:01:44.738 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:44 smithi132 ceph-mon[33500]: 4.2 scrub starts 2024-03-26T05:01:44.738 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:44 smithi132 ceph-mon[33500]: 4.2 scrub ok 2024-03-26T05:01:44.738 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:44 smithi132 ceph-mon[33500]: 3.18 scrub starts 2024-03-26T05:01:44.738 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:44 smithi132 ceph-mon[33500]: 3.18 scrub ok 2024-03-26T05:01:44.738 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:44 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1321700621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T05:01:44.738 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:44 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:01:44.738 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:44 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:01:44.905 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:44 smithi080 podman[75188]: 2024-03-26 05:01:44.796218392 +0000 UTC m=+0.018451559 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T05:01:44.905 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:44 smithi080 podman[75188]: 2024-03-26 05:01:44.913657549 +0000 UTC m=+0.135890703 volume create 1999c43d9e33008c07bd67e644a2c6e0ff54e4e840be89a07ed46518874d9e2a 2024-03-26T05:01:45.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:01:44 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:01:45.173 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 podman[75188]: 2024-03-26 05:01:45.005816081 +0000 UTC m=+0.228049234 container create c399a78cc74f7df33cfa32216983fbd1b3eee0d901b66c3623d9aac545673d1e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:01:45.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[29691]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:01:45.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[29691]: 5.d scrub starts 2024-03-26T05:01:45.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[29691]: 5.d scrub ok 2024-03-26T05:01:45.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[29691]: 5.11 scrub starts 2024-03-26T05:01:45.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[29691]: 5.11 scrub ok 2024-03-26T05:01:45.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[29691]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T05:01:45.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[29691]: 4.c scrub starts 2024-03-26T05:01:45.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[29691]: 4.c scrub ok 2024-03-26T05:01:45.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[29691]: 4.3 scrub starts 2024-03-26T05:01:45.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[29691]: 4.3 scrub ok 2024-03-26T05:01:45.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[40404]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:01:45.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[40404]: 5.d scrub starts 2024-03-26T05:01:45.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[40404]: 5.d scrub ok 2024-03-26T05:01:45.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[40404]: 5.11 scrub starts 2024-03-26T05:01:45.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[40404]: 5.11 scrub ok 2024-03-26T05:01:45.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[40404]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T05:01:45.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[40404]: 4.c scrub starts 2024-03-26T05:01:45.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[40404]: 4.c scrub ok 2024-03-26T05:01:45.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[40404]: 4.3 scrub starts 2024-03-26T05:01:45.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-mon[40404]: 4.3 scrub ok 2024-03-26T05:01:45.620 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 podman[75188]: 2024-03-26 05:01:45.283161473 +0000 UTC m=+0.505394636 container init c399a78cc74f7df33cfa32216983fbd1b3eee0d901b66c3623d9aac545673d1e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:01:45.620 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 podman[75188]: 2024-03-26 05:01:45.28567888 +0000 UTC m=+0.507912034 container start c399a78cc74f7df33cfa32216983fbd1b3eee0d901b66c3623d9aac545673d1e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:01:45.620 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[75215]: ts=2024-03-26T05:01:45.297Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T05:01:45.620 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[75215]: ts=2024-03-26T05:01:45.297Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T05:01:45.620 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[75215]: ts=2024-03-26T05:01:45.298Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.80 port=9094 2024-03-26T05:01:45.621 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[75215]: ts=2024-03-26T05:01:45.299Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T05:01:45.621 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[75215]: ts=2024-03-26T05:01:45.321Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T05:01:45.621 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[75215]: ts=2024-03-26T05:01:45.322Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T05:01:45.621 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[75215]: ts=2024-03-26T05:01:45.325Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T05:01:45.621 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[75215]: ts=2024-03-26T05:01:45.325Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T05:01:45.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:45 smithi132 ceph-mon[33500]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:01:45.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:45 smithi132 ceph-mon[33500]: 5.d scrub starts 2024-03-26T05:01:45.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:45 smithi132 ceph-mon[33500]: 5.d scrub ok 2024-03-26T05:01:45.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:45 smithi132 ceph-mon[33500]: 5.11 scrub starts 2024-03-26T05:01:45.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:45 smithi132 ceph-mon[33500]: 5.11 scrub ok 2024-03-26T05:01:45.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:45 smithi132 ceph-mon[33500]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T05:01:45.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:45 smithi132 ceph-mon[33500]: 4.c scrub starts 2024-03-26T05:01:45.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:45 smithi132 ceph-mon[33500]: 4.c scrub ok 2024-03-26T05:01:45.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:45 smithi132 ceph-mon[33500]: 4.3 scrub starts 2024-03-26T05:01:45.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:45 smithi132 ceph-mon[33500]: 4.3 scrub ok 2024-03-26T05:01:45.874 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:46.000 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 bash[75188]: c399a78cc74f7df33cfa32216983fbd1b3eee0d901b66c3623d9aac545673d1e 2024-03-26T05:01:46.000 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:45 smithi080 systemd[1]: Started Ceph alertmanager.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:01:46.000 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:45] ENGINE Bus STOPPING 2024-03-26T05:01:46.000 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:01:46.000 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:45] ENGINE Bus STOPPED 2024-03-26T05:01:46.000 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:45 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:45] ENGINE Bus STARTING 2024-03-26T05:01:46.329 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:46] ENGINE Serving on http://:::9283 2024-03-26T05:01:46.329 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:46] ENGINE Bus STARTED 2024-03-26T05:01:46.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:46.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[29691]: 5.f scrub starts 2024-03-26T05:01:46.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[29691]: 5.f scrub ok 2024-03-26T05:01:46.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T05:01:46.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[29691]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T05:01:46.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:46.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[40404]: 5.f scrub starts 2024-03-26T05:01:46.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[40404]: 5.f scrub ok 2024-03-26T05:01:46.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T05:01:46.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:46 smithi080 ceph-mon[40404]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T05:01:46.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:46 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:46.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:46 smithi132 ceph-mon[33500]: 5.f scrub starts 2024-03-26T05:01:46.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:46 smithi132 ceph-mon[33500]: 5.f scrub ok 2024-03-26T05:01:46.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:46 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:46 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:46 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:46 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:46 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:46 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:46 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T05:01:46.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:46 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:46.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:46 smithi132 ceph-mon[33500]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T05:01:47.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[29691]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:47.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[29691]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T05:01:47.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[29691]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T05:01:47.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[29691]: Deploying daemon grafana.a on smithi132 2024-03-26T05:01:47.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[29691]: 3.1a deep-scrub starts 2024-03-26T05:01:47.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[29691]: 3.1a deep-scrub ok 2024-03-26T05:01:47.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[29691]: 6.0 scrub starts 2024-03-26T05:01:47.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[29691]: 6.0 scrub ok 2024-03-26T05:01:47.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[29691]: 3.e scrub starts 2024-03-26T05:01:47.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[29691]: 3.e scrub ok 2024-03-26T05:01:47.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[40404]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:47.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[40404]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T05:01:47.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[40404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T05:01:47.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[40404]: Deploying daemon grafana.a on smithi132 2024-03-26T05:01:47.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[40404]: 3.1a deep-scrub starts 2024-03-26T05:01:47.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[40404]: 3.1a deep-scrub ok 2024-03-26T05:01:47.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[40404]: 6.0 scrub starts 2024-03-26T05:01:47.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[40404]: 6.0 scrub ok 2024-03-26T05:01:47.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[40404]: 3.e scrub starts 2024-03-26T05:01:47.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-mon[40404]: 3.e scrub ok 2024-03-26T05:01:47.620 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:47 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[75215]: ts=2024-03-26T05:01:47.299Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000458191s 2024-03-26T05:01:47.733 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:01:47.736 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:01:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:47 smithi132 ceph-mon[33500]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:47.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:47 smithi132 ceph-mon[33500]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T05:01:47.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:47 smithi132 ceph-mon[33500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T05:01:47.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:47 smithi132 ceph-mon[33500]: Deploying daemon grafana.a on smithi132 2024-03-26T05:01:47.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:47 smithi132 ceph-mon[33500]: 3.1a deep-scrub starts 2024-03-26T05:01:47.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:47 smithi132 ceph-mon[33500]: 3.1a deep-scrub ok 2024-03-26T05:01:47.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:47 smithi132 ceph-mon[33500]: 6.0 scrub starts 2024-03-26T05:01:47.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:47 smithi132 ceph-mon[33500]: 6.0 scrub ok 2024-03-26T05:01:47.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:47 smithi132 ceph-mon[33500]: 3.e scrub starts 2024-03-26T05:01:47.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:47 smithi132 ceph-mon[33500]: 3.e scrub ok 2024-03-26T05:01:48.401 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-03-26T05:01:47.712802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1298,"num_read_kb":1112,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":811,"ondisk_log_size":811,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":246504,"kb_used_data":9544,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749714712,"statfs":{"total":767960285184,"available":767707865088,"internally_reserved":0,"allocated":9773056,"data_stored":5565424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.401319"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":57,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348273+0000","last_change":"2024-03-26T05:00:28.973250+0000","last_active":"2024-03-26T05:01:44.348273+0000","last_peered":"2024-03-26T05:01:44.348273+0000","last_clean":"2024-03-26T05:01:44.348273+0000","last_became_active":"2024-03-26T05:00:28.972969+0000","last_became_peered":"2024-03-26T05:00:28.972969+0000","last_unstale":"2024-03-26T05:01:44.348273+0000","last_undegraded":"2024-03-26T05:01:44.348273+0000","last_fullsized":"2024-03-26T05:01:44.348273+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":68,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.646522+0000","last_change":"2024-03-26T05:01:39.812600+0000","last_active":"2024-03-26T05:01:44.646522+0000","last_peered":"2024-03-26T05:01:44.646522+0000","last_clean":"2024-03-26T05:01:44.646522+0000","last_became_active":"2024-03-26T05:00:26.953154+0000","last_became_peered":"2024-03-26T05:00:26.953154+0000","last_unstale":"2024-03-26T05:01:44.646522+0000","last_undegraded":"2024-03-26T05:01:44.646522+0000","last_fullsized":"2024-03-26T05:01:44.646522+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.812509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.812509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:45.654690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":77,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344740+0000","last_change":"2024-03-26T05:01:35.658900+0000","last_active":"2024-03-26T05:01:44.344740+0000","last_peered":"2024-03-26T05:01:44.344740+0000","last_clean":"2024-03-26T05:01:44.344740+0000","last_became_active":"2024-03-26T05:00:23.346607+0000","last_became_peered":"2024-03-26T05:00:23.346607+0000","last_unstale":"2024-03-26T05:01:44.344740+0000","last_undegraded":"2024-03-26T05:01:44.344740+0000","last_fullsized":"2024-03-26T05:01:44.344740+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:35.658796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.658796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:41.835654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'15","reported_seq":93,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348180+0000","last_change":"2024-03-26T05:00:24.943621+0000","last_active":"2024-03-26T05:01:44.348180+0000","last_peered":"2024-03-26T05:01:44.348180+0000","last_clean":"2024-03-26T05:01:44.348180+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:01:44.348180+0000","last_undegraded":"2024-03-26T05:01:44.348180+0000","last_fullsized":"2024-03-26T05:01:44.348180+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:45.003443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":56,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398391+0000","last_change":"2024-03-26T05:00:29.463053+0000","last_active":"2024-03-26T05:01:44.398391+0000","last_peered":"2024-03-26T05:01:44.398391+0000","last_clean":"2024-03-26T05:01:44.398391+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:01:44.398391+0000","last_undegraded":"2024-03-26T05:01:44.398391+0000","last_fullsized":"2024-03-26T05:01:44.398391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:56.816597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":61,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348086+0000","last_change":"2024-03-26T05:00:26.955355+0000","last_active":"2024-03-26T05:01:44.348086+0000","last_peered":"2024-03-26T05:01:44.348086+0000","last_clean":"2024-03-26T05:01:44.348086+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:01:44.348086+0000","last_undegraded":"2024-03-26T05:01:44.348086+0000","last_fullsized":"2024-03-26T05:01:44.348086+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.268685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":76,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398787+0000","last_change":"2024-03-26T05:01:37.201540+0000","last_active":"2024-03-26T05:01:44.398787+0000","last_peered":"2024-03-26T05:01:44.398787+0000","last_clean":"2024-03-26T05:01:44.398787+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:01:44.398787+0000","last_undegraded":"2024-03-26T05:01:44.398787+0000","last_fullsized":"2024-03-26T05:01:44.398787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.201446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.201446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:43.519218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'39","reported_seq":137,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398276+0000","last_change":"2024-03-26T05:00:24.932741+0000","last_active":"2024-03-26T05:01:44.398276+0000","last_peered":"2024-03-26T05:01:44.398276+0000","last_clean":"2024-03-26T05:01:44.398276+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:01:44.398276+0000","last_undegraded":"2024-03-26T05:01:44.398276+0000","last_fullsized":"2024-03-26T05:01:44.398276+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:53.767288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":65,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344313+0000","last_change":"2024-03-26T05:01:32.598168+0000","last_active":"2024-03-26T05:01:44.344313+0000","last_peered":"2024-03-26T05:01:44.344313+0000","last_clean":"2024-03-26T05:01:44.344313+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:01:44.344313+0000","last_undegraded":"2024-03-26T05:01:44.344313+0000","last_fullsized":"2024-03-26T05:01:44.344313+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:32.598087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:32.598087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.083903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":58,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.461712+0000","last_change":"2024-03-26T05:00:26.956404+0000","last_active":"2024-03-26T05:01:41.461712+0000","last_peered":"2024-03-26T05:01:41.461712+0000","last_clean":"2024-03-26T05:01:41.461712+0000","last_became_active":"2024-03-26T05:00:26.956013+0000","last_became_peered":"2024-03-26T05:00:26.956013+0000","last_unstale":"2024-03-26T05:01:41.461712+0000","last_undegraded":"2024-03-26T05:01:41.461712+0000","last_fullsized":"2024-03-26T05:01:41.461712+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:44.609011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":74,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.461258+0000","last_change":"2024-03-26T05:01:30.949017+0000","last_active":"2024-03-26T05:01:41.461258+0000","last_peered":"2024-03-26T05:01:41.461258+0000","last_clean":"2024-03-26T05:01:41.461258+0000","last_became_active":"2024-03-26T05:00:22.768098+0000","last_became_peered":"2024-03-26T05:00:22.768098+0000","last_unstale":"2024-03-26T05:01:41.461258+0000","last_undegraded":"2024-03-26T05:01:41.461258+0000","last_fullsized":"2024-03-26T05:01:41.461258+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.948866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.948866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.633607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'21","reported_seq":113,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348699+0000","last_change":"2024-03-26T05:01:30.476359+0000","last_active":"2024-03-26T05:01:44.348699+0000","last_peered":"2024-03-26T05:01:44.348699+0000","last_clean":"2024-03-26T05:01:44.348699+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:01:44.348699+0000","last_undegraded":"2024-03-26T05:01:44.348699+0000","last_fullsized":"2024-03-26T05:01:44.348699+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:01:30.476249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.476249+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.885741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":54,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.460817+0000","last_change":"2024-03-26T05:00:28.982522+0000","last_active":"2024-03-26T05:01:41.460817+0000","last_peered":"2024-03-26T05:01:41.460817+0000","last_clean":"2024-03-26T05:01:41.460817+0000","last_became_active":"2024-03-26T05:00:28.982276+0000","last_became_peered":"2024-03-26T05:00:28.982276+0000","last_unstale":"2024-03-26T05:01:41.460817+0000","last_undegraded":"2024-03-26T05:01:41.460817+0000","last_fullsized":"2024-03-26T05:01:41.460817+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.461612+0000","last_change":"2024-03-26T05:00:26.960760+0000","last_active":"2024-03-26T05:01:44.461612+0000","last_peered":"2024-03-26T05:01:44.461612+0000","last_clean":"2024-03-26T05:01:44.461612+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:01:44.461612+0000","last_undegraded":"2024-03-26T05:01:44.461612+0000","last_fullsized":"2024-03-26T05:01:44.461612+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":74,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.462805+0000","last_change":"2024-03-26T05:01:39.021449+0000","last_active":"2024-03-26T05:01:41.462805+0000","last_peered":"2024-03-26T05:01:41.462805+0000","last_clean":"2024-03-26T05:01:41.462805+0000","last_became_active":"2024-03-26T05:00:22.766675+0000","last_became_peered":"2024-03-26T05:00:22.766675+0000","last_unstale":"2024-03-26T05:01:41.462805+0000","last_undegraded":"2024-03-26T05:01:41.462805+0000","last_fullsized":"2024-03-26T05:01:41.462805+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.021315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.021315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:39.428392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'11","reported_seq":87,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348456+0000","last_change":"2024-03-26T05:00:24.943205+0000","last_active":"2024-03-26T05:01:44.348456+0000","last_peered":"2024-03-26T05:01:44.348456+0000","last_clean":"2024-03-26T05:01:44.348456+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:01:44.348456+0000","last_undegraded":"2024-03-26T05:01:44.348456+0000","last_fullsized":"2024-03-26T05:01:44.348456+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:50.151492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":57,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.349249+0000","last_change":"2024-03-26T05:00:29.461374+0000","last_active":"2024-03-26T05:01:44.349249+0000","last_peered":"2024-03-26T05:01:44.349249+0000","last_clean":"2024-03-26T05:01:44.349249+0000","last_became_active":"2024-03-26T05:00:29.461032+0000","last_became_peered":"2024-03-26T05:00:29.461032+0000","last_unstale":"2024-03-26T05:01:44.349249+0000","last_undegraded":"2024-03-26T05:01:44.349249+0000","last_fullsized":"2024-03-26T05:01:44.349249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":60,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.646965+0000","last_change":"2024-03-26T05:00:26.960665+0000","last_active":"2024-03-26T05:01:44.646965+0000","last_peered":"2024-03-26T05:01:44.646965+0000","last_clean":"2024-03-26T05:01:44.646965+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:01:44.646965+0000","last_undegraded":"2024-03-26T05:01:44.646965+0000","last_fullsized":"2024-03-26T05:01:44.646965+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":69,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344644+0000","last_change":"2024-03-26T05:00:22.777871+0000","last_active":"2024-03-26T05:01:44.344644+0000","last_peered":"2024-03-26T05:01:44.344644+0000","last_clean":"2024-03-26T05:01:44.344644+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:01:44.344644+0000","last_undegraded":"2024-03-26T05:01:44.344644+0000","last_fullsized":"2024-03-26T05:01:44.344644+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":95,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.461731+0000","last_change":"2024-03-26T05:01:24.617269+0000","last_active":"2024-03-26T05:01:41.461731+0000","last_peered":"2024-03-26T05:01:41.461731+0000","last_clean":"2024-03-26T05:01:41.461731+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:01:41.461731+0000","last_undegraded":"2024-03-26T05:01:41.461731+0000","last_fullsized":"2024-03-26T05:01:41.461731+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:24.617170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:24.617170+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:34.140661+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":64,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398996+0000","last_change":"2024-03-26T05:01:30.046528+0000","last_active":"2024-03-26T05:01:44.398996+0000","last_peered":"2024-03-26T05:01:44.398996+0000","last_clean":"2024-03-26T05:01:44.398996+0000","last_became_active":"2024-03-26T05:00:29.463238+0000","last_became_peered":"2024-03-26T05:00:29.463238+0000","last_unstale":"2024-03-26T05:01:44.398996+0000","last_undegraded":"2024-03-26T05:01:44.398996+0000","last_fullsized":"2024-03-26T05:01:44.398996+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.046425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.046425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:35.648365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":61,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348472+0000","last_change":"2024-03-26T05:00:26.962232+0000","last_active":"2024-03-26T05:01:44.348472+0000","last_peered":"2024-03-26T05:01:44.348472+0000","last_clean":"2024-03-26T05:01:44.348472+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:01:44.348472+0000","last_undegraded":"2024-03-26T05:01:44.348472+0000","last_fullsized":"2024-03-26T05:01:44.348472+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":69,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348709+0000","last_change":"2024-03-26T05:00:23.347429+0000","last_active":"2024-03-26T05:01:44.348709+0000","last_peered":"2024-03-26T05:01:44.348709+0000","last_clean":"2024-03-26T05:01:44.348709+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:01:44.348709+0000","last_undegraded":"2024-03-26T05:01:44.348709+0000","last_fullsized":"2024-03-26T05:01:44.348709+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'20","reported_seq":101,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398520+0000","last_change":"2024-03-26T05:00:24.937057+0000","last_active":"2024-03-26T05:01:44.398520+0000","last_peered":"2024-03-26T05:01:44.398520+0000","last_clean":"2024-03-26T05:01:44.398520+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:01:44.398520+0000","last_undegraded":"2024-03-26T05:01:44.398520+0000","last_fullsized":"2024-03-26T05:01:44.398520+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":56,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.461840+0000","last_change":"2024-03-26T05:00:28.981879+0000","last_active":"2024-03-26T05:01:44.461840+0000","last_peered":"2024-03-26T05:01:44.461840+0000","last_clean":"2024-03-26T05:01:44.461840+0000","last_became_active":"2024-03-26T05:00:28.981505+0000","last_became_peered":"2024-03-26T05:00:28.981505+0000","last_unstale":"2024-03-26T05:01:44.461840+0000","last_undegraded":"2024-03-26T05:01:44.461840+0000","last_fullsized":"2024-03-26T05:01:44.461840+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":87,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.401432+0000","last_change":"2024-03-26T05:00:26.954779+0000","last_active":"2024-03-26T05:01:44.401432+0000","last_peered":"2024-03-26T05:01:44.401432+0000","last_clean":"2024-03-26T05:01:44.401432+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:01:44.401432+0000","last_undegraded":"2024-03-26T05:01:44.401432+0000","last_fullsized":"2024-03-26T05:01:44.401432+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":69,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.646569+0000","last_change":"2024-03-26T05:00:22.773390+0000","last_active":"2024-03-26T05:01:44.646569+0000","last_peered":"2024-03-26T05:01:44.646569+0000","last_clean":"2024-03-26T05:01:44.646569+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:01:44.646569+0000","last_undegraded":"2024-03-26T05:01:44.646569+0000","last_fullsized":"2024-03-26T05:01:44.646569+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:50.999861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'16","reported_seq":100,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344656+0000","last_change":"2024-03-26T05:01:41.650400+0000","last_active":"2024-03-26T05:01:44.344656+0000","last_peered":"2024-03-26T05:01:44.344656+0000","last_clean":"2024-03-26T05:01:44.344656+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:01:44.344656+0000","last_undegraded":"2024-03-26T05:01:44.344656+0000","last_fullsized":"2024-03-26T05:01:44.344656+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:01:41.650310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.650310+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:08.831570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":56,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.647065+0000","last_change":"2024-03-26T05:00:28.981390+0000","last_active":"2024-03-26T05:01:44.647065+0000","last_peered":"2024-03-26T05:01:44.647065+0000","last_clean":"2024-03-26T05:01:44.647065+0000","last_became_active":"2024-03-26T05:00:28.981049+0000","last_became_peered":"2024-03-26T05:00:28.981049+0000","last_unstale":"2024-03-26T05:01:44.647065+0000","last_undegraded":"2024-03-26T05:01:44.647065+0000","last_fullsized":"2024-03-26T05:01:44.647065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:45.739756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":61,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344875+0000","last_change":"2024-03-26T05:00:26.963282+0000","last_active":"2024-03-26T05:01:44.344875+0000","last_peered":"2024-03-26T05:01:44.344875+0000","last_clean":"2024-03-26T05:01:44.344875+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:01:44.344875+0000","last_undegraded":"2024-03-26T05:01:44.344875+0000","last_fullsized":"2024-03-26T05:01:44.344875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":76,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.397995+0000","last_change":"2024-03-26T05:01:43.166558+0000","last_active":"2024-03-26T05:01:44.397995+0000","last_peered":"2024-03-26T05:01:44.397995+0000","last_clean":"2024-03-26T05:01:44.397995+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:01:44.397995+0000","last_undegraded":"2024-03-26T05:01:44.397995+0000","last_fullsized":"2024-03-26T05:01:44.397995+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:43.166422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:43.166422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:50.132493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'17","reported_seq":114,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.401408+0000","last_change":"2024-03-26T05:00:49.652267+0000","last_active":"2024-03-26T05:01:44.401408+0000","last_peered":"2024-03-26T05:01:44.401408+0000","last_clean":"2024-03-26T05:01:44.401408+0000","last_became_active":"2024-03-26T05:00:49.643308+0000","last_became_peered":"2024-03-26T05:00:49.643308+0000","last_unstale":"2024-03-26T05:01:44.401408+0000","last_undegraded":"2024-03-26T05:01:44.401408+0000","last_fullsized":"2024-03-26T05:01:44.401408+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.461549+0000","last_change":"2024-03-26T05:00:28.978762+0000","last_active":"2024-03-26T05:01:44.461549+0000","last_peered":"2024-03-26T05:01:44.461549+0000","last_clean":"2024-03-26T05:01:44.461549+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:01:44.461549+0000","last_undegraded":"2024-03-26T05:01:44.461549+0000","last_fullsized":"2024-03-26T05:01:44.461549+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:49.872411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.400637+0000","last_change":"2024-03-26T05:01:44.235422+0000","last_active":"2024-03-26T05:01:44.400637+0000","last_peered":"2024-03-26T05:01:44.400637+0000","last_clean":"2024-03-26T05:01:44.400637+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:01:44.400637+0000","last_undegraded":"2024-03-26T05:01:44.400637+0000","last_fullsized":"2024-03-26T05:01:44.400637+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:44.235328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.235328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.223031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":69,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344660+0000","last_change":"2024-03-26T05:00:22.775209+0000","last_active":"2024-03-26T05:01:44.344660+0000","last_peered":"2024-03-26T05:01:44.344660+0000","last_clean":"2024-03-26T05:01:44.344660+0000","last_became_active":"2024-03-26T05:00:22.774783+0000","last_became_peered":"2024-03-26T05:00:22.774783+0000","last_unstale":"2024-03-26T05:01:44.344660+0000","last_undegraded":"2024-03-26T05:01:44.344660+0000","last_fullsized":"2024-03-26T05:01:44.344660+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":70,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398034+0000","last_change":"2024-03-26T05:00:24.942166+0000","last_active":"2024-03-26T05:01:44.398034+0000","last_peered":"2024-03-26T05:01:44.398034+0000","last_clean":"2024-03-26T05:01:44.398034+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:01:44.398034+0000","last_undegraded":"2024-03-26T05:01:44.398034+0000","last_fullsized":"2024-03-26T05:01:44.398034+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":56,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398283+0000","last_change":"2024-03-26T05:00:29.461798+0000","last_active":"2024-03-26T05:01:44.398283+0000","last_peered":"2024-03-26T05:01:44.398283+0000","last_clean":"2024-03-26T05:01:44.398283+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:01:44.398283+0000","last_undegraded":"2024-03-26T05:01:44.398283+0000","last_fullsized":"2024-03-26T05:01:44.398283+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.461704+0000","last_change":"2024-03-26T05:00:26.959842+0000","last_active":"2024-03-26T05:01:44.461704+0000","last_peered":"2024-03-26T05:01:44.461704+0000","last_clean":"2024-03-26T05:01:44.461704+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:01:44.461704+0000","last_undegraded":"2024-03-26T05:01:44.461704+0000","last_fullsized":"2024-03-26T05:01:44.461704+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":74,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.462171+0000","last_change":"2024-03-26T05:01:29.997645+0000","last_active":"2024-03-26T05:01:41.462171+0000","last_peered":"2024-03-26T05:01:41.462171+0000","last_clean":"2024-03-26T05:01:41.462171+0000","last_became_active":"2024-03-26T05:00:22.771199+0000","last_became_peered":"2024-03-26T05:00:22.771199+0000","last_unstale":"2024-03-26T05:01:41.462171+0000","last_undegraded":"2024-03-26T05:01:41.462171+0000","last_fullsized":"2024-03-26T05:01:41.462171+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:35.309625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'17","reported_seq":95,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398274+0000","last_change":"2024-03-26T05:00:25.648453+0000","last_active":"2024-03-26T05:01:44.398274+0000","last_peered":"2024-03-26T05:01:44.398274+0000","last_clean":"2024-03-26T05:01:44.398274+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:01:44.398274+0000","last_undegraded":"2024-03-26T05:01:44.398274+0000","last_fullsized":"2024-03-26T05:01:44.398274+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":57,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.343884+0000","last_change":"2024-03-26T05:00:28.985476+0000","last_active":"2024-03-26T05:01:44.343884+0000","last_peered":"2024-03-26T05:01:44.343884+0000","last_clean":"2024-03-26T05:01:44.343884+0000","last_became_active":"2024-03-26T05:00:28.985158+0000","last_became_peered":"2024-03-26T05:00:28.985158+0000","last_unstale":"2024-03-26T05:01:44.343884+0000","last_undegraded":"2024-03-26T05:01:44.343884+0000","last_fullsized":"2024-03-26T05:01:44.343884+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":60,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398451+0000","last_change":"2024-03-26T05:00:26.960672+0000","last_active":"2024-03-26T05:01:44.398451+0000","last_peered":"2024-03-26T05:01:44.398451+0000","last_clean":"2024-03-26T05:01:44.398451+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:01:44.398451+0000","last_undegraded":"2024-03-26T05:01:44.398451+0000","last_fullsized":"2024-03-26T05:01:44.398451+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:55.490995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.461956+0000","last_change":"2024-03-26T05:00:22.773808+0000","last_active":"2024-03-26T05:01:44.461956+0000","last_peered":"2024-03-26T05:01:44.461956+0000","last_clean":"2024-03-26T05:01:44.461956+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:01:44.461956+0000","last_undegraded":"2024-03-26T05:01:44.461956+0000","last_fullsized":"2024-03-26T05:01:44.461956+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:49.214803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":80,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.646791+0000","last_change":"2024-03-26T05:00:24.939870+0000","last_active":"2024-03-26T05:01:44.646791+0000","last_peered":"2024-03-26T05:01:44.646791+0000","last_clean":"2024-03-26T05:01:44.646791+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:01:44.646791+0000","last_undegraded":"2024-03-26T05:01:44.646791+0000","last_fullsized":"2024-03-26T05:01:44.646791+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:46.453483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":56,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398039+0000","last_change":"2024-03-26T05:00:28.984567+0000","last_active":"2024-03-26T05:01:44.398039+0000","last_peered":"2024-03-26T05:01:44.398039+0000","last_clean":"2024-03-26T05:01:44.398039+0000","last_became_active":"2024-03-26T05:00:28.983417+0000","last_became_peered":"2024-03-26T05:00:28.983417+0000","last_unstale":"2024-03-26T05:01:44.398039+0000","last_undegraded":"2024-03-26T05:01:44.398039+0000","last_fullsized":"2024-03-26T05:01:44.398039+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:51.606343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":68,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.646802+0000","last_change":"2024-03-26T05:01:30.777153+0000","last_active":"2024-03-26T05:01:44.646802+0000","last_peered":"2024-03-26T05:01:44.646802+0000","last_clean":"2024-03-26T05:01:44.646802+0000","last_became_active":"2024-03-26T05:00:26.954653+0000","last_became_peered":"2024-03-26T05:00:26.954653+0000","last_unstale":"2024-03-26T05:01:44.646802+0000","last_undegraded":"2024-03-26T05:01:44.646802+0000","last_fullsized":"2024-03-26T05:01:44.646802+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.777070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:45.021844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":69,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348825+0000","last_change":"2024-03-26T05:00:22.767847+0000","last_active":"2024-03-26T05:01:44.348825+0000","last_peered":"2024-03-26T05:01:44.348825+0000","last_clean":"2024-03-26T05:01:44.348825+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:01:44.348825+0000","last_undegraded":"2024-03-26T05:01:44.348825+0000","last_fullsized":"2024-03-26T05:01:44.348825+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:47.148519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"95'16","reported_seq":97,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348747+0000","last_change":"2024-03-26T05:01:40.462389+0000","last_active":"2024-03-26T05:01:44.348747+0000","last_peered":"2024-03-26T05:01:44.348747+0000","last_clean":"2024-03-26T05:01:44.348747+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:01:44.348747+0000","last_undegraded":"2024-03-26T05:01:44.348747+0000","last_fullsized":"2024-03-26T05:01:44.348747+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.462271+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.765446+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":57,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344475+0000","last_change":"2024-03-26T05:00:28.984841+0000","last_active":"2024-03-26T05:01:44.344475+0000","last_peered":"2024-03-26T05:01:44.344475+0000","last_clean":"2024-03-26T05:01:44.344475+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:01:44.344475+0000","last_undegraded":"2024-03-26T05:01:44.344475+0000","last_fullsized":"2024-03-26T05:01:44.344475+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":83,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.461385+0000","last_change":"2024-03-26T05:00:26.956153+0000","last_active":"2024-03-26T05:01:41.461385+0000","last_peered":"2024-03-26T05:01:41.461385+0000","last_clean":"2024-03-26T05:01:41.461385+0000","last_became_active":"2024-03-26T05:00:26.955836+0000","last_became_peered":"2024-03-26T05:00:26.955836+0000","last_unstale":"2024-03-26T05:01:41.461385+0000","last_undegraded":"2024-03-26T05:01:41.461385+0000","last_fullsized":"2024-03-26T05:01:41.461385+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:54.166825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":75,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.461753+0000","last_change":"2024-03-26T05:01:26.785152+0000","last_active":"2024-03-26T05:01:44.461753+0000","last_peered":"2024-03-26T05:01:44.461753+0000","last_clean":"2024-03-26T05:01:44.461753+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:01:44.461753+0000","last_undegraded":"2024-03-26T05:01:44.461753+0000","last_fullsized":"2024-03-26T05:01:44.461753+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.785056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.785056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.302223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'22","reported_seq":111,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398469+0000","last_change":"2024-03-26T05:01:40.118101+0000","last_active":"2024-03-26T05:01:44.398469+0000","last_peered":"2024-03-26T05:01:44.398469+0000","last_clean":"2024-03-26T05:01:44.398469+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:01:44.398469+0000","last_undegraded":"2024-03-26T05:01:44.398469+0000","last_fullsized":"2024-03-26T05:01:44.398469+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:01:40.117953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.117953+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.646893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":57,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.349385+0000","last_change":"2024-03-26T05:00:28.979801+0000","last_active":"2024-03-26T05:01:44.349385+0000","last_peered":"2024-03-26T05:01:44.349385+0000","last_clean":"2024-03-26T05:01:44.349385+0000","last_became_active":"2024-03-26T05:00:28.979477+0000","last_became_peered":"2024-03-26T05:00:28.979477+0000","last_unstale":"2024-03-26T05:01:44.349385+0000","last_undegraded":"2024-03-26T05:01:44.349385+0000","last_fullsized":"2024-03-26T05:01:44.349385+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:56.897877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":88,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398810+0000","last_change":"2024-03-26T05:00:26.954972+0000","last_active":"2024-03-26T05:01:44.398810+0000","last_peered":"2024-03-26T05:01:44.398810+0000","last_clean":"2024-03-26T05:01:44.398810+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:01:44.398810+0000","last_undegraded":"2024-03-26T05:01:44.398810+0000","last_fullsized":"2024-03-26T05:01:44.398810+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":69,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344243+0000","last_change":"2024-03-26T05:00:22.777258+0000","last_active":"2024-03-26T05:01:44.344243+0000","last_peered":"2024-03-26T05:01:44.344243+0000","last_clean":"2024-03-26T05:01:44.344243+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:01:44.344243+0000","last_undegraded":"2024-03-26T05:01:44.344243+0000","last_fullsized":"2024-03-26T05:01:44.344243+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'15","reported_seq":90,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.461568+0000","last_change":"2024-03-26T05:00:25.648619+0000","last_active":"2024-03-26T05:01:41.461568+0000","last_peered":"2024-03-26T05:01:41.461568+0000","last_clean":"2024-03-26T05:01:41.461568+0000","last_became_active":"2024-03-26T05:00:25.648284+0000","last_became_peered":"2024-03-26T05:00:25.648284+0000","last_unstale":"2024-03-26T05:01:41.461568+0000","last_undegraded":"2024-03-26T05:01:41.461568+0000","last_fullsized":"2024-03-26T05:01:41.461568+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":62,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.461941+0000","last_change":"2024-03-26T05:01:31.568804+0000","last_active":"2024-03-26T05:01:41.461941+0000","last_peered":"2024-03-26T05:01:41.461941+0000","last_clean":"2024-03-26T05:01:41.461941+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:01:41.461941+0000","last_undegraded":"2024-03-26T05:01:41.461941+0000","last_fullsized":"2024-03-26T05:01:41.461941+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:31.568722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:31.568722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.465689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":60,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398928+0000","last_change":"2024-03-26T05:00:26.956233+0000","last_active":"2024-03-26T05:01:44.398928+0000","last_peered":"2024-03-26T05:01:44.398928+0000","last_clean":"2024-03-26T05:01:44.398928+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:01:44.398928+0000","last_undegraded":"2024-03-26T05:01:44.398928+0000","last_fullsized":"2024-03-26T05:01:44.398928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:54.400300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.461574+0000","last_change":"2024-03-26T05:00:22.770737+0000","last_active":"2024-03-26T05:01:44.461574+0000","last_peered":"2024-03-26T05:01:44.461574+0000","last_clean":"2024-03-26T05:01:44.461574+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:01:44.461574+0000","last_undegraded":"2024-03-26T05:01:44.461574+0000","last_fullsized":"2024-03-26T05:01:44.461574+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'21","reported_seq":113,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344778+0000","last_change":"2024-03-26T05:01:38.722775+0000","last_active":"2024-03-26T05:01:44.344778+0000","last_peered":"2024-03-26T05:01:44.344778+0000","last_clean":"2024-03-26T05:01:44.344778+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:01:44.344778+0000","last_undegraded":"2024-03-26T05:01:44.344778+0000","last_fullsized":"2024-03-26T05:01:44.344778+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:01:38.722652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:38.722652+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.850545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.461932+0000","last_change":"2024-03-26T05:00:29.462603+0000","last_active":"2024-03-26T05:01:44.461932+0000","last_peered":"2024-03-26T05:01:44.461932+0000","last_clean":"2024-03-26T05:01:44.461932+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:01:44.461932+0000","last_undegraded":"2024-03-26T05:01:44.461932+0000","last_fullsized":"2024-03-26T05:01:44.461932+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.502035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":58,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.462266+0000","last_change":"2024-03-26T05:00:26.953694+0000","last_active":"2024-03-26T05:01:41.462266+0000","last_peered":"2024-03-26T05:01:41.462266+0000","last_clean":"2024-03-26T05:01:41.462266+0000","last_became_active":"2024-03-26T05:00:26.953416+0000","last_became_peered":"2024-03-26T05:00:26.953416+0000","last_unstale":"2024-03-26T05:01:41.462266+0000","last_undegraded":"2024-03-26T05:01:41.462266+0000","last_fullsized":"2024-03-26T05:01:41.462266+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.182281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":81,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.400604+0000","last_change":"2024-03-26T05:01:37.375697+0000","last_active":"2024-03-26T05:01:44.400604+0000","last_peered":"2024-03-26T05:01:44.400604+0000","last_clean":"2024-03-26T05:01:44.400604+0000","last_became_active":"2024-03-26T05:00:22.771080+0000","last_became_peered":"2024-03-26T05:00:22.771080+0000","last_unstale":"2024-03-26T05:01:44.400604+0000","last_undegraded":"2024-03-26T05:01:44.400604+0000","last_fullsized":"2024-03-26T05:01:44.400604+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:01:37.375533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.375533+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:51.717267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'12","reported_seq":93,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398983+0000","last_change":"2024-03-26T05:01:27.964049+0000","last_active":"2024-03-26T05:01:44.398983+0000","last_peered":"2024-03-26T05:01:44.398983+0000","last_clean":"2024-03-26T05:01:44.398983+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:01:44.398983+0000","last_undegraded":"2024-03-26T05:01:44.398983+0000","last_fullsized":"2024-03-26T05:01:44.398983+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:01:27.963954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:27.963954+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:50.330132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":64,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.461521+0000","last_change":"2024-03-26T05:01:36.945807+0000","last_active":"2024-03-26T05:01:41.461521+0000","last_peered":"2024-03-26T05:01:41.461521+0000","last_clean":"2024-03-26T05:01:41.461521+0000","last_became_active":"2024-03-26T05:00:29.461059+0000","last_became_peered":"2024-03-26T05:00:29.461059+0000","last_unstale":"2024-03-26T05:01:41.461521+0000","last_undegraded":"2024-03-26T05:01:41.461521+0000","last_fullsized":"2024-03-26T05:01:41.461521+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:01:36.945708+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.001390+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'3","reported_seq":93,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.462201+0000","last_change":"2024-03-26T05:01:22.799027+0000","last_active":"2024-03-26T05:01:44.462201+0000","last_peered":"2024-03-26T05:01:44.462201+0000","last_clean":"2024-03-26T05:01:44.462201+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:01:44.462201+0000","last_undegraded":"2024-03-26T05:01:44.462201+0000","last_fullsized":"2024-03-26T05:01:44.462201+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-03-26T05:01:22.798897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:22.798897+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.103651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'27","reported_seq":119,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.460758+0000","last_change":"2024-03-26T05:01:29.048598+0000","last_active":"2024-03-26T05:01:41.460758+0000","last_peered":"2024-03-26T05:01:41.460758+0000","last_clean":"2024-03-26T05:01:41.460758+0000","last_became_active":"2024-03-26T05:00:25.651328+0000","last_became_peered":"2024-03-26T05:00:25.651328+0000","last_unstale":"2024-03-26T05:01:41.460758+0000","last_undegraded":"2024-03-26T05:01:41.460758+0000","last_fullsized":"2024-03-26T05:01:41.460758+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'27","last_scrub_stamp":"2024-03-26T05:01:29.048456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.048456+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:56.053198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":92,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.461928+0000","last_change":"2024-03-26T05:01:37.743111+0000","last_active":"2024-03-26T05:01:44.461928+0000","last_peered":"2024-03-26T05:01:44.461928+0000","last_clean":"2024-03-26T05:01:44.461928+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:01:44.461928+0000","last_undegraded":"2024-03-26T05:01:44.461928+0000","last_fullsized":"2024-03-26T05:01:44.461928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:37.742995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.742995+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:07.111791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":64,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398286+0000","last_change":"2024-03-26T05:01:29.011852+0000","last_active":"2024-03-26T05:01:44.398286+0000","last_peered":"2024-03-26T05:01:44.398286+0000","last_clean":"2024-03-26T05:01:44.398286+0000","last_became_active":"2024-03-26T05:00:28.985917+0000","last_became_peered":"2024-03-26T05:00:28.985917+0000","last_unstale":"2024-03-26T05:01:44.398286+0000","last_undegraded":"2024-03-26T05:01:44.398286+0000","last_fullsized":"2024-03-26T05:01:44.398286+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.011757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.011757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.742465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.462281+0000","last_change":"2024-03-26T05:00:22.775380+0000","last_active":"2024-03-26T05:01:44.462281+0000","last_peered":"2024-03-26T05:01:44.462281+0000","last_clean":"2024-03-26T05:01:44.462281+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:01:44.462281+0000","last_undegraded":"2024-03-26T05:01:44.462281+0000","last_fullsized":"2024-03-26T05:01:44.462281+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'24","reported_seq":112,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348775+0000","last_change":"2024-03-26T05:00:24.944015+0000","last_active":"2024-03-26T05:01:44.348775+0000","last_peered":"2024-03-26T05:01:44.348775+0000","last_clean":"2024-03-26T05:01:44.348775+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:01:44.348775+0000","last_undegraded":"2024-03-26T05:01:44.348775+0000","last_fullsized":"2024-03-26T05:01:44.348775+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:46.669169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":66,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.460247+0000","last_change":"2024-03-26T05:01:26.618364+0000","last_active":"2024-03-26T05:01:41.460247+0000","last_peered":"2024-03-26T05:01:41.460247+0000","last_clean":"2024-03-26T05:01:41.460247+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:01:41.460247+0000","last_undegraded":"2024-03-26T05:01:41.460247+0000","last_fullsized":"2024-03-26T05:01:41.460247+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.618218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.618218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.093176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.461378+0000","last_change":"2024-03-26T05:00:28.981813+0000","last_active":"2024-03-26T05:01:44.461378+0000","last_peered":"2024-03-26T05:01:44.461378+0000","last_clean":"2024-03-26T05:01:44.461378+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:01:44.461378+0000","last_undegraded":"2024-03-26T05:01:44.461378+0000","last_fullsized":"2024-03-26T05:01:44.461378+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.155885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":66,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.460530+0000","last_change":"2024-03-26T05:01:33.638557+0000","last_active":"2024-03-26T05:01:41.460530+0000","last_peered":"2024-03-26T05:01:41.460530+0000","last_clean":"2024-03-26T05:01:41.460530+0000","last_became_active":"2024-03-26T05:00:26.961730+0000","last_became_peered":"2024-03-26T05:00:26.961730+0000","last_unstale":"2024-03-26T05:01:41.460530+0000","last_undegraded":"2024-03-26T05:01:41.460530+0000","last_fullsized":"2024-03-26T05:01:41.460530+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:33.638469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:33.638469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:00.135833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.462290+0000","last_change":"2024-03-26T05:00:22.772945+0000","last_active":"2024-03-26T05:01:44.462290+0000","last_peered":"2024-03-26T05:01:44.462290+0000","last_clean":"2024-03-26T05:01:44.462290+0000","last_became_active":"2024-03-26T05:00:22.772590+0000","last_became_peered":"2024-03-26T05:00:22.772590+0000","last_unstale":"2024-03-26T05:01:44.462290+0000","last_undegraded":"2024-03-26T05:01:44.462290+0000","last_fullsized":"2024-03-26T05:01:44.462290+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:44.940408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'25","reported_seq":106,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.400941+0000","last_change":"2024-03-26T05:00:25.649083+0000","last_active":"2024-03-26T05:01:44.400941+0000","last_peered":"2024-03-26T05:01:44.400941+0000","last_clean":"2024-03-26T05:01:44.400941+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:01:44.400941+0000","last_undegraded":"2024-03-26T05:01:44.400941+0000","last_fullsized":"2024-03-26T05:01:44.400941+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":57,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.343935+0000","last_change":"2024-03-26T05:00:28.981411+0000","last_active":"2024-03-26T05:01:44.343935+0000","last_peered":"2024-03-26T05:01:44.343935+0000","last_clean":"2024-03-26T05:01:44.343935+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:01:44.343935+0000","last_undegraded":"2024-03-26T05:01:44.343935+0000","last_fullsized":"2024-03-26T05:01:44.343935+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:55.410289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":58,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.460443+0000","last_change":"2024-03-26T05:00:26.956293+0000","last_active":"2024-03-26T05:01:41.460443+0000","last_peered":"2024-03-26T05:01:41.460443+0000","last_clean":"2024-03-26T05:01:41.460443+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:01:41.460443+0000","last_undegraded":"2024-03-26T05:01:41.460443+0000","last_fullsized":"2024-03-26T05:01:41.460443+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:54.998598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":66,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.462078+0000","last_change":"2024-03-26T05:00:22.774369+0000","last_active":"2024-03-26T05:01:41.462078+0000","last_peered":"2024-03-26T05:01:41.462078+0000","last_clean":"2024-03-26T05:01:41.462078+0000","last_became_active":"2024-03-26T05:00:22.773770+0000","last_became_peered":"2024-03-26T05:00:22.773770+0000","last_unstale":"2024-03-26T05:01:41.462078+0000","last_undegraded":"2024-03-26T05:01:41.462078+0000","last_fullsized":"2024-03-26T05:01:41.462078+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'15","reported_seq":93,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344787+0000","last_change":"2024-03-26T05:00:24.936134+0000","last_active":"2024-03-26T05:01:44.344787+0000","last_peered":"2024-03-26T05:01:44.344787+0000","last_clean":"2024-03-26T05:01:44.344787+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:01:44.344787+0000","last_undegraded":"2024-03-26T05:01:44.344787+0000","last_fullsized":"2024-03-26T05:01:44.344787+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":56,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398503+0000","last_change":"2024-03-26T05:00:28.984072+0000","last_active":"2024-03-26T05:01:44.398503+0000","last_peered":"2024-03-26T05:01:44.398503+0000","last_clean":"2024-03-26T05:01:44.398503+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:01:44.398503+0000","last_undegraded":"2024-03-26T05:01:44.398503+0000","last_fullsized":"2024-03-26T05:01:44.398503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":85,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.461273+0000","last_change":"2024-03-26T05:01:35.934149+0000","last_active":"2024-03-26T05:01:41.461273+0000","last_peered":"2024-03-26T05:01:41.461273+0000","last_clean":"2024-03-26T05:01:41.461273+0000","last_became_active":"2024-03-26T05:00:21.760521+0000","last_became_peered":"2024-03-26T05:00:21.760521+0000","last_unstale":"2024-03-26T05:01:41.461273+0000","last_undegraded":"2024-03-26T05:01:41.461273+0000","last_fullsized":"2024-03-26T05:01:41.461273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:01:35.934044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.934044+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:02.210221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":69,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348540+0000","last_change":"2024-03-26T05:00:22.771224+0000","last_active":"2024-03-26T05:01:44.348540+0000","last_peered":"2024-03-26T05:01:44.348540+0000","last_clean":"2024-03-26T05:01:44.348540+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:01:44.348540+0000","last_undegraded":"2024-03-26T05:01:44.348540+0000","last_fullsized":"2024-03-26T05:01:44.348540+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:51.123388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'34","reported_seq":149,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.647033+0000","last_change":"2024-03-26T05:01:40.826966+0000","last_active":"2024-03-26T05:01:44.647033+0000","last_peered":"2024-03-26T05:01:44.647033+0000","last_clean":"2024-03-26T05:01:44.647033+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:01:44.647033+0000","last_undegraded":"2024-03-26T05:01:44.647033+0000","last_fullsized":"2024-03-26T05:01:44.647033+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'34","last_scrub_stamp":"2024-03-26T05:01:40.826856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.826856+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.349406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":61,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344353+0000","last_change":"2024-03-26T05:00:26.964781+0000","last_active":"2024-03-26T05:01:44.344353+0000","last_peered":"2024-03-26T05:01:44.344353+0000","last_clean":"2024-03-26T05:01:44.344353+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:01:44.344353+0000","last_undegraded":"2024-03-26T05:01:44.344353+0000","last_fullsized":"2024-03-26T05:01:44.344353+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":63,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.462390+0000","last_change":"2024-03-26T05:01:41.789755+0000","last_active":"2024-03-26T05:01:44.462390+0000","last_peered":"2024-03-26T05:01:44.462390+0000","last_clean":"2024-03-26T05:01:44.462390+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:01:44.462390+0000","last_undegraded":"2024-03-26T05:01:44.462390+0000","last_fullsized":"2024-03-26T05:01:44.462390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:41.789650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.789650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.302843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":68,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.646604+0000","last_change":"2024-03-26T05:00:23.346922+0000","last_active":"2024-03-26T05:01:44.646604+0000","last_peered":"2024-03-26T05:01:44.646604+0000","last_clean":"2024-03-26T05:01:44.646604+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:01:44.646604+0000","last_undegraded":"2024-03-26T05:01:44.646604+0000","last_fullsized":"2024-03-26T05:01:44.646604+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:48.374217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":83,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.460349+0000","last_change":"2024-03-26T05:01:37.678850+0000","last_active":"2024-03-26T05:01:41.460349+0000","last_peered":"2024-03-26T05:01:41.460349+0000","last_clean":"2024-03-26T05:01:41.460349+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:01:41.460349+0000","last_undegraded":"2024-03-26T05:01:41.460349+0000","last_fullsized":"2024-03-26T05:01:41.460349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.678736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.678736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.820757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'19","reported_seq":103,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.647152+0000","last_change":"2024-03-26T05:00:24.937826+0000","last_active":"2024-03-26T05:01:44.647152+0000","last_peered":"2024-03-26T05:01:44.647152+0000","last_clean":"2024-03-26T05:01:44.647152+0000","last_became_active":"2024-03-26T05:00:24.937145+0000","last_became_peered":"2024-03-26T05:00:24.937145+0000","last_unstale":"2024-03-26T05:01:44.647152+0000","last_undegraded":"2024-03-26T05:01:44.647152+0000","last_fullsized":"2024-03-26T05:01:44.647152+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:49.327553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":66,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.461194+0000","last_change":"2024-03-26T05:01:29.627760+0000","last_active":"2024-03-26T05:01:41.461194+0000","last_peered":"2024-03-26T05:01:41.461194+0000","last_clean":"2024-03-26T05:01:41.461194+0000","last_became_active":"2024-03-26T05:00:26.957936+0000","last_became_peered":"2024-03-26T05:00:26.957936+0000","last_unstale":"2024-03-26T05:01:41.461194+0000","last_undegraded":"2024-03-26T05:01:41.461194+0000","last_fullsized":"2024-03-26T05:01:41.461194+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.627680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.627680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.292786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":54,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.462460+0000","last_change":"2024-03-26T05:00:28.979759+0000","last_active":"2024-03-26T05:01:41.462460+0000","last_peered":"2024-03-26T05:01:41.462460+0000","last_clean":"2024-03-26T05:01:41.462460+0000","last_became_active":"2024-03-26T05:00:28.979373+0000","last_became_peered":"2024-03-26T05:00:28.979373+0000","last_unstale":"2024-03-26T05:01:41.462460+0000","last_undegraded":"2024-03-26T05:01:41.462460+0000","last_fullsized":"2024-03-26T05:01:41.462460+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:44.065816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":322,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.461507+0000","last_change":"2024-03-26T05:01:04.782428+0000","last_active":"2024-03-26T05:01:44.461507+0000","last_peered":"2024-03-26T05:01:44.461507+0000","last_clean":"2024-03-26T05:01:44.461507+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:01:44.461507+0000","last_undegraded":"2024-03-26T05:01:44.461507+0000","last_fullsized":"2024-03-26T05:01:44.461507+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-03-26T05:01:04.782336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:01:04.782336+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:15.009834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":76,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398317+0000","last_change":"2024-03-26T05:01:25.941429+0000","last_active":"2024-03-26T05:01:44.398317+0000","last_peered":"2024-03-26T05:01:44.398317+0000","last_clean":"2024-03-26T05:01:44.398317+0000","last_became_active":"2024-03-26T05:00:22.774298+0000","last_became_peered":"2024-03-26T05:00:22.774298+0000","last_unstale":"2024-03-26T05:01:44.398317+0000","last_undegraded":"2024-03-26T05:01:44.398317+0000","last_fullsized":"2024-03-26T05:01:44.398317+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:25.941344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:25.941344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:34.075498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":102,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.401251+0000","last_change":"2024-03-26T05:00:25.648924+0000","last_active":"2024-03-26T05:01:44.401251+0000","last_peered":"2024-03-26T05:01:44.401251+0000","last_clean":"2024-03-26T05:01:44.401251+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:01:44.401251+0000","last_undegraded":"2024-03-26T05:01:44.401251+0000","last_fullsized":"2024-03-26T05:01:44.401251+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.782291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.462029+0000","last_change":"2024-03-26T05:00:26.962047+0000","last_active":"2024-03-26T05:01:44.462029+0000","last_peered":"2024-03-26T05:01:44.462029+0000","last_clean":"2024-03-26T05:01:44.462029+0000","last_became_active":"2024-03-26T05:00:26.961724+0000","last_became_peered":"2024-03-26T05:00:26.961724+0000","last_unstale":"2024-03-26T05:01:44.462029+0000","last_undegraded":"2024-03-26T05:01:44.462029+0000","last_fullsized":"2024-03-26T05:01:44.462029+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":64,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.646818+0000","last_change":"2024-03-26T05:01:38.822538+0000","last_active":"2024-03-26T05:01:44.646818+0000","last_peered":"2024-03-26T05:01:44.646818+0000","last_clean":"2024-03-26T05:01:44.646818+0000","last_became_active":"2024-03-26T05:00:28.983202+0000","last_became_peered":"2024-03-26T05:00:28.983202+0000","last_unstale":"2024-03-26T05:01:44.646818+0000","last_undegraded":"2024-03-26T05:01:44.646818+0000","last_fullsized":"2024-03-26T05:01:44.646818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:38.822409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:38.822409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:41.940538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":77,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344195+0000","last_change":"2024-03-26T05:01:25.629227+0000","last_active":"2024-03-26T05:01:44.344195+0000","last_peered":"2024-03-26T05:01:44.344195+0000","last_clean":"2024-03-26T05:01:44.344195+0000","last_became_active":"2024-03-26T05:00:22.773081+0000","last_became_peered":"2024-03-26T05:00:22.773081+0000","last_unstale":"2024-03-26T05:01:44.344195+0000","last_undegraded":"2024-03-26T05:01:44.344195+0000","last_fullsized":"2024-03-26T05:01:44.344195+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:25.629101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:25.629101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:43.255919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":164,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.461442+0000","last_change":"2024-03-26T05:01:36.780629+0000","last_active":"2024-03-26T05:01:44.461442+0000","last_peered":"2024-03-26T05:01:44.461442+0000","last_clean":"2024-03-26T05:01:44.461442+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:01:44.461442+0000","last_undegraded":"2024-03-26T05:01:44.461442+0000","last_fullsized":"2024-03-26T05:01:44.461442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:01:36.780518+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.077309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":103,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344563+0000","last_change":"2024-03-26T05:00:49.653404+0000","last_active":"2024-03-26T05:01:44.344563+0000","last_peered":"2024-03-26T05:01:44.344563+0000","last_clean":"2024-03-26T05:01:44.344563+0000","last_became_active":"2024-03-26T05:00:49.642276+0000","last_became_peered":"2024-03-26T05:00:49.642276+0000","last_unstale":"2024-03-26T05:01:44.344563+0000","last_undegraded":"2024-03-26T05:01:44.344563+0000","last_fullsized":"2024-03-26T05:01:44.344563+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:48.856193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":66,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.973872+0000","last_change":"2024-03-26T05:01:41.973895+0000","last_active":"2024-03-26T05:01:41.973872+0000","last_peered":"2024-03-26T05:01:41.973872+0000","last_clean":"2024-03-26T05:01:41.973872+0000","last_became_active":"2024-03-26T05:00:26.956677+0000","last_became_peered":"2024-03-26T05:00:26.956677+0000","last_unstale":"2024-03-26T05:01:41.973872+0000","last_undegraded":"2024-03-26T05:01:41.973872+0000","last_fullsized":"2024-03-26T05:01:41.973872+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:41.973805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.973805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:54.658519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":56,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.646466+0000","last_change":"2024-03-26T05:00:29.461548+0000","last_active":"2024-03-26T05:01:44.646466+0000","last_peered":"2024-03-26T05:01:44.646466+0000","last_clean":"2024-03-26T05:01:44.646466+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:01:44.646466+0000","last_undegraded":"2024-03-26T05:01:44.646466+0000","last_fullsized":"2024-03-26T05:01:44.646466+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:46.388389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":76,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.646966+0000","last_change":"2024-03-26T05:01:35.745231+0000","last_active":"2024-03-26T05:01:44.646966+0000","last_peered":"2024-03-26T05:01:44.646966+0000","last_clean":"2024-03-26T05:01:44.646966+0000","last_became_active":"2024-03-26T05:00:23.346665+0000","last_became_peered":"2024-03-26T05:00:23.346665+0000","last_unstale":"2024-03-26T05:01:44.646966+0000","last_undegraded":"2024-03-26T05:01:44.646966+0000","last_fullsized":"2024-03-26T05:01:44.646966+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.745100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.314539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'25","reported_seq":113,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344553+0000","last_change":"2024-03-26T05:00:25.648598+0000","last_active":"2024-03-26T05:01:44.344553+0000","last_peered":"2024-03-26T05:01:44.344553+0000","last_clean":"2024-03-26T05:01:44.344553+0000","last_became_active":"2024-03-26T05:00:25.648253+0000","last_became_peered":"2024-03-26T05:00:25.648253+0000","last_unstale":"2024-03-26T05:01:44.344553+0000","last_undegraded":"2024-03-26T05:01:44.344553+0000","last_fullsized":"2024-03-26T05:01:44.344553+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.400989+0000","last_change":"2024-03-26T05:00:26.958976+0000","last_active":"2024-03-26T05:01:44.400989+0000","last_peered":"2024-03-26T05:01:44.400989+0000","last_clean":"2024-03-26T05:01:44.400989+0000","last_became_active":"2024-03-26T05:00:26.958689+0000","last_became_peered":"2024-03-26T05:00:26.958689+0000","last_unstale":"2024-03-26T05:01:44.400989+0000","last_undegraded":"2024-03-26T05:01:44.400989+0000","last_fullsized":"2024-03-26T05:01:44.400989+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":57,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344344+0000","last_change":"2024-03-26T05:00:28.987246+0000","last_active":"2024-03-26T05:01:44.344344+0000","last_peered":"2024-03-26T05:01:44.344344+0000","last_clean":"2024-03-26T05:01:44.344344+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:01:44.344344+0000","last_undegraded":"2024-03-26T05:01:44.344344+0000","last_fullsized":"2024-03-26T05:01:44.344344+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":66,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.461431+0000","last_change":"2024-03-26T05:00:23.347452+0000","last_active":"2024-03-26T05:01:41.461431+0000","last_peered":"2024-03-26T05:01:41.461431+0000","last_clean":"2024-03-26T05:01:41.461431+0000","last_became_active":"2024-03-26T05:00:23.346876+0000","last_became_peered":"2024-03-26T05:00:23.346876+0000","last_unstale":"2024-03-26T05:01:41.461431+0000","last_undegraded":"2024-03-26T05:01:41.461431+0000","last_fullsized":"2024-03-26T05:01:41.461431+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'16","reported_seq":60,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:42.753594+0000","last_change":"2024-03-26T05:01:42.753638+0000","last_active":"2024-03-26T05:01:42.753594+0000","last_peered":"2024-03-26T05:01:42.753594+0000","last_clean":"2024-03-26T05:01:42.753594+0000","last_became_active":"2024-03-26T05:00:49.641444+0000","last_became_peered":"2024-03-26T05:00:49.641444+0000","last_unstale":"2024-03-26T05:01:42.753594+0000","last_undegraded":"2024-03-26T05:01:42.753594+0000","last_fullsized":"2024-03-26T05:01:42.753594+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:01:42.753523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:42.753523+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.891497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":86,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344885+0000","last_change":"2024-03-26T05:00:26.962263+0000","last_active":"2024-03-26T05:01:44.344885+0000","last_peered":"2024-03-26T05:01:44.344885+0000","last_clean":"2024-03-26T05:01:44.344885+0000","last_became_active":"2024-03-26T05:00:26.961907+0000","last_became_peered":"2024-03-26T05:00:26.961907+0000","last_unstale":"2024-03-26T05:01:44.344885+0000","last_undegraded":"2024-03-26T05:01:44.344885+0000","last_fullsized":"2024-03-26T05:01:44.344885+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.462140+0000","last_change":"2024-03-26T05:00:29.461447+0000","last_active":"2024-03-26T05:01:44.462140+0000","last_peered":"2024-03-26T05:01:44.462140+0000","last_clean":"2024-03-26T05:01:44.462140+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:01:44.462140+0000","last_undegraded":"2024-03-26T05:01:44.462140+0000","last_fullsized":"2024-03-26T05:01:44.462140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:57.420912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":78,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.344627+0000","last_change":"2024-03-26T05:01:29.581397+0000","last_active":"2024-03-26T05:01:44.344627+0000","last_peered":"2024-03-26T05:01:44.344627+0000","last_clean":"2024-03-26T05:01:44.344627+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:01:44.344627+0000","last_undegraded":"2024-03-26T05:01:44.344627+0000","last_fullsized":"2024-03-26T05:01:44.344627+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:01:29.581309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.581309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.525431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'20","reported_seq":98,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348408+0000","last_change":"2024-03-26T05:00:24.936644+0000","last_active":"2024-03-26T05:01:44.348408+0000","last_peered":"2024-03-26T05:01:44.348408+0000","last_clean":"2024-03-26T05:01:44.348408+0000","last_became_active":"2024-03-26T05:00:24.936315+0000","last_became_peered":"2024-03-26T05:00:24.936315+0000","last_unstale":"2024-03-26T05:01:44.348408+0000","last_undegraded":"2024-03-26T05:01:44.348408+0000","last_fullsized":"2024-03-26T05:01:44.348408+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":85,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398171+0000","last_change":"2024-03-26T05:00:26.957814+0000","last_active":"2024-03-26T05:01:44.398171+0000","last_peered":"2024-03-26T05:01:44.398171+0000","last_clean":"2024-03-26T05:01:44.398171+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:01:44.398171+0000","last_undegraded":"2024-03-26T05:01:44.398171+0000","last_fullsized":"2024-03-26T05:01:44.398171+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:55.181714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":57,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348380+0000","last_change":"2024-03-26T05:00:28.980771+0000","last_active":"2024-03-26T05:01:44.348380+0000","last_peered":"2024-03-26T05:01:44.348380+0000","last_clean":"2024-03-26T05:01:44.348380+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:01:44.348380+0000","last_undegraded":"2024-03-26T05:01:44.348380+0000","last_fullsized":"2024-03-26T05:01:44.348380+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":68,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398826+0000","last_change":"2024-03-26T05:00:22.779020+0000","last_active":"2024-03-26T05:01:44.398826+0000","last_peered":"2024-03-26T05:01:44.398826+0000","last_clean":"2024-03-26T05:01:44.398826+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:01:44.398826+0000","last_undegraded":"2024-03-26T05:01:44.398826+0000","last_fullsized":"2024-03-26T05:01:44.398826+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:48.991339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"96'40","reported_seq":138,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398139+0000","last_change":"2024-03-26T05:01:42.152769+0000","last_active":"2024-03-26T05:01:44.398139+0000","last_peered":"2024-03-26T05:01:44.398139+0000","last_clean":"2024-03-26T05:01:44.398139+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:01:44.398139+0000","last_undegraded":"2024-03-26T05:01:44.398139+0000","last_fullsized":"2024-03-26T05:01:44.398139+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:01:42.152630+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:01:42.152630+0000","last_clean_scrub_stamp":"2024-03-26T05:01:42.152630+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.656601+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":61,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348237+0000","last_change":"2024-03-26T05:00:26.959009+0000","last_active":"2024-03-26T05:01:44.348237+0000","last_peered":"2024-03-26T05:01:44.348237+0000","last_clean":"2024-03-26T05:01:44.348237+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:01:44.348237+0000","last_undegraded":"2024-03-26T05:01:44.348237+0000","last_fullsized":"2024-03-26T05:01:44.348237+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:47.880056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":54,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.461167+0000","last_change":"2024-03-26T05:00:28.983093+0000","last_active":"2024-03-26T05:01:41.461167+0000","last_peered":"2024-03-26T05:01:41.461167+0000","last_clean":"2024-03-26T05:01:41.461167+0000","last_became_active":"2024-03-26T05:00:28.982825+0000","last_became_peered":"2024-03-26T05:00:28.982825+0000","last_unstale":"2024-03-26T05:01:41.461167+0000","last_undegraded":"2024-03-26T05:01:41.461167+0000","last_fullsized":"2024-03-26T05:01:41.461167+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.359012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":94,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.349262+0000","last_change":"2024-03-26T05:01:41.485196+0000","last_active":"2024-03-26T05:01:44.349262+0000","last_peered":"2024-03-26T05:01:44.349262+0000","last_clean":"2024-03-26T05:01:44.349262+0000","last_became_active":"2024-03-26T05:00:26.956596+0000","last_became_peered":"2024-03-26T05:00:26.956596+0000","last_unstale":"2024-03-26T05:01:44.349262+0000","last_undegraded":"2024-03-26T05:01:44.349262+0000","last_fullsized":"2024-03-26T05:01:44.349262+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:41.485086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.485086+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:59.948091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":68,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398060+0000","last_change":"2024-03-26T05:00:22.775050+0000","last_active":"2024-03-26T05:01:44.398060+0000","last_peered":"2024-03-26T05:01:44.398060+0000","last_clean":"2024-03-26T05:01:44.398060+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:01:44.398060+0000","last_undegraded":"2024-03-26T05:01:44.398060+0000","last_fullsized":"2024-03-26T05:01:44.398060+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:45.323836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":97,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.647224+0000","last_change":"2024-03-26T05:01:28.754168+0000","last_active":"2024-03-26T05:01:44.647224+0000","last_peered":"2024-03-26T05:01:44.647224+0000","last_clean":"2024-03-26T05:01:44.647224+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:01:44.647224+0000","last_undegraded":"2024-03-26T05:01:44.647224+0000","last_fullsized":"2024-03-26T05:01:44.647224+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:28.754068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:28.754068+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:40.695752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":56,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398337+0000","last_change":"2024-03-26T05:00:29.461578+0000","last_active":"2024-03-26T05:01:44.398337+0000","last_peered":"2024-03-26T05:01:44.398337+0000","last_clean":"2024-03-26T05:01:44.398337+0000","last_became_active":"2024-03-26T05:00:29.461236+0000","last_became_peered":"2024-03-26T05:00:29.461236+0000","last_unstale":"2024-03-26T05:01:44.398337+0000","last_undegraded":"2024-03-26T05:01:44.398337+0000","last_fullsized":"2024-03-26T05:01:44.398337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:49.354548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":58,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.462928+0000","last_change":"2024-03-26T05:00:26.956392+0000","last_active":"2024-03-26T05:01:41.462928+0000","last_peered":"2024-03-26T05:01:41.462928+0000","last_clean":"2024-03-26T05:01:41.462928+0000","last_became_active":"2024-03-26T05:00:26.955805+0000","last_became_peered":"2024-03-26T05:00:26.955805+0000","last_unstale":"2024-03-26T05:01:41.462928+0000","last_undegraded":"2024-03-26T05:01:41.462928+0000","last_fullsized":"2024-03-26T05:01:41.462928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":77,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.348269+0000","last_change":"2024-03-26T05:01:26.442833+0000","last_active":"2024-03-26T05:01:44.348269+0000","last_peered":"2024-03-26T05:01:44.348269+0000","last_clean":"2024-03-26T05:01:44.348269+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:01:44.348269+0000","last_undegraded":"2024-03-26T05:01:44.348269+0000","last_fullsized":"2024-03-26T05:01:44.348269+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.442722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.442722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:39.204290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'17","reported_seq":96,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.349138+0000","last_change":"2024-03-26T05:00:24.944403+0000","last_active":"2024-03-26T05:01:44.349138+0000","last_peered":"2024-03-26T05:01:44.349138+0000","last_clean":"2024-03-26T05:01:44.349138+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:01:44.349138+0000","last_undegraded":"2024-03-26T05:01:44.349138+0000","last_fullsized":"2024-03-26T05:01:44.349138+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":54,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:41.461041+0000","last_change":"2024-03-26T05:00:28.982075+0000","last_active":"2024-03-26T05:01:41.461041+0000","last_peered":"2024-03-26T05:01:41.461041+0000","last_clean":"2024-03-26T05:01:41.461041+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:01:41.461041+0000","last_undegraded":"2024-03-26T05:01:41.461041+0000","last_fullsized":"2024-03-26T05:01:41.461041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:46.440667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":68,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.647128+0000","last_change":"2024-03-26T05:01:37.774253+0000","last_active":"2024-03-26T05:01:44.647128+0000","last_peered":"2024-03-26T05:01:44.647128+0000","last_clean":"2024-03-26T05:01:44.647128+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:01:44.647128+0000","last_undegraded":"2024-03-26T05:01:44.647128+0000","last_fullsized":"2024-03-26T05:01:44.647128+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.774158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.774158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:01.227552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":75,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.400771+0000","last_change":"2024-03-26T05:01:28.386278+0000","last_active":"2024-03-26T05:01:44.400771+0000","last_peered":"2024-03-26T05:01:44.400771+0000","last_clean":"2024-03-26T05:01:44.400771+0000","last_became_active":"2024-03-26T05:00:22.768610+0000","last_became_peered":"2024-03-26T05:00:22.768610+0000","last_unstale":"2024-03-26T05:01:44.400771+0000","last_undegraded":"2024-03-26T05:01:44.400771+0000","last_fullsized":"2024-03-26T05:01:44.400771+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_clean_scrub_stamp":"2024-03-26T05:01:28.386172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.056496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":93,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.349204+0000","last_change":"2024-03-26T05:00:24.942796+0000","last_active":"2024-03-26T05:01:44.349204+0000","last_peered":"2024-03-26T05:01:44.349204+0000","last_clean":"2024-03-26T05:01:44.349204+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:01:44.349204+0000","last_undegraded":"2024-03-26T05:01:44.349204+0000","last_fullsized":"2024-03-26T05:01:44.349204+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":57,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.349373+0000","last_change":"2024-03-26T05:00:28.978772+0000","last_active":"2024-03-26T05:01:44.349373+0000","last_peered":"2024-03-26T05:01:44.349373+0000","last_clean":"2024-03-26T05:01:44.349373+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:01:44.349373+0000","last_undegraded":"2024-03-26T05:01:44.349373+0000","last_fullsized":"2024-03-26T05:01:44.349373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":91,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-26T05:01:43.782552+0000","last_change":"2024-03-26T05:01:43.782577+0000","last_active":"2024-03-26T05:01:43.782552+0000","last_peered":"2024-03-26T05:01:43.782552+0000","last_clean":"2024-03-26T05:01:43.782552+0000","last_became_active":"2024-03-26T05:00:26.957142+0000","last_became_peered":"2024-03-26T05:00:26.957142+0000","last_unstale":"2024-03-26T05:01:43.782552+0000","last_undegraded":"2024-03-26T05:01:43.782552+0000","last_fullsized":"2024-03-26T05:01:43.782552+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:43.782490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:43.782490+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.086882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":76,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.398497+0000","last_change":"2024-03-26T05:01:39.142373+0000","last_active":"2024-03-26T05:01:44.398497+0000","last_peered":"2024-03-26T05:01:44.398497+0000","last_clean":"2024-03-26T05:01:44.398497+0000","last_became_active":"2024-03-26T05:00:22.778719+0000","last_became_peered":"2024-03-26T05:00:22.778719+0000","last_unstale":"2024-03-26T05:01:44.398497+0000","last_undegraded":"2024-03-26T05:01:44.398497+0000","last_fullsized":"2024-03-26T05:01:44.398497+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.142249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.142249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.076132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":88,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-26T05:01:44.403891+0000","last_change":"2024-03-26T05:01:44.403960+0000","last_active":"2024-03-26T05:01:44.403891+0000","last_peered":"2024-03-26T05:01:44.403891+0000","last_clean":"2024-03-26T05:01:44.403891+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:01:44.403891+0000","last_undegraded":"2024-03-26T05:01:44.403891+0000","last_fullsized":"2024-03-26T05:01:44.403891+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:01:44.403798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.403798+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:04.259798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":602,"ondisk_log_size":602,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005077,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1064437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332122,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32716,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712436,"statfs":{"total":95995035648,"available":95961534464,"internally_reserved":0,"allocated":1572864,"data_stored":1062921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":473817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28588,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716564,"statfs":{"total":95995035648,"available":95965761536,"internally_reserved":0,"allocated":1601536,"data_stored":1064590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":472367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117033,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":473033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920805,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":481380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659167,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":472879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:01:48.408 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:01:48.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:48 smithi080 ceph-mon[40404]: 3.1b deep-scrub starts 2024-03-26T05:01:48.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:48 smithi080 ceph-mon[40404]: 3.1b deep-scrub ok 2024-03-26T05:01:48.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:48 smithi080 ceph-mon[29691]: 3.1b deep-scrub starts 2024-03-26T05:01:48.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:48 smithi080 ceph-mon[29691]: 3.1b deep-scrub ok 2024-03-26T05:01:48.662 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:48.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:48 smithi132 ceph-mon[33500]: 3.1b deep-scrub starts 2024-03-26T05:01:48.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:48 smithi132 ceph-mon[33500]: 3.1b deep-scrub ok 2024-03-26T05:01:49.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[29691]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 799 B/s rd, 0 op/s 2024-03-26T05:01:49.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[29691]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:01:49.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[29691]: 4.15 deep-scrub starts 2024-03-26T05:01:49.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[29691]: 4.15 deep-scrub ok 2024-03-26T05:01:49.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[29691]: 6.2 scrub starts 2024-03-26T05:01:49.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[29691]: 6.2 scrub ok 2024-03-26T05:01:49.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:49.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:49.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[29691]: 5.14 scrub starts 2024-03-26T05:01:49.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[29691]: 5.14 scrub ok 2024-03-26T05:01:49.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[40404]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 799 B/s rd, 0 op/s 2024-03-26T05:01:49.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[40404]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:01:49.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[40404]: 4.15 deep-scrub starts 2024-03-26T05:01:49.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[40404]: 4.15 deep-scrub ok 2024-03-26T05:01:49.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[40404]: 6.2 scrub starts 2024-03-26T05:01:49.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[40404]: 6.2 scrub ok 2024-03-26T05:01:49.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:49.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:49.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[40404]: 5.14 scrub starts 2024-03-26T05:01:49.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:49 smithi080 ceph-mon[40404]: 5.14 scrub ok 2024-03-26T05:01:49.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:49 smithi132 ceph-mon[33500]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 799 B/s rd, 0 op/s 2024-03-26T05:01:49.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:49 smithi132 ceph-mon[33500]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:01:49.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:49 smithi132 ceph-mon[33500]: 4.15 deep-scrub starts 2024-03-26T05:01:49.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:49 smithi132 ceph-mon[33500]: 4.15 deep-scrub ok 2024-03-26T05:01:49.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:49 smithi132 ceph-mon[33500]: 6.2 scrub starts 2024-03-26T05:01:49.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:49 smithi132 ceph-mon[33500]: 6.2 scrub ok 2024-03-26T05:01:49.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:01:49.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:49.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:49 smithi132 ceph-mon[33500]: 5.14 scrub starts 2024-03-26T05:01:49.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:49 smithi132 ceph-mon[33500]: 5.14 scrub ok 2024-03-26T05:01:50.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:50 smithi080 ceph-mon[40404]: 6.10 scrub starts 2024-03-26T05:01:50.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:50 smithi080 ceph-mon[40404]: 6.10 scrub ok 2024-03-26T05:01:50.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:50 smithi080 ceph-mon[29691]: 6.10 scrub starts 2024-03-26T05:01:50.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:50 smithi080 ceph-mon[29691]: 6.10 scrub ok 2024-03-26T05:01:50.674 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:01:50.676 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:01:50.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:50 smithi132 ceph-mon[33500]: 6.10 scrub starts 2024-03-26T05:01:50.905 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:50 smithi132 ceph-mon[33500]: 6.10 scrub ok 2024-03-26T05:01:51.443 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-26T05:01:49.713202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1303,"num_read_kb":1117,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":816,"ondisk_log_size":816,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":246676,"kb_used_data":9724,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749714540,"statfs":{"total":767960285184,"available":767707688960,"internally_reserved":0,"allocated":9957376,"data_stored":5681394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001795"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":59,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312740+0000","last_change":"2024-03-26T05:00:28.973250+0000","last_active":"2024-03-26T05:01:46.312740+0000","last_peered":"2024-03-26T05:01:46.312740+0000","last_clean":"2024-03-26T05:01:46.312740+0000","last_became_active":"2024-03-26T05:00:28.972969+0000","last_became_peered":"2024-03-26T05:00:28.972969+0000","last_unstale":"2024-03-26T05:01:46.312740+0000","last_undegraded":"2024-03-26T05:01:46.312740+0000","last_fullsized":"2024-03-26T05:01:46.312740+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407289+0000","last_change":"2024-03-26T05:01:39.812600+0000","last_active":"2024-03-26T05:01:46.407289+0000","last_peered":"2024-03-26T05:01:46.407289+0000","last_clean":"2024-03-26T05:01:46.407289+0000","last_became_active":"2024-03-26T05:00:26.953154+0000","last_became_peered":"2024-03-26T05:00:26.953154+0000","last_unstale":"2024-03-26T05:01:46.407289+0000","last_undegraded":"2024-03-26T05:01:46.407289+0000","last_fullsized":"2024-03-26T05:01:46.407289+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.812509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.812509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:45.654690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312858+0000","last_change":"2024-03-26T05:01:35.658900+0000","last_active":"2024-03-26T05:01:46.312858+0000","last_peered":"2024-03-26T05:01:46.312858+0000","last_clean":"2024-03-26T05:01:46.312858+0000","last_became_active":"2024-03-26T05:00:23.346607+0000","last_became_peered":"2024-03-26T05:00:23.346607+0000","last_unstale":"2024-03-26T05:01:46.312858+0000","last_undegraded":"2024-03-26T05:01:46.312858+0000","last_fullsized":"2024-03-26T05:01:46.312858+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:35.658796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.658796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:41.835654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'15","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312731+0000","last_change":"2024-03-26T05:00:24.943621+0000","last_active":"2024-03-26T05:01:46.312731+0000","last_peered":"2024-03-26T05:01:46.312731+0000","last_clean":"2024-03-26T05:01:46.312731+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:01:46.312731+0000","last_undegraded":"2024-03-26T05:01:46.312731+0000","last_fullsized":"2024-03-26T05:01:46.312731+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734617+0000","last_change":"2024-03-26T05:00:29.463053+0000","last_active":"2024-03-26T05:01:46.734617+0000","last_peered":"2024-03-26T05:01:46.734617+0000","last_clean":"2024-03-26T05:01:46.734617+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:01:46.734617+0000","last_undegraded":"2024-03-26T05:01:46.734617+0000","last_fullsized":"2024-03-26T05:01:46.734617+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:56.816597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":63,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312593+0000","last_change":"2024-03-26T05:00:26.955355+0000","last_active":"2024-03-26T05:01:46.312593+0000","last_peered":"2024-03-26T05:01:46.312593+0000","last_clean":"2024-03-26T05:01:46.312593+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:01:46.312593+0000","last_undegraded":"2024-03-26T05:01:46.312593+0000","last_fullsized":"2024-03-26T05:01:46.312593+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.268685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735275+0000","last_change":"2024-03-26T05:01:37.201540+0000","last_active":"2024-03-26T05:01:46.735275+0000","last_peered":"2024-03-26T05:01:46.735275+0000","last_clean":"2024-03-26T05:01:46.735275+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:01:46.735275+0000","last_undegraded":"2024-03-26T05:01:46.735275+0000","last_fullsized":"2024-03-26T05:01:46.735275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.201446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.201446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:43.519218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'39","reported_seq":139,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734709+0000","last_change":"2024-03-26T05:00:24.932741+0000","last_active":"2024-03-26T05:01:46.734709+0000","last_peered":"2024-03-26T05:01:46.734709+0000","last_clean":"2024-03-26T05:01:46.734709+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:01:46.734709+0000","last_undegraded":"2024-03-26T05:01:46.734709+0000","last_fullsized":"2024-03-26T05:01:46.734709+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:53.767288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":67,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.311993+0000","last_change":"2024-03-26T05:01:32.598168+0000","last_active":"2024-03-26T05:01:46.311993+0000","last_peered":"2024-03-26T05:01:46.311993+0000","last_clean":"2024-03-26T05:01:46.311993+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:01:46.311993+0000","last_undegraded":"2024-03-26T05:01:46.311993+0000","last_fullsized":"2024-03-26T05:01:46.311993+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:32.598087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:32.598087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.083903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":62,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028457+0000","last_change":"2024-03-26T05:00:26.956404+0000","last_active":"2024-03-26T05:01:47.028457+0000","last_peered":"2024-03-26T05:01:47.028457+0000","last_clean":"2024-03-26T05:01:47.028457+0000","last_became_active":"2024-03-26T05:00:26.956013+0000","last_became_peered":"2024-03-26T05:00:26.956013+0000","last_unstale":"2024-03-26T05:01:47.028457+0000","last_undegraded":"2024-03-26T05:01:47.028457+0000","last_fullsized":"2024-03-26T05:01:47.028457+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029144+0000","last_change":"2024-03-26T05:01:30.949017+0000","last_active":"2024-03-26T05:01:47.029144+0000","last_peered":"2024-03-26T05:01:47.029144+0000","last_clean":"2024-03-26T05:01:47.029144+0000","last_became_active":"2024-03-26T05:00:22.768098+0000","last_became_peered":"2024-03-26T05:00:22.768098+0000","last_unstale":"2024-03-26T05:01:47.029144+0000","last_undegraded":"2024-03-26T05:01:47.029144+0000","last_fullsized":"2024-03-26T05:01:47.029144+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.948866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.948866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.633607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'21","reported_seq":115,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.313107+0000","last_change":"2024-03-26T05:01:30.476359+0000","last_active":"2024-03-26T05:01:46.313107+0000","last_peered":"2024-03-26T05:01:46.313107+0000","last_clean":"2024-03-26T05:01:46.313107+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:01:46.313107+0000","last_undegraded":"2024-03-26T05:01:46.313107+0000","last_fullsized":"2024-03-26T05:01:46.313107+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:01:30.476249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.476249+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.885741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028842+0000","last_change":"2024-03-26T05:00:28.982522+0000","last_active":"2024-03-26T05:01:47.028842+0000","last_peered":"2024-03-26T05:01:47.028842+0000","last_clean":"2024-03-26T05:01:47.028842+0000","last_became_active":"2024-03-26T05:00:28.982276+0000","last_became_peered":"2024-03-26T05:00:28.982276+0000","last_unstale":"2024-03-26T05:01:47.028842+0000","last_undegraded":"2024-03-26T05:01:47.028842+0000","last_fullsized":"2024-03-26T05:01:47.028842+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":61,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.761933+0000","last_change":"2024-03-26T05:00:26.960760+0000","last_active":"2024-03-26T05:01:46.761933+0000","last_peered":"2024-03-26T05:01:46.761933+0000","last_clean":"2024-03-26T05:01:46.761933+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:01:46.761933+0000","last_undegraded":"2024-03-26T05:01:46.761933+0000","last_fullsized":"2024-03-26T05:01:46.761933+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029263+0000","last_change":"2024-03-26T05:01:39.021449+0000","last_active":"2024-03-26T05:01:47.029263+0000","last_peered":"2024-03-26T05:01:47.029263+0000","last_clean":"2024-03-26T05:01:47.029263+0000","last_became_active":"2024-03-26T05:00:22.766675+0000","last_became_peered":"2024-03-26T05:00:22.766675+0000","last_unstale":"2024-03-26T05:01:47.029263+0000","last_undegraded":"2024-03-26T05:01:47.029263+0000","last_fullsized":"2024-03-26T05:01:47.029263+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.021315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.021315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:39.428392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'11","reported_seq":89,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312889+0000","last_change":"2024-03-26T05:00:24.943205+0000","last_active":"2024-03-26T05:01:46.312889+0000","last_peered":"2024-03-26T05:01:46.312889+0000","last_clean":"2024-03-26T05:01:46.312889+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:01:46.312889+0000","last_undegraded":"2024-03-26T05:01:46.312889+0000","last_fullsized":"2024-03-26T05:01:46.312889+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:50.151492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":59,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.313830+0000","last_change":"2024-03-26T05:00:29.461374+0000","last_active":"2024-03-26T05:01:46.313830+0000","last_peered":"2024-03-26T05:01:46.313830+0000","last_clean":"2024-03-26T05:01:46.313830+0000","last_became_active":"2024-03-26T05:00:29.461032+0000","last_became_peered":"2024-03-26T05:00:29.461032+0000","last_unstale":"2024-03-26T05:01:46.313830+0000","last_undegraded":"2024-03-26T05:01:46.313830+0000","last_fullsized":"2024-03-26T05:01:46.313830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":62,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407829+0000","last_change":"2024-03-26T05:00:26.960665+0000","last_active":"2024-03-26T05:01:46.407829+0000","last_peered":"2024-03-26T05:01:46.407829+0000","last_clean":"2024-03-26T05:01:46.407829+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:01:46.407829+0000","last_undegraded":"2024-03-26T05:01:46.407829+0000","last_fullsized":"2024-03-26T05:01:46.407829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.742370+0000","last_change":"2024-03-26T05:01:47.742402+0000","last_active":"2024-03-26T05:01:47.742370+0000","last_peered":"2024-03-26T05:01:47.742370+0000","last_clean":"2024-03-26T05:01:47.742370+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:01:47.742370+0000","last_undegraded":"2024-03-26T05:01:47.742370+0000","last_fullsized":"2024-03-26T05:01:47.742370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:47.742292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:47.742292+0000","last_clean_scrub_stamp":"2024-03-26T05:01:47.742292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.854236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":99,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.409129+0000","last_change":"2024-03-26T05:01:24.617269+0000","last_active":"2024-03-26T05:01:46.409129+0000","last_peered":"2024-03-26T05:01:46.409129+0000","last_clean":"2024-03-26T05:01:46.409129+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:01:46.409129+0000","last_undegraded":"2024-03-26T05:01:46.409129+0000","last_fullsized":"2024-03-26T05:01:46.409129+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:24.617170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:24.617170+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:34.140661+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":66,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735279+0000","last_change":"2024-03-26T05:01:30.046528+0000","last_active":"2024-03-26T05:01:46.735279+0000","last_peered":"2024-03-26T05:01:46.735279+0000","last_clean":"2024-03-26T05:01:46.735279+0000","last_became_active":"2024-03-26T05:00:29.463238+0000","last_became_peered":"2024-03-26T05:00:29.463238+0000","last_unstale":"2024-03-26T05:01:46.735279+0000","last_undegraded":"2024-03-26T05:01:46.735279+0000","last_fullsized":"2024-03-26T05:01:46.735279+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.046425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.046425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:35.648365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":63,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.313384+0000","last_change":"2024-03-26T05:00:26.962232+0000","last_active":"2024-03-26T05:01:46.313384+0000","last_peered":"2024-03-26T05:01:46.313384+0000","last_clean":"2024-03-26T05:01:46.313384+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:01:46.313384+0000","last_undegraded":"2024-03-26T05:01:46.313384+0000","last_fullsized":"2024-03-26T05:01:46.313384+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.410443+0000","last_change":"2024-03-26T05:01:46.410472+0000","last_active":"2024-03-26T05:01:46.410443+0000","last_peered":"2024-03-26T05:01:46.410443+0000","last_clean":"2024-03-26T05:01:46.410443+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:01:46.410443+0000","last_undegraded":"2024-03-26T05:01:46.410443+0000","last_fullsized":"2024-03-26T05:01:46.410443+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:01:46.410356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:52.981688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'20","reported_seq":103,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735014+0000","last_change":"2024-03-26T05:00:24.937057+0000","last_active":"2024-03-26T05:01:46.735014+0000","last_peered":"2024-03-26T05:01:46.735014+0000","last_clean":"2024-03-26T05:01:46.735014+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:01:46.735014+0000","last_undegraded":"2024-03-26T05:01:46.735014+0000","last_fullsized":"2024-03-26T05:01:46.735014+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762118+0000","last_change":"2024-03-26T05:00:28.981879+0000","last_active":"2024-03-26T05:01:46.762118+0000","last_peered":"2024-03-26T05:01:46.762118+0000","last_clean":"2024-03-26T05:01:46.762118+0000","last_became_active":"2024-03-26T05:00:28.981505+0000","last_became_peered":"2024-03-26T05:00:28.981505+0000","last_unstale":"2024-03-26T05:01:46.762118+0000","last_undegraded":"2024-03-26T05:01:46.762118+0000","last_fullsized":"2024-03-26T05:01:46.762118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":90,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.227311+0000","last_change":"2024-03-26T05:00:26.954779+0000","last_active":"2024-03-26T05:01:47.227311+0000","last_peered":"2024-03-26T05:01:47.227311+0000","last_clean":"2024-03-26T05:01:47.227311+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:01:47.227311+0000","last_undegraded":"2024-03-26T05:01:47.227311+0000","last_fullsized":"2024-03-26T05:01:47.227311+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":71,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407341+0000","last_change":"2024-03-26T05:00:22.773390+0000","last_active":"2024-03-26T05:01:46.407341+0000","last_peered":"2024-03-26T05:01:46.407341+0000","last_clean":"2024-03-26T05:01:46.407341+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:01:46.407341+0000","last_undegraded":"2024-03-26T05:01:46.407341+0000","last_fullsized":"2024-03-26T05:01:46.407341+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:50.999861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'16","reported_seq":102,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312688+0000","last_change":"2024-03-26T05:01:41.650400+0000","last_active":"2024-03-26T05:01:46.312688+0000","last_peered":"2024-03-26T05:01:46.312688+0000","last_clean":"2024-03-26T05:01:46.312688+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:01:46.312688+0000","last_undegraded":"2024-03-26T05:01:46.312688+0000","last_fullsized":"2024-03-26T05:01:46.312688+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:01:41.650310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.650310+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:08.831570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407953+0000","last_change":"2024-03-26T05:00:28.981390+0000","last_active":"2024-03-26T05:01:46.407953+0000","last_peered":"2024-03-26T05:01:46.407953+0000","last_clean":"2024-03-26T05:01:46.407953+0000","last_became_active":"2024-03-26T05:00:28.981049+0000","last_became_peered":"2024-03-26T05:00:28.981049+0000","last_unstale":"2024-03-26T05:01:46.407953+0000","last_undegraded":"2024-03-26T05:01:46.407953+0000","last_fullsized":"2024-03-26T05:01:46.407953+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":63,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312795+0000","last_change":"2024-03-26T05:00:26.963282+0000","last_active":"2024-03-26T05:01:46.312795+0000","last_peered":"2024-03-26T05:01:46.312795+0000","last_clean":"2024-03-26T05:01:46.312795+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:01:46.312795+0000","last_undegraded":"2024-03-26T05:01:46.312795+0000","last_fullsized":"2024-03-26T05:01:46.312795+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734309+0000","last_change":"2024-03-26T05:01:43.166558+0000","last_active":"2024-03-26T05:01:46.734309+0000","last_peered":"2024-03-26T05:01:46.734309+0000","last_clean":"2024-03-26T05:01:46.734309+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:01:46.734309+0000","last_undegraded":"2024-03-26T05:01:46.734309+0000","last_fullsized":"2024-03-26T05:01:46.734309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:43.166422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:43.166422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:50.132493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'17","reported_seq":116,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.228086+0000","last_change":"2024-03-26T05:00:49.652267+0000","last_active":"2024-03-26T05:01:47.228086+0000","last_peered":"2024-03-26T05:01:47.228086+0000","last_clean":"2024-03-26T05:01:47.228086+0000","last_became_active":"2024-03-26T05:00:49.643308+0000","last_became_peered":"2024-03-26T05:00:49.643308+0000","last_unstale":"2024-03-26T05:01:47.228086+0000","last_undegraded":"2024-03-26T05:01:47.228086+0000","last_fullsized":"2024-03-26T05:01:47.228086+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":57,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.761903+0000","last_change":"2024-03-26T05:00:28.978762+0000","last_active":"2024-03-26T05:01:46.761903+0000","last_peered":"2024-03-26T05:01:46.761903+0000","last_clean":"2024-03-26T05:01:46.761903+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:01:46.761903+0000","last_undegraded":"2024-03-26T05:01:46.761903+0000","last_fullsized":"2024-03-26T05:01:46.761903+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:49.872411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":69,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.227221+0000","last_change":"2024-03-26T05:01:44.235422+0000","last_active":"2024-03-26T05:01:47.227221+0000","last_peered":"2024-03-26T05:01:47.227221+0000","last_clean":"2024-03-26T05:01:47.227221+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:01:47.227221+0000","last_undegraded":"2024-03-26T05:01:47.227221+0000","last_fullsized":"2024-03-26T05:01:47.227221+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:44.235328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.235328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.223031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":71,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312814+0000","last_change":"2024-03-26T05:00:22.775209+0000","last_active":"2024-03-26T05:01:46.312814+0000","last_peered":"2024-03-26T05:01:46.312814+0000","last_clean":"2024-03-26T05:01:46.312814+0000","last_became_active":"2024-03-26T05:00:22.774783+0000","last_became_peered":"2024-03-26T05:00:22.774783+0000","last_unstale":"2024-03-26T05:01:46.312814+0000","last_undegraded":"2024-03-26T05:01:46.312814+0000","last_fullsized":"2024-03-26T05:01:46.312814+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":72,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734370+0000","last_change":"2024-03-26T05:00:24.942166+0000","last_active":"2024-03-26T05:01:46.734370+0000","last_peered":"2024-03-26T05:01:46.734370+0000","last_clean":"2024-03-26T05:01:46.734370+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:01:46.734370+0000","last_undegraded":"2024-03-26T05:01:46.734370+0000","last_fullsized":"2024-03-26T05:01:46.734370+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734443+0000","last_change":"2024-03-26T05:00:29.461798+0000","last_active":"2024-03-26T05:01:46.734443+0000","last_peered":"2024-03-26T05:01:46.734443+0000","last_clean":"2024-03-26T05:01:46.734443+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:01:46.734443+0000","last_undegraded":"2024-03-26T05:01:46.734443+0000","last_fullsized":"2024-03-26T05:01:46.734443+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":61,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762160+0000","last_change":"2024-03-26T05:00:26.959842+0000","last_active":"2024-03-26T05:01:46.762160+0000","last_peered":"2024-03-26T05:01:46.762160+0000","last_clean":"2024-03-26T05:01:46.762160+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:01:46.762160+0000","last_undegraded":"2024-03-26T05:01:46.762160+0000","last_fullsized":"2024-03-26T05:01:46.762160+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029324+0000","last_change":"2024-03-26T05:01:29.997645+0000","last_active":"2024-03-26T05:01:47.029324+0000","last_peered":"2024-03-26T05:01:47.029324+0000","last_clean":"2024-03-26T05:01:47.029324+0000","last_became_active":"2024-03-26T05:00:22.771199+0000","last_became_peered":"2024-03-26T05:00:22.771199+0000","last_unstale":"2024-03-26T05:01:47.029324+0000","last_undegraded":"2024-03-26T05:01:47.029324+0000","last_fullsized":"2024-03-26T05:01:47.029324+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:35.309625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'17","reported_seq":97,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734568+0000","last_change":"2024-03-26T05:00:25.648453+0000","last_active":"2024-03-26T05:01:46.734568+0000","last_peered":"2024-03-26T05:01:46.734568+0000","last_clean":"2024-03-26T05:01:46.734568+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:01:46.734568+0000","last_undegraded":"2024-03-26T05:01:46.734568+0000","last_fullsized":"2024-03-26T05:01:46.734568+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":59,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.311777+0000","last_change":"2024-03-26T05:00:28.985476+0000","last_active":"2024-03-26T05:01:46.311777+0000","last_peered":"2024-03-26T05:01:46.311777+0000","last_clean":"2024-03-26T05:01:46.311777+0000","last_became_active":"2024-03-26T05:00:28.985158+0000","last_became_peered":"2024-03-26T05:00:28.985158+0000","last_unstale":"2024-03-26T05:01:46.311777+0000","last_undegraded":"2024-03-26T05:01:46.311777+0000","last_fullsized":"2024-03-26T05:01:46.311777+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":62,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734714+0000","last_change":"2024-03-26T05:00:26.960672+0000","last_active":"2024-03-26T05:01:46.734714+0000","last_peered":"2024-03-26T05:01:46.734714+0000","last_clean":"2024-03-26T05:01:46.734714+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:01:46.734714+0000","last_undegraded":"2024-03-26T05:01:46.734714+0000","last_fullsized":"2024-03-26T05:01:46.734714+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:55.490995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":69,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762279+0000","last_change":"2024-03-26T05:00:22.773808+0000","last_active":"2024-03-26T05:01:46.762279+0000","last_peered":"2024-03-26T05:01:46.762279+0000","last_clean":"2024-03-26T05:01:46.762279+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:01:46.762279+0000","last_undegraded":"2024-03-26T05:01:46.762279+0000","last_fullsized":"2024-03-26T05:01:46.762279+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:49.214803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":82,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407552+0000","last_change":"2024-03-26T05:00:24.939870+0000","last_active":"2024-03-26T05:01:46.407552+0000","last_peered":"2024-03-26T05:01:46.407552+0000","last_clean":"2024-03-26T05:01:46.407552+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:01:46.407552+0000","last_undegraded":"2024-03-26T05:01:46.407552+0000","last_fullsized":"2024-03-26T05:01:46.407552+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:46.453483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734292+0000","last_change":"2024-03-26T05:00:28.984567+0000","last_active":"2024-03-26T05:01:46.734292+0000","last_peered":"2024-03-26T05:01:46.734292+0000","last_clean":"2024-03-26T05:01:46.734292+0000","last_became_active":"2024-03-26T05:00:28.983417+0000","last_became_peered":"2024-03-26T05:00:28.983417+0000","last_unstale":"2024-03-26T05:01:46.734292+0000","last_undegraded":"2024-03-26T05:01:46.734292+0000","last_fullsized":"2024-03-26T05:01:46.734292+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:51.606343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407544+0000","last_change":"2024-03-26T05:01:30.777153+0000","last_active":"2024-03-26T05:01:46.407544+0000","last_peered":"2024-03-26T05:01:46.407544+0000","last_clean":"2024-03-26T05:01:46.407544+0000","last_became_active":"2024-03-26T05:00:26.954653+0000","last_became_peered":"2024-03-26T05:00:26.954653+0000","last_unstale":"2024-03-26T05:01:46.407544+0000","last_undegraded":"2024-03-26T05:01:46.407544+0000","last_fullsized":"2024-03-26T05:01:46.407544+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.777070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:45.021844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":71,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.313612+0000","last_change":"2024-03-26T05:00:22.767847+0000","last_active":"2024-03-26T05:01:46.313612+0000","last_peered":"2024-03-26T05:01:46.313612+0000","last_clean":"2024-03-26T05:01:46.313612+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:01:46.313612+0000","last_undegraded":"2024-03-26T05:01:46.313612+0000","last_fullsized":"2024-03-26T05:01:46.313612+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:47.148519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"95'16","reported_seq":99,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.313046+0000","last_change":"2024-03-26T05:01:40.462389+0000","last_active":"2024-03-26T05:01:46.313046+0000","last_peered":"2024-03-26T05:01:46.313046+0000","last_clean":"2024-03-26T05:01:46.313046+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:01:46.313046+0000","last_undegraded":"2024-03-26T05:01:46.313046+0000","last_fullsized":"2024-03-26T05:01:46.313046+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.462271+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.765446+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":59,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312126+0000","last_change":"2024-03-26T05:00:28.984841+0000","last_active":"2024-03-26T05:01:46.312126+0000","last_peered":"2024-03-26T05:01:46.312126+0000","last_clean":"2024-03-26T05:01:46.312126+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:01:46.312126+0000","last_undegraded":"2024-03-26T05:01:46.312126+0000","last_fullsized":"2024-03-26T05:01:46.312126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028686+0000","last_change":"2024-03-26T05:00:26.956153+0000","last_active":"2024-03-26T05:01:47.028686+0000","last_peered":"2024-03-26T05:01:47.028686+0000","last_clean":"2024-03-26T05:01:47.028686+0000","last_became_active":"2024-03-26T05:00:26.955836+0000","last_became_peered":"2024-03-26T05:00:26.955836+0000","last_unstale":"2024-03-26T05:01:47.028686+0000","last_undegraded":"2024-03-26T05:01:47.028686+0000","last_fullsized":"2024-03-26T05:01:47.028686+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:54.166825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":77,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.761998+0000","last_change":"2024-03-26T05:01:26.785152+0000","last_active":"2024-03-26T05:01:46.761998+0000","last_peered":"2024-03-26T05:01:46.761998+0000","last_clean":"2024-03-26T05:01:46.761998+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:01:46.761998+0000","last_undegraded":"2024-03-26T05:01:46.761998+0000","last_fullsized":"2024-03-26T05:01:46.761998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.785056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.785056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.302223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'22","reported_seq":113,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734580+0000","last_change":"2024-03-26T05:01:40.118101+0000","last_active":"2024-03-26T05:01:46.734580+0000","last_peered":"2024-03-26T05:01:46.734580+0000","last_clean":"2024-03-26T05:01:46.734580+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:01:46.734580+0000","last_undegraded":"2024-03-26T05:01:46.734580+0000","last_fullsized":"2024-03-26T05:01:46.734580+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:01:40.117953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.117953+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.646893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":59,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.313956+0000","last_change":"2024-03-26T05:00:28.979801+0000","last_active":"2024-03-26T05:01:46.313956+0000","last_peered":"2024-03-26T05:01:46.313956+0000","last_clean":"2024-03-26T05:01:46.313956+0000","last_became_active":"2024-03-26T05:00:28.979477+0000","last_became_peered":"2024-03-26T05:00:28.979477+0000","last_unstale":"2024-03-26T05:01:46.313956+0000","last_undegraded":"2024-03-26T05:01:46.313956+0000","last_fullsized":"2024-03-26T05:01:46.313956+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:56.897877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":99,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:49.006465+0000","last_change":"2024-03-26T05:01:49.006492+0000","last_active":"2024-03-26T05:01:49.006465+0000","last_peered":"2024-03-26T05:01:49.006465+0000","last_clean":"2024-03-26T05:01:49.006465+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:01:49.006465+0000","last_undegraded":"2024-03-26T05:01:49.006465+0000","last_fullsized":"2024-03-26T05:01:49.006465+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:49.006396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:49.006396+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.424499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":71,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312198+0000","last_change":"2024-03-26T05:00:22.777258+0000","last_active":"2024-03-26T05:01:46.312198+0000","last_peered":"2024-03-26T05:01:46.312198+0000","last_clean":"2024-03-26T05:01:46.312198+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:01:46.312198+0000","last_undegraded":"2024-03-26T05:01:46.312198+0000","last_fullsized":"2024-03-26T05:01:46.312198+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"98'20","reported_seq":107,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:48.070410+0000","last_change":"2024-03-26T05:01:48.070439+0000","last_active":"2024-03-26T05:01:48.070410+0000","last_peered":"2024-03-26T05:01:48.070410+0000","last_clean":"2024-03-26T05:01:48.070410+0000","last_became_active":"2024-03-26T05:00:25.648284+0000","last_became_peered":"2024-03-26T05:00:25.648284+0000","last_unstale":"2024-03-26T05:01:48.070410+0000","last_undegraded":"2024-03-26T05:01:48.070410+0000","last_fullsized":"2024-03-26T05:01:48.070410+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'20","last_scrub_stamp":"2024-03-26T05:01:48.070338+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:01:48.070338+0000","last_clean_scrub_stamp":"2024-03-26T05:01:48.070338+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.153022+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":66,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.409306+0000","last_change":"2024-03-26T05:01:31.568804+0000","last_active":"2024-03-26T05:01:46.409306+0000","last_peered":"2024-03-26T05:01:46.409306+0000","last_clean":"2024-03-26T05:01:46.409306+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:01:46.409306+0000","last_undegraded":"2024-03-26T05:01:46.409306+0000","last_fullsized":"2024-03-26T05:01:46.409306+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:31.568722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:31.568722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.465689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":62,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735223+0000","last_change":"2024-03-26T05:00:26.956233+0000","last_active":"2024-03-26T05:01:46.735223+0000","last_peered":"2024-03-26T05:01:46.735223+0000","last_clean":"2024-03-26T05:01:46.735223+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:01:46.735223+0000","last_undegraded":"2024-03-26T05:01:46.735223+0000","last_fullsized":"2024-03-26T05:01:46.735223+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:54.400300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":69,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.761902+0000","last_change":"2024-03-26T05:00:22.770737+0000","last_active":"2024-03-26T05:01:46.761902+0000","last_peered":"2024-03-26T05:01:46.761902+0000","last_clean":"2024-03-26T05:01:46.761902+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:01:46.761902+0000","last_undegraded":"2024-03-26T05:01:46.761902+0000","last_fullsized":"2024-03-26T05:01:46.761902+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'21","reported_seq":115,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312714+0000","last_change":"2024-03-26T05:01:38.722775+0000","last_active":"2024-03-26T05:01:46.312714+0000","last_peered":"2024-03-26T05:01:46.312714+0000","last_clean":"2024-03-26T05:01:46.312714+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:01:46.312714+0000","last_undegraded":"2024-03-26T05:01:46.312714+0000","last_fullsized":"2024-03-26T05:01:46.312714+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:01:38.722652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:38.722652+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.850545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":57,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762162+0000","last_change":"2024-03-26T05:00:29.462603+0000","last_active":"2024-03-26T05:01:46.762162+0000","last_peered":"2024-03-26T05:01:46.762162+0000","last_clean":"2024-03-26T05:01:46.762162+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:01:46.762162+0000","last_undegraded":"2024-03-26T05:01:46.762162+0000","last_fullsized":"2024-03-26T05:01:46.762162+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.502035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":62,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028536+0000","last_change":"2024-03-26T05:00:26.953694+0000","last_active":"2024-03-26T05:01:47.028536+0000","last_peered":"2024-03-26T05:01:47.028536+0000","last_clean":"2024-03-26T05:01:47.028536+0000","last_became_active":"2024-03-26T05:00:26.953416+0000","last_became_peered":"2024-03-26T05:00:26.953416+0000","last_unstale":"2024-03-26T05:01:47.028536+0000","last_undegraded":"2024-03-26T05:01:47.028536+0000","last_fullsized":"2024-03-26T05:01:47.028536+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.182281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":83,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.227443+0000","last_change":"2024-03-26T05:01:37.375697+0000","last_active":"2024-03-26T05:01:47.227443+0000","last_peered":"2024-03-26T05:01:47.227443+0000","last_clean":"2024-03-26T05:01:47.227443+0000","last_became_active":"2024-03-26T05:00:22.771080+0000","last_became_peered":"2024-03-26T05:00:22.771080+0000","last_unstale":"2024-03-26T05:01:47.227443+0000","last_undegraded":"2024-03-26T05:01:47.227443+0000","last_fullsized":"2024-03-26T05:01:47.227443+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:01:37.375533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.375533+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:51.717267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'12","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735330+0000","last_change":"2024-03-26T05:01:27.964049+0000","last_active":"2024-03-26T05:01:46.735330+0000","last_peered":"2024-03-26T05:01:46.735330+0000","last_clean":"2024-03-26T05:01:46.735330+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:01:46.735330+0000","last_undegraded":"2024-03-26T05:01:46.735330+0000","last_fullsized":"2024-03-26T05:01:46.735330+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:01:27.963954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:27.963954+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:50.330132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":68,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029083+0000","last_change":"2024-03-26T05:01:36.945807+0000","last_active":"2024-03-26T05:01:47.029083+0000","last_peered":"2024-03-26T05:01:47.029083+0000","last_clean":"2024-03-26T05:01:47.029083+0000","last_became_active":"2024-03-26T05:00:29.461059+0000","last_became_peered":"2024-03-26T05:00:29.461059+0000","last_unstale":"2024-03-26T05:01:47.029083+0000","last_undegraded":"2024-03-26T05:01:47.029083+0000","last_fullsized":"2024-03-26T05:01:47.029083+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:01:36.945708+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.001390+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'3","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762539+0000","last_change":"2024-03-26T05:01:22.799027+0000","last_active":"2024-03-26T05:01:46.762539+0000","last_peered":"2024-03-26T05:01:46.762539+0000","last_clean":"2024-03-26T05:01:46.762539+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:01:46.762539+0000","last_undegraded":"2024-03-26T05:01:46.762539+0000","last_fullsized":"2024-03-26T05:01:46.762539+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-03-26T05:01:22.798897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:22.798897+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.103651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'27","reported_seq":123,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028903+0000","last_change":"2024-03-26T05:01:29.048598+0000","last_active":"2024-03-26T05:01:47.028903+0000","last_peered":"2024-03-26T05:01:47.028903+0000","last_clean":"2024-03-26T05:01:47.028903+0000","last_became_active":"2024-03-26T05:00:25.651328+0000","last_became_peered":"2024-03-26T05:00:25.651328+0000","last_unstale":"2024-03-26T05:01:47.028903+0000","last_undegraded":"2024-03-26T05:01:47.028903+0000","last_fullsized":"2024-03-26T05:01:47.028903+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'27","last_scrub_stamp":"2024-03-26T05:01:29.048456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.048456+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:56.053198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762104+0000","last_change":"2024-03-26T05:01:37.743111+0000","last_active":"2024-03-26T05:01:46.762104+0000","last_peered":"2024-03-26T05:01:46.762104+0000","last_clean":"2024-03-26T05:01:46.762104+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:01:46.762104+0000","last_undegraded":"2024-03-26T05:01:46.762104+0000","last_fullsized":"2024-03-26T05:01:46.762104+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:37.742995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.742995+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:07.111791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":66,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734506+0000","last_change":"2024-03-26T05:01:29.011852+0000","last_active":"2024-03-26T05:01:46.734506+0000","last_peered":"2024-03-26T05:01:46.734506+0000","last_clean":"2024-03-26T05:01:46.734506+0000","last_became_active":"2024-03-26T05:00:28.985917+0000","last_became_peered":"2024-03-26T05:00:28.985917+0000","last_unstale":"2024-03-26T05:01:46.734506+0000","last_undegraded":"2024-03-26T05:01:46.734506+0000","last_fullsized":"2024-03-26T05:01:46.734506+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.011757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.011757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.742465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":77,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.855772+0000","last_change":"2024-03-26T05:01:46.855799+0000","last_active":"2024-03-26T05:01:46.855772+0000","last_peered":"2024-03-26T05:01:46.855772+0000","last_clean":"2024-03-26T05:01:46.855772+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:01:46.855772+0000","last_undegraded":"2024-03-26T05:01:46.855772+0000","last_fullsized":"2024-03-26T05:01:46.855772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:46.855699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:46.855699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:15.236244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'24","reported_seq":114,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.313031+0000","last_change":"2024-03-26T05:00:24.944015+0000","last_active":"2024-03-26T05:01:46.313031+0000","last_peered":"2024-03-26T05:01:46.313031+0000","last_clean":"2024-03-26T05:01:46.313031+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:01:46.313031+0000","last_undegraded":"2024-03-26T05:01:46.313031+0000","last_fullsized":"2024-03-26T05:01:46.313031+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:46.669169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.408911+0000","last_change":"2024-03-26T05:01:26.618364+0000","last_active":"2024-03-26T05:01:46.408911+0000","last_peered":"2024-03-26T05:01:46.408911+0000","last_clean":"2024-03-26T05:01:46.408911+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:01:46.408911+0000","last_undegraded":"2024-03-26T05:01:46.408911+0000","last_fullsized":"2024-03-26T05:01:46.408911+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.618218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.618218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.093176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":57,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.761762+0000","last_change":"2024-03-26T05:00:28.981813+0000","last_active":"2024-03-26T05:01:46.761762+0000","last_peered":"2024-03-26T05:01:46.761762+0000","last_clean":"2024-03-26T05:01:46.761762+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:01:46.761762+0000","last_undegraded":"2024-03-26T05:01:46.761762+0000","last_fullsized":"2024-03-26T05:01:46.761762+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.155885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.409105+0000","last_change":"2024-03-26T05:01:33.638557+0000","last_active":"2024-03-26T05:01:46.409105+0000","last_peered":"2024-03-26T05:01:46.409105+0000","last_clean":"2024-03-26T05:01:46.409105+0000","last_became_active":"2024-03-26T05:00:26.961730+0000","last_became_peered":"2024-03-26T05:00:26.961730+0000","last_unstale":"2024-03-26T05:01:46.409105+0000","last_undegraded":"2024-03-26T05:01:46.409105+0000","last_fullsized":"2024-03-26T05:01:46.409105+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:33.638469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:33.638469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:00.135833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":69,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762440+0000","last_change":"2024-03-26T05:00:22.772945+0000","last_active":"2024-03-26T05:01:46.762440+0000","last_peered":"2024-03-26T05:01:46.762440+0000","last_clean":"2024-03-26T05:01:46.762440+0000","last_became_active":"2024-03-26T05:00:22.772590+0000","last_became_peered":"2024-03-26T05:00:22.772590+0000","last_unstale":"2024-03-26T05:01:46.762440+0000","last_undegraded":"2024-03-26T05:01:46.762440+0000","last_fullsized":"2024-03-26T05:01:46.762440+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'25","reported_seq":108,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.227701+0000","last_change":"2024-03-26T05:00:25.649083+0000","last_active":"2024-03-26T05:01:47.227701+0000","last_peered":"2024-03-26T05:01:47.227701+0000","last_clean":"2024-03-26T05:01:47.227701+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:01:47.227701+0000","last_undegraded":"2024-03-26T05:01:47.227701+0000","last_fullsized":"2024-03-26T05:01:47.227701+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":59,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.311856+0000","last_change":"2024-03-26T05:00:28.981411+0000","last_active":"2024-03-26T05:01:46.311856+0000","last_peered":"2024-03-26T05:01:46.311856+0000","last_clean":"2024-03-26T05:01:46.311856+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:01:46.311856+0000","last_undegraded":"2024-03-26T05:01:46.311856+0000","last_fullsized":"2024-03-26T05:01:46.311856+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:55.410289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":62,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.409083+0000","last_change":"2024-03-26T05:00:26.956293+0000","last_active":"2024-03-26T05:01:46.409083+0000","last_peered":"2024-03-26T05:01:46.409083+0000","last_clean":"2024-03-26T05:01:46.409083+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:01:46.409083+0000","last_undegraded":"2024-03-26T05:01:46.409083+0000","last_fullsized":"2024-03-26T05:01:46.409083+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:54.998598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029139+0000","last_change":"2024-03-26T05:00:22.774369+0000","last_active":"2024-03-26T05:01:47.029139+0000","last_peered":"2024-03-26T05:01:47.029139+0000","last_clean":"2024-03-26T05:01:47.029139+0000","last_became_active":"2024-03-26T05:00:22.773770+0000","last_became_peered":"2024-03-26T05:00:22.773770+0000","last_unstale":"2024-03-26T05:01:47.029139+0000","last_undegraded":"2024-03-26T05:01:47.029139+0000","last_fullsized":"2024-03-26T05:01:47.029139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'15","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312654+0000","last_change":"2024-03-26T05:00:24.936134+0000","last_active":"2024-03-26T05:01:46.312654+0000","last_peered":"2024-03-26T05:01:46.312654+0000","last_clean":"2024-03-26T05:01:46.312654+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:01:46.312654+0000","last_undegraded":"2024-03-26T05:01:46.312654+0000","last_fullsized":"2024-03-26T05:01:46.312654+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734991+0000","last_change":"2024-03-26T05:00:28.984072+0000","last_active":"2024-03-26T05:01:46.734991+0000","last_peered":"2024-03-26T05:01:46.734991+0000","last_clean":"2024-03-26T05:01:46.734991+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:01:46.734991+0000","last_undegraded":"2024-03-26T05:01:46.734991+0000","last_fullsized":"2024-03-26T05:01:46.734991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":89,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028958+0000","last_change":"2024-03-26T05:01:35.934149+0000","last_active":"2024-03-26T05:01:47.028958+0000","last_peered":"2024-03-26T05:01:47.028958+0000","last_clean":"2024-03-26T05:01:47.028958+0000","last_became_active":"2024-03-26T05:00:21.760521+0000","last_became_peered":"2024-03-26T05:00:21.760521+0000","last_unstale":"2024-03-26T05:01:47.028958+0000","last_undegraded":"2024-03-26T05:01:47.028958+0000","last_fullsized":"2024-03-26T05:01:47.028958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:01:35.934044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.934044+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:02.210221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":71,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312887+0000","last_change":"2024-03-26T05:00:22.771224+0000","last_active":"2024-03-26T05:01:46.312887+0000","last_peered":"2024-03-26T05:01:46.312887+0000","last_clean":"2024-03-26T05:01:46.312887+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:01:46.312887+0000","last_undegraded":"2024-03-26T05:01:46.312887+0000","last_fullsized":"2024-03-26T05:01:46.312887+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:51.123388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'34","reported_seq":152,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407750+0000","last_change":"2024-03-26T05:01:40.826966+0000","last_active":"2024-03-26T05:01:46.407750+0000","last_peered":"2024-03-26T05:01:46.407750+0000","last_clean":"2024-03-26T05:01:46.407750+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:01:46.407750+0000","last_undegraded":"2024-03-26T05:01:46.407750+0000","last_fullsized":"2024-03-26T05:01:46.407750+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'34","last_scrub_stamp":"2024-03-26T05:01:40.826856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.826856+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.349406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":63,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312390+0000","last_change":"2024-03-26T05:00:26.964781+0000","last_active":"2024-03-26T05:01:46.312390+0000","last_peered":"2024-03-26T05:01:46.312390+0000","last_clean":"2024-03-26T05:01:46.312390+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:01:46.312390+0000","last_undegraded":"2024-03-26T05:01:46.312390+0000","last_fullsized":"2024-03-26T05:01:46.312390+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":65,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762516+0000","last_change":"2024-03-26T05:01:41.789755+0000","last_active":"2024-03-26T05:01:46.762516+0000","last_peered":"2024-03-26T05:01:46.762516+0000","last_clean":"2024-03-26T05:01:46.762516+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:01:46.762516+0000","last_undegraded":"2024-03-26T05:01:46.762516+0000","last_fullsized":"2024-03-26T05:01:46.762516+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:41.789650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.789650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.302843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407341+0000","last_change":"2024-03-26T05:00:23.346922+0000","last_active":"2024-03-26T05:01:46.407341+0000","last_peered":"2024-03-26T05:01:46.407341+0000","last_clean":"2024-03-26T05:01:46.407341+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:01:46.407341+0000","last_undegraded":"2024-03-26T05:01:46.407341+0000","last_fullsized":"2024-03-26T05:01:46.407341+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:48.374217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.408962+0000","last_change":"2024-03-26T05:01:37.678850+0000","last_active":"2024-03-26T05:01:46.408962+0000","last_peered":"2024-03-26T05:01:46.408962+0000","last_clean":"2024-03-26T05:01:46.408962+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:01:46.408962+0000","last_undegraded":"2024-03-26T05:01:46.408962+0000","last_fullsized":"2024-03-26T05:01:46.408962+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.678736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.678736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.820757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'19","reported_seq":105,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407898+0000","last_change":"2024-03-26T05:00:24.937826+0000","last_active":"2024-03-26T05:01:46.407898+0000","last_peered":"2024-03-26T05:01:46.407898+0000","last_clean":"2024-03-26T05:01:46.407898+0000","last_became_active":"2024-03-26T05:00:24.937145+0000","last_became_peered":"2024-03-26T05:00:24.937145+0000","last_unstale":"2024-03-26T05:01:46.407898+0000","last_undegraded":"2024-03-26T05:01:46.407898+0000","last_fullsized":"2024-03-26T05:01:46.407898+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:49.327553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.409299+0000","last_change":"2024-03-26T05:01:29.627760+0000","last_active":"2024-03-26T05:01:46.409299+0000","last_peered":"2024-03-26T05:01:46.409299+0000","last_clean":"2024-03-26T05:01:46.409299+0000","last_became_active":"2024-03-26T05:00:26.957936+0000","last_became_peered":"2024-03-26T05:00:26.957936+0000","last_unstale":"2024-03-26T05:01:46.409299+0000","last_undegraded":"2024-03-26T05:01:46.409299+0000","last_fullsized":"2024-03-26T05:01:46.409299+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.627680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.627680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.292786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029318+0000","last_change":"2024-03-26T05:00:28.979759+0000","last_active":"2024-03-26T05:01:47.029318+0000","last_peered":"2024-03-26T05:01:47.029318+0000","last_clean":"2024-03-26T05:01:47.029318+0000","last_became_active":"2024-03-26T05:00:28.979373+0000","last_became_peered":"2024-03-26T05:00:28.979373+0000","last_unstale":"2024-03-26T05:01:47.029318+0000","last_undegraded":"2024-03-26T05:01:47.029318+0000","last_fullsized":"2024-03-26T05:01:47.029318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":324,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.761880+0000","last_change":"2024-03-26T05:01:04.782428+0000","last_active":"2024-03-26T05:01:46.761880+0000","last_peered":"2024-03-26T05:01:46.761880+0000","last_clean":"2024-03-26T05:01:46.761880+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:01:46.761880+0000","last_undegraded":"2024-03-26T05:01:46.761880+0000","last_fullsized":"2024-03-26T05:01:46.761880+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-03-26T05:01:04.782336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:01:04.782336+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:15.009834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734559+0000","last_change":"2024-03-26T05:01:25.941429+0000","last_active":"2024-03-26T05:01:46.734559+0000","last_peered":"2024-03-26T05:01:46.734559+0000","last_clean":"2024-03-26T05:01:46.734559+0000","last_became_active":"2024-03-26T05:00:22.774298+0000","last_became_peered":"2024-03-26T05:00:22.774298+0000","last_unstale":"2024-03-26T05:01:46.734559+0000","last_undegraded":"2024-03-26T05:01:46.734559+0000","last_fullsized":"2024-03-26T05:01:46.734559+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:25.941344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:25.941344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:34.075498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":105,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.227977+0000","last_change":"2024-03-26T05:00:25.648924+0000","last_active":"2024-03-26T05:01:47.227977+0000","last_peered":"2024-03-26T05:01:47.227977+0000","last_clean":"2024-03-26T05:01:47.227977+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:01:47.227977+0000","last_undegraded":"2024-03-26T05:01:47.227977+0000","last_fullsized":"2024-03-26T05:01:47.227977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.782291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":61,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762259+0000","last_change":"2024-03-26T05:00:26.962047+0000","last_active":"2024-03-26T05:01:46.762259+0000","last_peered":"2024-03-26T05:01:46.762259+0000","last_clean":"2024-03-26T05:01:46.762259+0000","last_became_active":"2024-03-26T05:00:26.961724+0000","last_became_peered":"2024-03-26T05:00:26.961724+0000","last_unstale":"2024-03-26T05:01:46.762259+0000","last_undegraded":"2024-03-26T05:01:46.762259+0000","last_fullsized":"2024-03-26T05:01:46.762259+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":66,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407409+0000","last_change":"2024-03-26T05:01:38.822538+0000","last_active":"2024-03-26T05:01:46.407409+0000","last_peered":"2024-03-26T05:01:46.407409+0000","last_clean":"2024-03-26T05:01:46.407409+0000","last_became_active":"2024-03-26T05:00:28.983202+0000","last_became_peered":"2024-03-26T05:00:28.983202+0000","last_unstale":"2024-03-26T05:01:46.407409+0000","last_undegraded":"2024-03-26T05:01:46.407409+0000","last_fullsized":"2024-03-26T05:01:46.407409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:38.822409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:38.822409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:41.940538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312084+0000","last_change":"2024-03-26T05:01:25.629227+0000","last_active":"2024-03-26T05:01:46.312084+0000","last_peered":"2024-03-26T05:01:46.312084+0000","last_clean":"2024-03-26T05:01:46.312084+0000","last_became_active":"2024-03-26T05:00:22.773081+0000","last_became_peered":"2024-03-26T05:00:22.773081+0000","last_unstale":"2024-03-26T05:01:46.312084+0000","last_undegraded":"2024-03-26T05:01:46.312084+0000","last_fullsized":"2024-03-26T05:01:46.312084+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:25.629101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:25.629101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:43.255919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":171,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:48.249265+0000","last_change":"2024-03-26T05:01:36.780629+0000","last_active":"2024-03-26T05:01:48.249265+0000","last_peered":"2024-03-26T05:01:48.249265+0000","last_clean":"2024-03-26T05:01:48.249265+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:01:48.249265+0000","last_undegraded":"2024-03-26T05:01:48.249265+0000","last_fullsized":"2024-03-26T05:01:48.249265+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:01:36.780518+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.077309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":105,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312587+0000","last_change":"2024-03-26T05:00:49.653404+0000","last_active":"2024-03-26T05:01:46.312587+0000","last_peered":"2024-03-26T05:01:46.312587+0000","last_clean":"2024-03-26T05:01:46.312587+0000","last_became_active":"2024-03-26T05:00:49.642276+0000","last_became_peered":"2024-03-26T05:00:49.642276+0000","last_unstale":"2024-03-26T05:01:46.312587+0000","last_undegraded":"2024-03-26T05:01:46.312587+0000","last_fullsized":"2024-03-26T05:01:46.312587+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:48.856193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028588+0000","last_change":"2024-03-26T05:01:41.973895+0000","last_active":"2024-03-26T05:01:47.028588+0000","last_peered":"2024-03-26T05:01:47.028588+0000","last_clean":"2024-03-26T05:01:47.028588+0000","last_became_active":"2024-03-26T05:00:26.956677+0000","last_became_peered":"2024-03-26T05:00:26.956677+0000","last_unstale":"2024-03-26T05:01:47.028588+0000","last_undegraded":"2024-03-26T05:01:47.028588+0000","last_fullsized":"2024-03-26T05:01:47.028588+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:41.973805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.973805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:54.658519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407228+0000","last_change":"2024-03-26T05:00:29.461548+0000","last_active":"2024-03-26T05:01:46.407228+0000","last_peered":"2024-03-26T05:01:46.407228+0000","last_clean":"2024-03-26T05:01:46.407228+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:01:46.407228+0000","last_undegraded":"2024-03-26T05:01:46.407228+0000","last_fullsized":"2024-03-26T05:01:46.407228+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407689+0000","last_change":"2024-03-26T05:01:35.745231+0000","last_active":"2024-03-26T05:01:46.407689+0000","last_peered":"2024-03-26T05:01:46.407689+0000","last_clean":"2024-03-26T05:01:46.407689+0000","last_became_active":"2024-03-26T05:00:23.346665+0000","last_became_peered":"2024-03-26T05:00:23.346665+0000","last_unstale":"2024-03-26T05:01:46.407689+0000","last_undegraded":"2024-03-26T05:01:46.407689+0000","last_fullsized":"2024-03-26T05:01:46.407689+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.745100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.314539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'25","reported_seq":123,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312530+0000","last_change":"2024-03-26T05:01:44.739144+0000","last_active":"2024-03-26T05:01:46.312530+0000","last_peered":"2024-03-26T05:01:46.312530+0000","last_clean":"2024-03-26T05:01:46.312530+0000","last_became_active":"2024-03-26T05:00:25.648253+0000","last_became_peered":"2024-03-26T05:00:25.648253+0000","last_unstale":"2024-03-26T05:01:46.312530+0000","last_undegraded":"2024-03-26T05:01:46.312530+0000","last_fullsized":"2024-03-26T05:01:46.312530+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:01:44.739046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.739046+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:06.858172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":61,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.227221+0000","last_change":"2024-03-26T05:00:26.958976+0000","last_active":"2024-03-26T05:01:47.227221+0000","last_peered":"2024-03-26T05:01:47.227221+0000","last_clean":"2024-03-26T05:01:47.227221+0000","last_became_active":"2024-03-26T05:00:26.958689+0000","last_became_peered":"2024-03-26T05:00:26.958689+0000","last_unstale":"2024-03-26T05:01:47.227221+0000","last_undegraded":"2024-03-26T05:01:47.227221+0000","last_fullsized":"2024-03-26T05:01:47.227221+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":67,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735907+0000","last_change":"2024-03-26T05:01:46.735926+0000","last_active":"2024-03-26T05:01:46.735907+0000","last_peered":"2024-03-26T05:01:46.735907+0000","last_clean":"2024-03-26T05:01:46.735907+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:01:46.735907+0000","last_undegraded":"2024-03-26T05:01:46.735907+0000","last_fullsized":"2024-03-26T05:01:46.735907+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:46.735848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:46.735848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:09.041185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029185+0000","last_change":"2024-03-26T05:00:23.347452+0000","last_active":"2024-03-26T05:01:47.029185+0000","last_peered":"2024-03-26T05:01:47.029185+0000","last_clean":"2024-03-26T05:01:47.029185+0000","last_became_active":"2024-03-26T05:00:23.346876+0000","last_became_peered":"2024-03-26T05:00:23.346876+0000","last_unstale":"2024-03-26T05:01:47.029185+0000","last_undegraded":"2024-03-26T05:01:47.029185+0000","last_fullsized":"2024-03-26T05:01:47.029185+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'16","reported_seq":64,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.408955+0000","last_change":"2024-03-26T05:01:42.753638+0000","last_active":"2024-03-26T05:01:46.408955+0000","last_peered":"2024-03-26T05:01:46.408955+0000","last_clean":"2024-03-26T05:01:46.408955+0000","last_became_active":"2024-03-26T05:00:49.641444+0000","last_became_peered":"2024-03-26T05:00:49.641444+0000","last_unstale":"2024-03-26T05:01:46.408955+0000","last_undegraded":"2024-03-26T05:01:46.408955+0000","last_fullsized":"2024-03-26T05:01:46.408955+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:01:42.753523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:42.753523+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.891497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":89,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312739+0000","last_change":"2024-03-26T05:00:26.962263+0000","last_active":"2024-03-26T05:01:46.312739+0000","last_peered":"2024-03-26T05:01:46.312739+0000","last_clean":"2024-03-26T05:01:46.312739+0000","last_became_active":"2024-03-26T05:00:26.961907+0000","last_became_peered":"2024-03-26T05:00:26.961907+0000","last_unstale":"2024-03-26T05:01:46.312739+0000","last_undegraded":"2024-03-26T05:01:46.312739+0000","last_fullsized":"2024-03-26T05:01:46.312739+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":57,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762259+0000","last_change":"2024-03-26T05:00:29.461447+0000","last_active":"2024-03-26T05:01:46.762259+0000","last_peered":"2024-03-26T05:01:46.762259+0000","last_clean":"2024-03-26T05:01:46.762259+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:01:46.762259+0000","last_undegraded":"2024-03-26T05:01:46.762259+0000","last_fullsized":"2024-03-26T05:01:46.762259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:57.420912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":80,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312608+0000","last_change":"2024-03-26T05:01:29.581397+0000","last_active":"2024-03-26T05:01:46.312608+0000","last_peered":"2024-03-26T05:01:46.312608+0000","last_clean":"2024-03-26T05:01:46.312608+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:01:46.312608+0000","last_undegraded":"2024-03-26T05:01:46.312608+0000","last_fullsized":"2024-03-26T05:01:46.312608+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:01:29.581309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.581309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.525431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'20","reported_seq":100,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312925+0000","last_change":"2024-03-26T05:00:24.936644+0000","last_active":"2024-03-26T05:01:46.312925+0000","last_peered":"2024-03-26T05:01:46.312925+0000","last_clean":"2024-03-26T05:01:46.312925+0000","last_became_active":"2024-03-26T05:00:24.936315+0000","last_became_peered":"2024-03-26T05:00:24.936315+0000","last_unstale":"2024-03-26T05:01:46.312925+0000","last_undegraded":"2024-03-26T05:01:46.312925+0000","last_fullsized":"2024-03-26T05:01:46.312925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734642+0000","last_change":"2024-03-26T05:00:26.957814+0000","last_active":"2024-03-26T05:01:46.734642+0000","last_peered":"2024-03-26T05:01:46.734642+0000","last_clean":"2024-03-26T05:01:46.734642+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:01:46.734642+0000","last_undegraded":"2024-03-26T05:01:46.734642+0000","last_fullsized":"2024-03-26T05:01:46.734642+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:55.181714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":67,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:48.472273+0000","last_change":"2024-03-26T05:01:48.472310+0000","last_active":"2024-03-26T05:01:48.472273+0000","last_peered":"2024-03-26T05:01:48.472273+0000","last_clean":"2024-03-26T05:01:48.472273+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:01:48.472273+0000","last_undegraded":"2024-03-26T05:01:48.472273+0000","last_fullsized":"2024-03-26T05:01:48.472273+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:48.472188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:48.472188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:03.679495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735088+0000","last_change":"2024-03-26T05:00:22.779020+0000","last_active":"2024-03-26T05:01:46.735088+0000","last_peered":"2024-03-26T05:01:46.735088+0000","last_clean":"2024-03-26T05:01:46.735088+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:01:46.735088+0000","last_undegraded":"2024-03-26T05:01:46.735088+0000","last_fullsized":"2024-03-26T05:01:46.735088+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:48.991339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"96'40","reported_seq":140,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734554+0000","last_change":"2024-03-26T05:01:42.152769+0000","last_active":"2024-03-26T05:01:46.734554+0000","last_peered":"2024-03-26T05:01:46.734554+0000","last_clean":"2024-03-26T05:01:46.734554+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:01:46.734554+0000","last_undegraded":"2024-03-26T05:01:46.734554+0000","last_fullsized":"2024-03-26T05:01:46.734554+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:01:42.152630+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:01:42.152630+0000","last_clean_scrub_stamp":"2024-03-26T05:01:42.152630+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.656601+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":63,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312736+0000","last_change":"2024-03-26T05:00:26.959009+0000","last_active":"2024-03-26T05:01:46.312736+0000","last_peered":"2024-03-26T05:01:46.312736+0000","last_clean":"2024-03-26T05:01:46.312736+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:01:46.312736+0000","last_undegraded":"2024-03-26T05:01:46.312736+0000","last_fullsized":"2024-03-26T05:01:46.312736+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:47.880056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029034+0000","last_change":"2024-03-26T05:00:28.983093+0000","last_active":"2024-03-26T05:01:47.029034+0000","last_peered":"2024-03-26T05:01:47.029034+0000","last_clean":"2024-03-26T05:01:47.029034+0000","last_became_active":"2024-03-26T05:00:28.982825+0000","last_became_peered":"2024-03-26T05:00:28.982825+0000","last_unstale":"2024-03-26T05:01:47.029034+0000","last_undegraded":"2024-03-26T05:01:47.029034+0000","last_fullsized":"2024-03-26T05:01:47.029034+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.359012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":97,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.313909+0000","last_change":"2024-03-26T05:01:41.485196+0000","last_active":"2024-03-26T05:01:46.313909+0000","last_peered":"2024-03-26T05:01:46.313909+0000","last_clean":"2024-03-26T05:01:46.313909+0000","last_became_active":"2024-03-26T05:00:26.956596+0000","last_became_peered":"2024-03-26T05:00:26.956596+0000","last_unstale":"2024-03-26T05:01:46.313909+0000","last_undegraded":"2024-03-26T05:01:46.313909+0000","last_fullsized":"2024-03-26T05:01:46.313909+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:41.485086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.485086+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:59.948091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734406+0000","last_change":"2024-03-26T05:00:22.775050+0000","last_active":"2024-03-26T05:01:46.734406+0000","last_peered":"2024-03-26T05:01:46.734406+0000","last_clean":"2024-03-26T05:01:46.734406+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:01:46.734406+0000","last_undegraded":"2024-03-26T05:01:46.734406+0000","last_fullsized":"2024-03-26T05:01:46.734406+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":99,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.408019+0000","last_change":"2024-03-26T05:01:28.754168+0000","last_active":"2024-03-26T05:01:46.408019+0000","last_peered":"2024-03-26T05:01:46.408019+0000","last_clean":"2024-03-26T05:01:46.408019+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:01:46.408019+0000","last_undegraded":"2024-03-26T05:01:46.408019+0000","last_fullsized":"2024-03-26T05:01:46.408019+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:28.754068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:28.754068+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:40.695752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734539+0000","last_change":"2024-03-26T05:00:29.461578+0000","last_active":"2024-03-26T05:01:46.734539+0000","last_peered":"2024-03-26T05:01:46.734539+0000","last_clean":"2024-03-26T05:01:46.734539+0000","last_became_active":"2024-03-26T05:00:29.461236+0000","last_became_peered":"2024-03-26T05:00:29.461236+0000","last_unstale":"2024-03-26T05:01:46.734539+0000","last_undegraded":"2024-03-26T05:01:46.734539+0000","last_fullsized":"2024-03-26T05:01:46.734539+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:49.354548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028708+0000","last_change":"2024-03-26T05:01:45.085244+0000","last_active":"2024-03-26T05:01:47.028708+0000","last_peered":"2024-03-26T05:01:47.028708+0000","last_clean":"2024-03-26T05:01:47.028708+0000","last_became_active":"2024-03-26T05:00:26.955805+0000","last_became_peered":"2024-03-26T05:00:26.955805+0000","last_unstale":"2024-03-26T05:01:47.028708+0000","last_undegraded":"2024-03-26T05:01:47.028708+0000","last_fullsized":"2024-03-26T05:01:47.028708+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:45.085111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:45.085111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.981772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.312741+0000","last_change":"2024-03-26T05:01:26.442833+0000","last_active":"2024-03-26T05:01:46.312741+0000","last_peered":"2024-03-26T05:01:46.312741+0000","last_clean":"2024-03-26T05:01:46.312741+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:01:46.312741+0000","last_undegraded":"2024-03-26T05:01:46.312741+0000","last_fullsized":"2024-03-26T05:01:46.312741+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.442722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.442722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:39.204290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'17","reported_seq":98,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.313805+0000","last_change":"2024-03-26T05:00:24.944403+0000","last_active":"2024-03-26T05:01:46.313805+0000","last_peered":"2024-03-26T05:01:46.313805+0000","last_clean":"2024-03-26T05:01:46.313805+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:01:46.313805+0000","last_undegraded":"2024-03-26T05:01:46.313805+0000","last_fullsized":"2024-03-26T05:01:46.313805+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.409457+0000","last_change":"2024-03-26T05:00:28.982075+0000","last_active":"2024-03-26T05:01:46.409457+0000","last_peered":"2024-03-26T05:01:46.409457+0000","last_clean":"2024-03-26T05:01:46.409457+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:01:46.409457+0000","last_undegraded":"2024-03-26T05:01:46.409457+0000","last_fullsized":"2024-03-26T05:01:46.409457+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:46.440667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.407851+0000","last_change":"2024-03-26T05:01:37.774253+0000","last_active":"2024-03-26T05:01:46.407851+0000","last_peered":"2024-03-26T05:01:46.407851+0000","last_clean":"2024-03-26T05:01:46.407851+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:01:46.407851+0000","last_undegraded":"2024-03-26T05:01:46.407851+0000","last_fullsized":"2024-03-26T05:01:46.407851+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.774158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.774158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:01.227552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":77,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.227381+0000","last_change":"2024-03-26T05:01:28.386278+0000","last_active":"2024-03-26T05:01:47.227381+0000","last_peered":"2024-03-26T05:01:47.227381+0000","last_clean":"2024-03-26T05:01:47.227381+0000","last_became_active":"2024-03-26T05:00:22.768610+0000","last_became_peered":"2024-03-26T05:00:22.768610+0000","last_unstale":"2024-03-26T05:01:47.227381+0000","last_undegraded":"2024-03-26T05:01:47.227381+0000","last_fullsized":"2024-03-26T05:01:47.227381+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_clean_scrub_stamp":"2024-03-26T05:01:28.386172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.056496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.313544+0000","last_change":"2024-03-26T05:00:24.942796+0000","last_active":"2024-03-26T05:01:46.313544+0000","last_peered":"2024-03-26T05:01:46.313544+0000","last_clean":"2024-03-26T05:01:46.313544+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:01:46.313544+0000","last_undegraded":"2024-03-26T05:01:46.313544+0000","last_fullsized":"2024-03-26T05:01:46.313544+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":59,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.313962+0000","last_change":"2024-03-26T05:00:28.978772+0000","last_active":"2024-03-26T05:01:46.313962+0000","last_peered":"2024-03-26T05:01:46.313962+0000","last_clean":"2024-03-26T05:01:46.313962+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:01:46.313962+0000","last_undegraded":"2024-03-26T05:01:46.313962+0000","last_fullsized":"2024-03-26T05:01:46.313962+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":96,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.409220+0000","last_change":"2024-03-26T05:01:43.782577+0000","last_active":"2024-03-26T05:01:46.409220+0000","last_peered":"2024-03-26T05:01:46.409220+0000","last_clean":"2024-03-26T05:01:46.409220+0000","last_became_active":"2024-03-26T05:00:26.957142+0000","last_became_peered":"2024-03-26T05:00:26.957142+0000","last_unstale":"2024-03-26T05:01:46.409220+0000","last_undegraded":"2024-03-26T05:01:46.409220+0000","last_fullsized":"2024-03-26T05:01:46.409220+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:43.782490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:43.782490+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.086882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734806+0000","last_change":"2024-03-26T05:01:39.142373+0000","last_active":"2024-03-26T05:01:46.734806+0000","last_peered":"2024-03-26T05:01:46.734806+0000","last_clean":"2024-03-26T05:01:46.734806+0000","last_became_active":"2024-03-26T05:00:22.778719+0000","last_became_peered":"2024-03-26T05:00:22.778719+0000","last_unstale":"2024-03-26T05:01:46.734806+0000","last_undegraded":"2024-03-26T05:01:46.734806+0000","last_fullsized":"2024-03-26T05:01:46.734806+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.142249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.142249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.076132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":90,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.313467+0000","last_change":"2024-03-26T05:01:44.403960+0000","last_active":"2024-03-26T05:01:46.313467+0000","last_peered":"2024-03-26T05:01:46.313467+0000","last_clean":"2024-03-26T05:01:46.313467+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:01:46.313467+0000","last_undegraded":"2024-03-26T05:01:46.313467+0000","last_fullsized":"2024-03-26T05:01:46.313467+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:01:44.403798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.403798+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:04.259798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":607,"ondisk_log_size":607,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005078,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28596,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716556,"statfs":{"total":95995035648,"available":95965753344,"internally_reserved":0,"allocated":1613824,"data_stored":1079903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332123,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32740,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712412,"statfs":{"total":95995035648,"available":95961509888,"internally_reserved":0,"allocated":1597440,"data_stored":1078387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1056768,"data_stored":489283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1626112,"data_stored":1080056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":487833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117034,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":488499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920806,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":489088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659168,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":488345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:01:51.449 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T05:01:51.449 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T05:01:51.449 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T05:01:51.449 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T05:01:51.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:51 smithi080 ceph-mon[29691]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:51.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:51 smithi080 ceph-mon[29691]: 6.1e deep-scrub starts 2024-03-26T05:01:51.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:51 smithi080 ceph-mon[29691]: 6.1e deep-scrub ok 2024-03-26T05:01:51.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:51 smithi080 ceph-mon[29691]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:01:51.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:51 smithi080 ceph-mon[29691]: 4.b scrub starts 2024-03-26T05:01:51.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:51 smithi080 ceph-mon[29691]: 4.b scrub ok 2024-03-26T05:01:51.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:51 smithi080 ceph-mon[40404]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:51.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:51 smithi080 ceph-mon[40404]: 6.1e deep-scrub starts 2024-03-26T05:01:51.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:51 smithi080 ceph-mon[40404]: 6.1e deep-scrub ok 2024-03-26T05:01:51.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:51 smithi080 ceph-mon[40404]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:01:51.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:51 smithi080 ceph-mon[40404]: 4.b scrub starts 2024-03-26T05:01:51.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:51 smithi080 ceph-mon[40404]: 4.b scrub ok 2024-03-26T05:01:51.691 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:51.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:51 smithi132 ceph-mon[33500]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:51.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:51 smithi132 ceph-mon[33500]: 6.1e deep-scrub starts 2024-03-26T05:01:51.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:51 smithi132 ceph-mon[33500]: 6.1e deep-scrub ok 2024-03-26T05:01:51.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:51 smithi132 ceph-mon[33500]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:01:51.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:51 smithi132 ceph-mon[33500]: 4.b scrub starts 2024-03-26T05:01:51.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:51 smithi132 ceph-mon[33500]: 4.b scrub ok 2024-03-26T05:01:52.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[40404]: 6.19 scrub starts 2024-03-26T05:01:52.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[40404]: 6.19 scrub ok 2024-03-26T05:01:52.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[40404]: 5.1f scrub starts 2024-03-26T05:01:52.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[40404]: 5.1f scrub ok 2024-03-26T05:01:52.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[40404]: 5.1c scrub starts 2024-03-26T05:01:52.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[40404]: 5.1c scrub ok 2024-03-26T05:01:52.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[40404]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:52.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[40404]: 3.12 scrub starts 2024-03-26T05:01:52.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[40404]: 3.12 scrub ok 2024-03-26T05:01:52.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[29691]: 6.19 scrub starts 2024-03-26T05:01:52.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[29691]: 6.19 scrub ok 2024-03-26T05:01:52.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[29691]: 5.1f scrub starts 2024-03-26T05:01:52.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[29691]: 5.1f scrub ok 2024-03-26T05:01:52.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[29691]: 5.1c scrub starts 2024-03-26T05:01:52.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[29691]: 5.1c scrub ok 2024-03-26T05:01:52.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[29691]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:52.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[29691]: 3.12 scrub starts 2024-03-26T05:01:52.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:52 smithi080 ceph-mon[29691]: 3.12 scrub ok 2024-03-26T05:01:52.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:52 smithi132 ceph-mon[33500]: 6.19 scrub starts 2024-03-26T05:01:52.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:52 smithi132 ceph-mon[33500]: 6.19 scrub ok 2024-03-26T05:01:52.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:52 smithi132 ceph-mon[33500]: 5.1f scrub starts 2024-03-26T05:01:52.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:52 smithi132 ceph-mon[33500]: 5.1f scrub ok 2024-03-26T05:01:52.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:52 smithi132 ceph-mon[33500]: 5.1c scrub starts 2024-03-26T05:01:52.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:52 smithi132 ceph-mon[33500]: 5.1c scrub ok 2024-03-26T05:01:52.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:52 smithi132 ceph-mon[33500]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:52.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:52 smithi132 ceph-mon[33500]: 3.12 scrub starts 2024-03-26T05:01:52.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:52 smithi132 ceph-mon[33500]: 3.12 scrub ok 2024-03-26T05:01:53.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:53 smithi080 ceph-mon[29691]: 5.1d scrub starts 2024-03-26T05:01:53.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:53 smithi080 ceph-mon[29691]: 5.1d scrub ok 2024-03-26T05:01:53.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:53 smithi080 ceph-mon[40404]: 5.1d scrub starts 2024-03-26T05:01:53.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:53 smithi080 ceph-mon[40404]: 5.1d scrub ok 2024-03-26T05:01:53.691 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:01:53.692 INFO:teuthology.orchestra.run.smithi080.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T05:01:53.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:53 smithi132 ceph-mon[33500]: 5.1d scrub starts 2024-03-26T05:01:53.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:53 smithi132 ceph-mon[33500]: 5.1d scrub ok 2024-03-26T05:01:54.303 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:01:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:01:54.330 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T05:01:54.331 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T05:01:54.331 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-26T05:01:54.344 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-26T05:01:54.345 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'squid', 'sha1': '9f04c91b77e4619963d41730d8cf756e3a6152de'}} 2024-03-26T05:01:54.345 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-26T05:01:54.345 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-26T05:01:54.587 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[40404]: 3.c scrub starts 2024-03-26T05:01:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[40404]: 3.c scrub ok 2024-03-26T05:01:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:01:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:01:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:01:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:01:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2957360952' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T05:01:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[40404]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:01:54.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[29691]: 3.c scrub starts 2024-03-26T05:01:54.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[29691]: 3.c scrub ok 2024-03-26T05:01:54.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:01:54.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:01:54.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:01:54.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:01:54.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2957360952' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T05:01:54.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:54 smithi080 ceph-mon[29691]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:01:54.748 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:54 smithi132 ceph-mon[33500]: 3.c scrub starts 2024-03-26T05:01:54.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:54 smithi132 ceph-mon[33500]: 3.c scrub ok 2024-03-26T05:01:54.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:54 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:01:54.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:54 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:01:54.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:54 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:01:54.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:54 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:01:54.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:54 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2957360952' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T05:01:54.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:54 smithi132 ceph-mon[33500]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:01:55.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:01:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:01:55.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:55 smithi132 ceph-mon[33500]: 6.f scrub starts 2024-03-26T05:01:55.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:55 smithi132 ceph-mon[33500]: 6.f scrub ok 2024-03-26T05:01:55.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:55 smithi132 ceph-mon[33500]: 5.1a scrub starts 2024-03-26T05:01:55.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:55 smithi132 ceph-mon[33500]: 5.1a scrub ok 2024-03-26T05:01:55.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:55 smithi132 ceph-mon[33500]: 3.5 scrub starts 2024-03-26T05:01:55.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:55 smithi132 ceph-mon[33500]: 3.5 scrub ok 2024-03-26T05:01:55.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:55 smithi132 ceph-mon[33500]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T05:01:55.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:55 smithi132 ceph-mon[33500]: 4.e scrub starts 2024-03-26T05:01:55.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:55 smithi132 ceph-mon[33500]: 4.e scrub ok 2024-03-26T05:01:55.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:55 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:55.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:55 smithi132 ceph-mon[33500]: 4.1d scrub starts 2024-03-26T05:01:55.545 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:55 smithi132 ceph-mon[33500]: 4.1d scrub ok 2024-03-26T05:01:55.618 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[75215]: ts=2024-03-26T05:01:55.302Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003071485s 2024-03-26T05:01:55.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[29691]: 6.f scrub starts 2024-03-26T05:01:55.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[29691]: 6.f scrub ok 2024-03-26T05:01:55.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[29691]: 5.1a scrub starts 2024-03-26T05:01:55.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[29691]: 5.1a scrub ok 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[29691]: 3.5 scrub starts 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[29691]: 3.5 scrub ok 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[29691]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[29691]: 4.e scrub starts 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[29691]: 4.e scrub ok 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[29691]: 4.1d scrub starts 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[29691]: 4.1d scrub ok 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[40404]: 6.f scrub starts 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[40404]: 6.f scrub ok 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[40404]: 5.1a scrub starts 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[40404]: 5.1a scrub ok 2024-03-26T05:01:55.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[40404]: 3.5 scrub starts 2024-03-26T05:01:55.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[40404]: 3.5 scrub ok 2024-03-26T05:01:55.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[40404]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T05:01:55.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[40404]: 4.e scrub starts 2024-03-26T05:01:55.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[40404]: 4.e scrub ok 2024-03-26T05:01:55.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:55.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[40404]: 4.1d scrub starts 2024-03-26T05:01:55.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:55 smithi080 ceph-mon[40404]: 4.1d scrub ok 2024-03-26T05:01:56.604 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-mon[29691]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:01:56.604 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-mon[29691]: 4.11 scrub starts 2024-03-26T05:01:56.604 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-mon[29691]: 4.11 scrub ok 2024-03-26T05:01:56.604 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-mon[29691]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T05:01:56.604 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-mon[40404]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:01:56.605 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-mon[40404]: 4.11 scrub starts 2024-03-26T05:01:56.605 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-mon[40404]: 4.11 scrub ok 2024-03-26T05:01:56.605 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-mon[40404]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T05:01:56.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:56 smithi132 ceph-mon[33500]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:01:56.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:56 smithi132 ceph-mon[33500]: 4.11 scrub starts 2024-03-26T05:01:56.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:56 smithi132 ceph-mon[33500]: 4.11 scrub ok 2024-03-26T05:01:56.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:56 smithi132 ceph-mon[33500]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T05:01:56.868 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:56] ENGINE Bus STOPPING 2024-03-26T05:01:56.868 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:01:56.868 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:56] ENGINE Bus STOPPED 2024-03-26T05:01:56.868 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:56] ENGINE Bus STARTING 2024-03-26T05:01:56.868 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:56] ENGINE Serving on http://:::9283 2024-03-26T05:01:56.868 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:01:56 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:01:56] ENGINE Bus STARTED 2024-03-26T05:01:57.220 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-26T05:01:57.221 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T05:01:57.221 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:01:57.462 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:01:57.855 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[29691]: 5.a scrub starts 2024-03-26T05:01:57.855 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[29691]: 5.a scrub ok 2024-03-26T05:01:57.855 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[29691]: 6.1c scrub starts 2024-03-26T05:01:57.855 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[29691]: 6.1c scrub ok 2024-03-26T05:01:57.855 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[29691]: 6.7 scrub starts 2024-03-26T05:01:57.855 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[29691]: 6.7 scrub ok 2024-03-26T05:01:57.855 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[29691]: 4.a scrub starts 2024-03-26T05:01:57.855 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[29691]: 4.a scrub ok 2024-03-26T05:01:57.855 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' 2024-03-26T05:01:57.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[40404]: 5.a scrub starts 2024-03-26T05:01:57.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[40404]: 5.a scrub ok 2024-03-26T05:01:57.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[40404]: 6.1c scrub starts 2024-03-26T05:01:57.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[40404]: 6.1c scrub ok 2024-03-26T05:01:57.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[40404]: 6.7 scrub starts 2024-03-26T05:01:57.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[40404]: 6.7 scrub ok 2024-03-26T05:01:57.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[40404]: 4.a scrub starts 2024-03-26T05:01:57.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[40404]: 4.a scrub ok 2024-03-26T05:01:57.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:57 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' 2024-03-26T05:01:58.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:57 smithi132 ceph-mon[33500]: 5.a scrub starts 2024-03-26T05:01:58.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:57 smithi132 ceph-mon[33500]: 5.a scrub ok 2024-03-26T05:01:58.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:57 smithi132 ceph-mon[33500]: 6.1c scrub starts 2024-03-26T05:01:58.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:57 smithi132 ceph-mon[33500]: 6.1c scrub ok 2024-03-26T05:01:58.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:57 smithi132 ceph-mon[33500]: 6.7 scrub starts 2024-03-26T05:01:58.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:57 smithi132 ceph-mon[33500]: 6.7 scrub ok 2024-03-26T05:01:58.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:57 smithi132 ceph-mon[33500]: 4.a scrub starts 2024-03-26T05:01:58.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:57 smithi132 ceph-mon[33500]: 4.a scrub ok 2024-03-26T05:01:58.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:57 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' 2024-03-26T05:01:58.858 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:58 smithi080 ceph-mon[40404]: 6.1d scrub starts 2024-03-26T05:01:58.859 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:58 smithi080 ceph-mon[40404]: 6.1d scrub ok 2024-03-26T05:01:58.859 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:58 smithi080 ceph-mon[40404]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:01:58.859 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:58 smithi080 ceph-mon[40404]: 4.10 deep-scrub starts 2024-03-26T05:01:58.859 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:58 smithi080 ceph-mon[40404]: 4.10 deep-scrub ok 2024-03-26T05:01:58.860 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:58 smithi080 ceph-mon[29691]: 6.1d scrub starts 2024-03-26T05:01:58.860 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:58 smithi080 ceph-mon[29691]: 6.1d scrub ok 2024-03-26T05:01:58.860 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:58 smithi080 ceph-mon[29691]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:01:58.860 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:58 smithi080 ceph-mon[29691]: 4.10 deep-scrub starts 2024-03-26T05:01:58.861 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:58 smithi080 ceph-mon[29691]: 4.10 deep-scrub ok 2024-03-26T05:01:59.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:58 smithi132 ceph-mon[33500]: 6.1d scrub starts 2024-03-26T05:01:59.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:58 smithi132 ceph-mon[33500]: 6.1d scrub ok 2024-03-26T05:01:59.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:58 smithi132 ceph-mon[33500]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:01:59.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:58 smithi132 ceph-mon[33500]: 4.10 deep-scrub starts 2024-03-26T05:01:59.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:58 smithi132 ceph-mon[33500]: 4.10 deep-scrub ok 2024-03-26T05:01:59.517 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:01:59.519 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:01:59.720 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:59 smithi132 ceph-mon[33500]: 6.1 deep-scrub starts 2024-03-26T05:01:59.720 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:59 smithi132 ceph-mon[33500]: 6.1 deep-scrub ok 2024-03-26T05:01:59.720 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:59 smithi132 ceph-mon[33500]: 5.1b scrub starts 2024-03-26T05:01:59.720 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:01:59 smithi132 ceph-mon[33500]: 5.1b scrub ok 2024-03-26T05:01:59.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:59 smithi080 ceph-mon[40404]: 6.1 deep-scrub starts 2024-03-26T05:01:59.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:59 smithi080 ceph-mon[40404]: 6.1 deep-scrub ok 2024-03-26T05:01:59.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:59 smithi080 ceph-mon[40404]: 5.1b scrub starts 2024-03-26T05:01:59.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:01:59 smithi080 ceph-mon[40404]: 5.1b scrub ok 2024-03-26T05:01:59.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:59 smithi080 ceph-mon[29691]: 6.1 deep-scrub starts 2024-03-26T05:01:59.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:59 smithi080 ceph-mon[29691]: 6.1 deep-scrub ok 2024-03-26T05:01:59.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:59 smithi080 ceph-mon[29691]: 5.1b scrub starts 2024-03-26T05:01:59.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:01:59 smithi080 ceph-mon[29691]: 5.1b scrub ok 2024-03-26T05:02:00.287 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-03-26T05:01:57.715193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1308,"num_read_kb":1122,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":816,"ondisk_log_size":816,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":246668,"kb_used_data":9736,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749714548,"statfs":{"total":767960285184,"available":767707697152,"internally_reserved":0,"allocated":9969664,"data_stored":5689152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001990"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":61,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.385646+0000","last_change":"2024-03-26T05:00:28.973250+0000","last_active":"2024-03-26T05:01:54.385646+0000","last_peered":"2024-03-26T05:01:54.385646+0000","last_clean":"2024-03-26T05:01:54.385646+0000","last_became_active":"2024-03-26T05:00:28.972969+0000","last_became_peered":"2024-03-26T05:00:28.972969+0000","last_unstale":"2024-03-26T05:01:54.385646+0000","last_undegraded":"2024-03-26T05:01:54.385646+0000","last_fullsized":"2024-03-26T05:01:54.385646+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":72,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484324+0000","last_change":"2024-03-26T05:01:39.812600+0000","last_active":"2024-03-26T05:01:54.484324+0000","last_peered":"2024-03-26T05:01:54.484324+0000","last_clean":"2024-03-26T05:01:54.484324+0000","last_became_active":"2024-03-26T05:00:26.953154+0000","last_became_peered":"2024-03-26T05:00:26.953154+0000","last_unstale":"2024-03-26T05:01:54.484324+0000","last_undegraded":"2024-03-26T05:01:54.484324+0000","last_fullsized":"2024-03-26T05:01:54.484324+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.812509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.812509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:45.654690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383852+0000","last_change":"2024-03-26T05:01:35.658900+0000","last_active":"2024-03-26T05:01:54.383852+0000","last_peered":"2024-03-26T05:01:54.383852+0000","last_clean":"2024-03-26T05:01:54.383852+0000","last_became_active":"2024-03-26T05:00:23.346607+0000","last_became_peered":"2024-03-26T05:00:23.346607+0000","last_unstale":"2024-03-26T05:01:54.383852+0000","last_undegraded":"2024-03-26T05:01:54.383852+0000","last_fullsized":"2024-03-26T05:01:54.383852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:35.658796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.658796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:41.835654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'15","reported_seq":97,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.385637+0000","last_change":"2024-03-26T05:00:24.943621+0000","last_active":"2024-03-26T05:01:54.385637+0000","last_peered":"2024-03-26T05:01:54.385637+0000","last_clean":"2024-03-26T05:01:54.385637+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:01:54.385637+0000","last_undegraded":"2024-03-26T05:01:54.385637+0000","last_fullsized":"2024-03-26T05:01:54.385637+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734617+0000","last_change":"2024-03-26T05:00:29.463053+0000","last_active":"2024-03-26T05:01:46.734617+0000","last_peered":"2024-03-26T05:01:46.734617+0000","last_clean":"2024-03-26T05:01:46.734617+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:01:46.734617+0000","last_undegraded":"2024-03-26T05:01:46.734617+0000","last_fullsized":"2024-03-26T05:01:46.734617+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:56.816597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":65,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.385530+0000","last_change":"2024-03-26T05:00:26.955355+0000","last_active":"2024-03-26T05:01:54.385530+0000","last_peered":"2024-03-26T05:01:54.385530+0000","last_clean":"2024-03-26T05:01:54.385530+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:01:54.385530+0000","last_undegraded":"2024-03-26T05:01:54.385530+0000","last_fullsized":"2024-03-26T05:01:54.385530+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735275+0000","last_change":"2024-03-26T05:01:37.201540+0000","last_active":"2024-03-26T05:01:46.735275+0000","last_peered":"2024-03-26T05:01:46.735275+0000","last_clean":"2024-03-26T05:01:46.735275+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:01:46.735275+0000","last_undegraded":"2024-03-26T05:01:46.735275+0000","last_fullsized":"2024-03-26T05:01:46.735275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.201446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.201446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:43.519218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'39","reported_seq":139,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734709+0000","last_change":"2024-03-26T05:00:24.932741+0000","last_active":"2024-03-26T05:01:46.734709+0000","last_peered":"2024-03-26T05:01:46.734709+0000","last_clean":"2024-03-26T05:01:46.734709+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:01:46.734709+0000","last_undegraded":"2024-03-26T05:01:46.734709+0000","last_fullsized":"2024-03-26T05:01:46.734709+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:53.767288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":69,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383589+0000","last_change":"2024-03-26T05:01:32.598168+0000","last_active":"2024-03-26T05:01:54.383589+0000","last_peered":"2024-03-26T05:01:54.383589+0000","last_clean":"2024-03-26T05:01:54.383589+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:01:54.383589+0000","last_undegraded":"2024-03-26T05:01:54.383589+0000","last_fullsized":"2024-03-26T05:01:54.383589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:32.598087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:32.598087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.083903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":62,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028457+0000","last_change":"2024-03-26T05:00:26.956404+0000","last_active":"2024-03-26T05:01:47.028457+0000","last_peered":"2024-03-26T05:01:47.028457+0000","last_clean":"2024-03-26T05:01:47.028457+0000","last_became_active":"2024-03-26T05:00:26.956013+0000","last_became_peered":"2024-03-26T05:00:26.956013+0000","last_unstale":"2024-03-26T05:01:47.028457+0000","last_undegraded":"2024-03-26T05:01:47.028457+0000","last_fullsized":"2024-03-26T05:01:47.028457+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029144+0000","last_change":"2024-03-26T05:01:30.949017+0000","last_active":"2024-03-26T05:01:47.029144+0000","last_peered":"2024-03-26T05:01:47.029144+0000","last_clean":"2024-03-26T05:01:47.029144+0000","last_became_active":"2024-03-26T05:00:22.768098+0000","last_became_peered":"2024-03-26T05:00:22.768098+0000","last_unstale":"2024-03-26T05:01:47.029144+0000","last_undegraded":"2024-03-26T05:01:47.029144+0000","last_fullsized":"2024-03-26T05:01:47.029144+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.948866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.948866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.633607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'21","reported_seq":117,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.385906+0000","last_change":"2024-03-26T05:01:30.476359+0000","last_active":"2024-03-26T05:01:54.385906+0000","last_peered":"2024-03-26T05:01:54.385906+0000","last_clean":"2024-03-26T05:01:54.385906+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:01:54.385906+0000","last_undegraded":"2024-03-26T05:01:54.385906+0000","last_fullsized":"2024-03-26T05:01:54.385906+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:01:30.476249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.476249+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.885741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":66,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:51.098422+0000","last_change":"2024-03-26T05:01:51.098460+0000","last_active":"2024-03-26T05:01:51.098422+0000","last_peered":"2024-03-26T05:01:51.098422+0000","last_clean":"2024-03-26T05:01:51.098422+0000","last_became_active":"2024-03-26T05:00:28.982276+0000","last_became_peered":"2024-03-26T05:00:28.982276+0000","last_unstale":"2024-03-26T05:01:51.098422+0000","last_undegraded":"2024-03-26T05:01:51.098422+0000","last_fullsized":"2024-03-26T05:01:51.098422+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:51.098327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.098327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:51.933537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":69,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:53.941776+0000","last_change":"2024-03-26T05:01:53.941805+0000","last_active":"2024-03-26T05:01:53.941776+0000","last_peered":"2024-03-26T05:01:53.941776+0000","last_clean":"2024-03-26T05:01:53.941776+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:01:53.941776+0000","last_undegraded":"2024-03-26T05:01:53.941776+0000","last_fullsized":"2024-03-26T05:01:53.941776+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:53.941700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:53.941700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:05.433050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029263+0000","last_change":"2024-03-26T05:01:39.021449+0000","last_active":"2024-03-26T05:01:47.029263+0000","last_peered":"2024-03-26T05:01:47.029263+0000","last_clean":"2024-03-26T05:01:47.029263+0000","last_became_active":"2024-03-26T05:00:22.766675+0000","last_became_peered":"2024-03-26T05:00:22.766675+0000","last_unstale":"2024-03-26T05:01:47.029263+0000","last_undegraded":"2024-03-26T05:01:47.029263+0000","last_fullsized":"2024-03-26T05:01:47.029263+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.021315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.021315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:39.428392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'11","reported_seq":91,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.385888+0000","last_change":"2024-03-26T05:00:24.943205+0000","last_active":"2024-03-26T05:01:54.385888+0000","last_peered":"2024-03-26T05:01:54.385888+0000","last_clean":"2024-03-26T05:01:54.385888+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:01:54.385888+0000","last_undegraded":"2024-03-26T05:01:54.385888+0000","last_fullsized":"2024-03-26T05:01:54.385888+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":69,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.386957+0000","last_change":"2024-03-26T05:01:50.502541+0000","last_active":"2024-03-26T05:01:54.386957+0000","last_peered":"2024-03-26T05:01:54.386957+0000","last_clean":"2024-03-26T05:01:54.386957+0000","last_became_active":"2024-03-26T05:00:29.461032+0000","last_became_peered":"2024-03-26T05:00:29.461032+0000","last_unstale":"2024-03-26T05:01:54.386957+0000","last_undegraded":"2024-03-26T05:01:54.386957+0000","last_fullsized":"2024-03-26T05:01:54.386957+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:01:50.502459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:52.644841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":72,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484892+0000","last_change":"2024-03-26T05:01:51.781163+0000","last_active":"2024-03-26T05:01:54.484892+0000","last_peered":"2024-03-26T05:01:54.484892+0000","last_clean":"2024-03-26T05:01:54.484892+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:01:54.484892+0000","last_undegraded":"2024-03-26T05:01:54.484892+0000","last_fullsized":"2024-03-26T05:01:54.484892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:51.781081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.781081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:56.997361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383792+0000","last_change":"2024-03-26T05:01:47.742402+0000","last_active":"2024-03-26T05:01:54.383792+0000","last_peered":"2024-03-26T05:01:54.383792+0000","last_clean":"2024-03-26T05:01:54.383792+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:01:54.383792+0000","last_undegraded":"2024-03-26T05:01:54.383792+0000","last_fullsized":"2024-03-26T05:01:54.383792+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:47.742292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:47.742292+0000","last_clean_scrub_stamp":"2024-03-26T05:01:47.742292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.854236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":99,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.409129+0000","last_change":"2024-03-26T05:01:24.617269+0000","last_active":"2024-03-26T05:01:46.409129+0000","last_peered":"2024-03-26T05:01:46.409129+0000","last_clean":"2024-03-26T05:01:46.409129+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:01:46.409129+0000","last_undegraded":"2024-03-26T05:01:46.409129+0000","last_fullsized":"2024-03-26T05:01:46.409129+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:24.617170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:24.617170+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:34.140661+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":66,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735279+0000","last_change":"2024-03-26T05:01:30.046528+0000","last_active":"2024-03-26T05:01:46.735279+0000","last_peered":"2024-03-26T05:01:46.735279+0000","last_clean":"2024-03-26T05:01:46.735279+0000","last_became_active":"2024-03-26T05:00:29.463238+0000","last_became_peered":"2024-03-26T05:00:29.463238+0000","last_unstale":"2024-03-26T05:01:46.735279+0000","last_undegraded":"2024-03-26T05:01:46.735279+0000","last_fullsized":"2024-03-26T05:01:46.735279+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.046425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.046425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:35.648365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":73,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.386321+0000","last_change":"2024-03-26T05:01:51.535065+0000","last_active":"2024-03-26T05:01:54.386321+0000","last_peered":"2024-03-26T05:01:54.386321+0000","last_clean":"2024-03-26T05:01:54.386321+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:01:54.386321+0000","last_undegraded":"2024-03-26T05:01:54.386321+0000","last_fullsized":"2024-03-26T05:01:54.386321+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:51.534988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.534988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:12.499805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.386019+0000","last_change":"2024-03-26T05:01:46.410472+0000","last_active":"2024-03-26T05:01:54.386019+0000","last_peered":"2024-03-26T05:01:54.386019+0000","last_clean":"2024-03-26T05:01:54.386019+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:01:54.386019+0000","last_undegraded":"2024-03-26T05:01:54.386019+0000","last_fullsized":"2024-03-26T05:01:54.386019+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:01:46.410356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:52.981688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'20","reported_seq":103,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735014+0000","last_change":"2024-03-26T05:00:24.937057+0000","last_active":"2024-03-26T05:01:46.735014+0000","last_peered":"2024-03-26T05:01:46.735014+0000","last_clean":"2024-03-26T05:01:46.735014+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:01:46.735014+0000","last_undegraded":"2024-03-26T05:01:46.735014+0000","last_fullsized":"2024-03-26T05:01:46.735014+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762118+0000","last_change":"2024-03-26T05:00:28.981879+0000","last_active":"2024-03-26T05:01:46.762118+0000","last_peered":"2024-03-26T05:01:46.762118+0000","last_clean":"2024-03-26T05:01:46.762118+0000","last_became_active":"2024-03-26T05:00:28.981505+0000","last_became_peered":"2024-03-26T05:00:28.981505+0000","last_unstale":"2024-03-26T05:01:46.762118+0000","last_undegraded":"2024-03-26T05:01:46.762118+0000","last_fullsized":"2024-03-26T05:01:46.762118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":101,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.487429+0000","last_change":"2024-03-26T05:01:51.236091+0000","last_active":"2024-03-26T05:01:54.487429+0000","last_peered":"2024-03-26T05:01:54.487429+0000","last_clean":"2024-03-26T05:01:54.487429+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:01:54.487429+0000","last_undegraded":"2024-03-26T05:01:54.487429+0000","last_fullsized":"2024-03-26T05:01:54.487429+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:51.235994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.235994+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:51.261797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":73,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484403+0000","last_change":"2024-03-26T05:00:22.773390+0000","last_active":"2024-03-26T05:01:54.484403+0000","last_peered":"2024-03-26T05:01:54.484403+0000","last_clean":"2024-03-26T05:01:54.484403+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:01:54.484403+0000","last_undegraded":"2024-03-26T05:01:54.484403+0000","last_fullsized":"2024-03-26T05:01:54.484403+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'16","reported_seq":104,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.384010+0000","last_change":"2024-03-26T05:01:41.650400+0000","last_active":"2024-03-26T05:01:54.384010+0000","last_peered":"2024-03-26T05:01:54.384010+0000","last_clean":"2024-03-26T05:01:54.384010+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:01:54.384010+0000","last_undegraded":"2024-03-26T05:01:54.384010+0000","last_fullsized":"2024-03-26T05:01:54.384010+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:01:41.650310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.650310+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:08.831570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":60,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484983+0000","last_change":"2024-03-26T05:00:28.981390+0000","last_active":"2024-03-26T05:01:54.484983+0000","last_peered":"2024-03-26T05:01:54.484983+0000","last_clean":"2024-03-26T05:01:54.484983+0000","last_became_active":"2024-03-26T05:00:28.981049+0000","last_became_peered":"2024-03-26T05:00:28.981049+0000","last_unstale":"2024-03-26T05:01:54.484983+0000","last_undegraded":"2024-03-26T05:01:54.484983+0000","last_fullsized":"2024-03-26T05:01:54.484983+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":65,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.384127+0000","last_change":"2024-03-26T05:00:26.963282+0000","last_active":"2024-03-26T05:01:54.384127+0000","last_peered":"2024-03-26T05:01:54.384127+0000","last_clean":"2024-03-26T05:01:54.384127+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:01:54.384127+0000","last_undegraded":"2024-03-26T05:01:54.384127+0000","last_fullsized":"2024-03-26T05:01:54.384127+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734309+0000","last_change":"2024-03-26T05:01:43.166558+0000","last_active":"2024-03-26T05:01:46.734309+0000","last_peered":"2024-03-26T05:01:46.734309+0000","last_clean":"2024-03-26T05:01:46.734309+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:01:46.734309+0000","last_undegraded":"2024-03-26T05:01:46.734309+0000","last_fullsized":"2024-03-26T05:01:46.734309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:43.166422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:43.166422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:50.132493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'17","reported_seq":126,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.487424+0000","last_change":"2024-03-26T05:01:53.258532+0000","last_active":"2024-03-26T05:01:54.487424+0000","last_peered":"2024-03-26T05:01:54.487424+0000","last_clean":"2024-03-26T05:01:54.487424+0000","last_became_active":"2024-03-26T05:00:49.643308+0000","last_became_peered":"2024-03-26T05:00:49.643308+0000","last_unstale":"2024-03-26T05:01:54.487424+0000","last_undegraded":"2024-03-26T05:01:54.487424+0000","last_fullsized":"2024-03-26T05:01:54.487424+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:01:53.258422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:53.258422+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:22.142699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":57,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.761903+0000","last_change":"2024-03-26T05:00:28.978762+0000","last_active":"2024-03-26T05:01:46.761903+0000","last_peered":"2024-03-26T05:01:46.761903+0000","last_clean":"2024-03-26T05:01:46.761903+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:01:46.761903+0000","last_undegraded":"2024-03-26T05:01:46.761903+0000","last_fullsized":"2024-03-26T05:01:46.761903+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:49.872411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":71,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.486845+0000","last_change":"2024-03-26T05:01:44.235422+0000","last_active":"2024-03-26T05:01:54.486845+0000","last_peered":"2024-03-26T05:01:54.486845+0000","last_clean":"2024-03-26T05:01:54.486845+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:01:54.486845+0000","last_undegraded":"2024-03-26T05:01:54.486845+0000","last_fullsized":"2024-03-26T05:01:54.486845+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:44.235328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.235328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.223031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":73,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383795+0000","last_change":"2024-03-26T05:00:22.775209+0000","last_active":"2024-03-26T05:01:54.383795+0000","last_peered":"2024-03-26T05:01:54.383795+0000","last_clean":"2024-03-26T05:01:54.383795+0000","last_became_active":"2024-03-26T05:00:22.774783+0000","last_became_peered":"2024-03-26T05:00:22.774783+0000","last_unstale":"2024-03-26T05:01:54.383795+0000","last_undegraded":"2024-03-26T05:01:54.383795+0000","last_fullsized":"2024-03-26T05:01:54.383795+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":72,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734370+0000","last_change":"2024-03-26T05:00:24.942166+0000","last_active":"2024-03-26T05:01:46.734370+0000","last_peered":"2024-03-26T05:01:46.734370+0000","last_clean":"2024-03-26T05:01:46.734370+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:01:46.734370+0000","last_undegraded":"2024-03-26T05:01:46.734370+0000","last_fullsized":"2024-03-26T05:01:46.734370+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734443+0000","last_change":"2024-03-26T05:00:29.461798+0000","last_active":"2024-03-26T05:01:46.734443+0000","last_peered":"2024-03-26T05:01:46.734443+0000","last_clean":"2024-03-26T05:01:46.734443+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:01:46.734443+0000","last_undegraded":"2024-03-26T05:01:46.734443+0000","last_fullsized":"2024-03-26T05:01:46.734443+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":61,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762160+0000","last_change":"2024-03-26T05:00:26.959842+0000","last_active":"2024-03-26T05:01:46.762160+0000","last_peered":"2024-03-26T05:01:46.762160+0000","last_clean":"2024-03-26T05:01:46.762160+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:01:46.762160+0000","last_undegraded":"2024-03-26T05:01:46.762160+0000","last_fullsized":"2024-03-26T05:01:46.762160+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029324+0000","last_change":"2024-03-26T05:01:29.997645+0000","last_active":"2024-03-26T05:01:47.029324+0000","last_peered":"2024-03-26T05:01:47.029324+0000","last_clean":"2024-03-26T05:01:47.029324+0000","last_became_active":"2024-03-26T05:00:22.771199+0000","last_became_peered":"2024-03-26T05:00:22.771199+0000","last_unstale":"2024-03-26T05:01:47.029324+0000","last_undegraded":"2024-03-26T05:01:47.029324+0000","last_fullsized":"2024-03-26T05:01:47.029324+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:35.309625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'17","reported_seq":97,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734568+0000","last_change":"2024-03-26T05:00:25.648453+0000","last_active":"2024-03-26T05:01:46.734568+0000","last_peered":"2024-03-26T05:01:46.734568+0000","last_clean":"2024-03-26T05:01:46.734568+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:01:46.734568+0000","last_undegraded":"2024-03-26T05:01:46.734568+0000","last_fullsized":"2024-03-26T05:01:46.734568+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":69,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383249+0000","last_change":"2024-03-26T05:01:49.768067+0000","last_active":"2024-03-26T05:01:54.383249+0000","last_peered":"2024-03-26T05:01:54.383249+0000","last_clean":"2024-03-26T05:01:54.383249+0000","last_became_active":"2024-03-26T05:00:28.985158+0000","last_became_peered":"2024-03-26T05:00:28.985158+0000","last_unstale":"2024-03-26T05:01:54.383249+0000","last_undegraded":"2024-03-26T05:01:54.383249+0000","last_fullsized":"2024-03-26T05:01:54.383249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:49.767986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:49.767986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.936245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":62,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734714+0000","last_change":"2024-03-26T05:00:26.960672+0000","last_active":"2024-03-26T05:01:46.734714+0000","last_peered":"2024-03-26T05:01:46.734714+0000","last_clean":"2024-03-26T05:01:46.734714+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:01:46.734714+0000","last_undegraded":"2024-03-26T05:01:46.734714+0000","last_fullsized":"2024-03-26T05:01:46.734714+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:55.490995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":69,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762279+0000","last_change":"2024-03-26T05:00:22.773808+0000","last_active":"2024-03-26T05:01:46.762279+0000","last_peered":"2024-03-26T05:01:46.762279+0000","last_clean":"2024-03-26T05:01:46.762279+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:01:46.762279+0000","last_undegraded":"2024-03-26T05:01:46.762279+0000","last_fullsized":"2024-03-26T05:01:46.762279+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:49.214803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":84,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484609+0000","last_change":"2024-03-26T05:00:24.939870+0000","last_active":"2024-03-26T05:01:54.484609+0000","last_peered":"2024-03-26T05:01:54.484609+0000","last_clean":"2024-03-26T05:01:54.484609+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:01:54.484609+0000","last_undegraded":"2024-03-26T05:01:54.484609+0000","last_fullsized":"2024-03-26T05:01:54.484609+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734292+0000","last_change":"2024-03-26T05:00:28.984567+0000","last_active":"2024-03-26T05:01:46.734292+0000","last_peered":"2024-03-26T05:01:46.734292+0000","last_clean":"2024-03-26T05:01:46.734292+0000","last_became_active":"2024-03-26T05:00:28.983417+0000","last_became_peered":"2024-03-26T05:00:28.983417+0000","last_unstale":"2024-03-26T05:01:46.734292+0000","last_undegraded":"2024-03-26T05:01:46.734292+0000","last_fullsized":"2024-03-26T05:01:46.734292+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:51.606343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":72,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484720+0000","last_change":"2024-03-26T05:01:30.777153+0000","last_active":"2024-03-26T05:01:54.484720+0000","last_peered":"2024-03-26T05:01:54.484720+0000","last_clean":"2024-03-26T05:01:54.484720+0000","last_became_active":"2024-03-26T05:00:26.954653+0000","last_became_peered":"2024-03-26T05:00:26.954653+0000","last_unstale":"2024-03-26T05:01:54.484720+0000","last_undegraded":"2024-03-26T05:01:54.484720+0000","last_fullsized":"2024-03-26T05:01:54.484720+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.777070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:45.021844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":73,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.386566+0000","last_change":"2024-03-26T05:00:22.767847+0000","last_active":"2024-03-26T05:01:54.386566+0000","last_peered":"2024-03-26T05:01:54.386566+0000","last_clean":"2024-03-26T05:01:54.386566+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:01:54.386566+0000","last_undegraded":"2024-03-26T05:01:54.386566+0000","last_fullsized":"2024-03-26T05:01:54.386566+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"95'16","reported_seq":101,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.386127+0000","last_change":"2024-03-26T05:01:40.462389+0000","last_active":"2024-03-26T05:01:54.386127+0000","last_peered":"2024-03-26T05:01:54.386127+0000","last_clean":"2024-03-26T05:01:54.386127+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:01:54.386127+0000","last_undegraded":"2024-03-26T05:01:54.386127+0000","last_fullsized":"2024-03-26T05:01:54.386127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.462271+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.765446+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":61,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383711+0000","last_change":"2024-03-26T05:00:28.984841+0000","last_active":"2024-03-26T05:01:54.383711+0000","last_peered":"2024-03-26T05:01:54.383711+0000","last_clean":"2024-03-26T05:01:54.383711+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:01:54.383711+0000","last_undegraded":"2024-03-26T05:01:54.383711+0000","last_fullsized":"2024-03-26T05:01:54.383711+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":89,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:48.399782+0000","last_change":"2024-03-26T05:00:26.956153+0000","last_active":"2024-03-26T05:01:48.399782+0000","last_peered":"2024-03-26T05:01:48.399782+0000","last_clean":"2024-03-26T05:01:48.399782+0000","last_became_active":"2024-03-26T05:00:26.955836+0000","last_became_peered":"2024-03-26T05:00:26.955836+0000","last_unstale":"2024-03-26T05:01:48.399782+0000","last_undegraded":"2024-03-26T05:01:48.399782+0000","last_fullsized":"2024-03-26T05:01:48.399782+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:54.166825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":77,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.761998+0000","last_change":"2024-03-26T05:01:26.785152+0000","last_active":"2024-03-26T05:01:46.761998+0000","last_peered":"2024-03-26T05:01:46.761998+0000","last_clean":"2024-03-26T05:01:46.761998+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:01:46.761998+0000","last_undegraded":"2024-03-26T05:01:46.761998+0000","last_fullsized":"2024-03-26T05:01:46.761998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.785056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.785056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.302223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'22","reported_seq":113,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734580+0000","last_change":"2024-03-26T05:01:40.118101+0000","last_active":"2024-03-26T05:01:46.734580+0000","last_peered":"2024-03-26T05:01:46.734580+0000","last_clean":"2024-03-26T05:01:46.734580+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:01:46.734580+0000","last_undegraded":"2024-03-26T05:01:46.734580+0000","last_fullsized":"2024-03-26T05:01:46.734580+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:01:40.117953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.117953+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.646893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":61,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.387192+0000","last_change":"2024-03-26T05:00:28.979801+0000","last_active":"2024-03-26T05:01:54.387192+0000","last_peered":"2024-03-26T05:01:54.387192+0000","last_clean":"2024-03-26T05:01:54.387192+0000","last_became_active":"2024-03-26T05:00:28.979477+0000","last_became_peered":"2024-03-26T05:00:28.979477+0000","last_unstale":"2024-03-26T05:01:54.387192+0000","last_undegraded":"2024-03-26T05:01:54.387192+0000","last_fullsized":"2024-03-26T05:01:54.387192+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:56.897877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":100,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:49.006492+0000","last_change":"2024-03-26T05:01:49.006492+0000","last_active":"2024-03-26T05:01:49.006492+0000","last_peered":"2024-03-26T05:01:49.006492+0000","last_clean":"2024-03-26T05:01:49.006492+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:01:49.006492+0000","last_undegraded":"2024-03-26T05:01:49.006492+0000","last_fullsized":"2024-03-26T05:01:49.006492+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:49.006396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:49.006396+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.424499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383454+0000","last_change":"2024-03-26T05:01:51.835794+0000","last_active":"2024-03-26T05:01:54.383454+0000","last_peered":"2024-03-26T05:01:54.383454+0000","last_clean":"2024-03-26T05:01:54.383454+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:01:54.383454+0000","last_undegraded":"2024-03-26T05:01:54.383454+0000","last_fullsized":"2024-03-26T05:01:54.383454+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:51.835722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.835722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:11.405634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"98'20","reported_seq":107,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:48.070410+0000","last_change":"2024-03-26T05:01:48.070439+0000","last_active":"2024-03-26T05:01:48.070410+0000","last_peered":"2024-03-26T05:01:48.070410+0000","last_clean":"2024-03-26T05:01:48.070410+0000","last_became_active":"2024-03-26T05:00:25.648284+0000","last_became_peered":"2024-03-26T05:00:25.648284+0000","last_unstale":"2024-03-26T05:01:48.070410+0000","last_undegraded":"2024-03-26T05:01:48.070410+0000","last_fullsized":"2024-03-26T05:01:48.070410+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'20","last_scrub_stamp":"2024-03-26T05:01:48.070338+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:01:48.070338+0000","last_clean_scrub_stamp":"2024-03-26T05:01:48.070338+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.153022+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":66,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.409306+0000","last_change":"2024-03-26T05:01:31.568804+0000","last_active":"2024-03-26T05:01:46.409306+0000","last_peered":"2024-03-26T05:01:46.409306+0000","last_clean":"2024-03-26T05:01:46.409306+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:01:46.409306+0000","last_undegraded":"2024-03-26T05:01:46.409306+0000","last_fullsized":"2024-03-26T05:01:46.409306+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:31.568722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:31.568722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.465689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":62,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735223+0000","last_change":"2024-03-26T05:00:26.956233+0000","last_active":"2024-03-26T05:01:46.735223+0000","last_peered":"2024-03-26T05:01:46.735223+0000","last_clean":"2024-03-26T05:01:46.735223+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:01:46.735223+0000","last_undegraded":"2024-03-26T05:01:46.735223+0000","last_fullsized":"2024-03-26T05:01:46.735223+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:54.400300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":69,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.761902+0000","last_change":"2024-03-26T05:00:22.770737+0000","last_active":"2024-03-26T05:01:46.761902+0000","last_peered":"2024-03-26T05:01:46.761902+0000","last_clean":"2024-03-26T05:01:46.761902+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:01:46.761902+0000","last_undegraded":"2024-03-26T05:01:46.761902+0000","last_fullsized":"2024-03-26T05:01:46.761902+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'21","reported_seq":117,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.384038+0000","last_change":"2024-03-26T05:01:38.722775+0000","last_active":"2024-03-26T05:01:54.384038+0000","last_peered":"2024-03-26T05:01:54.384038+0000","last_clean":"2024-03-26T05:01:54.384038+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:01:54.384038+0000","last_undegraded":"2024-03-26T05:01:54.384038+0000","last_fullsized":"2024-03-26T05:01:54.384038+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:01:38.722652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:38.722652+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.850545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":57,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762162+0000","last_change":"2024-03-26T05:00:29.462603+0000","last_active":"2024-03-26T05:01:46.762162+0000","last_peered":"2024-03-26T05:01:46.762162+0000","last_clean":"2024-03-26T05:01:46.762162+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:01:46.762162+0000","last_undegraded":"2024-03-26T05:01:46.762162+0000","last_fullsized":"2024-03-26T05:01:46.762162+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.502035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":62,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028536+0000","last_change":"2024-03-26T05:00:26.953694+0000","last_active":"2024-03-26T05:01:47.028536+0000","last_peered":"2024-03-26T05:01:47.028536+0000","last_clean":"2024-03-26T05:01:47.028536+0000","last_became_active":"2024-03-26T05:00:26.953416+0000","last_became_peered":"2024-03-26T05:00:26.953416+0000","last_unstale":"2024-03-26T05:01:47.028536+0000","last_undegraded":"2024-03-26T05:01:47.028536+0000","last_fullsized":"2024-03-26T05:01:47.028536+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.182281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":85,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.486873+0000","last_change":"2024-03-26T05:01:37.375697+0000","last_active":"2024-03-26T05:01:54.486873+0000","last_peered":"2024-03-26T05:01:54.486873+0000","last_clean":"2024-03-26T05:01:54.486873+0000","last_became_active":"2024-03-26T05:00:22.771080+0000","last_became_peered":"2024-03-26T05:00:22.771080+0000","last_unstale":"2024-03-26T05:01:54.486873+0000","last_undegraded":"2024-03-26T05:01:54.486873+0000","last_fullsized":"2024-03-26T05:01:54.486873+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:01:37.375533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.375533+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:51.717267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'12","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735330+0000","last_change":"2024-03-26T05:01:27.964049+0000","last_active":"2024-03-26T05:01:46.735330+0000","last_peered":"2024-03-26T05:01:46.735330+0000","last_clean":"2024-03-26T05:01:46.735330+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:01:46.735330+0000","last_undegraded":"2024-03-26T05:01:46.735330+0000","last_fullsized":"2024-03-26T05:01:46.735330+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:01:27.963954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:27.963954+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:50.330132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":68,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029083+0000","last_change":"2024-03-26T05:01:36.945807+0000","last_active":"2024-03-26T05:01:47.029083+0000","last_peered":"2024-03-26T05:01:47.029083+0000","last_clean":"2024-03-26T05:01:47.029083+0000","last_became_active":"2024-03-26T05:00:29.461059+0000","last_became_peered":"2024-03-26T05:00:29.461059+0000","last_unstale":"2024-03-26T05:01:47.029083+0000","last_undegraded":"2024-03-26T05:01:47.029083+0000","last_fullsized":"2024-03-26T05:01:47.029083+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:01:36.945708+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.001390+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'3","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762539+0000","last_change":"2024-03-26T05:01:22.799027+0000","last_active":"2024-03-26T05:01:46.762539+0000","last_peered":"2024-03-26T05:01:46.762539+0000","last_clean":"2024-03-26T05:01:46.762539+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:01:46.762539+0000","last_undegraded":"2024-03-26T05:01:46.762539+0000","last_fullsized":"2024-03-26T05:01:46.762539+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-03-26T05:01:22.798897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:22.798897+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.103651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'27","reported_seq":123,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028903+0000","last_change":"2024-03-26T05:01:29.048598+0000","last_active":"2024-03-26T05:01:47.028903+0000","last_peered":"2024-03-26T05:01:47.028903+0000","last_clean":"2024-03-26T05:01:47.028903+0000","last_became_active":"2024-03-26T05:00:25.651328+0000","last_became_peered":"2024-03-26T05:00:25.651328+0000","last_unstale":"2024-03-26T05:01:47.028903+0000","last_undegraded":"2024-03-26T05:01:47.028903+0000","last_fullsized":"2024-03-26T05:01:47.028903+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'27","last_scrub_stamp":"2024-03-26T05:01:29.048456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.048456+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:56.053198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":96,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:48.399573+0000","last_change":"2024-03-26T05:01:37.743111+0000","last_active":"2024-03-26T05:01:48.399573+0000","last_peered":"2024-03-26T05:01:48.399573+0000","last_clean":"2024-03-26T05:01:48.399573+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:01:48.399573+0000","last_undegraded":"2024-03-26T05:01:48.399573+0000","last_fullsized":"2024-03-26T05:01:48.399573+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:37.742995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.742995+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:07.111791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":66,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734506+0000","last_change":"2024-03-26T05:01:29.011852+0000","last_active":"2024-03-26T05:01:46.734506+0000","last_peered":"2024-03-26T05:01:46.734506+0000","last_clean":"2024-03-26T05:01:46.734506+0000","last_became_active":"2024-03-26T05:00:28.985917+0000","last_became_peered":"2024-03-26T05:00:28.985917+0000","last_unstale":"2024-03-26T05:01:46.734506+0000","last_undegraded":"2024-03-26T05:01:46.734506+0000","last_fullsized":"2024-03-26T05:01:46.734506+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.011757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.011757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.742465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":77,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.855772+0000","last_change":"2024-03-26T05:01:46.855799+0000","last_active":"2024-03-26T05:01:46.855772+0000","last_peered":"2024-03-26T05:01:46.855772+0000","last_clean":"2024-03-26T05:01:46.855772+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:01:46.855772+0000","last_undegraded":"2024-03-26T05:01:46.855772+0000","last_fullsized":"2024-03-26T05:01:46.855772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:46.855699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:46.855699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:15.236244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'24","reported_seq":116,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.386179+0000","last_change":"2024-03-26T05:00:24.944015+0000","last_active":"2024-03-26T05:01:54.386179+0000","last_peered":"2024-03-26T05:01:54.386179+0000","last_clean":"2024-03-26T05:01:54.386179+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:01:54.386179+0000","last_undegraded":"2024-03-26T05:01:54.386179+0000","last_fullsized":"2024-03-26T05:01:54.386179+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.408911+0000","last_change":"2024-03-26T05:01:26.618364+0000","last_active":"2024-03-26T05:01:46.408911+0000","last_peered":"2024-03-26T05:01:46.408911+0000","last_clean":"2024-03-26T05:01:46.408911+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:01:46.408911+0000","last_undegraded":"2024-03-26T05:01:46.408911+0000","last_fullsized":"2024-03-26T05:01:46.408911+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.618218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.618218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.093176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":57,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.761762+0000","last_change":"2024-03-26T05:00:28.981813+0000","last_active":"2024-03-26T05:01:46.761762+0000","last_peered":"2024-03-26T05:01:46.761762+0000","last_clean":"2024-03-26T05:01:46.761762+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:01:46.761762+0000","last_undegraded":"2024-03-26T05:01:46.761762+0000","last_fullsized":"2024-03-26T05:01:46.761762+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.155885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.409105+0000","last_change":"2024-03-26T05:01:33.638557+0000","last_active":"2024-03-26T05:01:46.409105+0000","last_peered":"2024-03-26T05:01:46.409105+0000","last_clean":"2024-03-26T05:01:46.409105+0000","last_became_active":"2024-03-26T05:00:26.961730+0000","last_became_peered":"2024-03-26T05:00:26.961730+0000","last_unstale":"2024-03-26T05:01:46.409105+0000","last_undegraded":"2024-03-26T05:01:46.409105+0000","last_fullsized":"2024-03-26T05:01:46.409105+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:33.638469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:33.638469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:00.135833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":69,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762440+0000","last_change":"2024-03-26T05:00:22.772945+0000","last_active":"2024-03-26T05:01:46.762440+0000","last_peered":"2024-03-26T05:01:46.762440+0000","last_clean":"2024-03-26T05:01:46.762440+0000","last_became_active":"2024-03-26T05:00:22.772590+0000","last_became_peered":"2024-03-26T05:00:22.772590+0000","last_unstale":"2024-03-26T05:01:46.762440+0000","last_undegraded":"2024-03-26T05:01:46.762440+0000","last_fullsized":"2024-03-26T05:01:46.762440+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'25","reported_seq":110,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.487047+0000","last_change":"2024-03-26T05:00:25.649083+0000","last_active":"2024-03-26T05:01:54.487047+0000","last_peered":"2024-03-26T05:01:54.487047+0000","last_clean":"2024-03-26T05:01:54.487047+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:01:54.487047+0000","last_undegraded":"2024-03-26T05:01:54.487047+0000","last_fullsized":"2024-03-26T05:01:54.487047+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":61,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383299+0000","last_change":"2024-03-26T05:00:28.981411+0000","last_active":"2024-03-26T05:01:54.383299+0000","last_peered":"2024-03-26T05:01:54.383299+0000","last_clean":"2024-03-26T05:01:54.383299+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:01:54.383299+0000","last_undegraded":"2024-03-26T05:01:54.383299+0000","last_fullsized":"2024-03-26T05:01:54.383299+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:55.410289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":62,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.409083+0000","last_change":"2024-03-26T05:00:26.956293+0000","last_active":"2024-03-26T05:01:46.409083+0000","last_peered":"2024-03-26T05:01:46.409083+0000","last_clean":"2024-03-26T05:01:46.409083+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:01:46.409083+0000","last_undegraded":"2024-03-26T05:01:46.409083+0000","last_fullsized":"2024-03-26T05:01:46.409083+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:54.998598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:53.133075+0000","last_change":"2024-03-26T05:01:53.133109+0000","last_active":"2024-03-26T05:01:53.133075+0000","last_peered":"2024-03-26T05:01:53.133075+0000","last_clean":"2024-03-26T05:01:53.133075+0000","last_became_active":"2024-03-26T05:00:22.773770+0000","last_became_peered":"2024-03-26T05:00:22.773770+0000","last_unstale":"2024-03-26T05:01:53.133075+0000","last_undegraded":"2024-03-26T05:01:53.133075+0000","last_fullsized":"2024-03-26T05:01:53.133075+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:53.133006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:53.133006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:10.552053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'15","reported_seq":105,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383936+0000","last_change":"2024-03-26T05:01:50.807516+0000","last_active":"2024-03-26T05:01:54.383936+0000","last_peered":"2024-03-26T05:01:54.383936+0000","last_clean":"2024-03-26T05:01:54.383936+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:01:54.383936+0000","last_undegraded":"2024-03-26T05:01:54.383936+0000","last_fullsized":"2024-03-26T05:01:54.383936+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:50.807430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:50.807430+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:03.240799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734991+0000","last_change":"2024-03-26T05:00:28.984072+0000","last_active":"2024-03-26T05:01:46.734991+0000","last_peered":"2024-03-26T05:01:46.734991+0000","last_clean":"2024-03-26T05:01:46.734991+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:01:46.734991+0000","last_undegraded":"2024-03-26T05:01:46.734991+0000","last_fullsized":"2024-03-26T05:01:46.734991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":89,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028958+0000","last_change":"2024-03-26T05:01:35.934149+0000","last_active":"2024-03-26T05:01:47.028958+0000","last_peered":"2024-03-26T05:01:47.028958+0000","last_clean":"2024-03-26T05:01:47.028958+0000","last_became_active":"2024-03-26T05:00:21.760521+0000","last_became_peered":"2024-03-26T05:00:21.760521+0000","last_unstale":"2024-03-26T05:01:47.028958+0000","last_undegraded":"2024-03-26T05:01:47.028958+0000","last_fullsized":"2024-03-26T05:01:47.028958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:01:35.934044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.934044+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:02.210221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":73,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.386000+0000","last_change":"2024-03-26T05:00:22.771224+0000","last_active":"2024-03-26T05:01:54.386000+0000","last_peered":"2024-03-26T05:01:54.386000+0000","last_clean":"2024-03-26T05:01:54.386000+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:01:54.386000+0000","last_undegraded":"2024-03-26T05:01:54.386000+0000","last_fullsized":"2024-03-26T05:01:54.386000+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'34","reported_seq":155,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484733+0000","last_change":"2024-03-26T05:01:40.826966+0000","last_active":"2024-03-26T05:01:54.484733+0000","last_peered":"2024-03-26T05:01:54.484733+0000","last_clean":"2024-03-26T05:01:54.484733+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:01:54.484733+0000","last_undegraded":"2024-03-26T05:01:54.484733+0000","last_fullsized":"2024-03-26T05:01:54.484733+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'34","last_scrub_stamp":"2024-03-26T05:01:40.826856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.826856+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.349406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":65,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383764+0000","last_change":"2024-03-26T05:00:26.964781+0000","last_active":"2024-03-26T05:01:54.383764+0000","last_peered":"2024-03-26T05:01:54.383764+0000","last_clean":"2024-03-26T05:01:54.383764+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:01:54.383764+0000","last_undegraded":"2024-03-26T05:01:54.383764+0000","last_fullsized":"2024-03-26T05:01:54.383764+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":65,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762516+0000","last_change":"2024-03-26T05:01:41.789755+0000","last_active":"2024-03-26T05:01:46.762516+0000","last_peered":"2024-03-26T05:01:46.762516+0000","last_clean":"2024-03-26T05:01:46.762516+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:01:46.762516+0000","last_undegraded":"2024-03-26T05:01:46.762516+0000","last_fullsized":"2024-03-26T05:01:46.762516+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:41.789650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.789650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.302843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":72,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484398+0000","last_change":"2024-03-26T05:00:23.346922+0000","last_active":"2024-03-26T05:01:54.484398+0000","last_peered":"2024-03-26T05:01:54.484398+0000","last_clean":"2024-03-26T05:01:54.484398+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:01:54.484398+0000","last_undegraded":"2024-03-26T05:01:54.484398+0000","last_fullsized":"2024-03-26T05:01:54.484398+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.408962+0000","last_change":"2024-03-26T05:01:37.678850+0000","last_active":"2024-03-26T05:01:46.408962+0000","last_peered":"2024-03-26T05:01:46.408962+0000","last_clean":"2024-03-26T05:01:46.408962+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:01:46.408962+0000","last_undegraded":"2024-03-26T05:01:46.408962+0000","last_fullsized":"2024-03-26T05:01:46.408962+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.678736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.678736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.820757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'19","reported_seq":107,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484800+0000","last_change":"2024-03-26T05:00:24.937826+0000","last_active":"2024-03-26T05:01:54.484800+0000","last_peered":"2024-03-26T05:01:54.484800+0000","last_clean":"2024-03-26T05:01:54.484800+0000","last_became_active":"2024-03-26T05:00:24.937145+0000","last_became_peered":"2024-03-26T05:00:24.937145+0000","last_unstale":"2024-03-26T05:01:54.484800+0000","last_undegraded":"2024-03-26T05:01:54.484800+0000","last_fullsized":"2024-03-26T05:01:54.484800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.409299+0000","last_change":"2024-03-26T05:01:29.627760+0000","last_active":"2024-03-26T05:01:46.409299+0000","last_peered":"2024-03-26T05:01:46.409299+0000","last_clean":"2024-03-26T05:01:46.409299+0000","last_became_active":"2024-03-26T05:00:26.957936+0000","last_became_peered":"2024-03-26T05:00:26.957936+0000","last_unstale":"2024-03-26T05:01:46.409299+0000","last_undegraded":"2024-03-26T05:01:46.409299+0000","last_fullsized":"2024-03-26T05:01:46.409299+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.627680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.627680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.292786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029318+0000","last_change":"2024-03-26T05:00:28.979759+0000","last_active":"2024-03-26T05:01:47.029318+0000","last_peered":"2024-03-26T05:01:47.029318+0000","last_clean":"2024-03-26T05:01:47.029318+0000","last_became_active":"2024-03-26T05:00:28.979373+0000","last_became_peered":"2024-03-26T05:00:28.979373+0000","last_unstale":"2024-03-26T05:01:47.029318+0000","last_undegraded":"2024-03-26T05:01:47.029318+0000","last_fullsized":"2024-03-26T05:01:47.029318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":324,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.761880+0000","last_change":"2024-03-26T05:01:04.782428+0000","last_active":"2024-03-26T05:01:46.761880+0000","last_peered":"2024-03-26T05:01:46.761880+0000","last_clean":"2024-03-26T05:01:46.761880+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:01:46.761880+0000","last_undegraded":"2024-03-26T05:01:46.761880+0000","last_fullsized":"2024-03-26T05:01:46.761880+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-03-26T05:01:04.782336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:01:04.782336+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:15.009834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734559+0000","last_change":"2024-03-26T05:01:25.941429+0000","last_active":"2024-03-26T05:01:46.734559+0000","last_peered":"2024-03-26T05:01:46.734559+0000","last_clean":"2024-03-26T05:01:46.734559+0000","last_became_active":"2024-03-26T05:00:22.774298+0000","last_became_peered":"2024-03-26T05:00:22.774298+0000","last_unstale":"2024-03-26T05:01:46.734559+0000","last_undegraded":"2024-03-26T05:01:46.734559+0000","last_fullsized":"2024-03-26T05:01:46.734559+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:25.941344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:25.941344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:34.075498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":108,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.487316+0000","last_change":"2024-03-26T05:00:25.648924+0000","last_active":"2024-03-26T05:01:54.487316+0000","last_peered":"2024-03-26T05:01:54.487316+0000","last_clean":"2024-03-26T05:01:54.487316+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:01:54.487316+0000","last_undegraded":"2024-03-26T05:01:54.487316+0000","last_fullsized":"2024-03-26T05:01:54.487316+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":61,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762259+0000","last_change":"2024-03-26T05:00:26.962047+0000","last_active":"2024-03-26T05:01:46.762259+0000","last_peered":"2024-03-26T05:01:46.762259+0000","last_clean":"2024-03-26T05:01:46.762259+0000","last_became_active":"2024-03-26T05:00:26.961724+0000","last_became_peered":"2024-03-26T05:00:26.961724+0000","last_unstale":"2024-03-26T05:01:46.762259+0000","last_undegraded":"2024-03-26T05:01:46.762259+0000","last_fullsized":"2024-03-26T05:01:46.762259+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":68,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484534+0000","last_change":"2024-03-26T05:01:38.822538+0000","last_active":"2024-03-26T05:01:54.484534+0000","last_peered":"2024-03-26T05:01:54.484534+0000","last_clean":"2024-03-26T05:01:54.484534+0000","last_became_active":"2024-03-26T05:00:28.983202+0000","last_became_peered":"2024-03-26T05:00:28.983202+0000","last_unstale":"2024-03-26T05:01:54.484534+0000","last_undegraded":"2024-03-26T05:01:54.484534+0000","last_fullsized":"2024-03-26T05:01:54.484534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:38.822409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:38.822409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:41.940538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383575+0000","last_change":"2024-03-26T05:01:25.629227+0000","last_active":"2024-03-26T05:01:54.383575+0000","last_peered":"2024-03-26T05:01:54.383575+0000","last_clean":"2024-03-26T05:01:54.383575+0000","last_became_active":"2024-03-26T05:00:22.773081+0000","last_became_peered":"2024-03-26T05:00:22.773081+0000","last_unstale":"2024-03-26T05:01:54.383575+0000","last_undegraded":"2024-03-26T05:01:54.383575+0000","last_fullsized":"2024-03-26T05:01:54.383575+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:25.629101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:25.629101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:43.255919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":176,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:53.255274+0000","last_change":"2024-03-26T05:01:36.780629+0000","last_active":"2024-03-26T05:01:53.255274+0000","last_peered":"2024-03-26T05:01:53.255274+0000","last_clean":"2024-03-26T05:01:53.255274+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:01:53.255274+0000","last_undegraded":"2024-03-26T05:01:53.255274+0000","last_fullsized":"2024-03-26T05:01:53.255274+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:01:36.780518+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.077309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":107,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383945+0000","last_change":"2024-03-26T05:00:49.653404+0000","last_active":"2024-03-26T05:01:54.383945+0000","last_peered":"2024-03-26T05:01:54.383945+0000","last_clean":"2024-03-26T05:01:54.383945+0000","last_became_active":"2024-03-26T05:00:49.642276+0000","last_became_peered":"2024-03-26T05:00:49.642276+0000","last_unstale":"2024-03-26T05:01:54.383945+0000","last_undegraded":"2024-03-26T05:01:54.383945+0000","last_fullsized":"2024-03-26T05:01:54.383945+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028588+0000","last_change":"2024-03-26T05:01:41.973895+0000","last_active":"2024-03-26T05:01:47.028588+0000","last_peered":"2024-03-26T05:01:47.028588+0000","last_clean":"2024-03-26T05:01:47.028588+0000","last_became_active":"2024-03-26T05:00:26.956677+0000","last_became_peered":"2024-03-26T05:00:26.956677+0000","last_unstale":"2024-03-26T05:01:47.028588+0000","last_undegraded":"2024-03-26T05:01:47.028588+0000","last_fullsized":"2024-03-26T05:01:47.028588+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:41.973805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.973805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:54.658519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":60,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484255+0000","last_change":"2024-03-26T05:00:29.461548+0000","last_active":"2024-03-26T05:01:54.484255+0000","last_peered":"2024-03-26T05:01:54.484255+0000","last_clean":"2024-03-26T05:01:54.484255+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:01:54.484255+0000","last_undegraded":"2024-03-26T05:01:54.484255+0000","last_fullsized":"2024-03-26T05:01:54.484255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484683+0000","last_change":"2024-03-26T05:01:35.745231+0000","last_active":"2024-03-26T05:01:54.484683+0000","last_peered":"2024-03-26T05:01:54.484683+0000","last_clean":"2024-03-26T05:01:54.484683+0000","last_became_active":"2024-03-26T05:00:23.346665+0000","last_became_peered":"2024-03-26T05:00:23.346665+0000","last_unstale":"2024-03-26T05:01:54.484683+0000","last_undegraded":"2024-03-26T05:01:54.484683+0000","last_fullsized":"2024-03-26T05:01:54.484683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.745100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.314539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'25","reported_seq":125,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383696+0000","last_change":"2024-03-26T05:01:44.739144+0000","last_active":"2024-03-26T05:01:54.383696+0000","last_peered":"2024-03-26T05:01:54.383696+0000","last_clean":"2024-03-26T05:01:54.383696+0000","last_became_active":"2024-03-26T05:00:25.648253+0000","last_became_peered":"2024-03-26T05:00:25.648253+0000","last_unstale":"2024-03-26T05:01:54.383696+0000","last_undegraded":"2024-03-26T05:01:54.383696+0000","last_fullsized":"2024-03-26T05:01:54.383696+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:01:44.739046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.739046+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:06.858172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":63,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.487066+0000","last_change":"2024-03-26T05:00:26.958976+0000","last_active":"2024-03-26T05:01:54.487066+0000","last_peered":"2024-03-26T05:01:54.487066+0000","last_clean":"2024-03-26T05:01:54.487066+0000","last_became_active":"2024-03-26T05:00:26.958689+0000","last_became_peered":"2024-03-26T05:00:26.958689+0000","last_unstale":"2024-03-26T05:01:54.487066+0000","last_undegraded":"2024-03-26T05:01:54.487066+0000","last_fullsized":"2024-03-26T05:01:54.487066+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":69,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383262+0000","last_change":"2024-03-26T05:01:46.735926+0000","last_active":"2024-03-26T05:01:54.383262+0000","last_peered":"2024-03-26T05:01:54.383262+0000","last_clean":"2024-03-26T05:01:54.383262+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:01:54.383262+0000","last_undegraded":"2024-03-26T05:01:54.383262+0000","last_fullsized":"2024-03-26T05:01:54.383262+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:46.735848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:46.735848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:09.041185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.084891+0000","last_change":"2024-03-26T05:01:54.084913+0000","last_active":"2024-03-26T05:01:54.084891+0000","last_peered":"2024-03-26T05:01:54.084891+0000","last_clean":"2024-03-26T05:01:54.084891+0000","last_became_active":"2024-03-26T05:00:23.346876+0000","last_became_peered":"2024-03-26T05:00:23.346876+0000","last_unstale":"2024-03-26T05:01:54.084891+0000","last_undegraded":"2024-03-26T05:01:54.084891+0000","last_fullsized":"2024-03-26T05:01:54.084891+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:54.084818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:54.084818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:10.773277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'16","reported_seq":64,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.408955+0000","last_change":"2024-03-26T05:01:42.753638+0000","last_active":"2024-03-26T05:01:46.408955+0000","last_peered":"2024-03-26T05:01:46.408955+0000","last_clean":"2024-03-26T05:01:46.408955+0000","last_became_active":"2024-03-26T05:00:49.641444+0000","last_became_peered":"2024-03-26T05:00:49.641444+0000","last_unstale":"2024-03-26T05:01:46.408955+0000","last_undegraded":"2024-03-26T05:01:46.408955+0000","last_fullsized":"2024-03-26T05:01:46.408955+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:01:42.753523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:42.753523+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.891497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":92,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.384032+0000","last_change":"2024-03-26T05:00:26.962263+0000","last_active":"2024-03-26T05:01:54.384032+0000","last_peered":"2024-03-26T05:01:54.384032+0000","last_clean":"2024-03-26T05:01:54.384032+0000","last_became_active":"2024-03-26T05:00:26.961907+0000","last_became_peered":"2024-03-26T05:00:26.961907+0000","last_unstale":"2024-03-26T05:01:54.384032+0000","last_undegraded":"2024-03-26T05:01:54.384032+0000","last_fullsized":"2024-03-26T05:01:54.384032+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":57,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.762259+0000","last_change":"2024-03-26T05:00:29.461447+0000","last_active":"2024-03-26T05:01:46.762259+0000","last_peered":"2024-03-26T05:01:46.762259+0000","last_clean":"2024-03-26T05:01:46.762259+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:01:46.762259+0000","last_undegraded":"2024-03-26T05:01:46.762259+0000","last_fullsized":"2024-03-26T05:01:46.762259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:57.420912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":82,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.383949+0000","last_change":"2024-03-26T05:01:29.581397+0000","last_active":"2024-03-26T05:01:54.383949+0000","last_peered":"2024-03-26T05:01:54.383949+0000","last_clean":"2024-03-26T05:01:54.383949+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:01:54.383949+0000","last_undegraded":"2024-03-26T05:01:54.383949+0000","last_fullsized":"2024-03-26T05:01:54.383949+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:01:29.581309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.581309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.525431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'20","reported_seq":102,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.385910+0000","last_change":"2024-03-26T05:00:24.936644+0000","last_active":"2024-03-26T05:01:54.385910+0000","last_peered":"2024-03-26T05:01:54.385910+0000","last_clean":"2024-03-26T05:01:54.385910+0000","last_became_active":"2024-03-26T05:00:24.936315+0000","last_became_peered":"2024-03-26T05:00:24.936315+0000","last_unstale":"2024-03-26T05:01:54.385910+0000","last_undegraded":"2024-03-26T05:01:54.385910+0000","last_fullsized":"2024-03-26T05:01:54.385910+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":89,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:48.399485+0000","last_change":"2024-03-26T05:00:26.957814+0000","last_active":"2024-03-26T05:01:48.399485+0000","last_peered":"2024-03-26T05:01:48.399485+0000","last_clean":"2024-03-26T05:01:48.399485+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:01:48.399485+0000","last_undegraded":"2024-03-26T05:01:48.399485+0000","last_fullsized":"2024-03-26T05:01:48.399485+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:55.181714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":69,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.385757+0000","last_change":"2024-03-26T05:01:48.472310+0000","last_active":"2024-03-26T05:01:54.385757+0000","last_peered":"2024-03-26T05:01:54.385757+0000","last_clean":"2024-03-26T05:01:54.385757+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:01:54.385757+0000","last_undegraded":"2024-03-26T05:01:54.385757+0000","last_fullsized":"2024-03-26T05:01:54.385757+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:48.472188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:48.472188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:03.679495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.735088+0000","last_change":"2024-03-26T05:00:22.779020+0000","last_active":"2024-03-26T05:01:46.735088+0000","last_peered":"2024-03-26T05:01:46.735088+0000","last_clean":"2024-03-26T05:01:46.735088+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:01:46.735088+0000","last_undegraded":"2024-03-26T05:01:46.735088+0000","last_fullsized":"2024-03-26T05:01:46.735088+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:48.991339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"96'40","reported_seq":140,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734554+0000","last_change":"2024-03-26T05:01:42.152769+0000","last_active":"2024-03-26T05:01:46.734554+0000","last_peered":"2024-03-26T05:01:46.734554+0000","last_clean":"2024-03-26T05:01:46.734554+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:01:46.734554+0000","last_undegraded":"2024-03-26T05:01:46.734554+0000","last_fullsized":"2024-03-26T05:01:46.734554+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:01:42.152630+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:01:42.152630+0000","last_clean_scrub_stamp":"2024-03-26T05:01:42.152630+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.656601+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":65,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.385707+0000","last_change":"2024-03-26T05:00:26.959009+0000","last_active":"2024-03-26T05:01:54.385707+0000","last_peered":"2024-03-26T05:01:54.385707+0000","last_clean":"2024-03-26T05:01:54.385707+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:01:54.385707+0000","last_undegraded":"2024-03-26T05:01:54.385707+0000","last_fullsized":"2024-03-26T05:01:54.385707+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.029034+0000","last_change":"2024-03-26T05:00:28.983093+0000","last_active":"2024-03-26T05:01:47.029034+0000","last_peered":"2024-03-26T05:01:47.029034+0000","last_clean":"2024-03-26T05:01:47.029034+0000","last_became_active":"2024-03-26T05:00:28.982825+0000","last_became_peered":"2024-03-26T05:00:28.982825+0000","last_unstale":"2024-03-26T05:01:47.029034+0000","last_undegraded":"2024-03-26T05:01:47.029034+0000","last_fullsized":"2024-03-26T05:01:47.029034+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:52.359012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":100,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.387044+0000","last_change":"2024-03-26T05:01:41.485196+0000","last_active":"2024-03-26T05:01:54.387044+0000","last_peered":"2024-03-26T05:01:54.387044+0000","last_clean":"2024-03-26T05:01:54.387044+0000","last_became_active":"2024-03-26T05:00:26.956596+0000","last_became_peered":"2024-03-26T05:00:26.956596+0000","last_unstale":"2024-03-26T05:01:54.387044+0000","last_undegraded":"2024-03-26T05:01:54.387044+0000","last_fullsized":"2024-03-26T05:01:54.387044+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:41.485086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.485086+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:59.948091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734406+0000","last_change":"2024-03-26T05:00:22.775050+0000","last_active":"2024-03-26T05:01:46.734406+0000","last_peered":"2024-03-26T05:01:46.734406+0000","last_clean":"2024-03-26T05:01:46.734406+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:01:46.734406+0000","last_undegraded":"2024-03-26T05:01:46.734406+0000","last_fullsized":"2024-03-26T05:01:46.734406+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":101,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484984+0000","last_change":"2024-03-26T05:01:28.754168+0000","last_active":"2024-03-26T05:01:54.484984+0000","last_peered":"2024-03-26T05:01:54.484984+0000","last_clean":"2024-03-26T05:01:54.484984+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:01:54.484984+0000","last_undegraded":"2024-03-26T05:01:54.484984+0000","last_fullsized":"2024-03-26T05:01:54.484984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:28.754068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:28.754068+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:40.695752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734539+0000","last_change":"2024-03-26T05:00:29.461578+0000","last_active":"2024-03-26T05:01:46.734539+0000","last_peered":"2024-03-26T05:01:46.734539+0000","last_clean":"2024-03-26T05:01:46.734539+0000","last_became_active":"2024-03-26T05:00:29.461236+0000","last_became_peered":"2024-03-26T05:00:29.461236+0000","last_unstale":"2024-03-26T05:01:46.734539+0000","last_undegraded":"2024-03-26T05:01:46.734539+0000","last_fullsized":"2024-03-26T05:01:46.734539+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:49.354548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:47.028708+0000","last_change":"2024-03-26T05:01:45.085244+0000","last_active":"2024-03-26T05:01:47.028708+0000","last_peered":"2024-03-26T05:01:47.028708+0000","last_clean":"2024-03-26T05:01:47.028708+0000","last_became_active":"2024-03-26T05:00:26.955805+0000","last_became_peered":"2024-03-26T05:00:26.955805+0000","last_unstale":"2024-03-26T05:01:47.028708+0000","last_undegraded":"2024-03-26T05:01:47.028708+0000","last_fullsized":"2024-03-26T05:01:47.028708+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:45.085111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:45.085111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.981772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.385768+0000","last_change":"2024-03-26T05:01:26.442833+0000","last_active":"2024-03-26T05:01:54.385768+0000","last_peered":"2024-03-26T05:01:54.385768+0000","last_clean":"2024-03-26T05:01:54.385768+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:01:54.385768+0000","last_undegraded":"2024-03-26T05:01:54.385768+0000","last_fullsized":"2024-03-26T05:01:54.385768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.442722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.442722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:39.204290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'17","reported_seq":108,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.489354+0000","last_change":"2024-03-26T05:01:54.489384+0000","last_active":"2024-03-26T05:01:54.489354+0000","last_peered":"2024-03-26T05:01:54.489354+0000","last_clean":"2024-03-26T05:01:54.489354+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:01:54.489354+0000","last_undegraded":"2024-03-26T05:01:54.489354+0000","last_fullsized":"2024-03-26T05:01:54.489354+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:01:54.489297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:54.489297+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.631248+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":66,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:53.801100+0000","last_change":"2024-03-26T05:01:53.801119+0000","last_active":"2024-03-26T05:01:53.801100+0000","last_peered":"2024-03-26T05:01:53.801100+0000","last_clean":"2024-03-26T05:01:53.801100+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:01:53.801100+0000","last_undegraded":"2024-03-26T05:01:53.801100+0000","last_fullsized":"2024-03-26T05:01:53.801100+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:53.801041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:53.801041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:03.065236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":72,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.484826+0000","last_change":"2024-03-26T05:01:37.774253+0000","last_active":"2024-03-26T05:01:54.484826+0000","last_peered":"2024-03-26T05:01:54.484826+0000","last_clean":"2024-03-26T05:01:54.484826+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:01:54.484826+0000","last_undegraded":"2024-03-26T05:01:54.484826+0000","last_fullsized":"2024-03-26T05:01:54.484826+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.774158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.774158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:01.227552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":79,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.486930+0000","last_change":"2024-03-26T05:01:28.386278+0000","last_active":"2024-03-26T05:01:54.486930+0000","last_peered":"2024-03-26T05:01:54.486930+0000","last_clean":"2024-03-26T05:01:54.486930+0000","last_became_active":"2024-03-26T05:00:22.768610+0000","last_became_peered":"2024-03-26T05:00:22.768610+0000","last_unstale":"2024-03-26T05:01:54.486930+0000","last_undegraded":"2024-03-26T05:01:54.486930+0000","last_fullsized":"2024-03-26T05:01:54.486930+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_clean_scrub_stamp":"2024-03-26T05:01:28.386172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.056496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":97,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.386639+0000","last_change":"2024-03-26T05:00:24.942796+0000","last_active":"2024-03-26T05:01:54.386639+0000","last_peered":"2024-03-26T05:01:54.386639+0000","last_clean":"2024-03-26T05:01:54.386639+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:01:54.386639+0000","last_undegraded":"2024-03-26T05:01:54.386639+0000","last_fullsized":"2024-03-26T05:01:54.386639+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":61,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.387110+0000","last_change":"2024-03-26T05:00:28.978772+0000","last_active":"2024-03-26T05:01:54.387110+0000","last_peered":"2024-03-26T05:01:54.387110+0000","last_clean":"2024-03-26T05:01:54.387110+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:01:54.387110+0000","last_undegraded":"2024-03-26T05:01:54.387110+0000","last_fullsized":"2024-03-26T05:01:54.387110+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":97,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:48.399652+0000","last_change":"2024-03-26T05:01:43.782577+0000","last_active":"2024-03-26T05:01:48.399652+0000","last_peered":"2024-03-26T05:01:48.399652+0000","last_clean":"2024-03-26T05:01:48.399652+0000","last_became_active":"2024-03-26T05:00:26.957142+0000","last_became_peered":"2024-03-26T05:00:26.957142+0000","last_unstale":"2024-03-26T05:01:48.399652+0000","last_undegraded":"2024-03-26T05:01:48.399652+0000","last_fullsized":"2024-03-26T05:01:48.399652+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:43.782490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:43.782490+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.086882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-26T05:01:46.734806+0000","last_change":"2024-03-26T05:01:39.142373+0000","last_active":"2024-03-26T05:01:46.734806+0000","last_peered":"2024-03-26T05:01:46.734806+0000","last_clean":"2024-03-26T05:01:46.734806+0000","last_became_active":"2024-03-26T05:00:22.778719+0000","last_became_peered":"2024-03-26T05:00:22.778719+0000","last_unstale":"2024-03-26T05:01:46.734806+0000","last_undegraded":"2024-03-26T05:01:46.734806+0000","last_fullsized":"2024-03-26T05:01:46.734806+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.142249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.142249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.076132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":92,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-26T05:01:54.386413+0000","last_change":"2024-03-26T05:01:44.403960+0000","last_active":"2024-03-26T05:01:54.386413+0000","last_peered":"2024-03-26T05:01:54.386413+0000","last_clean":"2024-03-26T05:01:54.386413+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:01:54.386413+0000","last_undegraded":"2024-03-26T05:01:54.386413+0000","last_fullsized":"2024-03-26T05:01:54.386413+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:01:44.403798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.403798+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:04.259798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":607,"ondisk_log_size":607,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005079,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1079903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332124,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32736,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712416,"statfs":{"total":95995035648,"available":95961513984,"internally_reserved":0,"allocated":1597440,"data_stored":1078387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":489283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1626112,"data_stored":1080056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":487833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117035,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":488499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920807,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":496846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659169,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":488345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:02:00.292 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:02:00.530 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:00.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:00 smithi132 ceph-mon[33500]: 5.10 scrub starts 2024-03-26T05:02:00.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:00 smithi132 ceph-mon[33500]: 5.10 scrub ok 2024-03-26T05:02:00.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:00 smithi132 ceph-mon[33500]: 5.2 deep-scrub starts 2024-03-26T05:02:00.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:00 smithi132 ceph-mon[33500]: 5.2 deep-scrub ok 2024-03-26T05:02:00.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:00 smithi132 ceph-mon[33500]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:02:00.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:00 smithi132 ceph-mon[33500]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:00.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:00 smithi080 ceph-mon[29691]: 5.10 scrub starts 2024-03-26T05:02:00.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:00 smithi080 ceph-mon[29691]: 5.10 scrub ok 2024-03-26T05:02:00.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:00 smithi080 ceph-mon[29691]: 5.2 deep-scrub starts 2024-03-26T05:02:00.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:00 smithi080 ceph-mon[29691]: 5.2 deep-scrub ok 2024-03-26T05:02:00.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:00 smithi080 ceph-mon[29691]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:02:00.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:00 smithi080 ceph-mon[29691]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:00.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:00 smithi080 ceph-mon[40404]: 5.10 scrub starts 2024-03-26T05:02:00.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:00 smithi080 ceph-mon[40404]: 5.10 scrub ok 2024-03-26T05:02:00.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:00 smithi080 ceph-mon[40404]: 5.2 deep-scrub starts 2024-03-26T05:02:00.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:00 smithi080 ceph-mon[40404]: 5.2 deep-scrub ok 2024-03-26T05:02:00.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:00 smithi080 ceph-mon[40404]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:02:00.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:00 smithi080 ceph-mon[40404]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:01.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:01 smithi132 ceph-mon[33500]: 5.16 scrub starts 2024-03-26T05:02:01.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:01 smithi132 ceph-mon[33500]: 5.16 scrub ok 2024-03-26T05:02:01.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:01 smithi132 ceph-mon[33500]: 4.5 deep-scrub starts 2024-03-26T05:02:01.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:01 smithi132 ceph-mon[33500]: 4.5 deep-scrub ok 2024-03-26T05:02:01.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:01 smithi132 ceph-mon[33500]: 5.3 deep-scrub starts 2024-03-26T05:02:01.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:01 smithi132 ceph-mon[33500]: 5.3 deep-scrub ok 2024-03-26T05:02:01.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:01 smithi132 ceph-mon[33500]: 6.6 scrub starts 2024-03-26T05:02:01.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:01 smithi132 ceph-mon[33500]: 6.6 scrub ok 2024-03-26T05:02:01.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[29691]: 5.16 scrub starts 2024-03-26T05:02:01.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[29691]: 5.16 scrub ok 2024-03-26T05:02:01.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[29691]: 4.5 deep-scrub starts 2024-03-26T05:02:01.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[29691]: 4.5 deep-scrub ok 2024-03-26T05:02:01.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[29691]: 5.3 deep-scrub starts 2024-03-26T05:02:01.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[29691]: 5.3 deep-scrub ok 2024-03-26T05:02:01.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[29691]: 6.6 scrub starts 2024-03-26T05:02:01.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[29691]: 6.6 scrub ok 2024-03-26T05:02:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[40404]: 5.16 scrub starts 2024-03-26T05:02:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[40404]: 5.16 scrub ok 2024-03-26T05:02:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[40404]: 4.5 deep-scrub starts 2024-03-26T05:02:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[40404]: 4.5 deep-scrub ok 2024-03-26T05:02:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[40404]: 5.3 deep-scrub starts 2024-03-26T05:02:01.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[40404]: 5.3 deep-scrub ok 2024-03-26T05:02:01.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[40404]: 6.6 scrub starts 2024-03-26T05:02:01.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:01 smithi080 ceph-mon[40404]: 6.6 scrub ok 2024-03-26T05:02:02.454 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 podman[63823]: 2024-03-26 05:02:02.33090029 +0000 UTC m=+0.017310164 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-26T05:02:02.455 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 podman[63823]: 2024-03-26 05:02:02.462090406 +0000 UTC m=+0.148500263 container create 3b5402cd4a1ba9f216040ca62203b2bd505af7e4e1937624e7411c2a338aeebe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, description=Ceph Grafana Container, maintainer=Paul Cuzner , distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, release=239.1651231664, io.openshift.tags=base rhel8, vcs-type=git, io.buildah.version=1.32.0, name=ubi8, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897) 2024-03-26T05:02:02.484 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:02:02.487 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:02:02.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-mon[33500]: 4.12 scrub starts 2024-03-26T05:02:02.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-mon[33500]: 4.12 scrub ok 2024-03-26T05:02:02.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-mon[33500]: 5.4 scrub starts 2024-03-26T05:02:02.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-mon[33500]: 5.4 scrub ok 2024-03-26T05:02:02.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-mon[33500]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:02.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-mon[33500]: 6.13 scrub starts 2024-03-26T05:02:02.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-mon[33500]: 6.13 scrub ok 2024-03-26T05:02:02.774 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 podman[63823]: 2024-03-26 05:02:02.706943262 +0000 UTC m=+0.393353121 container init 3b5402cd4a1ba9f216040ca62203b2bd505af7e4e1937624e7411c2a338aeebe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a, com.redhat.component=ubi8-container, io.openshift.expose-services=, release=239.1651231664, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, description=Ceph Grafana Container, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0) 2024-03-26T05:02:02.774 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 podman[63823]: 2024-03-26 05:02:02.710084964 +0000 UTC m=+0.396494822 container start 3b5402cd4a1ba9f216040ca62203b2bd505af7e4e1937624e7411c2a338aeebe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a, io.openshift.expose-services=, io.openshift.tags=base rhel8, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, description=Ceph Grafana Container, vendor=Red Hat, Inc., maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-26T05:02:02.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[40404]: 4.12 scrub starts 2024-03-26T05:02:02.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[40404]: 4.12 scrub ok 2024-03-26T05:02:02.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[40404]: 5.4 scrub starts 2024-03-26T05:02:02.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[40404]: 5.4 scrub ok 2024-03-26T05:02:02.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[40404]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:02.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[40404]: 6.13 scrub starts 2024-03-26T05:02:02.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[40404]: 6.13 scrub ok 2024-03-26T05:02:02.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[29691]: 4.12 scrub starts 2024-03-26T05:02:02.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[29691]: 4.12 scrub ok 2024-03-26T05:02:02.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[29691]: 5.4 scrub starts 2024-03-26T05:02:02.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[29691]: 5.4 scrub ok 2024-03-26T05:02:02.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[29691]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:02.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[29691]: 6.13 scrub starts 2024-03-26T05:02:02.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:02 smithi080 ceph-mon[29691]: 6.13 scrub ok 2024-03-26T05:02:03.026 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 bash[63823]: 3b5402cd4a1ba9f216040ca62203b2bd505af7e4e1937624e7411c2a338aeebe 2024-03-26T05:02:03.026 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 systemd[1]: Started Ceph grafana.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:02:03.026 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.907229527Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T05:02:03.026 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.907435847Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.907453418Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.907460629Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.907466456Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.907472585Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.90747792Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.907483694Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.907489334Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.907495293Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.907500464Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.90750557Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.907510873Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=settings t=2024-03-26T05:02:02.907516178Z level=info msg="App mode production" 2024-03-26T05:02:03.027 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=sqlstore t=2024-03-26T05:02:02.907562963Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T05:02:03.028 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=sqlstore t=2024-03-26T05:02:02.907581944Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T05:02:03.028 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.908411985Z level=info msg="Starting DB migrations" 2024-03-26T05:02:03.028 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.908485772Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T05:02:03.028 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.909027707Z level=info msg="Executing migration" id="create user table" 2024-03-26T05:02:03.028 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.90948291Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.913301834Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.91706786Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.921014389Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.924938669Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.92863879Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.932566464Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.936715296Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.940999233Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.942785794Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.944609093Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.946502765Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.948354447Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.948798834Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T05:02:03.029 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.950635236Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.952439231Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.954310607Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.956093033Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.957994362Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.959842207Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.961651592Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.963527757Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.965335882Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.967470348Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.969271724Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.971119345Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.973333834Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.975120747Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.977034761Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T05:02:03.030 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.979144787Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.981340238Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.983216935Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.985091776Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.987027544Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.988879312Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.990716488Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.992628239Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.994501333Z level=info msg="Executing migration" id="create star table" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.994881048Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.996789166Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:02.998625584Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.000491603Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.002660024Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T05:02:03.031 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.004838546Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T05:02:03.032 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.008612576Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T05:02:03.032 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.009042659Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T05:02:03.032 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.013325918Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T05:02:03.032 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.017341184Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T05:02:03.032 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.021631171Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T05:02:03.032 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.022024424Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T05:02:03.032 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.025891564Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T05:02:03.032 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.029698811Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T05:02:03.032 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.03364137Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T05:02:03.032 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.034203775Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T05:02:03.210 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-03-26T05:02:01.716049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1313,"num_read_kb":1127,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":818,"ondisk_log_size":818,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":246876,"kb_used_data":9916,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749714340,"statfs":{"total":767960285184,"available":767707484160,"internally_reserved":0,"allocated":10153984,"data_stored":5805122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002022"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":63,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.315336+0000","last_change":"2024-03-26T05:00:28.973250+0000","last_active":"2024-03-26T05:01:56.315336+0000","last_peered":"2024-03-26T05:01:56.315336+0000","last_clean":"2024-03-26T05:01:56.315336+0000","last_became_active":"2024-03-26T05:00:28.972969+0000","last_became_peered":"2024-03-26T05:00:28.972969+0000","last_unstale":"2024-03-26T05:01:56.315336+0000","last_undegraded":"2024-03-26T05:01:56.315336+0000","last_fullsized":"2024-03-26T05:01:56.315336+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.806173+0000","last_change":"2024-03-26T05:01:39.812600+0000","last_active":"2024-03-26T05:01:56.806173+0000","last_peered":"2024-03-26T05:01:56.806173+0000","last_clean":"2024-03-26T05:01:56.806173+0000","last_became_active":"2024-03-26T05:00:26.953154+0000","last_became_peered":"2024-03-26T05:00:26.953154+0000","last_unstale":"2024-03-26T05:01:56.806173+0000","last_undegraded":"2024-03-26T05:01:56.806173+0000","last_fullsized":"2024-03-26T05:01:56.806173+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.812509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.812509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:45.654690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.319173+0000","last_change":"2024-03-26T05:01:35.658900+0000","last_active":"2024-03-26T05:01:56.319173+0000","last_peered":"2024-03-26T05:01:56.319173+0000","last_clean":"2024-03-26T05:01:56.319173+0000","last_became_active":"2024-03-26T05:00:23.346607+0000","last_became_peered":"2024-03-26T05:00:23.346607+0000","last_unstale":"2024-03-26T05:01:56.319173+0000","last_undegraded":"2024-03-26T05:01:56.319173+0000","last_fullsized":"2024-03-26T05:01:56.319173+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:35.658796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.658796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:41.835654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'15","reported_seq":99,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.314412+0000","last_change":"2024-03-26T05:00:24.943621+0000","last_active":"2024-03-26T05:01:56.314412+0000","last_peered":"2024-03-26T05:01:56.314412+0000","last_clean":"2024-03-26T05:01:56.314412+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:01:56.314412+0000","last_undegraded":"2024-03-26T05:01:56.314412+0000","last_fullsized":"2024-03-26T05:01:56.314412+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662186+0000","last_change":"2024-03-26T05:00:29.463053+0000","last_active":"2024-03-26T05:01:56.662186+0000","last_peered":"2024-03-26T05:01:56.662186+0000","last_clean":"2024-03-26T05:01:56.662186+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:01:56.662186+0000","last_undegraded":"2024-03-26T05:01:56.662186+0000","last_fullsized":"2024-03-26T05:01:56.662186+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:56.816597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":67,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.314272+0000","last_change":"2024-03-26T05:00:26.955355+0000","last_active":"2024-03-26T05:01:56.314272+0000","last_peered":"2024-03-26T05:01:56.314272+0000","last_clean":"2024-03-26T05:01:56.314272+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:01:56.314272+0000","last_undegraded":"2024-03-26T05:01:56.314272+0000","last_fullsized":"2024-03-26T05:01:56.314272+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662602+0000","last_change":"2024-03-26T05:01:37.201540+0000","last_active":"2024-03-26T05:01:56.662602+0000","last_peered":"2024-03-26T05:01:56.662602+0000","last_clean":"2024-03-26T05:01:56.662602+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:01:56.662602+0000","last_undegraded":"2024-03-26T05:01:56.662602+0000","last_fullsized":"2024-03-26T05:01:56.662602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.201446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.201446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:43.519218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'39","reported_seq":143,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662141+0000","last_change":"2024-03-26T05:00:24.932741+0000","last_active":"2024-03-26T05:01:56.662141+0000","last_peered":"2024-03-26T05:01:56.662141+0000","last_clean":"2024-03-26T05:01:56.662141+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:01:56.662141+0000","last_undegraded":"2024-03-26T05:01:56.662141+0000","last_fullsized":"2024-03-26T05:01:56.662141+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.318730+0000","last_change":"2024-03-26T05:01:32.598168+0000","last_active":"2024-03-26T05:01:56.318730+0000","last_peered":"2024-03-26T05:01:56.318730+0000","last_clean":"2024-03-26T05:01:56.318730+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:01:56.318730+0000","last_undegraded":"2024-03-26T05:01:56.318730+0000","last_fullsized":"2024-03-26T05:01:56.318730+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:32.598087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:32.598087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.083903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:58.072748+0000","last_change":"2024-03-26T05:01:58.072768+0000","last_active":"2024-03-26T05:01:58.072748+0000","last_peered":"2024-03-26T05:01:58.072748+0000","last_clean":"2024-03-26T05:01:58.072748+0000","last_became_active":"2024-03-26T05:00:26.956013+0000","last_became_peered":"2024-03-26T05:00:26.956013+0000","last_unstale":"2024-03-26T05:01:58.072748+0000","last_undegraded":"2024-03-26T05:01:58.072748+0000","last_fullsized":"2024-03-26T05:01:58.072748+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:58.072676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:58.072676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:19.079733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480521+0000","last_change":"2024-03-26T05:01:30.949017+0000","last_active":"2024-03-26T05:01:56.480521+0000","last_peered":"2024-03-26T05:01:56.480521+0000","last_clean":"2024-03-26T05:01:56.480521+0000","last_became_active":"2024-03-26T05:00:22.768098+0000","last_became_peered":"2024-03-26T05:00:22.768098+0000","last_unstale":"2024-03-26T05:01:56.480521+0000","last_undegraded":"2024-03-26T05:01:56.480521+0000","last_fullsized":"2024-03-26T05:01:56.480521+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.948866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.948866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.633607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'21","reported_seq":119,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.315565+0000","last_change":"2024-03-26T05:01:30.476359+0000","last_active":"2024-03-26T05:01:56.315565+0000","last_peered":"2024-03-26T05:01:56.315565+0000","last_clean":"2024-03-26T05:01:56.315565+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:01:56.315565+0000","last_undegraded":"2024-03-26T05:01:56.315565+0000","last_fullsized":"2024-03-26T05:01:56.315565+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:01:30.476249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.476249+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.885741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480129+0000","last_change":"2024-03-26T05:01:51.098460+0000","last_active":"2024-03-26T05:01:56.480129+0000","last_peered":"2024-03-26T05:01:56.480129+0000","last_clean":"2024-03-26T05:01:56.480129+0000","last_became_active":"2024-03-26T05:00:28.982276+0000","last_became_peered":"2024-03-26T05:00:28.982276+0000","last_unstale":"2024-03-26T05:01:56.480129+0000","last_undegraded":"2024-03-26T05:01:56.480129+0000","last_fullsized":"2024-03-26T05:01:56.480129+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:51.098327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.098327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:51.933537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":73,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.072579+0000","last_change":"2024-03-26T05:01:53.941805+0000","last_active":"2024-03-26T05:01:57.072579+0000","last_peered":"2024-03-26T05:01:57.072579+0000","last_clean":"2024-03-26T05:01:57.072579+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:01:57.072579+0000","last_undegraded":"2024-03-26T05:01:57.072579+0000","last_fullsized":"2024-03-26T05:01:57.072579+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:53.941700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:53.941700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:05.433050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480554+0000","last_change":"2024-03-26T05:01:39.021449+0000","last_active":"2024-03-26T05:01:56.480554+0000","last_peered":"2024-03-26T05:01:56.480554+0000","last_clean":"2024-03-26T05:01:56.480554+0000","last_became_active":"2024-03-26T05:00:22.766675+0000","last_became_peered":"2024-03-26T05:00:22.766675+0000","last_unstale":"2024-03-26T05:01:56.480554+0000","last_undegraded":"2024-03-26T05:01:56.480554+0000","last_fullsized":"2024-03-26T05:01:56.480554+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.021315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.021315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:39.428392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'11","reported_seq":93,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.314390+0000","last_change":"2024-03-26T05:00:24.943205+0000","last_active":"2024-03-26T05:01:56.314390+0000","last_peered":"2024-03-26T05:01:56.314390+0000","last_clean":"2024-03-26T05:01:56.314390+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:01:56.314390+0000","last_undegraded":"2024-03-26T05:01:56.314390+0000","last_fullsized":"2024-03-26T05:01:56.314390+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.317578+0000","last_change":"2024-03-26T05:01:50.502541+0000","last_active":"2024-03-26T05:01:56.317578+0000","last_peered":"2024-03-26T05:01:56.317578+0000","last_clean":"2024-03-26T05:01:56.317578+0000","last_became_active":"2024-03-26T05:00:29.461032+0000","last_became_peered":"2024-03-26T05:00:29.461032+0000","last_unstale":"2024-03-26T05:01:56.317578+0000","last_undegraded":"2024-03-26T05:01:56.317578+0000","last_fullsized":"2024-03-26T05:01:56.317578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:01:50.502459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:52.644841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.806727+0000","last_change":"2024-03-26T05:01:51.781163+0000","last_active":"2024-03-26T05:01:56.806727+0000","last_peered":"2024-03-26T05:01:56.806727+0000","last_clean":"2024-03-26T05:01:56.806727+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:01:56.806727+0000","last_undegraded":"2024-03-26T05:01:56.806727+0000","last_fullsized":"2024-03-26T05:01:56.806727+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:51.781081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.781081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:56.997361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.319164+0000","last_change":"2024-03-26T05:01:47.742402+0000","last_active":"2024-03-26T05:01:56.319164+0000","last_peered":"2024-03-26T05:01:56.319164+0000","last_clean":"2024-03-26T05:01:56.319164+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:01:56.319164+0000","last_undegraded":"2024-03-26T05:01:56.319164+0000","last_fullsized":"2024-03-26T05:01:56.319164+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:47.742292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:47.742292+0000","last_clean_scrub_stamp":"2024-03-26T05:01:47.742292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.854236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":103,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071989+0000","last_change":"2024-03-26T05:01:24.617269+0000","last_active":"2024-03-26T05:01:57.071989+0000","last_peered":"2024-03-26T05:01:57.071989+0000","last_clean":"2024-03-26T05:01:57.071989+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:01:57.071989+0000","last_undegraded":"2024-03-26T05:01:57.071989+0000","last_fullsized":"2024-03-26T05:01:57.071989+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:24.617170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:24.617170+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:34.140661+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662645+0000","last_change":"2024-03-26T05:01:30.046528+0000","last_active":"2024-03-26T05:01:56.662645+0000","last_peered":"2024-03-26T05:01:56.662645+0000","last_clean":"2024-03-26T05:01:56.662645+0000","last_became_active":"2024-03-26T05:00:29.463238+0000","last_became_peered":"2024-03-26T05:00:29.463238+0000","last_unstale":"2024-03-26T05:01:56.662645+0000","last_undegraded":"2024-03-26T05:01:56.662645+0000","last_fullsized":"2024-03-26T05:01:56.662645+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.046425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.046425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:35.648365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.314913+0000","last_change":"2024-03-26T05:01:51.535065+0000","last_active":"2024-03-26T05:01:56.314913+0000","last_peered":"2024-03-26T05:01:56.314913+0000","last_clean":"2024-03-26T05:01:56.314913+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:01:56.314913+0000","last_undegraded":"2024-03-26T05:01:56.314913+0000","last_fullsized":"2024-03-26T05:01:56.314913+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:51.534988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.534988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:12.499805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.315739+0000","last_change":"2024-03-26T05:01:46.410472+0000","last_active":"2024-03-26T05:01:56.315739+0000","last_peered":"2024-03-26T05:01:56.315739+0000","last_clean":"2024-03-26T05:01:56.315739+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:01:56.315739+0000","last_undegraded":"2024-03-26T05:01:56.315739+0000","last_fullsized":"2024-03-26T05:01:56.315739+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:01:46.410356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:52.981688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'20","reported_seq":115,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662443+0000","last_change":"2024-03-26T05:01:54.969276+0000","last_active":"2024-03-26T05:01:56.662443+0000","last_peered":"2024-03-26T05:01:56.662443+0000","last_clean":"2024-03-26T05:01:56.662443+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:01:56.662443+0000","last_undegraded":"2024-03-26T05:01:56.662443+0000","last_fullsized":"2024-03-26T05:01:56.662443+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:01:54.969172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:54.969172+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:13.767531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071524+0000","last_change":"2024-03-26T05:01:55.953306+0000","last_active":"2024-03-26T05:01:57.071524+0000","last_peered":"2024-03-26T05:01:57.071524+0000","last_clean":"2024-03-26T05:01:57.071524+0000","last_became_active":"2024-03-26T05:00:28.981505+0000","last_became_peered":"2024-03-26T05:00:28.981505+0000","last_unstale":"2024-03-26T05:01:57.071524+0000","last_undegraded":"2024-03-26T05:01:57.071524+0000","last_fullsized":"2024-03-26T05:01:57.071524+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:01:55.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:55.953223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:08.010799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":104,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.072523+0000","last_change":"2024-03-26T05:01:51.236091+0000","last_active":"2024-03-26T05:01:57.072523+0000","last_peered":"2024-03-26T05:01:57.072523+0000","last_clean":"2024-03-26T05:01:57.072523+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:01:57.072523+0000","last_undegraded":"2024-03-26T05:01:57.072523+0000","last_fullsized":"2024-03-26T05:01:57.072523+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:51.235994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.235994+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:51.261797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.806213+0000","last_change":"2024-03-26T05:00:22.773390+0000","last_active":"2024-03-26T05:01:56.806213+0000","last_peered":"2024-03-26T05:01:56.806213+0000","last_clean":"2024-03-26T05:01:56.806213+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:01:56.806213+0000","last_undegraded":"2024-03-26T05:01:56.806213+0000","last_fullsized":"2024-03-26T05:01:56.806213+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'16","reported_seq":106,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.318971+0000","last_change":"2024-03-26T05:01:41.650400+0000","last_active":"2024-03-26T05:01:56.318971+0000","last_peered":"2024-03-26T05:01:56.318971+0000","last_clean":"2024-03-26T05:01:56.318971+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:01:56.318971+0000","last_undegraded":"2024-03-26T05:01:56.318971+0000","last_fullsized":"2024-03-26T05:01:56.318971+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:01:41.650310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.650310+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:08.831570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.807245+0000","last_change":"2024-03-26T05:01:56.807267+0000","last_active":"2024-03-26T05:01:56.807245+0000","last_peered":"2024-03-26T05:01:56.807245+0000","last_clean":"2024-03-26T05:01:56.807245+0000","last_became_active":"2024-03-26T05:00:28.981049+0000","last_became_peered":"2024-03-26T05:00:28.981049+0000","last_unstale":"2024-03-26T05:01:56.807245+0000","last_undegraded":"2024-03-26T05:01:56.807245+0000","last_fullsized":"2024-03-26T05:01:56.807245+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:56.807184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:56.807184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:00.264512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":67,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.319131+0000","last_change":"2024-03-26T05:00:26.963282+0000","last_active":"2024-03-26T05:01:56.319131+0000","last_peered":"2024-03-26T05:01:56.319131+0000","last_clean":"2024-03-26T05:01:56.319131+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:01:56.319131+0000","last_undegraded":"2024-03-26T05:01:56.319131+0000","last_fullsized":"2024-03-26T05:01:56.319131+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.661886+0000","last_change":"2024-03-26T05:01:43.166558+0000","last_active":"2024-03-26T05:01:56.661886+0000","last_peered":"2024-03-26T05:01:56.661886+0000","last_clean":"2024-03-26T05:01:56.661886+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:01:56.661886+0000","last_undegraded":"2024-03-26T05:01:56.661886+0000","last_fullsized":"2024-03-26T05:01:56.661886+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:43.166422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:43.166422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:50.132493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'17","reported_seq":128,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.072371+0000","last_change":"2024-03-26T05:01:53.258532+0000","last_active":"2024-03-26T05:01:57.072371+0000","last_peered":"2024-03-26T05:01:57.072371+0000","last_clean":"2024-03-26T05:01:57.072371+0000","last_became_active":"2024-03-26T05:00:49.643308+0000","last_became_peered":"2024-03-26T05:00:49.643308+0000","last_unstale":"2024-03-26T05:01:57.072371+0000","last_undegraded":"2024-03-26T05:01:57.072371+0000","last_fullsized":"2024-03-26T05:01:57.072371+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:01:53.258422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:53.258422+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:22.142699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":61,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.072459+0000","last_change":"2024-03-26T05:00:28.978762+0000","last_active":"2024-03-26T05:01:57.072459+0000","last_peered":"2024-03-26T05:01:57.072459+0000","last_clean":"2024-03-26T05:01:57.072459+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:01:57.072459+0000","last_undegraded":"2024-03-26T05:01:57.072459+0000","last_fullsized":"2024-03-26T05:01:57.072459+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":73,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071691+0000","last_change":"2024-03-26T05:01:44.235422+0000","last_active":"2024-03-26T05:01:57.071691+0000","last_peered":"2024-03-26T05:01:57.071691+0000","last_clean":"2024-03-26T05:01:57.071691+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:01:57.071691+0000","last_undegraded":"2024-03-26T05:01:57.071691+0000","last_fullsized":"2024-03-26T05:01:57.071691+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:44.235328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.235328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.223031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.319113+0000","last_change":"2024-03-26T05:00:22.775209+0000","last_active":"2024-03-26T05:01:56.319113+0000","last_peered":"2024-03-26T05:01:56.319113+0000","last_clean":"2024-03-26T05:01:56.319113+0000","last_became_active":"2024-03-26T05:00:22.774783+0000","last_became_peered":"2024-03-26T05:00:22.774783+0000","last_unstale":"2024-03-26T05:01:56.319113+0000","last_undegraded":"2024-03-26T05:01:56.319113+0000","last_fullsized":"2024-03-26T05:01:56.319113+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"100'6","reported_seq":86,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:58.079420+0000","last_change":"2024-03-26T05:01:58.079447+0000","last_active":"2024-03-26T05:01:58.079420+0000","last_peered":"2024-03-26T05:01:58.079420+0000","last_clean":"2024-03-26T05:01:58.079420+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:01:58.079420+0000","last_undegraded":"2024-03-26T05:01:58.079420+0000","last_fullsized":"2024-03-26T05:01:58.079420+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'6","last_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:01:58.079363+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:08.138424+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662053+0000","last_change":"2024-03-26T05:00:29.461798+0000","last_active":"2024-03-26T05:01:56.662053+0000","last_peered":"2024-03-26T05:01:56.662053+0000","last_clean":"2024-03-26T05:01:56.662053+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:01:56.662053+0000","last_undegraded":"2024-03-26T05:01:56.662053+0000","last_fullsized":"2024-03-26T05:01:56.662053+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":73,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:58.932134+0000","last_change":"2024-03-26T05:01:58.932169+0000","last_active":"2024-03-26T05:01:58.932134+0000","last_peered":"2024-03-26T05:01:58.932134+0000","last_clean":"2024-03-26T05:01:58.932134+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:01:58.932134+0000","last_undegraded":"2024-03-26T05:01:58.932134+0000","last_fullsized":"2024-03-26T05:01:58.932134+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:58.932064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:58.932064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:04.564275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480708+0000","last_change":"2024-03-26T05:01:29.997645+0000","last_active":"2024-03-26T05:01:56.480708+0000","last_peered":"2024-03-26T05:01:56.480708+0000","last_clean":"2024-03-26T05:01:56.480708+0000","last_became_active":"2024-03-26T05:00:22.771199+0000","last_became_peered":"2024-03-26T05:00:22.771199+0000","last_unstale":"2024-03-26T05:01:56.480708+0000","last_undegraded":"2024-03-26T05:01:56.480708+0000","last_fullsized":"2024-03-26T05:01:56.480708+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:35.309625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'17","reported_seq":109,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662110+0000","last_change":"2024-03-26T05:01:55.996301+0000","last_active":"2024-03-26T05:01:56.662110+0000","last_peered":"2024-03-26T05:01:56.662110+0000","last_clean":"2024-03-26T05:01:56.662110+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:01:56.662110+0000","last_undegraded":"2024-03-26T05:01:56.662110+0000","last_fullsized":"2024-03-26T05:01:56.662110+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:01:55.996210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:55.996210+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:19.201138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.318392+0000","last_change":"2024-03-26T05:01:49.768067+0000","last_active":"2024-03-26T05:01:56.318392+0000","last_peered":"2024-03-26T05:01:56.318392+0000","last_clean":"2024-03-26T05:01:56.318392+0000","last_became_active":"2024-03-26T05:00:28.985158+0000","last_became_peered":"2024-03-26T05:00:28.985158+0000","last_unstale":"2024-03-26T05:01:56.318392+0000","last_undegraded":"2024-03-26T05:01:56.318392+0000","last_fullsized":"2024-03-26T05:01:56.318392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:49.767986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:49.767986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.936245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":66,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662334+0000","last_change":"2024-03-26T05:00:26.960672+0000","last_active":"2024-03-26T05:01:56.662334+0000","last_peered":"2024-03-26T05:01:56.662334+0000","last_clean":"2024-03-26T05:01:56.662334+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:01:56.662334+0000","last_undegraded":"2024-03-26T05:01:56.662334+0000","last_fullsized":"2024-03-26T05:01:56.662334+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":73,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.073065+0000","last_change":"2024-03-26T05:00:22.773808+0000","last_active":"2024-03-26T05:01:57.073065+0000","last_peered":"2024-03-26T05:01:57.073065+0000","last_clean":"2024-03-26T05:01:57.073065+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:01:57.073065+0000","last_undegraded":"2024-03-26T05:01:57.073065+0000","last_fullsized":"2024-03-26T05:01:57.073065+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":86,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.806429+0000","last_change":"2024-03-26T05:00:24.939870+0000","last_active":"2024-03-26T05:01:56.806429+0000","last_peered":"2024-03-26T05:01:56.806429+0000","last_clean":"2024-03-26T05:01:56.806429+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:01:56.806429+0000","last_undegraded":"2024-03-26T05:01:56.806429+0000","last_fullsized":"2024-03-26T05:01:56.806429+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.661901+0000","last_change":"2024-03-26T05:00:28.984567+0000","last_active":"2024-03-26T05:01:56.661901+0000","last_peered":"2024-03-26T05:01:56.661901+0000","last_clean":"2024-03-26T05:01:56.661901+0000","last_became_active":"2024-03-26T05:00:28.983417+0000","last_became_peered":"2024-03-26T05:00:28.983417+0000","last_unstale":"2024-03-26T05:01:56.661901+0000","last_undegraded":"2024-03-26T05:01:56.661901+0000","last_fullsized":"2024-03-26T05:01:56.661901+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.806395+0000","last_change":"2024-03-26T05:01:30.777153+0000","last_active":"2024-03-26T05:01:56.806395+0000","last_peered":"2024-03-26T05:01:56.806395+0000","last_clean":"2024-03-26T05:01:56.806395+0000","last_became_active":"2024-03-26T05:00:26.954653+0000","last_became_peered":"2024-03-26T05:00:26.954653+0000","last_unstale":"2024-03-26T05:01:56.806395+0000","last_undegraded":"2024-03-26T05:01:56.806395+0000","last_fullsized":"2024-03-26T05:01:56.806395+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.777070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:45.021844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.315141+0000","last_change":"2024-03-26T05:00:22.767847+0000","last_active":"2024-03-26T05:01:56.315141+0000","last_peered":"2024-03-26T05:01:56.315141+0000","last_clean":"2024-03-26T05:01:56.315141+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:01:56.315141+0000","last_undegraded":"2024-03-26T05:01:56.315141+0000","last_fullsized":"2024-03-26T05:01:56.315141+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"95'16","reported_seq":103,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.314634+0000","last_change":"2024-03-26T05:01:40.462389+0000","last_active":"2024-03-26T05:01:56.314634+0000","last_peered":"2024-03-26T05:01:56.314634+0000","last_clean":"2024-03-26T05:01:56.314634+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:01:56.314634+0000","last_undegraded":"2024-03-26T05:01:56.314634+0000","last_fullsized":"2024-03-26T05:01:56.314634+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.462271+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.765446+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":63,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.318848+0000","last_change":"2024-03-26T05:00:28.984841+0000","last_active":"2024-03-26T05:01:56.318848+0000","last_peered":"2024-03-26T05:01:56.318848+0000","last_clean":"2024-03-26T05:01:56.318848+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:01:56.318848+0000","last_undegraded":"2024-03-26T05:01:56.318848+0000","last_fullsized":"2024-03-26T05:01:56.318848+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":94,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480667+0000","last_change":"2024-03-26T05:00:26.956153+0000","last_active":"2024-03-26T05:01:56.480667+0000","last_peered":"2024-03-26T05:01:56.480667+0000","last_clean":"2024-03-26T05:01:56.480667+0000","last_became_active":"2024-03-26T05:00:26.955836+0000","last_became_peered":"2024-03-26T05:00:26.955836+0000","last_unstale":"2024-03-26T05:01:56.480667+0000","last_undegraded":"2024-03-26T05:01:56.480667+0000","last_fullsized":"2024-03-26T05:01:56.480667+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":81,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.072284+0000","last_change":"2024-03-26T05:01:26.785152+0000","last_active":"2024-03-26T05:01:57.072284+0000","last_peered":"2024-03-26T05:01:57.072284+0000","last_clean":"2024-03-26T05:01:57.072284+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:01:57.072284+0000","last_undegraded":"2024-03-26T05:01:57.072284+0000","last_fullsized":"2024-03-26T05:01:57.072284+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.785056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.785056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.302223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'22","reported_seq":117,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662279+0000","last_change":"2024-03-26T05:01:40.118101+0000","last_active":"2024-03-26T05:01:56.662279+0000","last_peered":"2024-03-26T05:01:56.662279+0000","last_clean":"2024-03-26T05:01:56.662279+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:01:56.662279+0000","last_undegraded":"2024-03-26T05:01:56.662279+0000","last_fullsized":"2024-03-26T05:01:56.662279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:01:40.117953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.117953+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.646893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":63,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.317230+0000","last_change":"2024-03-26T05:00:28.979801+0000","last_active":"2024-03-26T05:01:56.317230+0000","last_peered":"2024-03-26T05:01:56.317230+0000","last_clean":"2024-03-26T05:01:56.317230+0000","last_became_active":"2024-03-26T05:00:28.979477+0000","last_became_peered":"2024-03-26T05:00:28.979477+0000","last_unstale":"2024-03-26T05:01:56.317230+0000","last_undegraded":"2024-03-26T05:01:56.317230+0000","last_fullsized":"2024-03-26T05:01:56.317230+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:56.897877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":105,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662606+0000","last_change":"2024-03-26T05:01:49.006492+0000","last_active":"2024-03-26T05:01:56.662606+0000","last_peered":"2024-03-26T05:01:56.662606+0000","last_clean":"2024-03-26T05:01:56.662606+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:01:56.662606+0000","last_undegraded":"2024-03-26T05:01:56.662606+0000","last_fullsized":"2024-03-26T05:01:56.662606+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:49.006396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:49.006396+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.424499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.318615+0000","last_change":"2024-03-26T05:01:51.835794+0000","last_active":"2024-03-26T05:01:56.318615+0000","last_peered":"2024-03-26T05:01:56.318615+0000","last_clean":"2024-03-26T05:01:56.318615+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:01:56.318615+0000","last_undegraded":"2024-03-26T05:01:56.318615+0000","last_fullsized":"2024-03-26T05:01:56.318615+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:51.835722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.835722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:11.405634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"98'20","reported_seq":111,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480779+0000","last_change":"2024-03-26T05:01:48.070439+0000","last_active":"2024-03-26T05:01:56.480779+0000","last_peered":"2024-03-26T05:01:56.480779+0000","last_clean":"2024-03-26T05:01:56.480779+0000","last_became_active":"2024-03-26T05:00:25.648284+0000","last_became_peered":"2024-03-26T05:00:25.648284+0000","last_unstale":"2024-03-26T05:01:56.480779+0000","last_undegraded":"2024-03-26T05:01:56.480779+0000","last_fullsized":"2024-03-26T05:01:56.480779+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'20","last_scrub_stamp":"2024-03-26T05:01:48.070338+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:01:48.070338+0000","last_clean_scrub_stamp":"2024-03-26T05:01:48.070338+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.153022+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071416+0000","last_change":"2024-03-26T05:01:31.568804+0000","last_active":"2024-03-26T05:01:57.071416+0000","last_peered":"2024-03-26T05:01:57.071416+0000","last_clean":"2024-03-26T05:01:57.071416+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:01:57.071416+0000","last_undegraded":"2024-03-26T05:01:57.071416+0000","last_fullsized":"2024-03-26T05:01:57.071416+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:31.568722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:31.568722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.465689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":66,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662634+0000","last_change":"2024-03-26T05:00:26.956233+0000","last_active":"2024-03-26T05:01:56.662634+0000","last_peered":"2024-03-26T05:01:56.662634+0000","last_clean":"2024-03-26T05:01:56.662634+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:01:56.662634+0000","last_undegraded":"2024-03-26T05:01:56.662634+0000","last_fullsized":"2024-03-26T05:01:56.662634+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":73,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071956+0000","last_change":"2024-03-26T05:00:22.770737+0000","last_active":"2024-03-26T05:01:57.071956+0000","last_peered":"2024-03-26T05:01:57.071956+0000","last_clean":"2024-03-26T05:01:57.071956+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:01:57.071956+0000","last_undegraded":"2024-03-26T05:01:57.071956+0000","last_fullsized":"2024-03-26T05:01:57.071956+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'21","reported_seq":119,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.319038+0000","last_change":"2024-03-26T05:01:38.722775+0000","last_active":"2024-03-26T05:01:56.319038+0000","last_peered":"2024-03-26T05:01:56.319038+0000","last_clean":"2024-03-26T05:01:56.319038+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:01:56.319038+0000","last_undegraded":"2024-03-26T05:01:56.319038+0000","last_fullsized":"2024-03-26T05:01:56.319038+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:01:38.722652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:38.722652+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.850545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":61,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.072993+0000","last_change":"2024-03-26T05:00:29.462603+0000","last_active":"2024-03-26T05:01:57.072993+0000","last_peered":"2024-03-26T05:01:57.072993+0000","last_clean":"2024-03-26T05:01:57.072993+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:01:57.072993+0000","last_undegraded":"2024-03-26T05:01:57.072993+0000","last_fullsized":"2024-03-26T05:01:57.072993+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":66,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480701+0000","last_change":"2024-03-26T05:00:26.953694+0000","last_active":"2024-03-26T05:01:56.480701+0000","last_peered":"2024-03-26T05:01:56.480701+0000","last_clean":"2024-03-26T05:01:56.480701+0000","last_became_active":"2024-03-26T05:00:26.953416+0000","last_became_peered":"2024-03-26T05:00:26.953416+0000","last_unstale":"2024-03-26T05:01:56.480701+0000","last_undegraded":"2024-03-26T05:01:56.480701+0000","last_fullsized":"2024-03-26T05:01:56.480701+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":87,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071584+0000","last_change":"2024-03-26T05:01:37.375697+0000","last_active":"2024-03-26T05:01:57.071584+0000","last_peered":"2024-03-26T05:01:57.071584+0000","last_clean":"2024-03-26T05:01:57.071584+0000","last_became_active":"2024-03-26T05:00:22.771080+0000","last_became_peered":"2024-03-26T05:00:22.771080+0000","last_unstale":"2024-03-26T05:01:57.071584+0000","last_undegraded":"2024-03-26T05:01:57.071584+0000","last_fullsized":"2024-03-26T05:01:57.071584+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:01:37.375533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.375533+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:51.717267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'12","reported_seq":99,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662691+0000","last_change":"2024-03-26T05:01:27.964049+0000","last_active":"2024-03-26T05:01:56.662691+0000","last_peered":"2024-03-26T05:01:56.662691+0000","last_clean":"2024-03-26T05:01:56.662691+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:01:56.662691+0000","last_undegraded":"2024-03-26T05:01:56.662691+0000","last_fullsized":"2024-03-26T05:01:56.662691+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:01:27.963954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:27.963954+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:50.330132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":72,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480415+0000","last_change":"2024-03-26T05:01:36.945807+0000","last_active":"2024-03-26T05:01:56.480415+0000","last_peered":"2024-03-26T05:01:56.480415+0000","last_clean":"2024-03-26T05:01:56.480415+0000","last_became_active":"2024-03-26T05:00:29.461059+0000","last_became_peered":"2024-03-26T05:00:29.461059+0000","last_unstale":"2024-03-26T05:01:56.480415+0000","last_undegraded":"2024-03-26T05:01:56.480415+0000","last_fullsized":"2024-03-26T05:01:56.480415+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:01:36.945708+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.001390+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'3","reported_seq":99,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.073425+0000","last_change":"2024-03-26T05:01:22.799027+0000","last_active":"2024-03-26T05:01:57.073425+0000","last_peered":"2024-03-26T05:01:57.073425+0000","last_clean":"2024-03-26T05:01:57.073425+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:01:57.073425+0000","last_undegraded":"2024-03-26T05:01:57.073425+0000","last_fullsized":"2024-03-26T05:01:57.073425+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-03-26T05:01:22.798897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:22.798897+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.103651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'27","reported_seq":127,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480144+0000","last_change":"2024-03-26T05:01:29.048598+0000","last_active":"2024-03-26T05:01:56.480144+0000","last_peered":"2024-03-26T05:01:56.480144+0000","last_clean":"2024-03-26T05:01:56.480144+0000","last_became_active":"2024-03-26T05:00:25.651328+0000","last_became_peered":"2024-03-26T05:00:25.651328+0000","last_unstale":"2024-03-26T05:01:56.480144+0000","last_undegraded":"2024-03-26T05:01:56.480144+0000","last_fullsized":"2024-03-26T05:01:56.480144+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'27","last_scrub_stamp":"2024-03-26T05:01:29.048456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.048456+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:56.053198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":101,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.072231+0000","last_change":"2024-03-26T05:01:37.743111+0000","last_active":"2024-03-26T05:01:57.072231+0000","last_peered":"2024-03-26T05:01:57.072231+0000","last_clean":"2024-03-26T05:01:57.072231+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:01:57.072231+0000","last_undegraded":"2024-03-26T05:01:57.072231+0000","last_fullsized":"2024-03-26T05:01:57.072231+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:37.742995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.742995+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:07.111791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662181+0000","last_change":"2024-03-26T05:01:29.011852+0000","last_active":"2024-03-26T05:01:56.662181+0000","last_peered":"2024-03-26T05:01:56.662181+0000","last_clean":"2024-03-26T05:01:56.662181+0000","last_became_active":"2024-03-26T05:00:28.985917+0000","last_became_peered":"2024-03-26T05:00:28.985917+0000","last_unstale":"2024-03-26T05:01:56.662181+0000","last_undegraded":"2024-03-26T05:01:56.662181+0000","last_fullsized":"2024-03-26T05:01:56.662181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.011757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.011757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.742465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":81,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.073071+0000","last_change":"2024-03-26T05:01:46.855799+0000","last_active":"2024-03-26T05:01:57.073071+0000","last_peered":"2024-03-26T05:01:57.073071+0000","last_clean":"2024-03-26T05:01:57.073071+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:01:57.073071+0000","last_undegraded":"2024-03-26T05:01:57.073071+0000","last_fullsized":"2024-03-26T05:01:57.073071+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:46.855699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:46.855699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:15.236244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'24","reported_seq":118,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.316482+0000","last_change":"2024-03-26T05:00:24.944015+0000","last_active":"2024-03-26T05:01:56.316482+0000","last_peered":"2024-03-26T05:01:56.316482+0000","last_clean":"2024-03-26T05:01:56.316482+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:01:56.316482+0000","last_undegraded":"2024-03-26T05:01:56.316482+0000","last_fullsized":"2024-03-26T05:01:56.316482+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071673+0000","last_change":"2024-03-26T05:01:26.618364+0000","last_active":"2024-03-26T05:01:57.071673+0000","last_peered":"2024-03-26T05:01:57.071673+0000","last_clean":"2024-03-26T05:01:57.071673+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:01:57.071673+0000","last_undegraded":"2024-03-26T05:01:57.071673+0000","last_fullsized":"2024-03-26T05:01:57.071673+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.618218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.618218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.093176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":61,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071441+0000","last_change":"2024-03-26T05:00:28.981813+0000","last_active":"2024-03-26T05:01:57.071441+0000","last_peered":"2024-03-26T05:01:57.071441+0000","last_clean":"2024-03-26T05:01:57.071441+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:01:57.071441+0000","last_undegraded":"2024-03-26T05:01:57.071441+0000","last_fullsized":"2024-03-26T05:01:57.071441+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071971+0000","last_change":"2024-03-26T05:01:33.638557+0000","last_active":"2024-03-26T05:01:57.071971+0000","last_peered":"2024-03-26T05:01:57.071971+0000","last_clean":"2024-03-26T05:01:57.071971+0000","last_became_active":"2024-03-26T05:00:26.961730+0000","last_became_peered":"2024-03-26T05:00:26.961730+0000","last_unstale":"2024-03-26T05:01:57.071971+0000","last_undegraded":"2024-03-26T05:01:57.071971+0000","last_fullsized":"2024-03-26T05:01:57.071971+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:33.638469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:33.638469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:00.135833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":73,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.073224+0000","last_change":"2024-03-26T05:00:22.772945+0000","last_active":"2024-03-26T05:01:57.073224+0000","last_peered":"2024-03-26T05:01:57.073224+0000","last_clean":"2024-03-26T05:01:57.073224+0000","last_became_active":"2024-03-26T05:00:22.772590+0000","last_became_peered":"2024-03-26T05:00:22.772590+0000","last_unstale":"2024-03-26T05:01:57.073224+0000","last_undegraded":"2024-03-26T05:01:57.073224+0000","last_fullsized":"2024-03-26T05:01:57.073224+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'25","reported_seq":120,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071944+0000","last_change":"2024-03-26T05:01:56.315423+0000","last_active":"2024-03-26T05:01:57.071944+0000","last_peered":"2024-03-26T05:01:57.071944+0000","last_clean":"2024-03-26T05:01:57.071944+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:01:57.071944+0000","last_undegraded":"2024-03-26T05:01:57.071944+0000","last_fullsized":"2024-03-26T05:01:57.071944+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:01:56.315274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:56.315274+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:14.373193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":63,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.318471+0000","last_change":"2024-03-26T05:00:28.981411+0000","last_active":"2024-03-26T05:01:56.318471+0000","last_peered":"2024-03-26T05:01:56.318471+0000","last_clean":"2024-03-26T05:01:56.318471+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:01:56.318471+0000","last_undegraded":"2024-03-26T05:01:56.318471+0000","last_fullsized":"2024-03-26T05:01:56.318471+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071916+0000","last_change":"2024-03-26T05:01:55.822123+0000","last_active":"2024-03-26T05:01:57.071916+0000","last_peered":"2024-03-26T05:01:57.071916+0000","last_clean":"2024-03-26T05:01:57.071916+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:01:57.071916+0000","last_undegraded":"2024-03-26T05:01:57.071916+0000","last_fullsized":"2024-03-26T05:01:57.071916+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:55.822016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:55.822016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:02.183520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480446+0000","last_change":"2024-03-26T05:01:53.133109+0000","last_active":"2024-03-26T05:01:56.480446+0000","last_peered":"2024-03-26T05:01:56.480446+0000","last_clean":"2024-03-26T05:01:56.480446+0000","last_became_active":"2024-03-26T05:00:22.773770+0000","last_became_peered":"2024-03-26T05:00:22.773770+0000","last_unstale":"2024-03-26T05:01:56.480446+0000","last_undegraded":"2024-03-26T05:01:56.480446+0000","last_fullsized":"2024-03-26T05:01:56.480446+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:53.133006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:53.133006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:10.552053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'15","reported_seq":107,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.319267+0000","last_change":"2024-03-26T05:01:50.807516+0000","last_active":"2024-03-26T05:01:56.319267+0000","last_peered":"2024-03-26T05:01:56.319267+0000","last_clean":"2024-03-26T05:01:56.319267+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:01:56.319267+0000","last_undegraded":"2024-03-26T05:01:56.319267+0000","last_fullsized":"2024-03-26T05:01:56.319267+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:50.807430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:50.807430+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:03.240799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662412+0000","last_change":"2024-03-26T05:00:28.984072+0000","last_active":"2024-03-26T05:01:56.662412+0000","last_peered":"2024-03-26T05:01:56.662412+0000","last_clean":"2024-03-26T05:01:56.662412+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:01:56.662412+0000","last_undegraded":"2024-03-26T05:01:56.662412+0000","last_fullsized":"2024-03-26T05:01:56.662412+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":93,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480262+0000","last_change":"2024-03-26T05:01:35.934149+0000","last_active":"2024-03-26T05:01:56.480262+0000","last_peered":"2024-03-26T05:01:56.480262+0000","last_clean":"2024-03-26T05:01:56.480262+0000","last_became_active":"2024-03-26T05:00:21.760521+0000","last_became_peered":"2024-03-26T05:00:21.760521+0000","last_unstale":"2024-03-26T05:01:56.480262+0000","last_undegraded":"2024-03-26T05:01:56.480262+0000","last_fullsized":"2024-03-26T05:01:56.480262+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:01:35.934044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.934044+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:02.210221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.314494+0000","last_change":"2024-03-26T05:00:22.771224+0000","last_active":"2024-03-26T05:01:56.314494+0000","last_peered":"2024-03-26T05:01:56.314494+0000","last_clean":"2024-03-26T05:01:56.314494+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:01:56.314494+0000","last_undegraded":"2024-03-26T05:01:56.314494+0000","last_fullsized":"2024-03-26T05:01:56.314494+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'34","reported_seq":158,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.806681+0000","last_change":"2024-03-26T05:01:40.826966+0000","last_active":"2024-03-26T05:01:56.806681+0000","last_peered":"2024-03-26T05:01:56.806681+0000","last_clean":"2024-03-26T05:01:56.806681+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:01:56.806681+0000","last_undegraded":"2024-03-26T05:01:56.806681+0000","last_fullsized":"2024-03-26T05:01:56.806681+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'34","last_scrub_stamp":"2024-03-26T05:01:40.826856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.826856+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.349406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":67,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.318965+0000","last_change":"2024-03-26T05:00:26.964781+0000","last_active":"2024-03-26T05:01:56.318965+0000","last_peered":"2024-03-26T05:01:56.318965+0000","last_clean":"2024-03-26T05:01:56.318965+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:01:56.318965+0000","last_undegraded":"2024-03-26T05:01:56.318965+0000","last_fullsized":"2024-03-26T05:01:56.318965+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":69,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.073326+0000","last_change":"2024-03-26T05:01:41.789755+0000","last_active":"2024-03-26T05:01:57.073326+0000","last_peered":"2024-03-26T05:01:57.073326+0000","last_clean":"2024-03-26T05:01:57.073326+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:01:57.073326+0000","last_undegraded":"2024-03-26T05:01:57.073326+0000","last_fullsized":"2024-03-26T05:01:57.073326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:41.789650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.789650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.302843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.806189+0000","last_change":"2024-03-26T05:00:23.346922+0000","last_active":"2024-03-26T05:01:56.806189+0000","last_peered":"2024-03-26T05:01:56.806189+0000","last_clean":"2024-03-26T05:01:56.806189+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:01:56.806189+0000","last_undegraded":"2024-03-26T05:01:56.806189+0000","last_fullsized":"2024-03-26T05:01:56.806189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":91,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071805+0000","last_change":"2024-03-26T05:01:37.678850+0000","last_active":"2024-03-26T05:01:57.071805+0000","last_peered":"2024-03-26T05:01:57.071805+0000","last_clean":"2024-03-26T05:01:57.071805+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:01:57.071805+0000","last_undegraded":"2024-03-26T05:01:57.071805+0000","last_fullsized":"2024-03-26T05:01:57.071805+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.678736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.678736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.820757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'19","reported_seq":109,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.806788+0000","last_change":"2024-03-26T05:00:24.937826+0000","last_active":"2024-03-26T05:01:56.806788+0000","last_peered":"2024-03-26T05:01:56.806788+0000","last_clean":"2024-03-26T05:01:56.806788+0000","last_became_active":"2024-03-26T05:00:24.937145+0000","last_became_peered":"2024-03-26T05:00:24.937145+0000","last_unstale":"2024-03-26T05:01:56.806788+0000","last_undegraded":"2024-03-26T05:01:56.806788+0000","last_fullsized":"2024-03-26T05:01:56.806788+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.072127+0000","last_change":"2024-03-26T05:01:29.627760+0000","last_active":"2024-03-26T05:01:57.072127+0000","last_peered":"2024-03-26T05:01:57.072127+0000","last_clean":"2024-03-26T05:01:57.072127+0000","last_became_active":"2024-03-26T05:00:26.957936+0000","last_became_peered":"2024-03-26T05:00:26.957936+0000","last_unstale":"2024-03-26T05:01:57.072127+0000","last_undegraded":"2024-03-26T05:01:57.072127+0000","last_fullsized":"2024-03-26T05:01:57.072127+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.627680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.627680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.292786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480632+0000","last_change":"2024-03-26T05:01:56.143486+0000","last_active":"2024-03-26T05:01:56.480632+0000","last_peered":"2024-03-26T05:01:56.480632+0000","last_clean":"2024-03-26T05:01:56.480632+0000","last_became_active":"2024-03-26T05:00:28.979373+0000","last_became_peered":"2024-03-26T05:00:28.979373+0000","last_unstale":"2024-03-26T05:01:56.480632+0000","last_undegraded":"2024-03-26T05:01:56.480632+0000","last_fullsized":"2024-03-26T05:01:56.480632+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:56.143404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:56.143404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:11.838368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":328,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.073250+0000","last_change":"2024-03-26T05:01:04.782428+0000","last_active":"2024-03-26T05:01:57.073250+0000","last_peered":"2024-03-26T05:01:57.073250+0000","last_clean":"2024-03-26T05:01:57.073250+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:01:57.073250+0000","last_undegraded":"2024-03-26T05:01:57.073250+0000","last_fullsized":"2024-03-26T05:01:57.073250+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-03-26T05:01:04.782336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:01:04.782336+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:15.009834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662179+0000","last_change":"2024-03-26T05:01:25.941429+0000","last_active":"2024-03-26T05:01:56.662179+0000","last_peered":"2024-03-26T05:01:56.662179+0000","last_clean":"2024-03-26T05:01:56.662179+0000","last_became_active":"2024-03-26T05:00:22.774298+0000","last_became_peered":"2024-03-26T05:00:22.774298+0000","last_unstale":"2024-03-26T05:01:56.662179+0000","last_undegraded":"2024-03-26T05:01:56.662179+0000","last_fullsized":"2024-03-26T05:01:56.662179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:25.941344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:25.941344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:34.075498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":111,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.072185+0000","last_change":"2024-03-26T05:00:25.648924+0000","last_active":"2024-03-26T05:01:57.072185+0000","last_peered":"2024-03-26T05:01:57.072185+0000","last_clean":"2024-03-26T05:01:57.072185+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:01:57.072185+0000","last_undegraded":"2024-03-26T05:01:57.072185+0000","last_fullsized":"2024-03-26T05:01:57.072185+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":65,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.072884+0000","last_change":"2024-03-26T05:00:26.962047+0000","last_active":"2024-03-26T05:01:57.072884+0000","last_peered":"2024-03-26T05:01:57.072884+0000","last_clean":"2024-03-26T05:01:57.072884+0000","last_became_active":"2024-03-26T05:00:26.961724+0000","last_became_peered":"2024-03-26T05:00:26.961724+0000","last_unstale":"2024-03-26T05:01:57.072884+0000","last_undegraded":"2024-03-26T05:01:57.072884+0000","last_fullsized":"2024-03-26T05:01:57.072884+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.806458+0000","last_change":"2024-03-26T05:01:38.822538+0000","last_active":"2024-03-26T05:01:56.806458+0000","last_peered":"2024-03-26T05:01:56.806458+0000","last_clean":"2024-03-26T05:01:56.806458+0000","last_became_active":"2024-03-26T05:00:28.983202+0000","last_became_peered":"2024-03-26T05:00:28.983202+0000","last_unstale":"2024-03-26T05:01:56.806458+0000","last_undegraded":"2024-03-26T05:01:56.806458+0000","last_fullsized":"2024-03-26T05:01:56.806458+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:38.822409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:38.822409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:41.940538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.318634+0000","last_change":"2024-03-26T05:01:25.629227+0000","last_active":"2024-03-26T05:01:56.318634+0000","last_peered":"2024-03-26T05:01:56.318634+0000","last_clean":"2024-03-26T05:01:56.318634+0000","last_became_active":"2024-03-26T05:00:22.773081+0000","last_became_peered":"2024-03-26T05:00:22.773081+0000","last_unstale":"2024-03-26T05:01:56.318634+0000","last_undegraded":"2024-03-26T05:01:56.318634+0000","last_fullsized":"2024-03-26T05:01:56.318634+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:25.629101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:25.629101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:43.255919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":185,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:58.260291+0000","last_change":"2024-03-26T05:01:36.780629+0000","last_active":"2024-03-26T05:01:58.260291+0000","last_peered":"2024-03-26T05:01:58.260291+0000","last_clean":"2024-03-26T05:01:58.260291+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:01:58.260291+0000","last_undegraded":"2024-03-26T05:01:58.260291+0000","last_fullsized":"2024-03-26T05:01:58.260291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:01:36.780518+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.077309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":109,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.318875+0000","last_change":"2024-03-26T05:00:49.653404+0000","last_active":"2024-03-26T05:01:56.318875+0000","last_peered":"2024-03-26T05:01:56.318875+0000","last_clean":"2024-03-26T05:01:56.318875+0000","last_became_active":"2024-03-26T05:00:49.642276+0000","last_became_peered":"2024-03-26T05:00:49.642276+0000","last_unstale":"2024-03-26T05:01:56.318875+0000","last_undegraded":"2024-03-26T05:01:56.318875+0000","last_fullsized":"2024-03-26T05:01:56.318875+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480824+0000","last_change":"2024-03-26T05:01:41.973895+0000","last_active":"2024-03-26T05:01:56.480824+0000","last_peered":"2024-03-26T05:01:56.480824+0000","last_clean":"2024-03-26T05:01:56.480824+0000","last_became_active":"2024-03-26T05:00:26.956677+0000","last_became_peered":"2024-03-26T05:00:26.956677+0000","last_unstale":"2024-03-26T05:01:56.480824+0000","last_undegraded":"2024-03-26T05:01:56.480824+0000","last_fullsized":"2024-03-26T05:01:56.480824+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:41.973805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.973805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:54.658519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.803207+0000","last_change":"2024-03-26T05:01:57.803229+0000","last_active":"2024-03-26T05:01:57.803207+0000","last_peered":"2024-03-26T05:01:57.803207+0000","last_clean":"2024-03-26T05:01:57.803207+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:01:57.803207+0000","last_undegraded":"2024-03-26T05:01:57.803207+0000","last_fullsized":"2024-03-26T05:01:57.803207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:57.803148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:57.803148+0000","last_clean_scrub_stamp":"2024-03-26T05:01:57.803148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:11.528300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.806541+0000","last_change":"2024-03-26T05:01:35.745231+0000","last_active":"2024-03-26T05:01:56.806541+0000","last_peered":"2024-03-26T05:01:56.806541+0000","last_clean":"2024-03-26T05:01:56.806541+0000","last_became_active":"2024-03-26T05:00:23.346665+0000","last_became_peered":"2024-03-26T05:00:23.346665+0000","last_unstale":"2024-03-26T05:01:56.806541+0000","last_undegraded":"2024-03-26T05:01:56.806541+0000","last_fullsized":"2024-03-26T05:01:56.806541+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.745100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.314539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'25","reported_seq":127,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.319036+0000","last_change":"2024-03-26T05:01:44.739144+0000","last_active":"2024-03-26T05:01:56.319036+0000","last_peered":"2024-03-26T05:01:56.319036+0000","last_clean":"2024-03-26T05:01:56.319036+0000","last_became_active":"2024-03-26T05:00:25.648253+0000","last_became_peered":"2024-03-26T05:00:25.648253+0000","last_unstale":"2024-03-26T05:01:56.319036+0000","last_undegraded":"2024-03-26T05:01:56.319036+0000","last_fullsized":"2024-03-26T05:01:56.319036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:01:44.739046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.739046+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:06.858172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":73,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:59.261283+0000","last_change":"2024-03-26T05:01:59.261306+0000","last_active":"2024-03-26T05:01:59.261283+0000","last_peered":"2024-03-26T05:01:59.261283+0000","last_clean":"2024-03-26T05:01:59.261283+0000","last_became_active":"2024-03-26T05:00:26.958689+0000","last_became_peered":"2024-03-26T05:00:26.958689+0000","last_unstale":"2024-03-26T05:01:59.261283+0000","last_undegraded":"2024-03-26T05:01:59.261283+0000","last_fullsized":"2024-03-26T05:01:59.261283+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:01:59.261180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:28.535218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.318405+0000","last_change":"2024-03-26T05:01:46.735926+0000","last_active":"2024-03-26T05:01:56.318405+0000","last_peered":"2024-03-26T05:01:56.318405+0000","last_clean":"2024-03-26T05:01:56.318405+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:01:56.318405+0000","last_undegraded":"2024-03-26T05:01:56.318405+0000","last_fullsized":"2024-03-26T05:01:56.318405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:46.735848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:46.735848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:09.041185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480662+0000","last_change":"2024-03-26T05:01:54.084913+0000","last_active":"2024-03-26T05:01:56.480662+0000","last_peered":"2024-03-26T05:01:56.480662+0000","last_clean":"2024-03-26T05:01:56.480662+0000","last_became_active":"2024-03-26T05:00:23.346876+0000","last_became_peered":"2024-03-26T05:00:23.346876+0000","last_unstale":"2024-03-26T05:01:56.480662+0000","last_undegraded":"2024-03-26T05:01:56.480662+0000","last_fullsized":"2024-03-26T05:01:56.480662+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:54.084818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:54.084818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:10.773277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'16","reported_seq":68,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071814+0000","last_change":"2024-03-26T05:01:42.753638+0000","last_active":"2024-03-26T05:01:57.071814+0000","last_peered":"2024-03-26T05:01:57.071814+0000","last_clean":"2024-03-26T05:01:57.071814+0000","last_became_active":"2024-03-26T05:00:49.641444+0000","last_became_peered":"2024-03-26T05:00:49.641444+0000","last_unstale":"2024-03-26T05:01:57.071814+0000","last_undegraded":"2024-03-26T05:01:57.071814+0000","last_fullsized":"2024-03-26T05:01:57.071814+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:01:42.753523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:42.753523+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.891497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.319316+0000","last_change":"2024-03-26T05:00:26.962263+0000","last_active":"2024-03-26T05:01:56.319316+0000","last_peered":"2024-03-26T05:01:56.319316+0000","last_clean":"2024-03-26T05:01:56.319316+0000","last_became_active":"2024-03-26T05:00:26.961907+0000","last_became_peered":"2024-03-26T05:00:26.961907+0000","last_unstale":"2024-03-26T05:01:56.319316+0000","last_undegraded":"2024-03-26T05:01:56.319316+0000","last_fullsized":"2024-03-26T05:01:56.319316+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":61,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.072450+0000","last_change":"2024-03-26T05:00:29.461447+0000","last_active":"2024-03-26T05:01:57.072450+0000","last_peered":"2024-03-26T05:01:57.072450+0000","last_clean":"2024-03-26T05:01:57.072450+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:01:57.072450+0000","last_undegraded":"2024-03-26T05:01:57.072450+0000","last_fullsized":"2024-03-26T05:01:57.072450+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:01:57.420912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.318944+0000","last_change":"2024-03-26T05:01:29.581397+0000","last_active":"2024-03-26T05:01:56.318944+0000","last_peered":"2024-03-26T05:01:56.318944+0000","last_clean":"2024-03-26T05:01:56.318944+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:01:56.318944+0000","last_undegraded":"2024-03-26T05:01:56.318944+0000","last_fullsized":"2024-03-26T05:01:56.318944+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:01:29.581309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.581309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.525431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'20","reported_seq":104,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.316297+0000","last_change":"2024-03-26T05:00:24.936644+0000","last_active":"2024-03-26T05:01:56.316297+0000","last_peered":"2024-03-26T05:01:56.316297+0000","last_clean":"2024-03-26T05:01:56.316297+0000","last_became_active":"2024-03-26T05:00:24.936315+0000","last_became_peered":"2024-03-26T05:00:24.936315+0000","last_unstale":"2024-03-26T05:01:56.316297+0000","last_undegraded":"2024-03-26T05:01:56.316297+0000","last_fullsized":"2024-03-26T05:01:56.316297+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":94,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662121+0000","last_change":"2024-03-26T05:00:26.957814+0000","last_active":"2024-03-26T05:01:56.662121+0000","last_peered":"2024-03-26T05:01:56.662121+0000","last_clean":"2024-03-26T05:01:56.662121+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:01:56.662121+0000","last_undegraded":"2024-03-26T05:01:56.662121+0000","last_fullsized":"2024-03-26T05:01:56.662121+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.315420+0000","last_change":"2024-03-26T05:01:48.472310+0000","last_active":"2024-03-26T05:01:56.315420+0000","last_peered":"2024-03-26T05:01:56.315420+0000","last_clean":"2024-03-26T05:01:56.315420+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:01:56.315420+0000","last_undegraded":"2024-03-26T05:01:56.315420+0000","last_fullsized":"2024-03-26T05:01:56.315420+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:48.472188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:48.472188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:03.679495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662542+0000","last_change":"2024-03-26T05:00:22.779020+0000","last_active":"2024-03-26T05:01:56.662542+0000","last_peered":"2024-03-26T05:01:56.662542+0000","last_clean":"2024-03-26T05:01:56.662542+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:01:56.662542+0000","last_undegraded":"2024-03-26T05:01:56.662542+0000","last_fullsized":"2024-03-26T05:01:56.662542+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"96'40","reported_seq":144,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662012+0000","last_change":"2024-03-26T05:01:42.152769+0000","last_active":"2024-03-26T05:01:56.662012+0000","last_peered":"2024-03-26T05:01:56.662012+0000","last_clean":"2024-03-26T05:01:56.662012+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:01:56.662012+0000","last_undegraded":"2024-03-26T05:01:56.662012+0000","last_fullsized":"2024-03-26T05:01:56.662012+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:01:42.152630+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:01:42.152630+0000","last_clean_scrub_stamp":"2024-03-26T05:01:42.152630+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.656601+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":67,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.316077+0000","last_change":"2024-03-26T05:00:26.959009+0000","last_active":"2024-03-26T05:01:56.316077+0000","last_peered":"2024-03-26T05:01:56.316077+0000","last_clean":"2024-03-26T05:01:56.316077+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:01:56.316077+0000","last_undegraded":"2024-03-26T05:01:56.316077+0000","last_fullsized":"2024-03-26T05:01:56.316077+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480446+0000","last_change":"2024-03-26T05:00:28.983093+0000","last_active":"2024-03-26T05:01:56.480446+0000","last_peered":"2024-03-26T05:01:56.480446+0000","last_clean":"2024-03-26T05:01:56.480446+0000","last_became_active":"2024-03-26T05:00:28.982825+0000","last_became_peered":"2024-03-26T05:00:28.982825+0000","last_unstale":"2024-03-26T05:01:56.480446+0000","last_undegraded":"2024-03-26T05:01:56.480446+0000","last_fullsized":"2024-03-26T05:01:56.480446+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":103,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.317675+0000","last_change":"2024-03-26T05:01:41.485196+0000","last_active":"2024-03-26T05:01:56.317675+0000","last_peered":"2024-03-26T05:01:56.317675+0000","last_clean":"2024-03-26T05:01:56.317675+0000","last_became_active":"2024-03-26T05:00:26.956596+0000","last_became_peered":"2024-03-26T05:00:26.956596+0000","last_unstale":"2024-03-26T05:01:56.317675+0000","last_undegraded":"2024-03-26T05:01:56.317675+0000","last_fullsized":"2024-03-26T05:01:56.317675+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:41.485086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.485086+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:59.948091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.661973+0000","last_change":"2024-03-26T05:00:22.775050+0000","last_active":"2024-03-26T05:01:56.661973+0000","last_peered":"2024-03-26T05:01:56.661973+0000","last_clean":"2024-03-26T05:01:56.661973+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:01:56.661973+0000","last_undegraded":"2024-03-26T05:01:56.661973+0000","last_fullsized":"2024-03-26T05:01:56.661973+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":103,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.806922+0000","last_change":"2024-03-26T05:01:28.754168+0000","last_active":"2024-03-26T05:01:56.806922+0000","last_peered":"2024-03-26T05:01:56.806922+0000","last_clean":"2024-03-26T05:01:56.806922+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:01:56.806922+0000","last_undegraded":"2024-03-26T05:01:56.806922+0000","last_fullsized":"2024-03-26T05:01:56.806922+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:28.754068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:28.754068+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:40.695752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662194+0000","last_change":"2024-03-26T05:00:29.461578+0000","last_active":"2024-03-26T05:01:56.662194+0000","last_peered":"2024-03-26T05:01:56.662194+0000","last_clean":"2024-03-26T05:01:56.662194+0000","last_became_active":"2024-03-26T05:00:29.461236+0000","last_became_peered":"2024-03-26T05:00:29.461236+0000","last_unstale":"2024-03-26T05:01:56.662194+0000","last_undegraded":"2024-03-26T05:01:56.662194+0000","last_fullsized":"2024-03-26T05:01:56.662194+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.480804+0000","last_change":"2024-03-26T05:01:45.085244+0000","last_active":"2024-03-26T05:01:56.480804+0000","last_peered":"2024-03-26T05:01:56.480804+0000","last_clean":"2024-03-26T05:01:56.480804+0000","last_became_active":"2024-03-26T05:00:26.955805+0000","last_became_peered":"2024-03-26T05:00:26.955805+0000","last_unstale":"2024-03-26T05:01:56.480804+0000","last_undegraded":"2024-03-26T05:01:56.480804+0000","last_fullsized":"2024-03-26T05:01:56.480804+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:45.085111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:45.085111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.981772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.316157+0000","last_change":"2024-03-26T05:01:26.442833+0000","last_active":"2024-03-26T05:01:56.316157+0000","last_peered":"2024-03-26T05:01:56.316157+0000","last_clean":"2024-03-26T05:01:56.316157+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:01:56.316157+0000","last_undegraded":"2024-03-26T05:01:56.316157+0000","last_fullsized":"2024-03-26T05:01:56.316157+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.442722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.442722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:39.204290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'17","reported_seq":110,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.317485+0000","last_change":"2024-03-26T05:01:54.489384+0000","last_active":"2024-03-26T05:01:56.317485+0000","last_peered":"2024-03-26T05:01:56.317485+0000","last_clean":"2024-03-26T05:01:56.317485+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:01:56.317485+0000","last_undegraded":"2024-03-26T05:01:56.317485+0000","last_fullsized":"2024-03-26T05:01:56.317485+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:01:54.489297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:54.489297+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.631248+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071457+0000","last_change":"2024-03-26T05:01:53.801119+0000","last_active":"2024-03-26T05:01:57.071457+0000","last_peered":"2024-03-26T05:01:57.071457+0000","last_clean":"2024-03-26T05:01:57.071457+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:01:57.071457+0000","last_undegraded":"2024-03-26T05:01:57.071457+0000","last_fullsized":"2024-03-26T05:01:57.071457+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:53.801041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:53.801041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:03.065236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.806726+0000","last_change":"2024-03-26T05:01:37.774253+0000","last_active":"2024-03-26T05:01:56.806726+0000","last_peered":"2024-03-26T05:01:56.806726+0000","last_clean":"2024-03-26T05:01:56.806726+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:01:56.806726+0000","last_undegraded":"2024-03-26T05:01:56.806726+0000","last_fullsized":"2024-03-26T05:01:56.806726+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.774158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.774158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:01.227552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":81,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.071774+0000","last_change":"2024-03-26T05:01:28.386278+0000","last_active":"2024-03-26T05:01:57.071774+0000","last_peered":"2024-03-26T05:01:57.071774+0000","last_clean":"2024-03-26T05:01:57.071774+0000","last_became_active":"2024-03-26T05:00:22.768610+0000","last_became_peered":"2024-03-26T05:00:22.768610+0000","last_unstale":"2024-03-26T05:01:57.071774+0000","last_undegraded":"2024-03-26T05:01:57.071774+0000","last_fullsized":"2024-03-26T05:01:57.071774+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_clean_scrub_stamp":"2024-03-26T05:01:28.386172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.056496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":99,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.317494+0000","last_change":"2024-03-26T05:00:24.942796+0000","last_active":"2024-03-26T05:01:56.317494+0000","last_peered":"2024-03-26T05:01:56.317494+0000","last_clean":"2024-03-26T05:01:56.317494+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:01:56.317494+0000","last_undegraded":"2024-03-26T05:01:56.317494+0000","last_fullsized":"2024-03-26T05:01:56.317494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":63,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.317663+0000","last_change":"2024-03-26T05:00:28.978772+0000","last_active":"2024-03-26T05:01:56.317663+0000","last_peered":"2024-03-26T05:01:56.317663+0000","last_clean":"2024-03-26T05:01:56.317663+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:01:56.317663+0000","last_undegraded":"2024-03-26T05:01:56.317663+0000","last_fullsized":"2024-03-26T05:01:56.317663+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":102,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:57.072115+0000","last_change":"2024-03-26T05:01:43.782577+0000","last_active":"2024-03-26T05:01:57.072115+0000","last_peered":"2024-03-26T05:01:57.072115+0000","last_clean":"2024-03-26T05:01:57.072115+0000","last_became_active":"2024-03-26T05:00:26.957142+0000","last_became_peered":"2024-03-26T05:00:26.957142+0000","last_unstale":"2024-03-26T05:01:57.072115+0000","last_undegraded":"2024-03-26T05:01:57.072115+0000","last_fullsized":"2024-03-26T05:01:57.072115+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:43.782490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:43.782490+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.086882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.662338+0000","last_change":"2024-03-26T05:01:39.142373+0000","last_active":"2024-03-26T05:01:56.662338+0000","last_peered":"2024-03-26T05:01:56.662338+0000","last_clean":"2024-03-26T05:01:56.662338+0000","last_became_active":"2024-03-26T05:00:22.778719+0000","last_became_peered":"2024-03-26T05:00:22.778719+0000","last_unstale":"2024-03-26T05:01:56.662338+0000","last_undegraded":"2024-03-26T05:01:56.662338+0000","last_fullsized":"2024-03-26T05:01:56.662338+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.142249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.142249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.076132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":94,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T05:01:56.315002+0000","last_change":"2024-03-26T05:01:44.403960+0000","last_active":"2024-03-26T05:01:56.315002+0000","last_peered":"2024-03-26T05:01:56.315002+0000","last_clean":"2024-03-26T05:01:56.315002+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:01:56.315002+0000","last_undegraded":"2024-03-26T05:01:56.315002+0000","last_fullsized":"2024-03-26T05:01:56.315002+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:01:44.403798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.403798+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:04.259798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":609,"ondisk_log_size":609,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005080,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1638400,"data_stored":1095369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332125,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32764,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712388,"statfs":{"total":95995035648,"available":95961485312,"internally_reserved":0,"allocated":1622016,"data_stored":1093853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":504749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28632,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716520,"statfs":{"total":95995035648,"available":95965716480,"internally_reserved":0,"allocated":1650688,"data_stored":1095522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":503299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117036,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":503965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920808,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1052672,"data_stored":504554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659170,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":503811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:02:03.216 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T05:02:03.216 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd dump 2024-03-26T05:02:03.278 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.038211226Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.042038144Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.046005892Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.046434911Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.050603042Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.054890628Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.059134792Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.05943498Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.061247236Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.061745119Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.063131804Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.065014184Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.065488581Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T05:02:03.279 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.067340524Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T05:02:03.280 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.067739606Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T05:02:03.280 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.06923615Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T05:02:03.280 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.06954232Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T05:02:03.280 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.071438112Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T05:02:03.280 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.072086391Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T05:02:03.280 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.074216074Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T05:02:03.281 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.076373354Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T05:02:03.281 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.07851532Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T05:02:03.281 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.078955518Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T05:02:03.281 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.081044014Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T05:02:03.281 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.08315769Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T05:02:03.281 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.085370033Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T05:02:03.281 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.085775512Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T05:02:03.281 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.087935064Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T05:02:03.281 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.088807498Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T05:02:03.281 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.090985848Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T05:02:03.281 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.091448482Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T05:02:03.281 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.092237514Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T05:02:03.282 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.092647113Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T05:02:03.282 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.093159101Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T05:02:03.282 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.095045927Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T05:02:03.282 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.095371834Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T05:02:03.282 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.096237581Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T05:02:03.282 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.096672934Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T05:02:03.282 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.098123879Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T05:02:03.282 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.100000781Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T05:02:03.282 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.10043035Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T05:02:03.282 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.102642586Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T05:02:03.282 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.103530873Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T05:02:03.282 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.107458627Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.108506009Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.111385064Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.114352293Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.118526088Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.11889751Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.12178016Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.124448459Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.127202742Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.12753951Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.128388375Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.12923613Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.130115758Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.130654556Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T05:02:03.283 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.131445511Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.131910316Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.132360384Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.133204903Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.133648252Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.13452275Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.13538034Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.136284967Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.13886377Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.141674219Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.144589698Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.147376426Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.150216598Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.152971742Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T05:02:03.284 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.155809518Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T05:02:03.285 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.156149367Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T05:02:03.285 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.158711526Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T05:02:03.285 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.161448036Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T05:02:03.285 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.164246657Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T05:02:03.285 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.167117733Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T05:02:03.286 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.169979105Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T05:02:03.286 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.172933218Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T05:02:03.286 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.175667141Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T05:02:03.286 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.178306175Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T05:02:03.286 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.181154728Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T05:02:03.286 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.183963455Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T05:02:03.286 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.186782965Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.189237719Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.192062231Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.194789743Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.197514185Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.20022857Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.203001968Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.205710314Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.208405847Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.211011003Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.213700216Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.21413272Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.216766459Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.21951219Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T05:02:03.287 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.222178004Z level=info msg="Executing migration" id="create session table" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.222633886Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.222940411Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.225068507Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.225530097Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.228231548Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.230747745Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.23109575Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.231404769Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.231687003Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.23209196Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.234726834Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.235416913Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.235741859Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T05:02:03.288 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.236338118Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.237155145Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.237978668Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.238416056Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.239241175Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.240117959Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.240905286Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.241358562Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.241767612Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.242151809Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.242581715Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.242973085Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.244367503Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T05:02:03.289 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.24474512Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.245192727Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.245517487Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.245872442Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.246270636Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.246874073Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.247508992Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.248142164Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.248754216Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.249226492Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.249514905Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.249787089Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.250187128Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.250662306Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T05:02:03.290 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.251020401Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.251409848Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.251806908Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.252439496Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.253091116Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.253423647Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.253846347Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.254259548Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.254889661Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.255218008Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.255617753Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.256033438Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.256357864Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.256788995Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T05:02:03.291 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.257588748Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.258020432Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.258484263Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.258957202Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.259727042Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.259986697Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.260650538Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.261510391Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.262341838Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.263172218Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.264023548Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.264850829Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.266465278Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.2673162Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T05:02:03.292 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.268128101Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.26901648Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.270193715Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.271287696Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.272095536Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.272962903Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.273801335Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.274667788Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.275509763Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.276356156Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.277217416Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.278059374Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.27893474Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.279786429Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.280671524Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T05:02:03.293 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.281523538Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T05:02:03.294 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.282401605Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T05:02:03.294 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.2833526Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T05:02:03.294 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.286124622Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T05:02:03.294 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.288957002Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T05:02:03.294 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.289428634Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T05:02:03.294 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.29186177Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T05:02:03.468 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:03.530 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.294708701Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T05:02:03.530 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.297651085Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T05:02:03.530 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.300386079Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T05:02:03.530 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.300999045Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T05:02:03.530 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.303706227Z level=info msg="Executing migration" id="create team table" 2024-03-26T05:02:03.530 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.306618311Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T05:02:03.530 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.309442583Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T05:02:03.531 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.312512135Z level=info msg="Executing migration" id="create team member table" 2024-03-26T05:02:03.531 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.315353618Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T05:02:03.531 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.318066307Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T05:02:03.531 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.320845839Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T05:02:03.531 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.323752886Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T05:02:03.531 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.326603687Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T05:02:03.531 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.329083184Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T05:02:03.531 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.331731548Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T05:02:03.531 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.334488909Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T05:02:03.531 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.337303096Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T05:02:03.531 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.339826443Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T05:02:03.532 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.342443083Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T05:02:03.532 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.345278518Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T05:02:03.532 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.347770666Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T05:02:03.532 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.350466449Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T05:02:03.532 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.353140649Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T05:02:03.532 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.355745898Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T05:02:03.532 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.358486997Z level=info msg="Executing migration" id="create tag table" 2024-03-26T05:02:03.532 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.36122609Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T05:02:03.532 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.361703378Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.364496367Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.367308351Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.370182745Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.372084708Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.372878968Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.373749996Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.37455698Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.374984822Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.375746226Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.376606167Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.377371644Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.378145823Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T05:02:03.533 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.3789367Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.379770776Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.380635091Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.381115649Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.381925152Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.382759225Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.383597911Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.384056056Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.384862593Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.385711339Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.386589859Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.387457935Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.388337966Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T05:02:03.534 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.38919278Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.390016493Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.390468351Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.391283198Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.391612984Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.392169658Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.393014798Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.393462056Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.394276011Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.394717163Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.395211991Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.396059468Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.396515877Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T05:02:03.535 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.39911627Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T05:02:03.536 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.40213512Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T05:02:03.536 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.404882496Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T05:02:03.536 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.405335791Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T05:02:03.536 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.408183528Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T05:02:03.536 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.408637847Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T05:02:03.536 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.412073571Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T05:02:03.536 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.412531848Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T05:02:03.536 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.415572092Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T05:02:03.537 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.416034369Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T05:02:03.537 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.418858451Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T05:02:03.537 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.421717829Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T05:02:03.537 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.424590379Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T05:02:03.537 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.42724173Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T05:02:03.537 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.430675146Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.43403398Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.436868832Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.43937813Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.440214911Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.442977679Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.445719281Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.448487972Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.448956259Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.451610515Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.454139012Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.456875182Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.457725889Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T05:02:03.538 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.460561192Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.463323752Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.466042524Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.466952908Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.469589565Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.472198658Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.47496778Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.475315654Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.47609161Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.47695771Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.477769201Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.478106494Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.478978242Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.479823777Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.480698895Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T05:02:03.539 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.481571325Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.482468849Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.483340221Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.484101822Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.48493827Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.485779441Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.486723664Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.487513928Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.488419565Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.489280475Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.4901258Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.491000017Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.491431151Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.49229462Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.493166569Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T05:02:03.540 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.494011182Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.494337888Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.495174338Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.496010548Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.496090759Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.496884561Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.497257661Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.498069625Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.498187701Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.498296845Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.498680576Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.499172303Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.500012868Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T05:02:03.541 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.500414651Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.501304224Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.502172084Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.504911509Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.505397707Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.508150169Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.510729122Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.515101224Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.517979634Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.520764759Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.525038268Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.52937554Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.532004633Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T05:02:03.542 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.534647642Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T05:02:03.543 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.537351536Z level=info msg="Executing migration" id="create permission table" 2024-03-26T05:02:03.543 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.537803457Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T05:02:03.543 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.540488323Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T05:02:03.543 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.543231569Z level=info msg="Executing migration" id="create role table" 2024-03-26T05:02:03.794 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.545832442Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T05:02:03.794 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.546782508Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T05:02:03.794 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.549389621Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.552141861Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.554759703Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.555243839Z level=info msg="Executing migration" id="create team role table" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.558047109Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.560627276Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.563077197Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.563550486Z level=info msg="Executing migration" id="create user role table" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.56436126Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.565207315Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.566112381Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.56670584Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.56754875Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.568464659Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T05:02:03.795 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.569182417Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T05:02:03.796 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.570199754Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T05:02:03.796 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.570691981Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T05:02:03.796 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.571424205Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.57186692Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.572331905Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.573106517Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.573585474Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.57458495Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.575017773Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.575479169Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.575763133Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.576206431Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.576691832Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.577101329Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.5774284Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T05:02:03.797 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.577775965Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.578196452Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.578655369Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.579685929Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.580011705Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.580457427Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.580902478Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.581359013Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.582366235Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.582818746Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.583266486Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.583365042Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.583698453Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T05:02:03.798 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.583796114Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.584099973Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.584527985Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.584925036Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.585364164Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.585792941Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.586214576Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.586636433Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.587073867Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.587498592Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.587906638Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.58837899Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.589278579Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.592519457Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T05:02:03.799 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.593545512Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.594568646Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.594886718Z level=info msg="Executing migration" id="add share column" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.595887121Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.596340689Z level=info msg="Executing migration" id="create file table" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.596774312Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.59756601Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.59806967Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.598662252Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.599560523Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.600449727Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.601161268Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.601510261Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.602339352Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T05:02:03.800 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.603437003Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.603781665Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.604639451Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.60508133Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.606035861Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.606779346Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.607840876Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.608874905Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.609741878Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.619476234Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.620430728Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.621179423Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.624659095Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.625508482Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T05:02:03.801 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.626398978Z level=info msg="Executing migration" id="create folder table" 2024-03-26T05:02:03.802 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.626868024Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T05:02:03.802 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.629873073Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T05:02:03.802 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.632705025Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T05:02:03.802 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.635445017Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T05:02:03.802 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=migrator t=2024-03-26T05:02:03.635932288Z level=info msg="migrations completed" performed=485 skipped=0 duration=727.466809ms 2024-03-26T05:02:03.802 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=sqlstore t=2024-03-26T05:02:03.636351582Z level=info msg="Created default organization" 2024-03-26T05:02:03.802 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=plugin.loader t=2024-03-26T05:02:03.693320221Z level=info msg="Plugin registered" pluginID=input 2024-03-26T05:02:03.802 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=plugin.loader t=2024-03-26T05:02:03.721681679Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-26T05:02:03.802 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=plugin.loader t=2024-03-26T05:02:03.721714108Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-26T05:02:03.802 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=secrets t=2024-03-26T05:02:03.721747548Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T05:02:03.802 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=query_data t=2024-03-26T05:02:03.723972235Z level=info msg="Query Service initialization" 2024-03-26T05:02:03.802 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=live.push_http t=2024-03-26T05:02:03.726573352Z level=info msg="Live Push Gateway initialization" 2024-03-26T05:02:04.111 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:02:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:02:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:02:04.111 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: 3.11 scrub starts 2024-03-26T05:02:04.111 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: 3.11 scrub ok 2024-03-26T05:02:04.111 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:02:04.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: 6.e scrub starts 2024-03-26T05:02:04.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: 6.e scrub ok 2024-03-26T05:02:04.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: Deploying daemon node-exporter.a on smithi080 2024-03-26T05:02:04.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: 5.1e scrub starts 2024-03-26T05:02:04.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: 5.1e scrub ok 2024-03-26T05:02:04.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:04.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: 3.11 scrub starts 2024-03-26T05:02:04.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: 3.11 scrub ok 2024-03-26T05:02:04.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:02:04.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: 6.e scrub starts 2024-03-26T05:02:04.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: 6.e scrub ok 2024-03-26T05:02:04.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: Deploying daemon node-exporter.a on smithi080 2024-03-26T05:02:04.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: 5.1e scrub starts 2024-03-26T05:02:04.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: 5.1e scrub ok 2024-03-26T05:02:04.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:03 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:04.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: 3.11 scrub starts 2024-03-26T05:02:04.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: 3.11 scrub ok 2024-03-26T05:02:04.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:02:04.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: 6.e scrub starts 2024-03-26T05:02:04.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: 6.e scrub ok 2024-03-26T05:02:04.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: Deploying daemon node-exporter.a on smithi080 2024-03-26T05:02:04.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: 5.1e scrub starts 2024-03-26T05:02:04.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: 5.1e scrub ok 2024-03-26T05:02:04.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:04.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:03 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:04.752 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=infra.usagestats.collector t=2024-03-26T05:02:04.33139613Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T05:02:04.752 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=provisioning.datasources t=2024-03-26T05:02:04.332361703Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T05:02:04.752 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=provisioning.datasources t=2024-03-26T05:02:04.34326759Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T05:02:04.752 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=provisioning.alerting t=2024-03-26T05:02:04.354224806Z level=info msg="starting to provision alerting" 2024-03-26T05:02:04.752 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=provisioning.alerting t=2024-03-26T05:02:04.354261385Z level=info msg="finished to provision alerting" 2024-03-26T05:02:04.752 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=ngalert.state.manager t=2024-03-26T05:02:04.354443181Z level=info msg="Warming state cache for startup" 2024-03-26T05:02:04.752 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=grafanaStorageLogger t=2024-03-26T05:02:04.354687607Z level=info msg="storage starting" 2024-03-26T05:02:04.752 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=http.server t=2024-03-26T05:02:04.358009334Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T05:02:04.752 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=ngalert.state.manager t=2024-03-26T05:02:04.36185789Z level=info msg="State cache has been initialized" states=0 duration=7.412773ms 2024-03-26T05:02:04.752 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=ngalert.multiorg.alertmanager t=2024-03-26T05:02:04.361926085Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T05:02:04.752 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=ticker t=2024-03-26T05:02:04.361956612Z level=info msg=starting first_tick=2024-03-26T05:02:10Z 2024-03-26T05:02:04.753 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=sqlstore.transactions t=2024-03-26T05:02:04.372606341Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T05:02:04.753 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=sqlstore.transactions t=2024-03-26T05:02:04.372953645Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T05:02:04.753 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=sqlstore.transactions t=2024-03-26T05:02:04.383745448Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-26T05:02:05.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-mon[33500]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:02:05.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-mon[33500]: 3.0 scrub starts 2024-03-26T05:02:05.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-mon[33500]: 3.0 scrub ok 2024-03-26T05:02:05.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:02:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:02:05.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:04 smithi080 ceph-mon[40404]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:02:05.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:04 smithi080 ceph-mon[40404]: 3.0 scrub starts 2024-03-26T05:02:05.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:04 smithi080 ceph-mon[40404]: 3.0 scrub ok 2024-03-26T05:02:05.179 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:04 smithi080 ceph-mon[29691]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:02:05.179 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:04 smithi080 ceph-mon[29691]: 3.0 scrub starts 2024-03-26T05:02:05.179 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:04 smithi080 ceph-mon[29691]: 3.0 scrub ok 2024-03-26T05:02:05.657 INFO:teuthology.orchestra.run.smithi080.stdout:epoch 100 2024-03-26T05:02:05.657 INFO:teuthology.orchestra.run.smithi080.stdout:fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T05:02:05.658 INFO:teuthology.orchestra.run.smithi080.stdout:created 2024-03-26T04:54:51.106954+0000 2024-03-26T05:02:05.658 INFO:teuthology.orchestra.run.smithi080.stdout:modified 2024-03-26T05:01:56.310994+0000 2024-03-26T05:02:05.658 INFO:teuthology.orchestra.run.smithi080.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-26T05:02:05.658 INFO:teuthology.orchestra.run.smithi080.stdout:crush_version 18 2024-03-26T05:02:05.658 INFO:teuthology.orchestra.run.smithi080.stdout:full_ratio 0.95 2024-03-26T05:02:05.658 INFO:teuthology.orchestra.run.smithi080.stdout:backfillfull_ratio 0.9 2024-03-26T05:02:05.658 INFO:teuthology.orchestra.run.smithi080.stdout:nearfull_ratio 0.85 2024-03-26T05:02:05.658 INFO:teuthology.orchestra.run.smithi080.stdout:require_min_compat_client luminous 2024-03-26T05:02:05.658 INFO:teuthology.orchestra.run.smithi080.stdout:min_compat_client luminous 2024-03-26T05:02:05.658 INFO:teuthology.orchestra.run.smithi080.stdout:require_osd_release squid 2024-03-26T05:02:05.658 INFO:teuthology.orchestra.run.smithi080.stdout:stretch_mode_enabled false 2024-03-26T05:02:05.658 INFO:teuthology.orchestra.run.smithi080.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 71 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 73 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:max_osd 8 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.80:6802/1840686622,v1:172.21.15.80:6803/1840686622] [v2:172.21.15.80:6804/1840686622,v1:172.21.15.80:6805/1840686622] exists,up 35a52544-0b0b-47d6-8d6a-571c48ceb366 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 up in weight 1 up_from 13 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.80:6810/1726833996,v1:172.21.15.80:6811/1726833996] [v2:172.21.15.80:6812/1726833996,v1:172.21.15.80:6813/1726833996] exists,up efa5ea33-4f11-49b7-a3f3-f60d5329db23 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.80:6818/128713757,v1:172.21.15.80:6819/128713757] [v2:172.21.15.80:6820/128713757,v1:172.21.15.80:6821/128713757] exists,up 536bec9b-9636-4e67-b452-ed1a55c22ab6 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 up in weight 1 up_from 27 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.80:6826/418771441,v1:172.21.15.80:6827/418771441] [v2:172.21.15.80:6828/418771441,v1:172.21.15.80:6829/418771441] exists,up 640cfdb6-b9c5-446d-8312-504ae12c7818 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 up in weight 1 up_from 33 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6800/1441071423,v1:172.21.15.132:6801/1441071423] [v2:172.21.15.132:6802/1441071423,v1:172.21.15.132:6803/1441071423] exists,up 0cc7822a-ed73-4c79-a30d-1e151ddd6639 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 up in weight 1 up_from 41 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6808/794504625,v1:172.21.15.132:6809/794504625] [v2:172.21.15.132:6810/794504625,v1:172.21.15.132:6811/794504625] exists,up acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc 2024-03-26T05:02:05.659 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 up in weight 1 up_from 51 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6816/2209078972,v1:172.21.15.132:6817/2209078972] [v2:172.21.15.132:6818/2209078972,v1:172.21.15.132:6819/2209078972] exists,up 2970bbab-11b2-4d1b-b57e-037bdac089b5 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 up in weight 1 up_from 61 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6824/3655388282,v1:172.21.15.132:6825/3655388282] [v2:172.21.15.132:6826/3655388282,v1:172.21.15.132:6827/3655388282] exists,up 12f4b466-28f9-469b-8fd0-65b91c6dc22d 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:pg_upmap_items 4.2 [1,2] 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:pg_upmap_items 4.6 [1,5] 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:pg_upmap_items 4.1f [1,2] 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:0/3992539385 expires 2024-03-27T05:01:03.689739+0000 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:6800/2770889532 expires 2024-03-27T04:55:48.542948+0000 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:6801/2466284312 expires 2024-03-27T04:55:21.711776+0000 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:0/490056571 expires 2024-03-27T04:55:21.711776+0000 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:0/1986135300 expires 2024-03-27T04:55:21.711776+0000 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:0/3796611170 expires 2024-03-27T04:55:21.711776+0000 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:0/1993138634 expires 2024-03-27T04:55:48.542948+0000 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:0/1832498181 expires 2024-03-27T05:01:03.689739+0000 2024-03-26T05:02:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:6800/2466284312 expires 2024-03-27T04:55:21.711776+0000 2024-03-26T05:02:05.661 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:0/1650870847 expires 2024-03-27T05:01:03.689739+0000 2024-03-26T05:02:05.661 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:6801/1418683843 expires 2024-03-27T05:01:03.689739+0000 2024-03-26T05:02:05.661 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:0/3147301987 expires 2024-03-27T05:01:03.689739+0000 2024-03-26T05:02:05.661 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:6801/2770889532 expires 2024-03-27T04:55:48.542948+0000 2024-03-26T05:02:05.661 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:0/571905941 expires 2024-03-27T04:55:48.542948+0000 2024-03-26T05:02:05.661 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:0/1451012560 expires 2024-03-27T04:55:48.542948+0000 2024-03-26T05:02:05.661 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:0/1235125152 expires 2024-03-27T05:01:03.689739+0000 2024-03-26T05:02:05.661 INFO:teuthology.orchestra.run.smithi080.stdout:blocklist 172.21.15.80:6800/1418683843 expires 2024-03-27T05:01:03.689739+0000 2024-03-26T05:02:05.952 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:05 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:05.952 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:05 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2172925536' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T05:02:05.953 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:05 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:05.953 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:05 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2172925536' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T05:02:06.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:05 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:06.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:05 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2172925536' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T05:02:06.518 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.80:6802/1840686622,v1:172.21.15.80:6803/1840686622] [v2:172.21.15.80:6804/1840686622,v1:172.21.15.80:6805/1840686622] exists,up 35a52544-0b0b-47d6-8d6a-571c48ceb366', 'osd.1 up in weight 1 up_from 13 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.80:6810/1726833996,v1:172.21.15.80:6811/1726833996] [v2:172.21.15.80:6812/1726833996,v1:172.21.15.80:6813/1726833996] exists,up efa5ea33-4f11-49b7-a3f3-f60d5329db23', 'osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.80:6818/128713757,v1:172.21.15.80:6819/128713757] [v2:172.21.15.80:6820/128713757,v1:172.21.15.80:6821/128713757] exists,up 536bec9b-9636-4e67-b452-ed1a55c22ab6', 'osd.3 up in weight 1 up_from 27 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.80:6826/418771441,v1:172.21.15.80:6827/418771441] [v2:172.21.15.80:6828/418771441,v1:172.21.15.80:6829/418771441] exists,up 640cfdb6-b9c5-446d-8312-504ae12c7818', 'osd.4 up in weight 1 up_from 33 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6800/1441071423,v1:172.21.15.132:6801/1441071423] [v2:172.21.15.132:6802/1441071423,v1:172.21.15.132:6803/1441071423] exists,up 0cc7822a-ed73-4c79-a30d-1e151ddd6639', 'osd.5 up in weight 1 up_from 41 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6808/794504625,v1:172.21.15.132:6809/794504625] [v2:172.21.15.132:6810/794504625,v1:172.21.15.132:6811/794504625] exists,up acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc', 'osd.6 up in weight 1 up_from 51 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6816/2209078972,v1:172.21.15.132:6817/2209078972] [v2:172.21.15.132:6818/2209078972,v1:172.21.15.132:6819/2209078972] exists,up 2970bbab-11b2-4d1b-b57e-037bdac089b5', 'osd.7 up in weight 1 up_from 61 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6824/3655388282,v1:172.21.15.132:6825/3655388282] [v2:172.21.15.132:6826/3655388282,v1:172.21.15.132:6827/3655388282] exists,up 12f4b466-28f9-469b-8fd0-65b91c6dc22d'] 2024-03-26T05:02:06.519 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon mon.a config show 2024-03-26T05:02:06.770 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:07.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[29691]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:07.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[29691]: 3.d scrub starts 2024-03-26T05:02:07.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[29691]: 3.d scrub ok 2024-03-26T05:02:07.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[29691]: 6.1a scrub starts 2024-03-26T05:02:07.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[29691]: 6.1a scrub ok 2024-03-26T05:02:07.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[29691]: 3.8 scrub starts 2024-03-26T05:02:07.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[29691]: 3.8 scrub ok 2024-03-26T05:02:07.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[40404]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:07.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[40404]: 3.d scrub starts 2024-03-26T05:02:07.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[40404]: 3.d scrub ok 2024-03-26T05:02:07.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[40404]: 6.1a scrub starts 2024-03-26T05:02:07.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[40404]: 6.1a scrub ok 2024-03-26T05:02:07.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[40404]: 3.8 scrub starts 2024-03-26T05:02:07.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:07 smithi080 ceph-mon[40404]: 3.8 scrub ok 2024-03-26T05:02:07.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:07 smithi132 ceph-mon[33500]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:07.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:07 smithi132 ceph-mon[33500]: 3.d scrub starts 2024-03-26T05:02:07.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:07 smithi132 ceph-mon[33500]: 3.d scrub ok 2024-03-26T05:02:07.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:07 smithi132 ceph-mon[33500]: 6.1a scrub starts 2024-03-26T05:02:07.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:07 smithi132 ceph-mon[33500]: 6.1a scrub ok 2024-03-26T05:02:07.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:07 smithi132 ceph-mon[33500]: 3.8 scrub starts 2024-03-26T05:02:07.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:07 smithi132 ceph-mon[33500]: 3.8 scrub ok 2024-03-26T05:02:08.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:08 smithi080 ceph-mon[29691]: 4.7 scrub starts 2024-03-26T05:02:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:08 smithi080 ceph-mon[29691]: 4.7 scrub ok 2024-03-26T05:02:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:08 smithi080 ceph-mon[29691]: 3.15 scrub starts 2024-03-26T05:02:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:08 smithi080 ceph-mon[29691]: 3.15 scrub ok 2024-03-26T05:02:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:08 smithi080 ceph-mon[29691]: 6.16 scrub starts 2024-03-26T05:02:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:08 smithi080 ceph-mon[29691]: 6.16 scrub ok 2024-03-26T05:02:08.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:08 smithi080 ceph-mon[40404]: 4.7 scrub starts 2024-03-26T05:02:08.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:08 smithi080 ceph-mon[40404]: 4.7 scrub ok 2024-03-26T05:02:08.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:08 smithi080 ceph-mon[40404]: 3.15 scrub starts 2024-03-26T05:02:08.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:08 smithi080 ceph-mon[40404]: 3.15 scrub ok 2024-03-26T05:02:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:08 smithi080 ceph-mon[40404]: 6.16 scrub starts 2024-03-26T05:02:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:08 smithi080 ceph-mon[40404]: 6.16 scrub ok 2024-03-26T05:02:08.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:08 smithi132 ceph-mon[33500]: 4.7 scrub starts 2024-03-26T05:02:08.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:08 smithi132 ceph-mon[33500]: 4.7 scrub ok 2024-03-26T05:02:08.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:08 smithi132 ceph-mon[33500]: 3.15 scrub starts 2024-03-26T05:02:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:08 smithi132 ceph-mon[33500]: 3.15 scrub ok 2024-03-26T05:02:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:08 smithi132 ceph-mon[33500]: 6.16 scrub starts 2024-03-26T05:02:08.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:08 smithi132 ceph-mon[33500]: 6.16 scrub ok 2024-03-26T05:02:08.868 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:08 smithi080 bash[76548]: Writing manifest to image destination 2024-03-26T05:02:09.315 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 podman[76548]: 2024-03-26 05:02:09.01610309 +0000 UTC m=+4.049126568 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T05:02:09.315 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 podman[76548]: 2024-03-26 05:02:09.223404796 +0000 UTC m=+4.256428317 container create cb2f32aef494cc3676fbb30e776b930dc07143735b1f50c585ca2391196b1590 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:02:09.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-mon[29691]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 898 B/s rd, 0 op/s 2024-03-26T05:02:09.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-mon[29691]: 3.19 scrub starts 2024-03-26T05:02:09.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-mon[29691]: 3.19 scrub ok 2024-03-26T05:02:09.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-mon[29691]: 6.12 scrub starts 2024-03-26T05:02:09.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-mon[29691]: 6.12 scrub ok 2024-03-26T05:02:09.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-mon[40404]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 898 B/s rd, 0 op/s 2024-03-26T05:02:09.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-mon[40404]: 3.19 scrub starts 2024-03-26T05:02:09.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-mon[40404]: 3.19 scrub ok 2024-03-26T05:02:09.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-mon[40404]: 6.12 scrub starts 2024-03-26T05:02:09.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-mon[40404]: 6.12 scrub ok 2024-03-26T05:02:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:09 smithi132 ceph-mon[33500]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 898 B/s rd, 0 op/s 2024-03-26T05:02:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:09 smithi132 ceph-mon[33500]: 3.19 scrub starts 2024-03-26T05:02:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:09 smithi132 ceph-mon[33500]: 3.19 scrub ok 2024-03-26T05:02:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:09 smithi132 ceph-mon[33500]: 6.12 scrub starts 2024-03-26T05:02:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:09 smithi132 ceph-mon[33500]: 6.12 scrub ok 2024-03-26T05:02:09.939 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 podman[76548]: 2024-03-26 05:02:09.65566774 +0000 UTC m=+4.688691218 container init cb2f32aef494cc3676fbb30e776b930dc07143735b1f50c585ca2391196b1590 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:02:09.939 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 podman[76548]: 2024-03-26 05:02:09.658565626 +0000 UTC m=+4.691589109 container start cb2f32aef494cc3676fbb30e776b930dc07143735b1f50c585ca2391196b1590 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.662Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.662Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.662Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.662Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T05:02:09.940 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T05:02:09.941 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T05:02:09.941 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T05:02:09.941 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T05:02:09.941 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T05:02:09.941 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T05:02:09.941 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T05:02:09.941 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T05:02:09.942 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T05:02:09.943 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T05:02:09.943 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T05:02:09.943 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T05:02:09.943 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.663Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T05:02:09.943 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T05:02:09.943 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T05:02:09.943 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T05:02:09.943 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T05:02:09.943 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T05:02:09.944 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T05:02:09.944 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T05:02:09.944 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T05:02:09.944 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T05:02:09.944 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T05:02:09.944 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T05:02:09.944 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T05:02:09.944 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T05:02:09.945 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T05:02:09.945 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T05:02:09.945 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a[76757]: ts=2024-03-26T05:02:09.664Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T05:02:09.945 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 bash[76548]: cb2f32aef494cc3676fbb30e776b930dc07143735b1f50c585ca2391196b1590 2024-03-26T05:02:09.945 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:02:09 smithi080 systemd[1]: Started Ceph node-exporter.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:02:09.963 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:02:09.963 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "mon.a", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster": "ceph", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "admin_socket_mode": "", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "auth_client_required": "cephx, none", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "auth_cluster_required": "cephx", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "auth_debug": "true", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "auth_service_required": "cephx", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "auth_supported": "", 2024-03-26T05:02:09.964 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_aio": "true", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_aio_poll_ms": "250", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_aio_reap_max": "16", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_async_discard": "false", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_block_size": "4096", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_debug_aio": "false", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_debug_inflight_ios": "false", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_enable_discard": "false", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_flock_retry": "3", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-26T05:02:09.965 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_inject_crash": "0", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_ioring": "false", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_ioring_hipri": "false", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bdev_type": "", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_alloc_size": "1048576", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_allocator": "hybrid", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_buffered_io": "true", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_check_for_zeros": "false", 2024-03-26T05:02:09.966 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_compact_log_sync": "false", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_max_log_runway": "4194304", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_max_prefetch": "1048576", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_min_flush_size": "524288", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_min_log_runway": "1048576", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_replay_recovery": "false", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-26T05:02:09.967 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_sync_write": "false", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_allocation_from_file": "true", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_allocator": "hybrid", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-26T05:02:09.968 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_block_create": "true", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_block_db_create": "false", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_block_db_path": "", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_block_db_size": "0", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_block_path": "", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_block_preallocate_file": "false", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_block_size": "107374182400", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_block_wal_create": "false", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_block_wal_path": "", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_block_wal_size": "100663296", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_bluefs": "true", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-26T05:02:09.969 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-26T05:02:09.970 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-26T05:02:09.970 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-26T05:02:09.970 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-26T05:02:09.970 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-26T05:02:09.970 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_autotune": "true", 2024-03-26T05:02:09.970 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-26T05:02:09.970 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-26T05:02:09.970 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-26T05:02:09.970 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-26T05:02:09.970 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_size": "0", 2024-03-26T05:02:09.970 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_type": "2q", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_clone_cow": "true", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-26T05:02:09.971 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_compression_mode": "none", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_csum_type": "crc32c", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_freelist": "false", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-26T05:02:09.972 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-26T05:02:09.973 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-26T05:02:09.973 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-26T05:02:09.973 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-26T05:02:09.973 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-26T05:02:09.973 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_small_allocations": "0", 2024-03-26T05:02:09.973 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-26T05:02:09.973 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_default_buffered_read": "true", 2024-03-26T05:02:09.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_default_buffered_write": "false", 2024-03-26T05:02:09.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-26T05:02:09.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-26T05:02:09.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-26T05:02:09.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-26T05:02:09.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-26T05:02:09.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-26T05:02:09.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-26T05:02:09.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-26T05:02:09.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-26T05:02:09.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fail_eio": "false", 2024-03-26T05:02:09.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_on_mount": "false", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_on_umount": "false", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-26T05:02:09.975 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_ignore_data_csum": "false", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_log_op_age": "5.000000", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_max_alloc_size": "0", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_max_blob_size": "0", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-26T05:02:09.976 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_max_deferred_txc": "32", 2024-03-26T05:02:09.977 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_min_alloc_size": "0", 2024-03-26T05:02:09.977 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-26T05:02:09.977 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-26T05:02:09.977 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_nid_prealloc": "1024", 2024-03-26T05:02:09.977 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-26T05:02:09.977 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-26T05:02:09.977 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-26T05:02:09.977 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-26T05:02:09.977 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_retry_disk_reads": "3", 2024-03-26T05:02:09.977 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_rocksdb_cf": "true", 2024-03-26T05:02:09.977 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-26T05:02:09.978 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-26T05:02:09.978 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-26T05:02:09.978 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_spdk_mem": "512", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_tracing": "false", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-26T05:02:09.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_zero_block_detection": "false", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-26T05:02:09.980 INFO:teuthology.orchestra.run.smithi080.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-26T05:02:09.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-26T05:02:09.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-26T05:02:09.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-26T05:02:09.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-26T05:02:09.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-26T05:02:09.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-26T05:02:09.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cephx_cluster_require_signatures": "false", 2024-03-26T05:02:09.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cephx_cluster_require_version": "2", 2024-03-26T05:02:09.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cephx_require_signatures": "false", 2024-03-26T05:02:09.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cephx_require_version": "2", 2024-03-26T05:02:09.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cephx_service_require_signatures": "false", 2024-03-26T05:02:09.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cephx_service_require_version": "2", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "cephx_sign_messages": "true", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "chdir": "", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "client_acl_type": "", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "client_asio_thread_count": "2", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "client_cache_mid": "0.750000", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "client_cache_size": "16384", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "client_caps_release_delay": "5", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "client_check_pool_perm": "true", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "client_debug_force_sync_read": "false", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "client_debug_getattr_caps": "false", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "client_debug_inject_tick_delay": "0", 2024-03-26T05:02:09.982 INFO:teuthology.orchestra.run.smithi080.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_die_on_failed_remount": "false", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_dirsize_rbytes": "true", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_force_lazyio": "false", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_fs": "", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_inject_release_failure": "false", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_max_inline_size": "4096", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_mds_namespace": "", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_metadata": "", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_mount_gid": "-1", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_mount_timeout": "300", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_mount_uid": "-1", 2024-03-26T05:02:09.983 INFO:teuthology.orchestra.run.smithi080.stdout: "client_mountpoint": "/", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_notify_timeout": "10", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_oc": "true", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_oc_max_dirty": "104857600", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_oc_max_objects": "1000", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_oc_size": "209715200", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_oc_target_dirty": "8388608", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_permissions": "true", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_quota": "true", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_quota_df": "true", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_readahead_max_bytes": "0", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_readahead_max_periods": "4", 2024-03-26T05:02:09.984 INFO:teuthology.orchestra.run.smithi080.stdout: "client_readahead_min": "131072", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "client_reconnect_stale": "false", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "client_shutdown_timeout": "30", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "client_snapdir": ".snap", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "client_tick_interval": "1", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "client_trace": "", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "client_try_dentry_invalidate": "false", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "client_use_faked_inos": "false", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "client_use_random_mds": "false", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "clog_to_graylog": "false", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "clog_to_graylog_port": "12201", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "clog_to_monitors": "default=true", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "clog_to_syslog": "false", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-26T05:02:09.985 INFO:teuthology.orchestra.run.smithi080.stdout: "clog_to_syslog_level": "info", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "-", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_network": "", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_network_interface": "", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "compressor_zlib_isal": "false", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "compressor_zlib_level": "5", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "compressor_zlib_winsize": "-15", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "compressor_zstd_level": "1", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "crimson_alien_op_num_threads": "6", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-26T05:02:09.986 INFO:teuthology.orchestra.run.smithi080.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "crimson_seastar_cpu_cores": "", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_location": "", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_location_hook": "", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_location_hook_timeout": "10", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "daemonize": "false", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "daos_pool": "tank", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_alienstore": "0/5", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_allow_any_pool_priority": "false", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_asok": "1/5", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_asok_assert_abort": "false", 2024-03-26T05:02:09.987 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_asserts_on_shutdown": "true", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_auth": "1/5", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_bdev": "1/3", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_bluefs": "1/5", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_bluestore": "1/5", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_buffer": "0/1", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_ceph_exporter": "1/5", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_cephfs_mirror": "0/5", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_cephsqlite": "0/5", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_client": "0/5", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_compressor": "1/5", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_context": "0/1", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_crimson_interrupt": "0/5", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_crush": "1/1", 2024-03-26T05:02:09.988 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_crypto": "1/5", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_cyanstore": "0/5", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_deliberately_leak_memory": "false", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_disable_randomized_ping": "false", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_dpdk": "1/5", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_eventtrace": "1/5", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_filer": "0/1", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_filestore": "1/3", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_finisher": "1/1", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_fuse": "1/5", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_heartbeat_testing_span": "0", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_heartbeatmap": "1/5", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_javaclient": "1/5", 2024-03-26T05:02:09.989 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_journal": "1/3", 2024-03-26T05:02:09.990 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_journaler": "0/5", 2024-03-26T05:02:09.990 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_kstore": "1/5", 2024-03-26T05:02:09.990 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_lockdep": "0/1", 2024-03-26T05:02:09.990 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_mclock": "1/5", 2024-03-26T05:02:09.990 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_mds": "1/5", 2024-03-26T05:02:09.990 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_mds_balancer": "1/5", 2024-03-26T05:02:09.990 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_mds_locker": "1/5", 2024-03-26T05:02:09.990 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_mds_log": "1/5", 2024-03-26T05:02:09.990 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_mds_log_expire": "1/5", 2024-03-26T05:02:09.990 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_mds_migrator": "1/5", 2024-03-26T05:02:09.990 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_mds_quiesce": "3/5", 2024-03-26T05:02:09.990 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_memstore": "1/5", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_mgr": "2/5", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_mgrc": "1/5", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_mon": "20/20", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_monc": "0/10", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_ms": "1/1", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_none": "0/5", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_objclass": "0/5", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_objectcacher": "0/5", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_objecter": "0/1", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_optracker": "0/5", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_osd": "1/5", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_paxos": "20/20", 2024-03-26T05:02:09.991 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_perfcounter": "1/5", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_prioritycache": "1/5", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rados": "0/5", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rbd": "0/5", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rbd_mirror": "0/5", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rbd_pwl": "0/5", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rbd_replay": "0/5", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_refs": "0/0", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_reserver": "1/1", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rgw": "1/5", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rgw_access": "1/5", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rgw_datacache": "1/5", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rgw_dbstore": "1/5", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rgw_flight": "1/5", 2024-03-26T05:02:09.992 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rgw_sync": "1/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_rocksdb": "4/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore": "0/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_backref": "0/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_cache": "0/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_cleaner": "0/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_device": "0/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_epm": "0/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_journal": "0/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_lba": "0/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_odata": "0/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_omap": "0/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_onode": "0/5", 2024-03-26T05:02:09.993 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_t": "0/5", 2024-03-26T05:02:09.994 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_seastore_tm": "0/5", 2024-03-26T05:02:09.994 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_striper": "0/1", 2024-03-26T05:02:09.994 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_test": "0/5", 2024-03-26T05:02:09.994 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_throttle": "1/1", 2024-03-26T05:02:09.994 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_timer": "0/1", 2024-03-26T05:02:09.994 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_tp": "0/5", 2024-03-26T05:02:09.994 INFO:teuthology.orchestra.run.smithi080.stdout: "debug_trace": "1/5", 2024-03-26T05:02:09.994 INFO:teuthology.orchestra.run.smithi080.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-03-26T05:02:09.994 INFO:teuthology.orchestra.run.smithi080.stdout: "device_failure_prediction_mode": "none", 2024-03-26T05:02:09.994 INFO:teuthology.orchestra.run.smithi080.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-26T05:02:09.994 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "err_to_graylog": "false", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "err_to_journald": "false", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "err_to_stderr": "true", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "err_to_syslog": "false", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "event_tracing": "false", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "exporter_addr": "0.0.0.0", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "exporter_http_port": "9926", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "exporter_prio_limit": "5", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "exporter_sort_metrics": "true", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "exporter_stats_period": "5", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "fake_statfs_for_testing": "0", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "fatal_signal_handlers": "true", 2024-03-26T05:02:09.995 INFO:teuthology.orchestra.run.smithi080.stdout: "filer_max_purge_ops": "10", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filer_max_truncate_ops": "128", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_apply_finisher_threads": "1", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_blackhole": "false", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_btrfs_clone_range": "true", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_btrfs_snap": "true", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_caller_concurrency": "10", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_collect_device_partition_information": "true", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_commit_timeout": "600.000000", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_debug_inject_read_err": "false", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_debug_omap_check": "false", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_debug_verify_split": "false", 2024-03-26T05:02:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_dump_file": "", 2024-03-26T05:02:09.997 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-26T05:02:09.997 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-26T05:02:09.997 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_fadvise": "true", 2024-03-26T05:02:09.997 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_fail_eio": "true", 2024-03-26T05:02:09.997 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_fd_cache_shards": "16", 2024-03-26T05:02:09.997 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_fd_cache_size": "128", 2024-03-26T05:02:09.997 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_fiemap": "false", 2024-03-26T05:02:09.997 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_fiemap_threshold": "4096", 2024-03-26T05:02:09.997 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-26T05:02:09.997 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-26T05:02:09.997 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_inject_stall": "0", 2024-03-26T05:02:09.997 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_journal_parallel": "false", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_journal_trailing": "false", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_journal_writeahead": "false", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_kill_at": "0", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_inline_xattrs": "0", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-26T05:02:09.998 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_xattr_value_size": "0", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_merge_threshold": "-10", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_odsync_write": "false", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_omap_backend": "rocksdb", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_omap_backend_path": "", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_op_thread_timeout": "60", 2024-03-26T05:02:09.999 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_op_threads": "2", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_punch_hole": "false", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_queue_max_ops": "50", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-26T05:02:10.000 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_seek_data_hole": "false", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_sloppy_crc": "false", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_splice": "false", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_split_multiple": "2", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_split_rand_factor": "20", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_update_to": "1000", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-26T05:02:10.001 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_enable": "true", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_xfs_extsize": "false", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "filestore_zfs_snap": "false", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "fio_dir": "/tmp/fio", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "fsid": "dd09e9cc-eb2c-11ee-95d0-87774f69a715", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_allow_other": "true", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_atomic_o_trunc": "true", 2024-03-26T05:02:10.002 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_big_writes": "true", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_debug": "false", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_default_permissions": "false", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_disable_pagecache": "false", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_max_write": "0", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_multithreaded": "true", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_require_active_mds": "true", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_set_user_groups": "true", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_splice_move": "true", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_splice_read": "true", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_splice_write": "true", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "fuse_use_invalidate_cb": "true", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-26T05:02:10.003 INFO:teuthology.orchestra.run.smithi080.stdout: "gss_target_name": "ceph", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_file": "", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_inject_failure": "0", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_interval": "5", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "host": "smithi080", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-26T05:02:10.004 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "inject_early_sigterm": "false", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "jaeger_agent_port": "6799", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "jaeger_tracing_enable": "false", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_aio": "true", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_align_min_size": "65536", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_block_align": "true", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_block_size": "4096", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_dio": "true", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_discard": "false", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_force_aio": "false", 2024-03-26T05:02:10.005 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_ignore_corruption": "false", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_max_write_bytes": "10485760", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_max_write_entries": "100", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_replay_from": "0", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_write_header_frequency": "0", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "journal_zero_on_create": "false", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "journaler_prefetch_periods": "10", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "journaler_prezero_periods": "5", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "journaler_write_head_interval": "15", 2024-03-26T05:02:10.006 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "keyfile": "", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "kstore_backend": "rocksdb", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "kstore_default_stripe_size": "65536", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "kstore_fsck_on_mount": "false", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "kstore_max_bytes": "67108864", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "kstore_max_ops": "512", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "kstore_nid_prealloc": "1024", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "kstore_onode_map_size": "1024", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "kstore_sync_submit_transaction": "false", 2024-03-26T05:02:10.007 INFO:teuthology.orchestra.run.smithi080.stdout: "kstore_sync_transaction": "false", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "librados_thread_count": "2", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "lockdep": "false", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "lockdep_force_backtrace": "false", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "log_coarse_timestamps": "true", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "log_flush_on_exit": "false", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "log_graylog_host": "127.0.0.1", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "log_graylog_port": "12201", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "log_max_new": "1000", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "log_max_recent": "10000", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "log_stderr_prefix": "", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "log_stop_at_utilization": "0.970000", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "log_to_file": "true", 2024-03-26T05:02:10.008 INFO:teuthology.orchestra.run.smithi080.stdout: "log_to_graylog": "false", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "log_to_journald": "false", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "log_to_stderr": "false", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "log_to_syslog": "false", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "max_rotating_auth_attempts": "10", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_action_on_write_error": "1", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_alternate_name_max": "8192", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_asio_thread_count": "2", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_export_pin": "true", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_fragment_dirs": "true", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_fragment_interval": "5", 2024-03-26T05:02:10.009 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_interval": "10", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_max": "-1", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_max_until": "-1", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_merge_size": "50", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_midchunk": "0.300000", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_min_start": "0.200000", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_minchunk": "0.001000", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_mode": "0", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_need_max": "1.200000", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_need_min": "0.800000", 2024-03-26T05:02:10.010 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_overload_epochs": "2", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_split_bits": "3", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_split_size": "10000", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_target_decay": "10.000000", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_beacon_grace": "15.000000", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_beacon_interval": "4.000000", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-26T05:02:10.011 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_cache_mid": "0.700000", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_cache_release_free_interval": "10", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_cache_reservation": "0.050000", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_cache_trim_interval": "1", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_cache_trim_threshold": "262144", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_client_prealloc_inos": "1000", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_connect_bootstrapping": "false", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_damage_table_max_entries": "10000", 2024-03-26T05:02:10.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_debug_auth_pins": "false", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_debug_frag": "false", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_debug_scatterstat": "false", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_debug_subtrees": "false", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_decay_halflife": "5.000000", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_default_dir_hash": "2", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_defer_session_stale": "true", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_deny_all_reconnect": "false", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_dir_keys_per_op": "16384", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_dir_max_commit_size": "10", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_dir_max_entries": "0", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_dir_prefetch": "true", 2024-03-26T05:02:10.013 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_dump_cache_on_map": "false", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_early_reply": "true", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_enable_op_tracker": "true", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_enforce_unique_name": "true", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_export_ephemeral_random": "true", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-26T05:02:10.014 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_health_summarize_threshold": "10", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_inject_health_dummy": "false", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_inject_migrator_session_race": "false", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-26T05:02:10.015 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_join_fs": "", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_journal_format": "1", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_kill_create_at": "0", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_kill_export_at": "0", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_kill_import_at": "0", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_kill_journal_at": "0", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_kill_journal_expire_at": "0", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_kill_journal_replay_at": "0", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_kill_link_at": "0", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_kill_mdstable_at": "0", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_kill_openc_at": "0", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_kill_rename_at": "0", 2024-03-26T05:02:10.016 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_kill_shutdown_at": "0", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_event_large_threshold": "524288", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_events_per_segment": "1024", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_max_events": "-1", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_max_segments": "128", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_pause": "false", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_segment_size": "0", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_trim_threshold": "128", 2024-03-26T05:02:10.017 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log_warn_factor": "2.000000", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_max_caps_per_client": "1048576", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_max_completed_flushes": "100000", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_max_completed_requests": "100000", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_max_export_size": "20971520", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_max_file_recover": "32", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_max_purge_files": "64", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_max_purge_ops": "8192", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_max_snaps_per_dir": "100", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_metrics_update_interval": "2", 2024-03-26T05:02:10.018 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_min_caps_per_client": "100", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_min_caps_working_set": "10000", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_numa_node": "-1", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_op_complaint_time": "30.000000", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_op_history_duration": "600", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_op_history_size": "20", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_op_history_slow_op_size": "20", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_op_log_threshold": "5", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_ping_grace": "15", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_ping_interval": "5", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-26T05:02:10.019 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_recall_max_caps": "30000", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_recall_warning_threshold": "262144", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_replay_interval": "1.000000", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_root_ino_gid": "0", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_root_ino_uid": "0", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-26T05:02:10.020 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_shutdown_check": "0", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_skip_ino": "0", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_snap_max_uid": "4294967294", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_snap_min_uid": "0", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_snap_rstat": "false", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_standby_replay_damaged": "false", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_symlink_recovery": "true", 2024-03-26T05:02:10.021 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_thrash_exports": "0", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_thrash_fragments": "0", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_tick_interval": "5.000000", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_valgrind_exit": "false", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_verify_backtrace": "1", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_verify_scatter": "false", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_wipe_sessions": "false", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "mempool_debug": "false", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "memstore_device_bytes": "1073741824", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "memstore_page_set": "false", 2024-03-26T05:02:10.022 INFO:teuthology.orchestra.run.smithi080.stdout: "memstore_page_size": "65536", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_client_bytes": "134217728", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_client_messages": "512", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_disabled_modules": "", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_max_pg_num_change": "128", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_mds_bytes": "134217728", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_mds_messages": "128", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_mon_bytes": "134217728", 2024-03-26T05:02:10.023 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_mon_messages": "128", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_osd_bytes": "536870912", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_osd_messages": "8192", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_pool": "true", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_standby_modules": "true", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_stats_period": "5", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_stats_threshold": "5", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_tick_period": "2", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_allow_pool_delete": "true", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_allow_pool_size_one": "false", 2024-03-26T05:02:10.024 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_auth_validate_all_caps": "true", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_client_bytes": "104857600", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_client_directed_command_retry": "2", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_client_hunt_parallel": "3", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_client_log_interval": "1.000000", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_client_ping_interval": "10.000000", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-26T05:02:10.025 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_client_target_rank": "-1", 2024-03-26T05:02:10.026 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-26T05:02:10.026 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-26T05:02:10.026 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-26T05:02:10.026 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cluster_log_file_level": "debug", 2024-03-26T05:02:10.026 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cluster_log_to_file": "true", 2024-03-26T05:02:10.026 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-26T05:02:10.026 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-26T05:02:10.026 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-26T05:02:10.026 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cluster_log_to_journald": "true", 2024-03-26T05:02:10.026 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-26T05:02:10.026 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-26T05:02:10.026 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_compact_on_bootstrap": "false", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_compact_on_start": "false", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_compact_on_trim": "true", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_cpu_threads": "4", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_crush_min_required_version": "hammer", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_daemon_bytes": "419430400", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_data_avail_crit": "5", 2024-03-26T05:02:10.027 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_data_avail_warn": "5", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_data_size_warn": "16106127360", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_debug_dump_json": "false", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_debug_dump_transactions": "false", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_debug_extra_checks": "false", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_debug_no_require_reef": "false", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_debug_no_require_squid": "false", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-26T05:02:10.028 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_down_mkfs_grace": "60", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_election_default_strategy": "1", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_election_timeout": "5.000000", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_elector_ping_divisor": "2", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_enable_op_tracker": "true", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_fake_pool_delete": "false", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_force_quorum_join": "false", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_globalid_prealloc": "10000", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_health_detail_to_clog": "true", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_health_log_update_period": "5", 2024-03-26T05:02:10.029 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_health_max_detail": "50", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_health_to_clog": "true", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_health_to_clog_interval": "600", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_host": "[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0]", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_host_override": "", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_initial_members": "", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_lease": "5.000000", 2024-03-26T05:02:10.030 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_log_full_interval": "50", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_log_max": "10000", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_log_max_summary": "50", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_max_log_epochs": "500", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_max_osd": "10000", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_max_pg_per_osd": "10000", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_max_pool_pg_num": "65536", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-26T05:02:10.031 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_mds_force_trim_to": "0", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_mds_skip_sanity": "false", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_memory_autotune": "true", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_memory_target": "2147483648", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_mgr_beacon_grace": "30", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_mgr_digest_period": "5", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_mgr_inactive_grace": "60", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_min_osdmap_epochs": "50", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_op_complaint_time": "30", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_op_history_duration": "600", 2024-03-26T05:02:10.032 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_op_history_size": "20", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_op_history_slow_op_size": "20", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_op_log_threshold": "5", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_auto_mark_in": "false", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_cache_size": "500", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-26T05:02:10.033 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_down_out_interval": "600", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_force_trim_to": "0", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-26T05:02:10.034 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_min_down_reporters": "2", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_report_timeout": "900", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-26T05:02:10.035 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_pg_stuck_threshold": "60", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_probe_timeout": "2.000000", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_reweight_max_change": "0.050000", 2024-03-26T05:02:10.036 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_reweight_max_osds": "4", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_scrub_interval": "86400", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_scrub_max_keys": "100", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_scrub_timeout": "300", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_session_timeout": "300", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_smart_report_timeout": "5", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_stat_smooth_intervals": "6", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-26T05:02:10.037 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_stretch_pool_min_size": "2", 2024-03-26T05:02:10.038 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_stretch_pool_size": "4", 2024-03-26T05:02:10.038 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-26T05:02:10.038 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-26T05:02:10.038 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_sync_debug": "false", 2024-03-26T05:02:10.038 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-26T05:02:10.038 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-26T05:02:10.038 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_sync_provider_kill_at": "0", 2024-03-26T05:02:10.039 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_sync_requester_kill_at": "0", 2024-03-26T05:02:10.039 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_sync_timeout": "60.000000", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_target_pg_per_osd": "100", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_tick_interval": "5", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_timecheck_interval": "300.000000", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_use_min_delay_socket": "false", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_older_version_delay": "604800", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_misplaced": "false", 2024-03-26T05:02:10.040 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_older_version": "true", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "monmap": "", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-26T05:02:10.041 INFO:teuthology.orchestra.run.smithi080.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "motr_tracing_enabled": "false", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_op_threads": "3", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_cm": "false", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_device_name": "", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_dscp": "96", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_local_gid": "", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-26T05:02:10.042 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_port_num": "1", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_sl": "3", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_support_srq": "true", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_rdma_type": "ib", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_async_reap_threshold": "5", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_bind_before_connect": "false", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_bind_ipv4": "true", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_bind_ipv6": "false", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_bind_msgr1": "true", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_bind_msgr2": "true", 2024-03-26T05:02:10.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_bind_port_max": "7568", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_bind_port_min": "6800", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_bind_retry_count": "3", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_bind_retry_delay": "5", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_blackhole_client": "false", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_blackhole_mds": "false", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_blackhole_mgr": "false", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_blackhole_mon": "false", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_blackhole_osd": "false", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_client_mode": "crc secure", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_cluster_mode": "crc secure", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_cluster_type": "", 2024-03-26T05:02:10.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_compress_secure": "false", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_connection_idle_timeout": "900", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_connection_ready_timeout": "10", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_crc_data": "true", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_crc_header": "true", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_die_on_bad_msg": "false", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_die_on_bug": "true", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_die_on_old_message": "true", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_die_on_skipped_message": "false", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_coremask": "0xF", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_enable_tso": "true", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-26T05:02:10.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_hugepages": "", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_lro": "true", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_memory_channel": "4", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_pmd": "", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_port_id": "0", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_dump_on_send": "false", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_initial_backoff": "0.200000", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_inject_delay_max": "1.000000", 2024-03-26T05:02:10.046 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_inject_delay_type": "", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_inject_network_congestion": "0", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_inject_socket_failures": "0", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_learn_addr_from_peer": "true", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_max_accept_failures": "4", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_max_backoff": "15.000000", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_mon_client_mode": "secure crc", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_mon_service_mode": "secure crc", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_osd_compress_min_size": "1024", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_osd_compress_mode": "none", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-26T05:02:10.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_pq_min_cost": "65536", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_public_type": "", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_service_mode": "crc secure", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_tcp_listen_backlog": "512", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_tcp_nodelay": "true", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_tcp_rcvbuf": "0", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "ms_type": "async+posix", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "no_config_file": "false", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "objecter_completion_locks_per_session": "32", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "objecter_inflight_ops": "1024", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-26T05:02:10.048 INFO:teuthology.orchestra.run.smithi080.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "objecter_tick_interval": "5.000000", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "objecter_timeout": "10.000000", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "objectstore_blackhole": "false", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "openssl_engine_opts": "", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_agent_delay_time": "5.000000", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_agent_hist_halflife": "1000", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_agent_max_low_ops": "2", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_agent_max_ops": "4", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_agent_slop": "0.020000", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-26T05:02:10.049 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_asio_thread_count": "2", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_async_recovery_min_cost": "100", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_backfill_scan_max": "512", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_backfill_scan_min": "64", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_backoff_on_degraded": "false", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_backoff_on_peering": "false", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_backoff_on_unfound": "true", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_beacon_report_interval": "300", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_bench_duration": "30", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_bench_max_block_size": "67108864", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-26T05:02:10.050 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_blkin_trace_all": "false", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_check_for_log_corruption": "false", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_class_update_on_start": "true", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_client_message_cap": "256", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_client_message_size_cap": "524288000", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_client_op_priority": "63", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_client_watch_timeout": "30", 2024-03-26T05:02:10.051 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_command_max_records": "256", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_compact_on_start": "false", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_crush_update_on_start": "true", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_crush_update_weight_set": "true", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_feed_pullee": "-1", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-26T05:02:10.052 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-26T05:02:10.053 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_misdirected_ops": "false", 2024-03-26T05:02:10.053 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_no_acting_change": "false", 2024-03-26T05:02:10.053 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_no_purge_strays": "false", 2024-03-26T05:02:10.053 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_op_order": "false", 2024-03-26T05:02:10.053 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-26T05:02:10.053 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-26T05:02:10.053 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-26T05:02:10.053 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-26T05:02:10.053 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_shutdown": "false", 2024-03-26T05:02:10.053 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-26T05:02:10.053 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-26T05:02:10.053 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_trim_objects": "false", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_verify_snaps": "false", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_deep_scrub_keys": "1024", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_deep_scrub_stride": "524288", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_default_notify_timeout": "30", 2024-03-26T05:02:10.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_delete_sleep": "0.000000", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_discard_disconnected_ops": "true", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_enable_op_tracker": "true", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_fast_info": "true", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_fast_shutdown": "true", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-26T05:02:10.055 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_function_tracing": "false", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_heartbeat_grace": "20", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_heartbeat_interval": "6", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_heartbeat_min_peers": "10", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_heartbeat_min_size": "2000", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_heartbeat_stale": "600", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_hit_set_max_size": "100000", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_hit_set_min_size": "1000", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-26T05:02:10.056 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_journal_size": "5120", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_kill_backfill_at": "0", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_map_cache_size": "50", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_map_dedup": "true", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_map_message_max": "40", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_map_share_max_epochs": "40", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_attr_name_len": "100", 2024-03-26T05:02:10.057 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_attr_size": "0", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_backfills": "1", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_markdown_count": "5", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_markdown_period": "600", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_object_name_len": "2048", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_object_namespace_len": "256", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_object_size": "134217728", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_pg_blocked_by": "16", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_pg_log_entries": "10000", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_pgls": "1024", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_push_cost": "8388608", 2024-03-26T05:02:10.058 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_push_objects": "10", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_scrubs": "3", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_trimming_pgs": "2", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_write_op_reply_len": "64", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_max_write_size": "90", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-26T05:02:10.059 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_profile": "balanced", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_memory_base": "805306368", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_memory_cache_min": "134217728", 2024-03-26T05:02:10.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_memory_target": "4294967296", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_memory_target_autotune": "false", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_min_pg_log_entries": "250", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_min_recovery_priority": "0", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mon_report_interval": "5", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_num_cache_shards": "32", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_num_op_tracker_shard": "32", 2024-03-26T05:02:10.061 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_numa_auto_affinity": "true", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_numa_node": "-1", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_numa_prefer_iface": "true", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_objecter_finishers": "1", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_objectstore": "bluestore", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_objectstore_fuse": "false", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_objectstore_tracing": "false", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_complaint_time": "30.000000", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_history_duration": "600", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_history_size": "20", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_history_slow_op_size": "20", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-26T05:02:10.062 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_log_threshold": "5", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_num_shards": "0", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_num_shards_hdd": "5", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_num_shards_ssd": "8", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_pq_min_cost": "65536", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_queue_cut_off": "high", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_op_thread_timeout": "15", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_open_classes_on_start": "true", 2024-03-26T05:02:10.063 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_os_flags": "0", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_peering_op_priority": "255", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pg_delete_cost": "1048576", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pg_delete_priority": "5", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pg_log_trim_max": "10000", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pg_log_trim_min": "100", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-26T05:02:10.064 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_crimson": "false", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_flags": "0", 2024-03-26T05:02:10.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_min_size": "0", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_pg_num": "32", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_pgp_num": "0", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_read_ratio": "70", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_size": "3", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_default_type": "replicated", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_push_per_object_cost": "1000", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-26T05:02:10.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recover_clone_overlap": "true", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_cost": "20971520", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_max_active": "0", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_max_single_start": "1", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_op_priority": "3", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_priority": "5", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-26T05:02:10.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_sleep": "0.000000", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_repair_during_recovery": "false", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_requested_scrub_priority": "120", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_auto_repair": "false", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_begin_hour": "0", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_begin_week_day": "0", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_chunk_max": "25", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_chunk_min": "5", 2024-03-26T05:02:10.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_cost": "52428800", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_during_recovery": "false", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_end_hour": "0", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_end_week_day": "0", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_event_cost": "4096", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_invalid_stats": "true", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_max_preemptions": "5", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-26T05:02:10.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_priority": "5", 2024-03-26T05:02:10.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-26T05:02:10.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_sleep": "0.000000", 2024-03-26T05:02:10.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-26T05:02:10.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-26T05:02:10.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-26T05:02:10.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-26T05:02:10.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_skip_data_digest": "false", 2024-03-26T05:02:10.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_smart_report_timeout": "5", 2024-03-26T05:02:10.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_snap_trim_cost": "1048576", 2024-03-26T05:02:10.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_snap_trim_priority": "5", 2024-03-26T05:02:10.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-26T05:02:10.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_target_transaction_size": "30", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-26T05:02:10.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_tracing": "false", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_use_stale_snap": "false", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "osdc_blkin_trace_all": "false", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "paxos_kill_at": "0", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "paxos_max_join_drift": "10", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "paxos_min": "500", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "paxos_min_wait": "0.050000", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "paxos_propose_interval": "1.000000", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "paxos_service_trim_max": "500", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-26T05:02:10.072 INFO:teuthology.orchestra.run.smithi080.stdout: "paxos_service_trim_min": "10", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "paxos_stash_full_interval": "25", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "paxos_trim_max": "500", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "paxos_trim_min": "250", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "perf": "true", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "pid_file": "", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "-", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrv": "", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "public_bind_addr": "-", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "public_network": "", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "public_network_interface": "", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "qat_compressor_enabled": "false", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "qat_compressor_session_max_number": "256", 2024-03-26T05:02:10.073 INFO:teuthology.orchestra.run.smithi080.stdout: "rados_mon_op_timeout": "0", 2024-03-26T05:02:10.074 INFO:teuthology.orchestra.run.smithi080.stdout: "rados_osd_op_timeout": "0", 2024-03-26T05:02:10.074 INFO:teuthology.orchestra.run.smithi080.stdout: "rados_tracing": "false", 2024-03-26T05:02:10.074 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_atime_update_interval": "60", 2024-03-26T05:02:10.074 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-26T05:02:10.074 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_balance_parent_reads": "false", 2024-03-26T05:02:10.074 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_balance_snap_reads": "false", 2024-03-26T05:02:10.074 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_blkin_trace_all": "false", 2024-03-26T05:02:10.074 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-26T05:02:10.074 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_cache": "true", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_cache_policy": "writearound", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_cache_size": "33554432", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_clone_copy_on_read": "false", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_compression_hint": "none", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_concurrent_management_ops": "10", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-26T05:02:10.075 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_default_clone_format": "auto", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_default_data_pool": "", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_default_features": "61", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_default_format": "2", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_default_map_options": "", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_default_order": "22", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_default_pool": "rbd", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_default_stripe_count": "0", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_default_stripe_unit": "0", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-26T05:02:10.076 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_enable_alloc_hint": "true", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_io_scheduler": "simple", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_journal_order": "24", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_journal_pool": "", 2024-03-26T05:02:10.077 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_journal_splay_width": "4", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_localize_parent_reads": "false", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_localize_snap_reads": "false", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-26T05:02:10.078 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-26T05:02:10.079 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_mtime_update_interval": "60", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_non_blocking_aio": "true", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_op_thread_timeout": "60", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_op_threads": "1", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_parent_cache_enabled": "false", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_plugins": "", 2024-03-26T05:02:10.080 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_bps_burst": "0", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_bps_limit": "0", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_exclude_ops": "0", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_iops_burst": "0", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_iops_limit": "0", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-26T05:02:10.081 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_read_from_replica_policy": "default", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_skip_partial_discard": "true", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-26T05:02:10.082 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_tracing": "false", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_validate_names": "true", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd_validate_pool": "true", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "restapi_base_url": "", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "restapi_log_level": "", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_acl_grants_max_num": "100", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_admin_entry": "admin", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_backend_store": "rados", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_barbican_url": "", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_beast_enable_async": "true", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_bucket_counters_cache": "false", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-26T05:02:10.083 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_cache_enabled": "true", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_cache_expiry_interval": "900", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_cache_lru_size": "10000", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_config_store": "rados", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_content_length_compat": "false", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_copy_obj_progress": "true", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_cors_rules_max_num": "100", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_cross_domain_policy": "", 2024-03-26T05:02:10.084 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_kmip_addr": "", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_kmip_password": "", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_kmip_username": "", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_require_ssl": "true", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-26T05:02:10.085 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_vault_addr": "", 2024-03-26T05:02:10.086 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_vault_auth": "token", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_vault_namespace": "", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_vault_prefix": "", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_vault_token_file": "", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_curl_buffersize": "524288", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_curl_low_speed_time": "300", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-26T05:02:10.087 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_d4n_port": "6379", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_data_log_changes_size": "1000", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_data_log_num_shards": "128", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_data_log_window": "30", 2024-03-26T05:02:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-26T05:02:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dns_name": "", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dns_s3website_name": "", 2024-03-26T05:02:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_dynamic_resharding": "true", 2024-03-26T05:02:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-26T05:02:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_enable_gc_threads": "true", 2024-03-26T05:02:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_enable_lc_threads": "true", 2024-03-26T05:02:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_enable_ops_log": "false", 2024-03-26T05:02:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_enable_quota_threads": "true", 2024-03-26T05:02:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_enable_static_website": "false", 2024-03-26T05:02:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_enable_usage_log": "false", 2024-03-26T05:02:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_enforce_swift_acls": "true", 2024-03-26T05:02:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_exit_timeout_secs": "120", 2024-03-26T05:02:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_expose_bucket": "false", 2024-03-26T05:02:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_extended_http_attrs": "", 2024-03-26T05:02:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_filter": "none", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_frontends": "beast port=7480", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_gc_max_deferred": "50", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_gc_max_objs": "32", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_gc_processor_period": "3600", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-26T05:02:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_init_timeout": "300", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_kafka_connection_idle": "300", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_kafka_message_timeout": "5000", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_admin_domain": "", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_admin_password": "", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_admin_password_path": "", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_admin_project": "", 2024-03-26T05:02:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_admin_tenant": "", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_admin_token": "", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_admin_token_path": "", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_admin_user": "", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_api_version": "2", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_barbican_domain": "", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_barbican_password": "", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_barbican_project": "", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_barbican_user": "", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-26T05:02:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_url": "", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_lc_debug_interval": "-1", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_lc_lock_max_time": "90", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_lc_max_objs": "32", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_lc_max_rules": "1000", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_lc_max_worker": "3", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_lc_thread_delay": "0", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_ldap_dnattr": "uid", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-26T05:02:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_ldap_searchfilter": "", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_log_http_headers": "", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_log_object_name_utc": "false", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_attr_name_len": "0", 2024-03-26T05:02:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_attr_size": "0", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_chunk_size": "4194304", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_listing_results": "1000", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_notify_retries": "10", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_put_param_size": "1048576", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_put_size": "5368709120", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_max_slo_entries": "1000", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_md_log_max_shards": "64", 2024-03-26T05:02:10.097 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_mp_lock_max_time": "600", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-26T05:02:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_max_gc": "300", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_num_async_rados_threads": "32", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_num_control_oids": "8", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_numa_node": "-1", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_objexp_chunk_size": "100", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_objexp_gc_interval": "600", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-26T05:02:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_op_thread_timeout": "600", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_op_tracing": "false", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_opa_token": "", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_opa_url": "", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_opa_verify_ssl": "true", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_ops_log_rados": "false", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_ops_log_socket_path": "", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_period_push_interval": "2.000000", 2024-03-26T05:02:10.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_posix_cache_lanes": "3", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_posix_cache_partitions": "3", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_print_continue": "true", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_rados_tracing": "false", 2024-03-26T05:02:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_realm": "", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_realm_id": "", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_region": "", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_request_uri": "", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_reshard_batch_size": "64", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_reshard_max_aio": "128", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_reshard_num_logs": "16", 2024-03-26T05:02:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_reshard_thread_interval": "600", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_resolve_cname": "false", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_run_sync_thread": "true", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_scheduler_type": "throttler", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_script_uri": "", 2024-03-26T05:02:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_service_provider_name": "", 2024-03-26T05:02:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-26T05:02:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sts_client_id": "", 2024-03-26T05:02:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sts_client_secret": "", 2024-03-26T05:02:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sts_entry": "sts", 2024-03-26T05:02:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sts_key": "sts", 2024-03-26T05:02:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-26T05:02:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sts_min_session_duration": "900", 2024-03-26T05:02:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sts_token_introspection_url": "", 2024-03-26T05:02:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_swift_account_in_url": "false", 2024-03-26T05:02:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_swift_auth_entry": "auth", 2024-03-26T05:02:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_swift_auth_url": "", 2024-03-26T05:02:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_swift_custom_header": "", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_swift_need_stats": "true", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_swift_tenant_name": "", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_swift_token_expiration": "86400", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_swift_url": "", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_swift_url_prefix": "swift", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sync_lease_period": "120", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-26T05:02:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_thread_pool_size": "512", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_torrent_comment": "", 2024-03-26T05:02:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_torrent_createby": "", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_torrent_encoding": "", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_torrent_flag": "false", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_torrent_origin": "", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_torrent_tracker": "", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_trust_forwarded_https": "false", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_usage_max_shards": "32", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_usage_max_user_shards": "1", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_use_opa_authz": "false", 2024-03-26T05:02:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_user_counters_cache": "false", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_user_max_buckets": "1000", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_user_policies_max_num": "100", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_user_unique_email": "true", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_verify_ssl": "true", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-26T05:02:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_zone": "", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_zone_id": "", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_zonegroup": "", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_zonegroup_id": "", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rgwlc_auto_session_clear": "true", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_block_size": "4096", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-26T05:02:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_cache_size": "536870912", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_index_type": "binary_search", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-26T05:02:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_partition_filters": "false", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_perf": "false", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "run_dir": "/var/run/ceph", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_block_create": "true", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_cache_lru_size": "67108864", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_default_max_object_size": "16777216", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_device_size": "53687091200", 2024-03-26T05:02:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_journal_batch_capacity": "16", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_main_device_type": "SSD", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "seastore_segment_size": "67108864", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "set_keepcaps": "false", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "setgroup": "ceph", 2024-03-26T05:02:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "setuser": "ceph", 2024-03-26T05:02:10.113 INFO:teuthology.orchestra.run.smithi080.stdout: "setuser_match_path": "", 2024-03-26T05:02:10.113 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-26T05:02:10.113 INFO:teuthology.orchestra.run.smithi080.stdout: "thp": "false", 2024-03-26T05:02:10.113 INFO:teuthology.orchestra.run.smithi080.stdout: "threadpool_default_timeout": "60", 2024-03-26T05:02:10.113 INFO:teuthology.orchestra.run.smithi080.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-26T05:02:10.113 INFO:teuthology.orchestra.run.smithi080.stdout: "throttler_perf_counter": "true" 2024-03-26T05:02:10.113 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:02:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:10 smithi132 ceph-mon[33500]: 6.8 scrub starts 2024-03-26T05:02:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:10 smithi132 ceph-mon[33500]: 6.8 scrub ok 2024-03-26T05:02:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:10 smithi132 ceph-mon[33500]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:10 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:10 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:10 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:10 smithi132 ceph-mon[33500]: Deploying daemon node-exporter.b on smithi132 2024-03-26T05:02:11.045 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:10 smithi132 bash[64108]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T05:02:11.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[40404]: 6.8 scrub starts 2024-03-26T05:02:11.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[40404]: 6.8 scrub ok 2024-03-26T05:02:11.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[40404]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:11.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:11.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:11.057 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:11.057 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[40404]: Deploying daemon node-exporter.b on smithi132 2024-03-26T05:02:11.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[29691]: 6.8 scrub starts 2024-03-26T05:02:11.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[29691]: 6.8 scrub ok 2024-03-26T05:02:11.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[29691]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:11.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:11.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:11.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:11.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:10 smithi080 ceph-mon[29691]: Deploying daemon node-exporter.b on smithi132 2024-03-26T05:02:11.089 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-26T05:02:11.336 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:11 smithi132 ceph-mon[33500]: 6.d scrub starts 2024-03-26T05:02:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:11 smithi132 ceph-mon[33500]: 6.d scrub ok 2024-03-26T05:02:12.044 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:11 smithi132 bash[64108]: Getting image source signatures 2024-03-26T05:02:12.044 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:11 smithi132 bash[64108]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T05:02:12.044 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:11 smithi132 bash[64108]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T05:02:12.044 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:11 smithi132 bash[64108]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T05:02:12.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:11 smithi080 ceph-mon[29691]: 6.d scrub starts 2024-03-26T05:02:12.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:11 smithi080 ceph-mon[29691]: 6.d scrub ok 2024-03-26T05:02:12.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:11 smithi080 ceph-mon[40404]: 6.d scrub starts 2024-03-26T05:02:12.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:11 smithi080 ceph-mon[40404]: 6.d scrub ok 2024-03-26T05:02:12.938 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a: {} 2024-03-26T05:02:12.949 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b: {} 2024-03-26T05:02:12.959 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c: {} 2024-03-26T05:02:12.960 INFO:teuthology.orchestra.run.smithi080.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T05:02:12.960 INFO:teuthology.orchestra.run.smithi080.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T05:02:12.960 INFO:teuthology.orchestra.run.smithi080.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T05:02:12.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:12 smithi080 ceph-mon[40404]: 6.15 scrub starts 2024-03-26T05:02:12.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:12 smithi080 ceph-mon[40404]: 6.15 scrub ok 2024-03-26T05:02:12.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:12 smithi080 ceph-mon[40404]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:12.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:12 smithi080 ceph-mon[40404]: 3.7 scrub starts 2024-03-26T05:02:12.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:12 smithi080 ceph-mon[40404]: 3.7 scrub ok 2024-03-26T05:02:12.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:12 smithi080 ceph-mon[29691]: 6.15 scrub starts 2024-03-26T05:02:12.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:12 smithi080 ceph-mon[29691]: 6.15 scrub ok 2024-03-26T05:02:12.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:12 smithi080 ceph-mon[29691]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:12.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:12 smithi080 ceph-mon[29691]: 3.7 scrub starts 2024-03-26T05:02:12.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:12 smithi080 ceph-mon[29691]: 3.7 scrub ok 2024-03-26T05:02:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:12 smithi132 ceph-mon[33500]: 6.15 scrub starts 2024-03-26T05:02:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:12 smithi132 ceph-mon[33500]: 6.15 scrub ok 2024-03-26T05:02:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:12 smithi132 ceph-mon[33500]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:12 smithi132 ceph-mon[33500]: 3.7 scrub starts 2024-03-26T05:02:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:12 smithi132 ceph-mon[33500]: 3.7 scrub ok 2024-03-26T05:02:13.793 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:13 smithi132 bash[64108]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T05:02:13.793 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:13 smithi132 bash[64108]: Writing manifest to image destination 2024-03-26T05:02:13.853 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-26T05:02:13.862 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T05:02:13.864 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T05:02:13.864 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-26T05:02:13.864 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-26T05:02:13.864 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-26T05:02:13.865 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:02:13.865 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:02:13.865 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-26T05:02:13.865 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T05:02:13.866 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-26T05:02:13.866 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:13.866 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-26T05:02:13.866 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:02:13.867 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-26T05:02:13.867 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T05:02:13.867 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-26T05:02:13.867 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:02:13.899 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:13.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:14.002 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:14.036 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:14.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:14.087 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:13 smithi132 podman[64108]: 2024-03-26 05:02:13.871433921 +0000 UTC m=+2.992473873 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T05:02:14.087 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:13 smithi132 podman[64108]: 2024-03-26 05:02:13.986960145 +0000 UTC m=+3.108000122 container create f91e694fff90be92cce0c8b796b51c27e84372e1b80ef2a16c70fb8a5dff3326 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:02:14.088 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:13 smithi132 ceph-mon[33500]: 5.15 scrub starts 2024-03-26T05:02:14.088 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:13 smithi132 ceph-mon[33500]: 5.15 scrub ok 2024-03-26T05:02:14.088 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:13 smithi132 ceph-mon[33500]: 6.3 scrub starts 2024-03-26T05:02:14.088 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:13 smithi132 ceph-mon[33500]: 6.3 scrub ok 2024-03-26T05:02:14.088 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:13 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/656488108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:02:14.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:13 smithi080 ceph-mon[29691]: 5.15 scrub starts 2024-03-26T05:02:14.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:13 smithi080 ceph-mon[29691]: 5.15 scrub ok 2024-03-26T05:02:14.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:13 smithi080 ceph-mon[29691]: 6.3 scrub starts 2024-03-26T05:02:14.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:13 smithi080 ceph-mon[29691]: 6.3 scrub ok 2024-03-26T05:02:14.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:13 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/656488108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:02:14.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:13 smithi080 ceph-mon[40404]: 5.15 scrub starts 2024-03-26T05:02:14.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:13 smithi080 ceph-mon[40404]: 5.15 scrub ok 2024-03-26T05:02:14.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:13 smithi080 ceph-mon[40404]: 6.3 scrub starts 2024-03-26T05:02:14.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:13 smithi080 ceph-mon[40404]: 6.3 scrub ok 2024-03-26T05:02:14.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:13 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/656488108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:02:14.120 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:02:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:02:14.138 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:14.152 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:14.152 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:14.165 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:14.178 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:14.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:14.280 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:14.303 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:14.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:14.405 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:14.430 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:14.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:14.531 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 podman[64108]: 2024-03-26 05:02:14.231505905 +0000 UTC m=+3.352545840 container init f91e694fff90be92cce0c8b796b51c27e84372e1b80ef2a16c70fb8a5dff3326 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 podman[64108]: 2024-03-26 05:02:14.234341789 +0000 UTC m=+3.355381734 container start f91e694fff90be92cce0c8b796b51c27e84372e1b80ef2a16c70fb8a5dff3326 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.238Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.238Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.238Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.238Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T05:02:14.544 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T05:02:14.545 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T05:02:14.545 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T05:02:14.545 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T05:02:14.545 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T05:02:14.545 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T05:02:14.545 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T05:02:14.545 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T05:02:14.546 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.239Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T05:02:14.547 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.240Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T05:02:14.548 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.240Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T05:02:14.548 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.240Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T05:02:14.548 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.240Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T05:02:14.548 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.240Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T05:02:14.548 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.240Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T05:02:14.548 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.240Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T05:02:14.548 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.240Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T05:02:14.548 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b[64158]: ts=2024-03-26T05:02:14.240Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T05:02:14.548 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 bash[64108]: f91e694fff90be92cce0c8b796b51c27e84372e1b80ef2a16c70fb8a5dff3326 2024-03-26T05:02:14.548 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:02:14 smithi132 systemd[1]: Started Ceph node-exporter.b for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:02:14.556 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:14.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:14.658 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:14.683 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:14.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:14.785 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:14.808 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:14.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:14.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:14.813 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[40404]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:14.813 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[40404]: 6.c scrub starts 2024-03-26T05:02:14.813 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[40404]: 6.c scrub ok 2024-03-26T05:02:14.813 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:14.813 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:14.814 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:14.814 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:14.814 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:14.814 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:02:14.814 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[29691]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:14.814 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[29691]: 6.c scrub starts 2024-03-26T05:02:14.814 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[29691]: 6.c scrub ok 2024-03-26T05:02:14.814 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:14.814 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:14.814 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:14.814 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:14.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:14.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:14 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:02:14.910 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:14.933 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:14.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:14.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:15.035 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:15.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:02:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-mon[33500]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-mon[33500]: 6.c scrub starts 2024-03-26T05:02:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-mon[33500]: 6.c scrub ok 2024-03-26T05:02:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:14 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:02:15.059 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:15.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:15.161 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:15.186 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:15.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:15.287 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:15.313 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:15.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:15.414 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:15.440 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:15.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:15.542 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:15.566 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:15.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:15.668 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:15.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:15.692 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:15.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:15.793 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:15.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:15.818 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:15.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:15.919 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:15.943 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:15.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:16.044 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:16.067 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:16.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:16.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:15 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:16.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:15 smithi080 ceph-mon[29691]: 3.17 scrub starts 2024-03-26T05:02:16.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:15 smithi080 ceph-mon[29691]: 3.17 scrub ok 2024-03-26T05:02:16.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:15 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:16.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:15 smithi080 ceph-mon[40404]: 3.17 scrub starts 2024-03-26T05:02:16.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:15 smithi080 ceph-mon[40404]: 3.17 scrub ok 2024-03-26T05:02:16.170 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:16.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:15 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:16.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:15 smithi132 ceph-mon[33500]: 3.17 scrub starts 2024-03-26T05:02:16.179 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:15 smithi132 ceph-mon[33500]: 3.17 scrub ok 2024-03-26T05:02:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:16.195 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:16.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:16.296 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:16.320 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:16.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:16.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:16.422 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:16.446 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:16.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:16.548 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:16.573 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:16.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:16.647 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:16.648 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:16.674 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:16.698 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:16.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:16.800 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:16.825 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:16.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:16.927 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:16.952 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:16.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:17.054 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:17.057 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:16 smithi132 ceph-mon[33500]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:17.057 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:16 smithi132 ceph-mon[33500]: 5.5 scrub starts 2024-03-26T05:02:17.058 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:16 smithi132 ceph-mon[33500]: 5.5 scrub ok 2024-03-26T05:02:17.058 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:16 smithi132 ceph-mon[33500]: 6.11 deep-scrub starts 2024-03-26T05:02:17.058 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:16 smithi132 ceph-mon[33500]: 6.11 deep-scrub ok 2024-03-26T05:02:17.077 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:17.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:17.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:17.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:16 smithi080 ceph-mon[29691]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:17.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:16 smithi080 ceph-mon[29691]: 5.5 scrub starts 2024-03-26T05:02:17.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:16 smithi080 ceph-mon[29691]: 5.5 scrub ok 2024-03-26T05:02:17.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:16 smithi080 ceph-mon[29691]: 6.11 deep-scrub starts 2024-03-26T05:02:17.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:16 smithi080 ceph-mon[29691]: 6.11 deep-scrub ok 2024-03-26T05:02:17.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:16 smithi080 ceph-mon[40404]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:17.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:16 smithi080 ceph-mon[40404]: 5.5 scrub starts 2024-03-26T05:02:17.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:16 smithi080 ceph-mon[40404]: 5.5 scrub ok 2024-03-26T05:02:17.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:16 smithi080 ceph-mon[40404]: 6.11 deep-scrub starts 2024-03-26T05:02:17.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:16 smithi080 ceph-mon[40404]: 6.11 deep-scrub ok 2024-03-26T05:02:17.179 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:17.203 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:17.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:17.305 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:17.328 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:17.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:17.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:17.430 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:17.453 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:17.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:17.555 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:17.580 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:17.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:17.681 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:17.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:17.706 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:17.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:17.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:17.832 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:17.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:17.934 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:17.946 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:17 smithi080 ceph-mon[29691]: 4.1 scrub starts 2024-03-26T05:02:17.947 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:17 smithi080 ceph-mon[29691]: 4.1 scrub ok 2024-03-26T05:02:17.947 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:17 smithi080 ceph-mon[29691]: 4.6 scrub starts 2024-03-26T05:02:17.947 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:17 smithi080 ceph-mon[29691]: 4.6 scrub ok 2024-03-26T05:02:17.947 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:17 smithi080 ceph-mon[29691]: 4.19 scrub starts 2024-03-26T05:02:17.947 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:17 smithi080 ceph-mon[29691]: 4.19 scrub ok 2024-03-26T05:02:17.947 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:17 smithi080 ceph-mon[40404]: 4.1 scrub starts 2024-03-26T05:02:17.947 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:17 smithi080 ceph-mon[40404]: 4.1 scrub ok 2024-03-26T05:02:17.947 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:17 smithi080 ceph-mon[40404]: 4.6 scrub starts 2024-03-26T05:02:17.947 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:17 smithi080 ceph-mon[40404]: 4.6 scrub ok 2024-03-26T05:02:17.947 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:17 smithi080 ceph-mon[40404]: 4.19 scrub starts 2024-03-26T05:02:17.947 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:17 smithi080 ceph-mon[40404]: 4.19 scrub ok 2024-03-26T05:02:17.953 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:17 smithi132 ceph-mon[33500]: 4.1 scrub starts 2024-03-26T05:02:17.953 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:17 smithi132 ceph-mon[33500]: 4.1 scrub ok 2024-03-26T05:02:17.953 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:17 smithi132 ceph-mon[33500]: 4.6 scrub starts 2024-03-26T05:02:17.953 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:17 smithi132 ceph-mon[33500]: 4.6 scrub ok 2024-03-26T05:02:17.953 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:17 smithi132 ceph-mon[33500]: 4.19 scrub starts 2024-03-26T05:02:17.954 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:17 smithi132 ceph-mon[33500]: 4.19 scrub ok 2024-03-26T05:02:17.958 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:17.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:17.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:18.060 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:18.083 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:18.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:18.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:18.185 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:18.211 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:18.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:18.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:18.313 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:18.337 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:18.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:18.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:18.439 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:18.463 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:18.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:18.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:18.565 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:18.589 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:18.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:18.690 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:18.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:18.715 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:18.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:18.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:18.839 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:18.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:18.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:18.941 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:18.966 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:18.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:19.067 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:19.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:19.093 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:19.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:19.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:18 smithi080 ceph-mon[40404]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:19.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:18 smithi080 ceph-mon[40404]: 5.17 deep-scrub starts 2024-03-26T05:02:19.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:18 smithi080 ceph-mon[40404]: 5.17 deep-scrub ok 2024-03-26T05:02:19.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:18 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:18 smithi080 ceph-mon[29691]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:18 smithi080 ceph-mon[29691]: 5.17 deep-scrub starts 2024-03-26T05:02:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:18 smithi080 ceph-mon[29691]: 5.17 deep-scrub ok 2024-03-26T05:02:19.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:18 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:19.194 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:19.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:19.220 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:19.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:19.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:18 smithi132 ceph-mon[33500]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:19.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:18 smithi132 ceph-mon[33500]: 5.17 deep-scrub starts 2024-03-26T05:02:19.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:18 smithi132 ceph-mon[33500]: 5.17 deep-scrub ok 2024-03-26T05:02:19.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:18 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:19.321 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:19.347 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:19.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:19.448 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:19.471 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:19.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:19.573 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:19.598 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:19.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:19.701 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:19.725 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:19.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:19.821 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:02:19.822 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:02:19.822 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 0 2024-03-26T05:02:19.822 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:02:19.827 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:19.851 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:19.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:19.917 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:02:19.953 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:19.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:19.978 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:19.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:20.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:20.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:20.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:20.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/247582869' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:02:20.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[29691]: 6.9 scrub starts 2024-03-26T05:02:20.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[29691]: 6.9 scrub ok 2024-03-26T05:02:20.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[29691]: 5.0 scrub starts 2024-03-26T05:02:20.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[29691]: 5.0 scrub ok 2024-03-26T05:02:20.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:20.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:20.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:20.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/247582869' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:02:20.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[40404]: 6.9 scrub starts 2024-03-26T05:02:20.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[40404]: 6.9 scrub ok 2024-03-26T05:02:20.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[40404]: 5.0 scrub starts 2024-03-26T05:02:20.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:19 smithi080 ceph-mon[40404]: 5.0 scrub ok 2024-03-26T05:02:20.079 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:20.115 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:20.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:20.216 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:20.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:20.240 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:20.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:20.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:20.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:20.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:20.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:19 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/247582869' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:02:20.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:19 smithi132 ceph-mon[33500]: 6.9 scrub starts 2024-03-26T05:02:20.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:19 smithi132 ceph-mon[33500]: 6.9 scrub ok 2024-03-26T05:02:20.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:19 smithi132 ceph-mon[33500]: 5.0 scrub starts 2024-03-26T05:02:20.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:19 smithi132 ceph-mon[33500]: 5.0 scrub ok 2024-03-26T05:02:20.341 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:20.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:20.367 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:20.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:20.468 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:20.492 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:20.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:20.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:20.621 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:20.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:20.723 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:20.750 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:20.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:20.852 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:20.881 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:20.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:20.982 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:21.010 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:21.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:21.112 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:21.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:20 smithi080 ceph-mon[29691]: 1.0 scrub starts 2024-03-26T05:02:21.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:20 smithi080 ceph-mon[29691]: 1.0 scrub ok 2024-03-26T05:02:21.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:20 smithi080 ceph-mon[29691]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:21.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:20 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:02:21.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:20 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/247582869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:02:21.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:20 smithi080 ceph-mon[29691]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T05:02:21.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:20 smithi080 ceph-mon[40404]: 1.0 scrub starts 2024-03-26T05:02:21.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:20 smithi080 ceph-mon[40404]: 1.0 scrub ok 2024-03-26T05:02:21.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:20 smithi080 ceph-mon[40404]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:21.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:20 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:02:21.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:20 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/247582869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:02:21.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:20 smithi080 ceph-mon[40404]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T05:02:21.137 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:21.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:21.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:21.239 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:21.270 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:21.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:21.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:20 smithi132 ceph-mon[33500]: 1.0 scrub starts 2024-03-26T05:02:21.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:20 smithi132 ceph-mon[33500]: 1.0 scrub ok 2024-03-26T05:02:21.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:20 smithi132 ceph-mon[33500]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:21.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:20 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:02:21.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:20 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/247582869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:02:21.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:20 smithi132 ceph-mon[33500]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T05:02:21.371 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:21.399 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:21.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:21.500 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:21.526 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:21.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:21.622 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:02:21.628 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:21.640 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:02:21.654 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:21.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:21.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:21.660 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:02:21.679 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:02:21.699 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:02:21.720 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:02:21.739 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:02:21.756 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:21.759 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:02:21.760 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:02:21.760 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:02:21.760 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:02:21.760 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:02:21.761 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:02:21.761 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:02:21.761 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:02:21.761 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:02:21.782 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:21.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:21.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:21.884 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:21.926 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:21.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:22.028 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:22.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:22.053 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:22.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:22.153 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:22.178 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:22.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:22.279 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:22.307 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:22.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:22.409 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:22.436 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:22.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:22.538 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:22.564 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:22.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:22.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:22 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3186833835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:02:22.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:22 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:02:22.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:22 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:02:22.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:22 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3186833835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:02:22.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:22 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:02:22.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:22 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:02:22.666 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:22.692 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:22.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:22.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:22.793 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:22.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:22 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3186833835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:02:22.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:22 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:02:22.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:22 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:02:22.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:22.820 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:22.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:22.921 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:22.944 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:22.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:23.045 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:23.068 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:23.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:23.170 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:23.193 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:23.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:23.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:23.275 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:02:23.275 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:02:23.275 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 102, 2024-03-26T05:02:23.275 INFO:teuthology.orchestra.run.smithi080.stdout: "fsid": "dd09e9cc-eb2c-11ee-95d0-87774f69a715", 2024-03-26T05:02:23.275 INFO:teuthology.orchestra.run.smithi080.stdout: "created": "2024-03-26T04:54:51.106954+0000", 2024-03-26T05:02:23.275 INFO:teuthology.orchestra.run.smithi080.stdout: "modified": "2024-03-26T05:02:22.322895+0000", 2024-03-26T05:02:23.275 INFO:teuthology.orchestra.run.smithi080.stdout: "last_up_change": "2024-03-26T05:00:08.801055+0000", 2024-03-26T05:02:23.275 INFO:teuthology.orchestra.run.smithi080.stdout: "last_in_change": "2024-03-26T04:59:50.693588+0000", 2024-03-26T05:02:23.275 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T05:02:23.275 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_num": 5801984, 2024-03-26T05:02:23.275 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_set": [ 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: "noscrub", 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: "pglog_hardlimit", 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: "purged_snapdirs", 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: "recovery_deletes", 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: "sortbitwise" 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_version": 18, 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_snapshot": "", 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_max": 6, 2024-03-26T05:02:23.276 INFO:teuthology.orchestra.run.smithi080.stdout: "max_osd": 8, 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "require_min_compat_client": "luminous", 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "min_compat_client": "luminous", 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "require_osd_release": "squid", 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "allow_crimson": false, 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "pools": [ 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 1, 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": ".mgr", 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T04:58:06.876819+0000", 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:02:23.277 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 1, 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 1, 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 1, 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 1, 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 1, 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:02:23.278 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "22", 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:02:23.279 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:02:23.280 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:02:23.281 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: "options": { 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_max": 32, 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_min": 1 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": {} 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:02:23.282 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:02:23.283 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 7.8899998664855957, 2024-03-26T05:02:23.283 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 7.8899998664855957, 2024-03-26T05:02:23.283 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.37999999523162842, 2024-03-26T05:02:23.283 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 3, 2024-03-26T05:02:23.283 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 3, 2024-03-26T05:02:23.283 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:02:23.283 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:02:23.283 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:02:23.283 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.283 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.283 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.283 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 2, 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "datapool", 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:20.705417+0000", 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 8193, 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:02:23.284 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 3, 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 3, 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 3, 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 3, 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 3, 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "71", 2024-03-26T05:02:23.285 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 2, 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 71, 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:02:23.286 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.287 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd": {} 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.288 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.6500000953674316, 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.6500000953674316, 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 3, 2024-03-26T05:02:23.289 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": ".rgw.root", 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:21.150368+0000", 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:02:23.290 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:02:23.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.292 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "69", 2024-03-26T05:02:23.292 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:02:23.292 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:02:23.292 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:02:23.293 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:02:23.294 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:02:23.295 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.5, 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.5, 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.5, 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.5, 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 4, 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.log", 2024-03-26T05:02:23.296 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:23.638065+0000", 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:02:23.297 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "71", 2024-03-26T05:02:23.298 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:02:23.299 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:02:23.300 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.301 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.25, 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.25, 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2.25, 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2.25, 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.302 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 5, 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.control", 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:25.181000+0000", 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:02:23.303 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.304 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "73", 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:02:23.305 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:02:23.306 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:02:23.307 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.25, 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.25, 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.25, 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.25, 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.308 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 6, 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.meta", 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:27.249998+0000", 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:02:23.309 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:02:23.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "75", 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:02:23.311 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.312 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "options": { 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_bias": 4 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.313 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:02:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:02:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:02:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:02:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.75, 2024-03-26T05:02:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.75, 2024-03-26T05:02:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:02:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.75, 2024-03-26T05:02:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.75, 2024-03-26T05:02:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:02:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:02:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 0, 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366", 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 8, 2024-03-26T05:02:23.315 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 81, 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6802", 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6803", 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:02:23.316 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6804", 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6805", 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:02:23.317 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6808", 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6809", 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.318 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6806", 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6807", 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.319 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6803/1840686622", 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6805/1840686622", 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6809/1840686622", 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6807/1840686622", 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 1, 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23", 2024-03-26T05:02:23.320 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 13, 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 73, 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.321 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6810", 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6811", 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.322 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6812", 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6813", 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6816", 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6817", 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.324 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6814", 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6815", 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6811/1726833996", 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6813/1726833996", 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6817/1726833996", 2024-03-26T05:02:23.325 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6815/1726833996", 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 2, 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "536bec9b-9636-4e67-b452-ed1a55c22ab6", 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:02:23.326 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 18, 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 81, 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6818", 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.327 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6819", 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6820", 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.328 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6821", 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6824", 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6825", 2024-03-26T05:02:23.329 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6822", 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6823", 2024-03-26T05:02:23.330 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6819/128713757", 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6821/128713757", 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6825/128713757", 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6823/128713757", 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.331 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 3, 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "640cfdb6-b9c5-446d-8312-504ae12c7818", 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 27, 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 73, 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.332 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6826", 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6827", 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6828", 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6829", 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.334 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6832", 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6833", 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.335 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6830", 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6831", 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6827/418771441", 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6829/418771441", 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6833/418771441", 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6831/418771441", 2024-03-26T05:02:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 4, 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639", 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:02:23.337 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 33, 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 73, 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6800", 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6801", 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:02:23.338 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6802", 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6803", 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:02:23.339 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6806", 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6807", 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.340 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6804", 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6805", 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.341 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6801/1441071423", 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6803/1441071423", 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6807/1441071423", 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6805/1441071423", 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 5, 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc", 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:02:23.342 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 41, 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 73, 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6808", 2024-03-26T05:02:23.343 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:02:23.344 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.344 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.344 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.344 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6809", 2024-03-26T05:02:23.344 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:02:23.344 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.344 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.344 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:23.345 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.345 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:02:23.345 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6810", 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6811", 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.346 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6814", 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6815", 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.347 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6812", 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6813", 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.348 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6809/794504625", 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6811/794504625", 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6815/794504625", 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6813/794504625", 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 6, 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5", 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:02:23.349 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 51, 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 81, 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6816", 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:02:23.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6817", 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6818", 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:02:23.351 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6819", 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6822", 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:02:23.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6823", 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6820", 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:02:23.353 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6821", 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6817/2209078972", 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6819/2209078972", 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6823/2209078972", 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6821/2209078972", 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:02:23.354 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 7, 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d", 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 61, 2024-03-26T05:02:23.355 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 73, 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6824", 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6825", 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:02:23.356 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6826", 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6827", 2024-03-26T05:02:23.357 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6830", 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6831", 2024-03-26T05:02:23.358 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6828", 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6829", 2024-03-26T05:02:23.359 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6825/3655388282", 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6827/3655388282", 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6831/3655388282", 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6829/3655388282", 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:02:23.360 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_xinfo": [ 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 0, 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:01:43.846649+0000", 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 1, 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:02:23.361 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:01:02.788528+0000", 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 2, 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:01:53.375113+0000", 2024-03-26T05:02:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 3, 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:02:21.834980+0000", 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 4, 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:01:19.004475+0000", 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 5, 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:02:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:01:40.057036+0000", 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 6, 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:00:57.224185+0000", 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:02:23.365 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 7, 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:01:10.331820+0000", 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap": [], 2024-03-26T05:02:23.366 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap_items": [ 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.2", 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.6", 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:02:23.367 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 5 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.1f", 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:02:23.368 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap_primaries": [], 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_temp": [], 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_temp": [], 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "blocklist": { 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3992539385": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/2770889532": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/2466284312": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/490056571": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1986135300": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3796611170": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1993138634": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1832498181": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/2466284312": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1650870847": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:02:23.369 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/1418683843": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3147301987": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/2770889532": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/571905941": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1451012560": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1235125152": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/1418683843": "2024-03-27T05:01:03.689739+0000" 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: "range_blocklist": {}, 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profiles": { 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: "default": { 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: "crush-failure-domain": "osd", 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: "k": "2", 2024-03-26T05:02:23.370 INFO:teuthology.orchestra.run.smithi080.stdout: "m": "1", 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "plugin": "jerasure", 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "technique": "reed_sol_van" 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps_queue": [], 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "new_removed_snaps": [], 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "new_purged_snaps": [], 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_node_flags": {}, 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "device_class_flags": {}, 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode": { 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode_enabled": false, 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_bucket_count": 0, 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "degraded_stretch_mode": 0, 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "recovering_stretch_mode": 0, 2024-03-26T05:02:23.371 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode_bucket": 0 2024-03-26T05:02:23.372 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:23.372 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:02:23.394 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:23.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:23.496 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:23.520 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:23.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:23.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:23 smithi080 ceph-mon[29691]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:23.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:23 smithi080 ceph-mon[29691]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T05:02:23.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:23 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/689460895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T05:02:23.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:23 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3136313366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:02:23.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:23 smithi080 ceph-mon[40404]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:23.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:23 smithi080 ceph-mon[40404]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T05:02:23.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:23 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/689460895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T05:02:23.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:23 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3136313366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:02:23.622 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:23.644 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:23.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:23.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:23.746 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:23.770 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:23.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:23.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:23 smithi132 ceph-mon[33500]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:23.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:23 smithi132 ceph-mon[33500]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T05:02:23.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:23 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/689460895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T05:02:23.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:23 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3136313366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:02:23.871 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:23.895 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:23.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:23.996 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:24.020 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:24.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:24.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:24.121 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:24.144 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:24.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:24.207 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T05:02:24.246 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:24.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:24.276 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:24.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:24.333 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:02:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:02:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:02:24.378 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:24.403 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:24.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:24.443 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:02:24.506 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:24.529 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:24.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:24.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:24.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:24 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/689460895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-26T05:02:24.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:24 smithi080 ceph-mon[40404]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T05:02:24.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:24 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/689460895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-26T05:02:24.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:24 smithi080 ceph-mon[29691]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T05:02:24.631 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:24.655 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:24.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:24.757 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:24.760 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/689460895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-26T05:02:24.761 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:24 smithi132 ceph-mon[33500]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T05:02:24.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:24.783 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:24.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:24.886 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:24.896 INFO:tasks.thrashosds.thrasher:j is {'epoch': 102, 'fsid': 'dd09e9cc-eb2c-11ee-95d0-87774f69a715', 'created': '2024-03-26T04:54:51.106954+0000', 'modified': '2024-03-26T05:02:22.322895+0000', 'last_up_change': '2024-03-26T05:00:08.801055+0000', 'last_in_change': '2024-03-26T04:59:50.693588+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T04:58:06.876819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T05:00:20.705417+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T05:00:21.150368+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T05:00:23.638065+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T05:00:25.181000+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T05:00:27.249998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '35a52544-0b0b-47d6-8d6a-571c48ceb366', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6802', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6803', 'nonce': 1840686622}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6804', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6805', 'nonce': 1840686622}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6808', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6809', 'nonce': 1840686622}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6806', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6807', 'nonce': 1840686622}]}, 'public_addr': '172.21.15.80:6803/1840686622', 'cluster_addr': '172.21.15.80:6805/1840686622', 'heartbeat_back_addr': '172.21.15.80:6809/1840686622', 'heartbeat_front_addr': '172.21.15.80:6807/1840686622', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efa5ea33-4f11-49b7-a3f3-f60d5329db23', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6810', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6811', 'nonce': 1726833996}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6812', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6813', 'nonce': 1726833996}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6816', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6817', 'nonce': 1726833996}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6814', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6815', 'nonce': 1726833996}]}, 'public_addr': '172.21.15.80:6811/1726833996', 'cluster_addr': '172.21.15.80:6813/1726833996', 'heartbeat_back_addr': '172.21.15.80:6817/1726833996', 'heartbeat_front_addr': '172.21.15.80:6815/1726833996', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '536bec9b-9636-4e67-b452-ed1a55c22ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6818', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6819', 'nonce': 128713757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6820', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6821', 'nonce': 128713757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6824', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6825', 'nonce': 128713757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6822', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6823', 'nonce': 128713757}]}, 'public_addr': '172.21.15.80:6819/128713757', 'cluster_addr': '172.21.15.80:6821/128713757', 'heartbeat_back_addr': '172.21.15.80:6825/128713757', 'heartbeat_front_addr': '172.21.15.80:6823/128713757', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '640cfdb6-b9c5-446d-8312-504ae12c7818', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6826', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6827', 'nonce': 418771441}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6828', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6829', 'nonce': 418771441}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6832', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6833', 'nonce': 418771441}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6830', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6831', 'nonce': 418771441}]}, 'public_addr': '172.21.15.80:6827/418771441', 'cluster_addr': '172.21.15.80:6829/418771441', 'heartbeat_back_addr': '172.21.15.80:6833/418771441', 'heartbeat_front_addr': '172.21.15.80:6831/418771441', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0cc7822a-ed73-4c79-a30d-1e151ddd6639', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6800', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6801', 'nonce': 1441071423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6802', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6803', 'nonce': 1441071423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6806', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6807', 'nonce': 1441071423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6804', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6805', 'nonce': 1441071423}]}, 'public_addr': '172.21.15.132:6801/1441071423', 'cluster_addr': '172.21.15.132:6803/1441071423', 'heartbeat_back_addr': '172.21.15.132:6807/1441071423', 'heartbeat_front_addr': '172.21.15.132:6805/1441071423', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6808', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6809', 'nonce': 794504625}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6810', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6811', 'nonce': 794504625}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6814', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6815', 'nonce': 794504625}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6812', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6813', 'nonce': 794504625}]}, 'public_addr': '172.21.15.132:6809/794504625', 'cluster_addr': '172.21.15.132:6811/794504625', 'heartbeat_back_addr': '172.21.15.132:6815/794504625', 'heartbeat_front_addr': '172.21.15.132:6813/794504625', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2970bbab-11b2-4d1b-b57e-037bdac089b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6816', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6817', 'nonce': 2209078972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6818', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6819', 'nonce': 2209078972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6822', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6823', 'nonce': 2209078972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6820', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6821', 'nonce': 2209078972}]}, 'public_addr': '172.21.15.132:6817/2209078972', 'cluster_addr': '172.21.15.132:6819/2209078972', 'heartbeat_back_addr': '172.21.15.132:6823/2209078972', 'heartbeat_front_addr': '172.21.15.132:6821/2209078972', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '12f4b466-28f9-469b-8fd0-65b91c6dc22d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6824', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6825', 'nonce': 3655388282}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6826', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6827', 'nonce': 3655388282}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6830', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6831', 'nonce': 3655388282}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6828', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6829', 'nonce': 3655388282}]}, 'public_addr': '172.21.15.132:6825/3655388282', 'cluster_addr': '172.21.15.132:6827/3655388282', 'heartbeat_back_addr': '172.21.15.132:6831/3655388282', 'heartbeat_front_addr': '172.21.15.132:6829/3655388282', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:01:43.846649+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:01:02.788528+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:01:53.375113+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:02:21.834980+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:01:19.004475+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:01:40.057036+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:00:57.224185+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:01:10.331820+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.80:0/3992539385': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/2770889532': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:6801/2466284312': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/490056571': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1986135300': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/3796611170': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1993138634': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1832498181': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/2466284312': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1650870847': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6801/1418683843': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:0/3147301987': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6801/2770889532': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/571905941': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1451012560': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1235125152': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/1418683843': '2024-03-27T05:01:03.689739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T05:02:24.897 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:02:24.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:24.911 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:24.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:25.011 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:25.036 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:25.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:25.037 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'unique_pool_0' created 2024-03-26T05:02:25.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:02:24 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:02:25.137 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:25.161 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:25.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:25.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:25.263 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:25.288 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:25.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:25.380 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:02:25.390 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:25.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:25.450 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:25.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:25.549 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:25 smithi080 ceph-mon[29691]: pgmap v58: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:25.550 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:25 smithi080 ceph-mon[29691]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:25.550 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:25 smithi080 ceph-mon[29691]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T05:02:25.550 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:25 smithi080 ceph-mon[40404]: pgmap v58: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:25.550 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:25 smithi080 ceph-mon[40404]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:25.550 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:25 smithi080 ceph-mon[40404]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T05:02:25.552 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:25.580 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:25.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:25.589 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-26T05:02:25.642 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:25.683 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:25.706 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:25.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:25.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:25 smithi132 ceph-mon[33500]: pgmap v58: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:25.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:25 smithi132 ceph-mon[33500]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:25.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:25 smithi132 ceph-mon[33500]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T05:02:25.808 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:25.832 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:25.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:25.934 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:25.950 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:25.964 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:25.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:26.066 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:26.094 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:26.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:26.196 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:26.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:26.225 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:26.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:26.326 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:26.354 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:26.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:26.456 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:26.480 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:26.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:26.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:26.582 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:26.610 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:26.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:26.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:26.611 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:26 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:26.611 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:26 smithi080 ceph-mon[40404]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T05:02:26.612 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:26 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:26.612 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:26 smithi080 ceph-mon[29691]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T05:02:26.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:26.738 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:26.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:26.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:26 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:26.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:26 smithi132 ceph-mon[33500]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T05:02:26.841 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:26.866 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:26.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:26.967 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:26.990 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:26.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:26.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:27.092 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:27.117 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:27.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:27.158 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:27.218 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:27.246 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:27.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:27.347 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:27.371 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:27.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:27.473 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:27.501 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:27.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:27.531 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:27 smithi080 ceph-mon[29691]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:27.531 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:27 smithi080 ceph-mon[40404]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:27.603 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:27.629 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:27.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:27.730 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:27.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:27.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:27 smithi132 ceph-mon[33500]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:27.859 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:27.887 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:27.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:27.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:27.989 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:28.012 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:28.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:28.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:28.114 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:28.138 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:28.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:28.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:28.198 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:28.239 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:28.268 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:28.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:28.369 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:28.394 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:28.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:28.495 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:28.519 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:28.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:28.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:28 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:02:28.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:28 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:02:28.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:28 smithi080 ceph-mon[29691]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T05:02:28.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:28 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:02:28.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:28 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:02:28.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:28 smithi080 ceph-mon[40404]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T05:02:28.621 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:28.649 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:28.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:28.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:28.751 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:28.779 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:28.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:28.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:28.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:28 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:02:28.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:28 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:02:28.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:28 smithi132 ceph-mon[33500]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T05:02:28.881 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:28.908 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:28.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:28.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:29.010 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:29.035 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:29.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:29.136 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:29.160 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:29.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:29.261 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:29.291 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:29.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:29.392 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:29.421 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:29.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:29.523 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:29.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:29.547 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:29.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:29.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:29 smithi080 ceph-mon[29691]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:02:29.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:29 smithi080 ceph-mon[40404]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:02:29.649 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:29.673 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:29.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:29.751 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:29.778 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:29.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:29 smithi132 ceph-mon[33500]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:02:29.805 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:29.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:29.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:29.907 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:29.931 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:29.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:30.032 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:30.059 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:30.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:30.161 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:30.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:30.190 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:30.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:30.291 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:30.316 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:30.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:30.418 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:30.446 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:30.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:30.548 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:30.573 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:30.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:30.675 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:30.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:30.701 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:30.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:30.799 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:02:30.799 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:02:30.803 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:30.828 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:30.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:30.930 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:30.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:30.959 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:30.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:31.060 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:31.084 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:31.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:31.185 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:31.215 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:31.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:31.316 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:31.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:31.342 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:31.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:31.401 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:31 smithi080 ceph-mon[40404]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:02:31.401 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:31 smithi080 ceph-mon[29691]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:02:31.443 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:31.472 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:31.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:31.573 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:31.601 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:31.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:31.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:31.703 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:31.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:31.772 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:31.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:31.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:31 smithi132 ceph-mon[33500]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:02:31.873 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:31.897 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:31.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:31.998 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:32.024 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:32.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:32.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:32.126 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:32.151 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:32.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:32.253 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:32.276 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:32.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:32.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:32.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:32.402 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:32.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:32.504 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:32.530 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:32.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:32.631 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:32.656 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:32.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:32.757 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:32.781 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:32.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:32.882 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:32.907 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:32.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:32.978 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T05:02:33.009 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:33.032 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:33.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:33.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:33.134 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:33.158 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:33.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:33.260 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:33.284 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:33.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:33.386 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:33.410 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:33.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:33.511 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:33 smithi080 ceph-mon[29691]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T05:02:33.511 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:33 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:02:33.511 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:33 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:02:33.511 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:33 smithi080 ceph-mon[29691]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T05:02:33.511 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:33 smithi080 ceph-mon[40404]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T05:02:33.511 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:33 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:02:33.511 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:33 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:02:33.511 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:33 smithi080 ceph-mon[40404]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T05:02:33.512 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:33.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:33.536 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:33.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:33.637 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:33.662 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:33.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:33.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:33.764 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:33.787 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:33.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:33.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:33 smithi132 ceph-mon[33500]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T05:02:33.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:33 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:02:33.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:33 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:02:33.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:33 smithi132 ceph-mon[33500]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T05:02:33.889 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:33.913 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:33.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:34.015 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:34.040 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:34.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:34.118 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:02:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:02:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:02:34.142 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:34.155 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:02:34.158 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:02:34.167 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:34.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:34.269 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:34.294 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:34.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:34.368 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:02:34.396 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:34.405 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:34 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2809652853' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:02:34.405 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:34 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:34.405 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:34 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:34.406 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:34 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2809652853' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:02:34.406 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:34 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:34.406 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:34 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:34.421 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:34.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:34.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:34.523 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:34.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:34.548 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:34.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:34.649 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:34.674 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:34.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:34.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:34.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:34 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2809652853' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:02:34.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:34 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:34.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:34 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:34.777 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:34.800 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:34.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:34.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:34.902 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:34.928 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:34.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:35.029 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:35.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:02:34 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:02:35.054 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:35.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:35.156 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:35.179 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:35.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:35.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:35.281 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:35.307 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:35.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:35.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:35.405 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:35 smithi080 ceph-mon[40404]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:02:35.405 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:35 smithi080 ceph-mon[40404]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:02:35.405 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:35 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:02:35.405 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:35 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2809652853' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:02:35.405 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:35 smithi080 ceph-mon[40404]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T05:02:35.405 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:35 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:35.405 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:35 smithi080 ceph-mon[29691]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:02:35.405 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:35 smithi080 ceph-mon[29691]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:02:35.405 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:35 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:02:35.406 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:35 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2809652853' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:02:35.406 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:35 smithi080 ceph-mon[29691]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T05:02:35.406 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:35 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:35.408 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:35.433 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:35.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:35.535 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:35.559 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:35.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:35.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:35.662 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:35.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:35.687 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:35.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:35.788 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:35.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:35 smithi132 ceph-mon[33500]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:02:35.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:35 smithi132 ceph-mon[33500]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:02:35.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:35 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:02:35.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:35 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2809652853' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:02:35.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:35 smithi132 ceph-mon[33500]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T05:02:35.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:35 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:35.803 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-03-26T05:02:33.723709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1343,"num_read_kb":1157,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":827,"ondisk_log_size":827,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244776,"kb_used_data":7804,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716440,"statfs":{"total":767960285184,"available":767709634560,"internally_reserved":0,"allocated":7991296,"data_stored":4989026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001979"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358209+0000","last_change":"2024-03-26T05:02:06.562353+0000","last_active":"2024-03-26T05:02:27.358209+0000","last_peered":"2024-03-26T05:02:27.358209+0000","last_clean":"2024-03-26T05:02:27.358209+0000","last_became_active":"2024-03-26T05:00:28.972969+0000","last_became_peered":"2024-03-26T05:00:28.972969+0000","last_unstale":"2024-03-26T05:02:27.358209+0000","last_undegraded":"2024-03-26T05:02:27.358209+0000","last_fullsized":"2024-03-26T05:02:27.358209+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:06.562257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:06.562257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:11.607057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653013+0000","last_change":"2024-03-26T05:01:39.812600+0000","last_active":"2024-03-26T05:02:27.653013+0000","last_peered":"2024-03-26T05:02:27.653013+0000","last_clean":"2024-03-26T05:02:27.653013+0000","last_became_active":"2024-03-26T05:00:26.953154+0000","last_became_peered":"2024-03-26T05:00:26.953154+0000","last_unstale":"2024-03-26T05:02:27.653013+0000","last_undegraded":"2024-03-26T05:02:27.653013+0000","last_fullsized":"2024-03-26T05:02:27.653013+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.812509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.812509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:45.654690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967422+0000","last_change":"2024-03-26T05:01:35.658900+0000","last_active":"2024-03-26T05:02:27.967422+0000","last_peered":"2024-03-26T05:02:27.967422+0000","last_clean":"2024-03-26T05:02:27.967422+0000","last_became_active":"2024-03-26T05:00:23.346607+0000","last_became_peered":"2024-03-26T05:00:23.346607+0000","last_unstale":"2024-03-26T05:02:27.967422+0000","last_undegraded":"2024-03-26T05:02:27.967422+0000","last_fullsized":"2024-03-26T05:02:27.967422+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:35.658796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.658796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:41.835654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'15","reported_seq":111,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358155+0000","last_change":"2024-03-26T05:00:24.943621+0000","last_active":"2024-03-26T05:02:27.358155+0000","last_peered":"2024-03-26T05:02:27.358155+0000","last_clean":"2024-03-26T05:02:27.358155+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:02:27.358155+0000","last_undegraded":"2024-03-26T05:02:27.358155+0000","last_fullsized":"2024-03-26T05:02:27.358155+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:29.538816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":74,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358429+0000","last_change":"2024-03-26T05:00:29.463053+0000","last_active":"2024-03-26T05:02:27.358429+0000","last_peered":"2024-03-26T05:02:27.358429+0000","last_clean":"2024-03-26T05:02:27.358429+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:02:27.358429+0000","last_undegraded":"2024-03-26T05:02:27.358429+0000","last_fullsized":"2024-03-26T05:02:27.358429+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":79,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358063+0000","last_change":"2024-03-26T05:00:26.955355+0000","last_active":"2024-03-26T05:02:27.358063+0000","last_peered":"2024-03-26T05:02:27.358063+0000","last_clean":"2024-03-26T05:02:27.358063+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:02:27.358063+0000","last_undegraded":"2024-03-26T05:02:27.358063+0000","last_fullsized":"2024-03-26T05:02:27.358063+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:29.538968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358400+0000","last_change":"2024-03-26T05:01:37.201540+0000","last_active":"2024-03-26T05:02:27.358400+0000","last_peered":"2024-03-26T05:02:27.358400+0000","last_clean":"2024-03-26T05:02:27.358400+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:02:27.358400+0000","last_undegraded":"2024-03-26T05:02:27.358400+0000","last_fullsized":"2024-03-26T05:02:27.358400+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.201446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.201446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:43.519218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'39","reported_seq":163,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358438+0000","last_change":"2024-03-26T05:02:17.029181+0000","last_active":"2024-03-26T05:02:27.358438+0000","last_peered":"2024-03-26T05:02:27.358438+0000","last_clean":"2024-03-26T05:02:27.358438+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:02:27.358438+0000","last_undegraded":"2024-03-26T05:02:27.358438+0000","last_fullsized":"2024-03-26T05:02:27.358438+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'39","last_scrub_stamp":"2024-03-26T05:02:17.029087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:17.029087+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:35.317893+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.966920+0000","last_change":"2024-03-26T05:01:32.598168+0000","last_active":"2024-03-26T05:02:27.966920+0000","last_peered":"2024-03-26T05:02:27.966920+0000","last_clean":"2024-03-26T05:02:27.966920+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:02:27.966920+0000","last_undegraded":"2024-03-26T05:02:27.966920+0000","last_fullsized":"2024-03-26T05:02:27.966920+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:32.598087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:32.598087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.083903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967783+0000","last_change":"2024-03-26T05:01:58.072768+0000","last_active":"2024-03-26T05:02:27.967783+0000","last_peered":"2024-03-26T05:02:27.967783+0000","last_clean":"2024-03-26T05:02:27.967783+0000","last_became_active":"2024-03-26T05:00:26.956013+0000","last_became_peered":"2024-03-26T05:00:26.956013+0000","last_unstale":"2024-03-26T05:02:27.967783+0000","last_undegraded":"2024-03-26T05:02:27.967783+0000","last_fullsized":"2024-03-26T05:02:27.967783+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:58.072676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:58.072676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:19.079733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967833+0000","last_change":"2024-03-26T05:01:30.949017+0000","last_active":"2024-03-26T05:02:27.967833+0000","last_peered":"2024-03-26T05:02:27.967833+0000","last_clean":"2024-03-26T05:02:27.967833+0000","last_became_active":"2024-03-26T05:00:22.768098+0000","last_became_peered":"2024-03-26T05:00:22.768098+0000","last_unstale":"2024-03-26T05:02:27.967833+0000","last_undegraded":"2024-03-26T05:02:27.967833+0000","last_fullsized":"2024-03-26T05:02:27.967833+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.948866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.948866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.633607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'21","reported_seq":131,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358443+0000","last_change":"2024-03-26T05:01:30.476359+0000","last_active":"2024-03-26T05:02:27.358443+0000","last_peered":"2024-03-26T05:02:27.358443+0000","last_clean":"2024-03-26T05:02:27.358443+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:02:27.358443+0000","last_undegraded":"2024-03-26T05:02:27.358443+0000","last_fullsized":"2024-03-26T05:02:27.358443+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:01:30.476249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.476249+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.885741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967251+0000","last_change":"2024-03-26T05:01:51.098460+0000","last_active":"2024-03-26T05:02:27.967251+0000","last_peered":"2024-03-26T05:02:27.967251+0000","last_clean":"2024-03-26T05:02:27.967251+0000","last_became_active":"2024-03-26T05:00:28.982276+0000","last_became_peered":"2024-03-26T05:00:28.982276+0000","last_unstale":"2024-03-26T05:02:27.967251+0000","last_undegraded":"2024-03-26T05:02:27.967251+0000","last_fullsized":"2024-03-26T05:02:27.967251+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:51.098327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.098327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:51.933537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968281+0000","last_change":"2024-03-26T05:01:53.941805+0000","last_active":"2024-03-26T05:02:27.968281+0000","last_peered":"2024-03-26T05:02:27.968281+0000","last_clean":"2024-03-26T05:02:27.968281+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:02:27.968281+0000","last_undegraded":"2024-03-26T05:02:27.968281+0000","last_fullsized":"2024-03-26T05:02:27.968281+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:53.941700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:53.941700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:05.433050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967928+0000","last_change":"2024-03-26T05:01:39.021449+0000","last_active":"2024-03-26T05:02:27.967928+0000","last_peered":"2024-03-26T05:02:27.967928+0000","last_clean":"2024-03-26T05:02:27.967928+0000","last_became_active":"2024-03-26T05:00:22.766675+0000","last_became_peered":"2024-03-26T05:00:22.766675+0000","last_unstale":"2024-03-26T05:02:27.967928+0000","last_undegraded":"2024-03-26T05:02:27.967928+0000","last_fullsized":"2024-03-26T05:02:27.967928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.021315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.021315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:39.428392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'11","reported_seq":105,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358386+0000","last_change":"2024-03-26T05:00:24.943205+0000","last_active":"2024-03-26T05:02:27.358386+0000","last_peered":"2024-03-26T05:02:27.358386+0000","last_clean":"2024-03-26T05:02:27.358386+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:02:27.358386+0000","last_undegraded":"2024-03-26T05:02:27.358386+0000","last_fullsized":"2024-03-26T05:02:27.358386+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:29.538892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.359259+0000","last_change":"2024-03-26T05:01:50.502541+0000","last_active":"2024-03-26T05:02:27.359259+0000","last_peered":"2024-03-26T05:02:27.359259+0000","last_clean":"2024-03-26T05:02:27.359259+0000","last_became_active":"2024-03-26T05:00:29.461032+0000","last_became_peered":"2024-03-26T05:00:29.461032+0000","last_unstale":"2024-03-26T05:02:27.359259+0000","last_undegraded":"2024-03-26T05:02:27.359259+0000","last_fullsized":"2024-03-26T05:02:27.359259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:01:50.502459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:52.644841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653549+0000","last_change":"2024-03-26T05:01:51.781163+0000","last_active":"2024-03-26T05:02:27.653549+0000","last_peered":"2024-03-26T05:02:27.653549+0000","last_clean":"2024-03-26T05:02:27.653549+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:02:27.653549+0000","last_undegraded":"2024-03-26T05:02:27.653549+0000","last_fullsized":"2024-03-26T05:02:27.653549+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:51.781081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.781081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:56.997361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967225+0000","last_change":"2024-03-26T05:01:47.742402+0000","last_active":"2024-03-26T05:02:27.967225+0000","last_peered":"2024-03-26T05:02:27.967225+0000","last_clean":"2024-03-26T05:02:27.967225+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:02:27.967225+0000","last_undegraded":"2024-03-26T05:02:27.967225+0000","last_fullsized":"2024-03-26T05:02:27.967225+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:47.742292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:47.742292+0000","last_clean_scrub_stamp":"2024-03-26T05:01:47.742292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.854236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":115,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.726743+0000","last_change":"2024-03-26T05:01:24.617269+0000","last_active":"2024-03-26T05:02:27.726743+0000","last_peered":"2024-03-26T05:02:27.726743+0000","last_clean":"2024-03-26T05:02:27.726743+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:02:27.726743+0000","last_undegraded":"2024-03-26T05:02:27.726743+0000","last_fullsized":"2024-03-26T05:02:27.726743+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:24.617170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:24.617170+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:34.140661+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358775+0000","last_change":"2024-03-26T05:01:30.046528+0000","last_active":"2024-03-26T05:02:27.358775+0000","last_peered":"2024-03-26T05:02:27.358775+0000","last_clean":"2024-03-26T05:02:27.358775+0000","last_became_active":"2024-03-26T05:00:29.463238+0000","last_became_peered":"2024-03-26T05:00:29.463238+0000","last_unstale":"2024-03-26T05:02:27.358775+0000","last_undegraded":"2024-03-26T05:02:27.358775+0000","last_fullsized":"2024-03-26T05:02:27.358775+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.046425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.046425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:35.648365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358914+0000","last_change":"2024-03-26T05:01:51.535065+0000","last_active":"2024-03-26T05:02:27.358914+0000","last_peered":"2024-03-26T05:02:27.358914+0000","last_clean":"2024-03-26T05:02:27.358914+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:02:27.358914+0000","last_undegraded":"2024-03-26T05:02:27.358914+0000","last_fullsized":"2024-03-26T05:02:27.358914+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:51.534988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.534988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:12.499805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358538+0000","last_change":"2024-03-26T05:01:46.410472+0000","last_active":"2024-03-26T05:02:27.358538+0000","last_peered":"2024-03-26T05:02:27.358538+0000","last_clean":"2024-03-26T05:02:27.358538+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:02:27.358538+0000","last_undegraded":"2024-03-26T05:02:27.358538+0000","last_fullsized":"2024-03-26T05:02:27.358538+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:01:46.410356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:52.981688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'20","reported_seq":127,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358180+0000","last_change":"2024-03-26T05:01:54.969276+0000","last_active":"2024-03-26T05:02:27.358180+0000","last_peered":"2024-03-26T05:02:27.358180+0000","last_clean":"2024-03-26T05:02:27.358180+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:02:27.358180+0000","last_undegraded":"2024-03-26T05:02:27.358180+0000","last_fullsized":"2024-03-26T05:02:27.358180+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:01:54.969172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:54.969172+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:13.767531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.969025+0000","last_change":"2024-03-26T05:01:55.953306+0000","last_active":"2024-03-26T05:02:27.969025+0000","last_peered":"2024-03-26T05:02:27.969025+0000","last_clean":"2024-03-26T05:02:27.969025+0000","last_became_active":"2024-03-26T05:00:28.981505+0000","last_became_peered":"2024-03-26T05:00:28.981505+0000","last_unstale":"2024-03-26T05:02:27.969025+0000","last_undegraded":"2024-03-26T05:02:27.969025+0000","last_fullsized":"2024-03-26T05:02:27.969025+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:01:55.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:55.953223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:08.010799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":122,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.495328+0000","last_change":"2024-03-26T05:01:51.236091+0000","last_active":"2024-03-26T05:02:27.495328+0000","last_peered":"2024-03-26T05:02:27.495328+0000","last_clean":"2024-03-26T05:02:27.495328+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:02:27.495328+0000","last_undegraded":"2024-03-26T05:02:27.495328+0000","last_fullsized":"2024-03-26T05:02:27.495328+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:51.235994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.235994+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:51.261797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653108+0000","last_change":"2024-03-26T05:02:07.853288+0000","last_active":"2024-03-26T05:02:27.653108+0000","last_peered":"2024-03-26T05:02:27.653108+0000","last_clean":"2024-03-26T05:02:27.653108+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:02:27.653108+0000","last_undegraded":"2024-03-26T05:02:27.653108+0000","last_fullsized":"2024-03-26T05:02:27.653108+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:02:07.853157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:07.853157+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:20.700130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'16","reported_seq":119,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967476+0000","last_change":"2024-03-26T05:01:41.650400+0000","last_active":"2024-03-26T05:02:27.967476+0000","last_peered":"2024-03-26T05:02:27.967476+0000","last_clean":"2024-03-26T05:02:27.967476+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:02:27.967476+0000","last_undegraded":"2024-03-26T05:02:27.967476+0000","last_fullsized":"2024-03-26T05:02:27.967476+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:01:41.650310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.650310+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:08.831570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653651+0000","last_change":"2024-03-26T05:01:56.807267+0000","last_active":"2024-03-26T05:02:27.653651+0000","last_peered":"2024-03-26T05:02:27.653651+0000","last_clean":"2024-03-26T05:02:27.653651+0000","last_became_active":"2024-03-26T05:00:28.981049+0000","last_became_peered":"2024-03-26T05:00:28.981049+0000","last_unstale":"2024-03-26T05:02:27.653651+0000","last_undegraded":"2024-03-26T05:02:27.653651+0000","last_fullsized":"2024-03-26T05:02:27.653651+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:56.807184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:56.807184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:00.264512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967566+0000","last_change":"2024-03-26T05:02:02.895526+0000","last_active":"2024-03-26T05:02:27.967566+0000","last_peered":"2024-03-26T05:02:27.967566+0000","last_clean":"2024-03-26T05:02:27.967566+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:02:27.967566+0000","last_undegraded":"2024-03-26T05:02:27.967566+0000","last_fullsized":"2024-03-26T05:02:27.967566+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:02.895388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:02:02.895388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:23.861159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.357604+0000","last_change":"2024-03-26T05:01:43.166558+0000","last_active":"2024-03-26T05:02:27.357604+0000","last_peered":"2024-03-26T05:02:27.357604+0000","last_clean":"2024-03-26T05:02:27.357604+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:02:27.357604+0000","last_undegraded":"2024-03-26T05:02:27.357604+0000","last_fullsized":"2024-03-26T05:02:27.357604+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:43.166422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:43.166422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:50.132493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'17","reported_seq":140,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.495265+0000","last_change":"2024-03-26T05:01:53.258532+0000","last_active":"2024-03-26T05:02:27.495265+0000","last_peered":"2024-03-26T05:02:27.495265+0000","last_clean":"2024-03-26T05:02:27.495265+0000","last_became_active":"2024-03-26T05:00:49.643308+0000","last_became_peered":"2024-03-26T05:00:49.643308+0000","last_unstale":"2024-03-26T05:02:27.495265+0000","last_undegraded":"2024-03-26T05:02:27.495265+0000","last_fullsized":"2024-03-26T05:02:27.495265+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:01:53.258422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:53.258422+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:22.142699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968189+0000","last_change":"2024-03-26T05:02:08.031235+0000","last_active":"2024-03-26T05:02:27.968189+0000","last_peered":"2024-03-26T05:02:27.968189+0000","last_clean":"2024-03-26T05:02:27.968189+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:02:27.968189+0000","last_undegraded":"2024-03-26T05:02:27.968189+0000","last_fullsized":"2024-03-26T05:02:27.968189+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:08.031138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:08.031138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:34.436715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.494751+0000","last_change":"2024-03-26T05:01:44.235422+0000","last_active":"2024-03-26T05:02:27.494751+0000","last_peered":"2024-03-26T05:02:27.494751+0000","last_clean":"2024-03-26T05:02:27.494751+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:02:27.494751+0000","last_undegraded":"2024-03-26T05:02:27.494751+0000","last_fullsized":"2024-03-26T05:02:27.494751+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:44.235328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.235328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.223031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967510+0000","last_change":"2024-03-26T05:02:15.044319+0000","last_active":"2024-03-26T05:02:27.967510+0000","last_peered":"2024-03-26T05:02:27.967510+0000","last_clean":"2024-03-26T05:02:27.967510+0000","last_became_active":"2024-03-26T05:00:22.774783+0000","last_became_peered":"2024-03-26T05:00:22.774783+0000","last_unstale":"2024-03-26T05:02:27.967510+0000","last_undegraded":"2024-03-26T05:02:27.967510+0000","last_fullsized":"2024-03-26T05:02:27.967510+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:15.044239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:15.044239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:35.582786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"100'6","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.357731+0000","last_change":"2024-03-26T05:01:58.079447+0000","last_active":"2024-03-26T05:02:27.357731+0000","last_peered":"2024-03-26T05:02:27.357731+0000","last_clean":"2024-03-26T05:02:27.357731+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:02:27.357731+0000","last_undegraded":"2024-03-26T05:02:27.357731+0000","last_fullsized":"2024-03-26T05:02:27.357731+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'6","last_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:01:58.079363+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:08.138424+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358843+0000","last_change":"2024-03-26T05:02:02.058626+0000","last_active":"2024-03-26T05:02:27.358843+0000","last_peered":"2024-03-26T05:02:27.358843+0000","last_clean":"2024-03-26T05:02:27.358843+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:02:27.358843+0000","last_undegraded":"2024-03-26T05:02:27.358843+0000","last_fullsized":"2024-03-26T05:02:27.358843+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:02.058523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:02.058523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:23.465853+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968001+0000","last_change":"2024-03-26T05:01:58.932169+0000","last_active":"2024-03-26T05:02:27.968001+0000","last_peered":"2024-03-26T05:02:27.968001+0000","last_clean":"2024-03-26T05:02:27.968001+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:02:27.968001+0000","last_undegraded":"2024-03-26T05:02:27.968001+0000","last_fullsized":"2024-03-26T05:02:27.968001+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:58.932064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:58.932064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:04.564275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967706+0000","last_change":"2024-03-26T05:01:29.997645+0000","last_active":"2024-03-26T05:02:27.967706+0000","last_peered":"2024-03-26T05:02:27.967706+0000","last_clean":"2024-03-26T05:02:27.967706+0000","last_became_active":"2024-03-26T05:00:22.771199+0000","last_became_peered":"2024-03-26T05:00:22.771199+0000","last_unstale":"2024-03-26T05:02:27.967706+0000","last_undegraded":"2024-03-26T05:02:27.967706+0000","last_fullsized":"2024-03-26T05:02:27.967706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:35.309625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'17","reported_seq":121,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358466+0000","last_change":"2024-03-26T05:01:55.996301+0000","last_active":"2024-03-26T05:02:27.358466+0000","last_peered":"2024-03-26T05:02:27.358466+0000","last_clean":"2024-03-26T05:02:27.358466+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:02:27.358466+0000","last_undegraded":"2024-03-26T05:02:27.358466+0000","last_fullsized":"2024-03-26T05:02:27.358466+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:01:55.996210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:55.996210+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:19.201138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.966653+0000","last_change":"2024-03-26T05:01:49.768067+0000","last_active":"2024-03-26T05:02:27.966653+0000","last_peered":"2024-03-26T05:02:27.966653+0000","last_clean":"2024-03-26T05:02:27.966653+0000","last_became_active":"2024-03-26T05:00:28.985158+0000","last_became_peered":"2024-03-26T05:00:28.985158+0000","last_unstale":"2024-03-26T05:02:27.966653+0000","last_undegraded":"2024-03-26T05:02:27.966653+0000","last_fullsized":"2024-03-26T05:02:27.966653+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:49.767986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:49.767986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.936245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358334+0000","last_change":"2024-03-26T05:00:26.960672+0000","last_active":"2024-03-26T05:02:27.358334+0000","last_peered":"2024-03-26T05:02:27.358334+0000","last_clean":"2024-03-26T05:02:27.358334+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:02:27.358334+0000","last_undegraded":"2024-03-26T05:02:27.358334+0000","last_fullsized":"2024-03-26T05:02:27.358334+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":93,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.969374+0000","last_change":"2024-03-26T05:02:07.036740+0000","last_active":"2024-03-26T05:02:27.969374+0000","last_peered":"2024-03-26T05:02:27.969374+0000","last_clean":"2024-03-26T05:02:27.969374+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:02:27.969374+0000","last_undegraded":"2024-03-26T05:02:27.969374+0000","last_fullsized":"2024-03-26T05:02:27.969374+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:07.036606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:07.036606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:36.636020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":106,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653378+0000","last_change":"2024-03-26T05:02:00.829744+0000","last_active":"2024-03-26T05:02:27.653378+0000","last_peered":"2024-03-26T05:02:27.653378+0000","last_clean":"2024-03-26T05:02:27.653378+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:02:27.653378+0000","last_undegraded":"2024-03-26T05:02:27.653378+0000","last_fullsized":"2024-03-26T05:02:27.653378+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-26T05:02:00.829653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:00.829653+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:11.975637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.357694+0000","last_change":"2024-03-26T05:02:16.073043+0000","last_active":"2024-03-26T05:02:27.357694+0000","last_peered":"2024-03-26T05:02:27.357694+0000","last_clean":"2024-03-26T05:02:27.357694+0000","last_became_active":"2024-03-26T05:00:28.983417+0000","last_became_peered":"2024-03-26T05:00:28.983417+0000","last_unstale":"2024-03-26T05:02:27.357694+0000","last_undegraded":"2024-03-26T05:02:27.357694+0000","last_fullsized":"2024-03-26T05:02:27.357694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:02:16.072965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:42.013976+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653234+0000","last_change":"2024-03-26T05:01:30.777153+0000","last_active":"2024-03-26T05:02:27.653234+0000","last_peered":"2024-03-26T05:02:27.653234+0000","last_clean":"2024-03-26T05:02:27.653234+0000","last_became_active":"2024-03-26T05:00:26.954653+0000","last_became_peered":"2024-03-26T05:00:26.954653+0000","last_unstale":"2024-03-26T05:02:27.653234+0000","last_undegraded":"2024-03-26T05:02:27.653234+0000","last_fullsized":"2024-03-26T05:02:27.653234+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:01:30.777070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:45.021844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358615+0000","last_change":"2024-03-26T05:00:22.767847+0000","last_active":"2024-03-26T05:02:27.358615+0000","last_peered":"2024-03-26T05:02:27.358615+0000","last_clean":"2024-03-26T05:02:27.358615+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:02:27.358615+0000","last_undegraded":"2024-03-26T05:02:27.358615+0000","last_fullsized":"2024-03-26T05:02:27.358615+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:29.538854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"95'16","reported_seq":115,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358677+0000","last_change":"2024-03-26T05:01:40.462389+0000","last_active":"2024-03-26T05:02:27.358677+0000","last_peered":"2024-03-26T05:02:27.358677+0000","last_clean":"2024-03-26T05:02:27.358677+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:02:27.358677+0000","last_undegraded":"2024-03-26T05:02:27.358677+0000","last_fullsized":"2024-03-26T05:02:27.358677+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.462271+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.765446+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967232+0000","last_change":"2024-03-26T05:02:07.972990+0000","last_active":"2024-03-26T05:02:27.967232+0000","last_peered":"2024-03-26T05:02:27.967232+0000","last_clean":"2024-03-26T05:02:27.967232+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:02:27.967232+0000","last_undegraded":"2024-03-26T05:02:27.967232+0000","last_fullsized":"2024-03-26T05:02:27.967232+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:07.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:07.972913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:35.115542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":120,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968039+0000","last_change":"2024-03-26T05:02:11.976655+0000","last_active":"2024-03-26T05:02:27.968039+0000","last_peered":"2024-03-26T05:02:27.968039+0000","last_clean":"2024-03-26T05:02:27.968039+0000","last_became_active":"2024-03-26T05:00:26.955836+0000","last_became_peered":"2024-03-26T05:00:26.955836+0000","last_unstale":"2024-03-26T05:02:27.968039+0000","last_undegraded":"2024-03-26T05:02:27.968039+0000","last_fullsized":"2024-03-26T05:02:27.968039+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:02:11.976542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:02:11.976542+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:33.073044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":93,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968457+0000","last_change":"2024-03-26T05:01:26.785152+0000","last_active":"2024-03-26T05:02:27.968457+0000","last_peered":"2024-03-26T05:02:27.968457+0000","last_clean":"2024-03-26T05:02:27.968457+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:02:27.968457+0000","last_undegraded":"2024-03-26T05:02:27.968457+0000","last_fullsized":"2024-03-26T05:02:27.968457+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.785056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.785056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.302223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'22","reported_seq":129,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358036+0000","last_change":"2024-03-26T05:01:40.118101+0000","last_active":"2024-03-26T05:02:27.358036+0000","last_peered":"2024-03-26T05:02:27.358036+0000","last_clean":"2024-03-26T05:02:27.358036+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:02:27.358036+0000","last_undegraded":"2024-03-26T05:02:27.358036+0000","last_fullsized":"2024-03-26T05:02:27.358036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:01:40.117953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.117953+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.646893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":75,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.359324+0000","last_change":"2024-03-26T05:00:28.979801+0000","last_active":"2024-03-26T05:02:27.359324+0000","last_peered":"2024-03-26T05:02:27.359324+0000","last_clean":"2024-03-26T05:02:27.359324+0000","last_became_active":"2024-03-26T05:00:28.979477+0000","last_became_peered":"2024-03-26T05:00:28.979477+0000","last_unstale":"2024-03-26T05:02:27.359324+0000","last_undegraded":"2024-03-26T05:02:27.359324+0000","last_fullsized":"2024-03-26T05:02:27.359324+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:00:27.943471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:29.538987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":123,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358333+0000","last_change":"2024-03-26T05:01:49.006492+0000","last_active":"2024-03-26T05:02:27.358333+0000","last_peered":"2024-03-26T05:02:27.358333+0000","last_clean":"2024-03-26T05:02:27.358333+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:02:27.358333+0000","last_undegraded":"2024-03-26T05:02:27.358333+0000","last_fullsized":"2024-03-26T05:02:27.358333+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:49.006396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:49.006396+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.424499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.966916+0000","last_change":"2024-03-26T05:01:51.835794+0000","last_active":"2024-03-26T05:02:27.966916+0000","last_peered":"2024-03-26T05:02:27.966916+0000","last_clean":"2024-03-26T05:02:27.966916+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:02:27.966916+0000","last_undegraded":"2024-03-26T05:02:27.966916+0000","last_fullsized":"2024-03-26T05:02:27.966916+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:51.835722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:51.835722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:11.405634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"98'20","reported_seq":123,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968164+0000","last_change":"2024-03-26T05:01:48.070439+0000","last_active":"2024-03-26T05:02:27.968164+0000","last_peered":"2024-03-26T05:02:27.968164+0000","last_clean":"2024-03-26T05:02:27.968164+0000","last_became_active":"2024-03-26T05:00:25.648284+0000","last_became_peered":"2024-03-26T05:00:25.648284+0000","last_unstale":"2024-03-26T05:02:27.968164+0000","last_undegraded":"2024-03-26T05:02:27.968164+0000","last_fullsized":"2024-03-26T05:02:27.968164+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'20","last_scrub_stamp":"2024-03-26T05:01:48.070338+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:01:48.070338+0000","last_clean_scrub_stamp":"2024-03-26T05:01:48.070338+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.153022+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.726994+0000","last_change":"2024-03-26T05:01:31.568804+0000","last_active":"2024-03-26T05:02:27.726994+0000","last_peered":"2024-03-26T05:02:27.726994+0000","last_clean":"2024-03-26T05:02:27.726994+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:02:27.726994+0000","last_undegraded":"2024-03-26T05:02:27.726994+0000","last_fullsized":"2024-03-26T05:02:27.726994+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:31.568722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:31.568722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:38.465689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358723+0000","last_change":"2024-03-26T05:02:18.010095+0000","last_active":"2024-03-26T05:02:27.358723+0000","last_peered":"2024-03-26T05:02:27.358723+0000","last_clean":"2024-03-26T05:02:27.358723+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:02:27.358723+0000","last_undegraded":"2024-03-26T05:02:27.358723+0000","last_fullsized":"2024-03-26T05:02:27.358723+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:02:18.010011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:44.716920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":93,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967769+0000","last_change":"2024-03-26T05:02:01.976368+0000","last_active":"2024-03-26T05:02:27.967769+0000","last_peered":"2024-03-26T05:02:27.967769+0000","last_clean":"2024-03-26T05:02:27.967769+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:02:27.967769+0000","last_undegraded":"2024-03-26T05:02:27.967769+0000","last_fullsized":"2024-03-26T05:02:27.967769+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:01.976283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:01.976283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:26.431871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'21","reported_seq":131,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967510+0000","last_change":"2024-03-26T05:01:38.722775+0000","last_active":"2024-03-26T05:02:27.967510+0000","last_peered":"2024-03-26T05:02:27.967510+0000","last_clean":"2024-03-26T05:02:27.967510+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:02:27.967510+0000","last_undegraded":"2024-03-26T05:02:27.967510+0000","last_fullsized":"2024-03-26T05:02:27.967510+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:01:38.722652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:38.722652+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.850545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.969269+0000","last_change":"2024-03-26T05:02:11.010815+0000","last_active":"2024-03-26T05:02:27.969269+0000","last_peered":"2024-03-26T05:02:27.969269+0000","last_clean":"2024-03-26T05:02:27.969269+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:02:27.969269+0000","last_undegraded":"2024-03-26T05:02:27.969269+0000","last_fullsized":"2024-03-26T05:02:27.969269+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:11.010698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:11.010698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:18.366168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968238+0000","last_change":"2024-03-26T05:01:59.983345+0000","last_active":"2024-03-26T05:02:27.968238+0000","last_peered":"2024-03-26T05:02:27.968238+0000","last_clean":"2024-03-26T05:02:27.968238+0000","last_became_active":"2024-03-26T05:00:26.953416+0000","last_became_peered":"2024-03-26T05:00:26.953416+0000","last_unstale":"2024-03-26T05:02:27.968238+0000","last_undegraded":"2024-03-26T05:02:27.968238+0000","last_fullsized":"2024-03-26T05:02:27.968238+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:59.983240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:59.983240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:03.736375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.494352+0000","last_change":"2024-03-26T05:01:37.375697+0000","last_active":"2024-03-26T05:02:27.494352+0000","last_peered":"2024-03-26T05:02:27.494352+0000","last_clean":"2024-03-26T05:02:27.494352+0000","last_became_active":"2024-03-26T05:00:22.771080+0000","last_became_peered":"2024-03-26T05:00:22.771080+0000","last_unstale":"2024-03-26T05:02:27.494352+0000","last_undegraded":"2024-03-26T05:02:27.494352+0000","last_fullsized":"2024-03-26T05:02:27.494352+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:01:37.375533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.375533+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:51.717267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'12","reported_seq":111,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358907+0000","last_change":"2024-03-26T05:01:27.964049+0000","last_active":"2024-03-26T05:02:27.358907+0000","last_peered":"2024-03-26T05:02:27.358907+0000","last_clean":"2024-03-26T05:02:27.358907+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:02:27.358907+0000","last_undegraded":"2024-03-26T05:02:27.358907+0000","last_fullsized":"2024-03-26T05:02:27.358907+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:01:27.963954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:27.963954+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:50.330132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653128+0000","last_change":"2024-03-26T05:02:24.355011+0000","last_active":"2024-03-26T05:02:27.653128+0000","last_peered":"2024-03-26T05:02:27.653128+0000","last_clean":"2024-03-26T05:02:27.653128+0000","last_became_active":"2024-03-26T05:02:24.354658+0000","last_became_peered":"2024-03-26T05:02:24.354658+0000","last_unstale":"2024-03-26T05:02:27.653128+0000","last_undegraded":"2024-03-26T05:02:27.653128+0000","last_fullsized":"2024-03-26T05:02:27.653128+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:49.467695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967897+0000","last_change":"2024-03-26T05:01:36.945807+0000","last_active":"2024-03-26T05:02:27.967897+0000","last_peered":"2024-03-26T05:02:27.967897+0000","last_clean":"2024-03-26T05:02:27.967897+0000","last_became_active":"2024-03-26T05:00:29.461059+0000","last_became_peered":"2024-03-26T05:00:29.461059+0000","last_unstale":"2024-03-26T05:02:27.967897+0000","last_undegraded":"2024-03-26T05:02:27.967897+0000","last_fullsized":"2024-03-26T05:02:27.967897+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:01:36.945708+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.001390+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'3","reported_seq":111,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968772+0000","last_change":"2024-03-26T05:01:22.799027+0000","last_active":"2024-03-26T05:02:27.968772+0000","last_peered":"2024-03-26T05:02:27.968772+0000","last_clean":"2024-03-26T05:02:27.968772+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:02:27.968772+0000","last_undegraded":"2024-03-26T05:02:27.968772+0000","last_fullsized":"2024-03-26T05:02:27.968772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-03-26T05:01:22.798897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:22.798897+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.103651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'27","reported_seq":139,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967362+0000","last_change":"2024-03-26T05:01:29.048598+0000","last_active":"2024-03-26T05:02:27.967362+0000","last_peered":"2024-03-26T05:02:27.967362+0000","last_clean":"2024-03-26T05:02:27.967362+0000","last_became_active":"2024-03-26T05:00:25.651328+0000","last_became_peered":"2024-03-26T05:00:25.651328+0000","last_unstale":"2024-03-26T05:02:27.967362+0000","last_undegraded":"2024-03-26T05:02:27.967362+0000","last_fullsized":"2024-03-26T05:02:27.967362+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'27","last_scrub_stamp":"2024-03-26T05:01:29.048456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.048456+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:56.053198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":119,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968122+0000","last_change":"2024-03-26T05:01:37.743111+0000","last_active":"2024-03-26T05:02:27.968122+0000","last_peered":"2024-03-26T05:02:27.968122+0000","last_clean":"2024-03-26T05:02:27.968122+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:02:27.968122+0000","last_undegraded":"2024-03-26T05:02:27.968122+0000","last_fullsized":"2024-03-26T05:02:27.968122+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:37.742995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.742995+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:07.111791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358858+0000","last_change":"2024-03-26T05:02:24.354594+0000","last_active":"2024-03-26T05:02:27.358858+0000","last_peered":"2024-03-26T05:02:27.358858+0000","last_clean":"2024-03-26T05:02:27.358858+0000","last_became_active":"2024-03-26T05:02:24.352844+0000","last_became_peered":"2024-03-26T05:02:24.352844+0000","last_unstale":"2024-03-26T05:02:27.358858+0000","last_undegraded":"2024-03-26T05:02:27.358858+0000","last_fullsized":"2024-03-26T05:02:27.358858+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:42.492540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358670+0000","last_change":"2024-03-26T05:01:29.011852+0000","last_active":"2024-03-26T05:02:27.358670+0000","last_peered":"2024-03-26T05:02:27.358670+0000","last_clean":"2024-03-26T05:02:27.358670+0000","last_became_active":"2024-03-26T05:00:28.985917+0000","last_became_peered":"2024-03-26T05:00:28.985917+0000","last_unstale":"2024-03-26T05:02:27.358670+0000","last_undegraded":"2024-03-26T05:02:27.358670+0000","last_fullsized":"2024-03-26T05:02:27.358670+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.011757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.011757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.742465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":93,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.969387+0000","last_change":"2024-03-26T05:01:46.855799+0000","last_active":"2024-03-26T05:02:27.969387+0000","last_peered":"2024-03-26T05:02:27.969387+0000","last_clean":"2024-03-26T05:02:27.969387+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:02:27.969387+0000","last_undegraded":"2024-03-26T05:02:27.969387+0000","last_fullsized":"2024-03-26T05:02:27.969387+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:46.855699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:46.855699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:15.236244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'24","reported_seq":130,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358583+0000","last_change":"2024-03-26T05:00:24.944015+0000","last_active":"2024-03-26T05:02:27.358583+0000","last_peered":"2024-03-26T05:02:27.358583+0000","last_clean":"2024-03-26T05:02:27.358583+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:02:27.358583+0000","last_undegraded":"2024-03-26T05:02:27.358583+0000","last_fullsized":"2024-03-26T05:02:27.358583+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:29.538836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.726582+0000","last_change":"2024-03-26T05:01:26.618364+0000","last_active":"2024-03-26T05:02:27.726582+0000","last_peered":"2024-03-26T05:02:27.726582+0000","last_clean":"2024-03-26T05:02:27.726582+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:02:27.726582+0000","last_undegraded":"2024-03-26T05:02:27.726582+0000","last_fullsized":"2024-03-26T05:02:27.726582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.618218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.618218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.093176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.966714+0000","last_change":"2024-03-26T05:02:24.356216+0000","last_active":"2024-03-26T05:02:27.966714+0000","last_peered":"2024-03-26T05:02:27.966714+0000","last_clean":"2024-03-26T05:02:27.966714+0000","last_became_active":"2024-03-26T05:02:24.355930+0000","last_became_peered":"2024-03-26T05:02:24.355930+0000","last_unstale":"2024-03-26T05:02:27.966714+0000","last_undegraded":"2024-03-26T05:02:27.966714+0000","last_fullsized":"2024-03-26T05:02:27.966714+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:32.331976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967489+0000","last_change":"2024-03-26T05:02:09.031840+0000","last_active":"2024-03-26T05:02:27.967489+0000","last_peered":"2024-03-26T05:02:27.967489+0000","last_clean":"2024-03-26T05:02:27.967489+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:02:27.967489+0000","last_undegraded":"2024-03-26T05:02:27.967489+0000","last_fullsized":"2024-03-26T05:02:27.967489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:09.031740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:09.031740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:16.950509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.726757+0000","last_change":"2024-03-26T05:01:33.638557+0000","last_active":"2024-03-26T05:02:27.726757+0000","last_peered":"2024-03-26T05:02:27.726757+0000","last_clean":"2024-03-26T05:02:27.726757+0000","last_became_active":"2024-03-26T05:00:26.961730+0000","last_became_peered":"2024-03-26T05:00:26.961730+0000","last_unstale":"2024-03-26T05:02:27.726757+0000","last_undegraded":"2024-03-26T05:02:27.726757+0000","last_fullsized":"2024-03-26T05:02:27.726757+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:33.638469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:33.638469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:00.135833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":93,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.969609+0000","last_change":"2024-03-26T05:02:06.041987+0000","last_active":"2024-03-26T05:02:27.969609+0000","last_peered":"2024-03-26T05:02:27.969609+0000","last_clean":"2024-03-26T05:02:27.969609+0000","last_became_active":"2024-03-26T05:00:22.772590+0000","last_became_peered":"2024-03-26T05:00:22.772590+0000","last_unstale":"2024-03-26T05:02:27.969609+0000","last_undegraded":"2024-03-26T05:02:27.969609+0000","last_fullsized":"2024-03-26T05:02:27.969609+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:06.041876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:06.041876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:30.074331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'25","reported_seq":132,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.494684+0000","last_change":"2024-03-26T05:01:56.315423+0000","last_active":"2024-03-26T05:02:27.494684+0000","last_peered":"2024-03-26T05:02:27.494684+0000","last_clean":"2024-03-26T05:02:27.494684+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:02:27.494684+0000","last_undegraded":"2024-03-26T05:02:27.494684+0000","last_fullsized":"2024-03-26T05:02:27.494684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:01:56.315274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:56.315274+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:14.373193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.726443+0000","last_change":"2024-03-26T05:02:24.353904+0000","last_active":"2024-03-26T05:02:27.726443+0000","last_peered":"2024-03-26T05:02:27.726443+0000","last_clean":"2024-03-26T05:02:27.726443+0000","last_became_active":"2024-03-26T05:02:24.353523+0000","last_became_peered":"2024-03-26T05:02:24.353523+0000","last_unstale":"2024-03-26T05:02:27.726443+0000","last_undegraded":"2024-03-26T05:02:27.726443+0000","last_fullsized":"2024-03-26T05:02:27.726443+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:42.522996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.966749+0000","last_change":"2024-03-26T05:02:18.965195+0000","last_active":"2024-03-26T05:02:27.966749+0000","last_peered":"2024-03-26T05:02:27.966749+0000","last_clean":"2024-03-26T05:02:27.966749+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:02:27.966749+0000","last_undegraded":"2024-03-26T05:02:27.966749+0000","last_fullsized":"2024-03-26T05:02:27.966749+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:18.965063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:18.965063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:26.718732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.726665+0000","last_change":"2024-03-26T05:01:55.822123+0000","last_active":"2024-03-26T05:02:27.726665+0000","last_peered":"2024-03-26T05:02:27.726665+0000","last_clean":"2024-03-26T05:02:27.726665+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:02:27.726665+0000","last_undegraded":"2024-03-26T05:02:27.726665+0000","last_fullsized":"2024-03-26T05:02:27.726665+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:55.822016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:55.822016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:02.183520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967707+0000","last_change":"2024-03-26T05:01:53.133109+0000","last_active":"2024-03-26T05:02:27.967707+0000","last_peered":"2024-03-26T05:02:27.967707+0000","last_clean":"2024-03-26T05:02:27.967707+0000","last_became_active":"2024-03-26T05:00:22.773770+0000","last_became_peered":"2024-03-26T05:00:22.773770+0000","last_unstale":"2024-03-26T05:02:27.967707+0000","last_undegraded":"2024-03-26T05:02:27.967707+0000","last_fullsized":"2024-03-26T05:02:27.967707+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:53.133006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:53.133006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:10.552053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'15","reported_seq":119,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967370+0000","last_change":"2024-03-26T05:01:50.807516+0000","last_active":"2024-03-26T05:02:27.967370+0000","last_peered":"2024-03-26T05:02:27.967370+0000","last_clean":"2024-03-26T05:02:27.967370+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:02:27.967370+0000","last_undegraded":"2024-03-26T05:02:27.967370+0000","last_fullsized":"2024-03-26T05:02:27.967370+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:50.807430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:50.807430+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:03.240799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.495078+0000","last_change":"2024-03-26T05:02:24.353664+0000","last_active":"2024-03-26T05:02:27.495078+0000","last_peered":"2024-03-26T05:02:27.495078+0000","last_clean":"2024-03-26T05:02:27.495078+0000","last_became_active":"2024-03-26T05:02:24.353248+0000","last_became_peered":"2024-03-26T05:02:24.353248+0000","last_unstale":"2024-03-26T05:02:27.495078+0000","last_undegraded":"2024-03-26T05:02:27.495078+0000","last_fullsized":"2024-03-26T05:02:27.495078+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:24.326552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358573+0000","last_change":"2024-03-26T05:02:01.000226+0000","last_active":"2024-03-26T05:02:27.358573+0000","last_peered":"2024-03-26T05:02:27.358573+0000","last_clean":"2024-03-26T05:02:27.358573+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:02:27.358573+0000","last_undegraded":"2024-03-26T05:02:27.358573+0000","last_fullsized":"2024-03-26T05:02:27.358573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:01.000137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:01.000137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:25.297327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":106,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967834+0000","last_change":"2024-03-26T05:01:35.934149+0000","last_active":"2024-03-26T05:02:27.967834+0000","last_peered":"2024-03-26T05:02:27.967834+0000","last_clean":"2024-03-26T05:02:27.967834+0000","last_became_active":"2024-03-26T05:00:21.760521+0000","last_became_peered":"2024-03-26T05:00:21.760521+0000","last_unstale":"2024-03-26T05:02:27.967834+0000","last_undegraded":"2024-03-26T05:02:27.967834+0000","last_fullsized":"2024-03-26T05:02:27.967834+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:01:35.934044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.934044+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:02.210221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358525+0000","last_change":"2024-03-26T05:00:22.771224+0000","last_active":"2024-03-26T05:02:27.358525+0000","last_peered":"2024-03-26T05:02:27.358525+0000","last_clean":"2024-03-26T05:02:27.358525+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:02:27.358525+0000","last_undegraded":"2024-03-26T05:02:27.358525+0000","last_fullsized":"2024-03-26T05:02:27.358525+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:00:21.748689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'34","reported_seq":177,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:28.407328+0000","last_change":"2024-03-26T05:01:40.826966+0000","last_active":"2024-03-26T05:02:28.407328+0000","last_peered":"2024-03-26T05:02:28.407328+0000","last_clean":"2024-03-26T05:02:28.407328+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:02:28.407328+0000","last_undegraded":"2024-03-26T05:02:28.407328+0000","last_fullsized":"2024-03-26T05:02:28.407328+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'34","last_scrub_stamp":"2024-03-26T05:01:40.826856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:40.826856+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.349406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967184+0000","last_change":"2024-03-26T05:02:16.053713+0000","last_active":"2024-03-26T05:02:27.967184+0000","last_peered":"2024-03-26T05:02:27.967184+0000","last_clean":"2024-03-26T05:02:27.967184+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:02:27.967184+0000","last_undegraded":"2024-03-26T05:02:27.967184+0000","last_fullsized":"2024-03-26T05:02:27.967184+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:16.053635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:02:16.053635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:42.127130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967020+0000","last_change":"2024-03-26T05:02:24.356951+0000","last_active":"2024-03-26T05:02:27.967020+0000","last_peered":"2024-03-26T05:02:27.967020+0000","last_clean":"2024-03-26T05:02:27.967020+0000","last_became_active":"2024-03-26T05:02:24.356658+0000","last_became_peered":"2024-03-26T05:02:24.356658+0000","last_unstale":"2024-03-26T05:02:27.967020+0000","last_undegraded":"2024-03-26T05:02:27.967020+0000","last_fullsized":"2024-03-26T05:02:27.967020+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:25.323866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.969716+0000","last_change":"2024-03-26T05:01:41.789755+0000","last_active":"2024-03-26T05:02:27.969716+0000","last_peered":"2024-03-26T05:02:27.969716+0000","last_clean":"2024-03-26T05:02:27.969716+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:02:27.969716+0000","last_undegraded":"2024-03-26T05:02:27.969716+0000","last_fullsized":"2024-03-26T05:02:27.969716+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:41.789650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.789650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:53.302843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653159+0000","last_change":"2024-03-26T05:02:03.856196+0000","last_active":"2024-03-26T05:02:27.653159+0000","last_peered":"2024-03-26T05:02:27.653159+0000","last_clean":"2024-03-26T05:02:27.653159+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:02:27.653159+0000","last_undegraded":"2024-03-26T05:02:27.653159+0000","last_fullsized":"2024-03-26T05:02:27.653159+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:03.856107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:03.856107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:32.599673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":104,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.726641+0000","last_change":"2024-03-26T05:01:37.678850+0000","last_active":"2024-03-26T05:02:27.726641+0000","last_peered":"2024-03-26T05:02:27.726641+0000","last_clean":"2024-03-26T05:02:27.726641+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:02:27.726641+0000","last_undegraded":"2024-03-26T05:02:27.726641+0000","last_fullsized":"2024-03-26T05:02:27.726641+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.678736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.678736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:44.820757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'19","reported_seq":129,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653654+0000","last_change":"2024-03-26T05:02:06.840443+0000","last_active":"2024-03-26T05:02:27.653654+0000","last_peered":"2024-03-26T05:02:27.653654+0000","last_clean":"2024-03-26T05:02:27.653654+0000","last_became_active":"2024-03-26T05:00:24.937145+0000","last_became_peered":"2024-03-26T05:00:24.937145+0000","last_unstale":"2024-03-26T05:02:27.653654+0000","last_undegraded":"2024-03-26T05:02:27.653654+0000","last_fullsized":"2024-03-26T05:02:27.653654+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-26T05:02:06.840354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:06.840354+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:22.821039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.726920+0000","last_change":"2024-03-26T05:01:29.627760+0000","last_active":"2024-03-26T05:02:27.726920+0000","last_peered":"2024-03-26T05:02:27.726920+0000","last_clean":"2024-03-26T05:02:27.726920+0000","last_became_active":"2024-03-26T05:00:26.957936+0000","last_became_peered":"2024-03-26T05:00:26.957936+0000","last_unstale":"2024-03-26T05:02:27.726920+0000","last_undegraded":"2024-03-26T05:02:27.726920+0000","last_fullsized":"2024-03-26T05:02:27.726920+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:29.627680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.627680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.292786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967457+0000","last_change":"2024-03-26T05:02:24.356413+0000","last_active":"2024-03-26T05:02:27.967457+0000","last_peered":"2024-03-26T05:02:27.967457+0000","last_clean":"2024-03-26T05:02:27.967457+0000","last_became_active":"2024-03-26T05:02:24.356129+0000","last_became_peered":"2024-03-26T05:02:24.356129+0000","last_unstale":"2024-03-26T05:02:27.967457+0000","last_undegraded":"2024-03-26T05:02:27.967457+0000","last_fullsized":"2024-03-26T05:02:27.967457+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:51.901888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967888+0000","last_change":"2024-03-26T05:01:56.143486+0000","last_active":"2024-03-26T05:02:27.967888+0000","last_peered":"2024-03-26T05:02:27.967888+0000","last_clean":"2024-03-26T05:02:27.967888+0000","last_became_active":"2024-03-26T05:00:28.979373+0000","last_became_peered":"2024-03-26T05:00:28.979373+0000","last_unstale":"2024-03-26T05:02:27.967888+0000","last_undegraded":"2024-03-26T05:02:27.967888+0000","last_fullsized":"2024-03-26T05:02:27.967888+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:56.143404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:56.143404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:11.838368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":349,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967697+0000","last_change":"2024-03-26T05:02:19.051942+0000","last_active":"2024-03-26T05:02:27.967697+0000","last_peered":"2024-03-26T05:02:27.967697+0000","last_clean":"2024-03-26T05:02:27.967697+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:02:27.967697+0000","last_undegraded":"2024-03-26T05:02:27.967697+0000","last_fullsized":"2024-03-26T05:02:27.967697+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:02:19.051810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:02:19.051810+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:27.395025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358716+0000","last_change":"2024-03-26T05:01:25.941429+0000","last_active":"2024-03-26T05:02:27.358716+0000","last_peered":"2024-03-26T05:02:27.358716+0000","last_clean":"2024-03-26T05:02:27.358716+0000","last_became_active":"2024-03-26T05:00:22.774298+0000","last_became_peered":"2024-03-26T05:00:22.774298+0000","last_unstale":"2024-03-26T05:02:27.358716+0000","last_undegraded":"2024-03-26T05:02:27.358716+0000","last_fullsized":"2024-03-26T05:02:27.358716+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:25.941344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:25.941344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:34.075498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'24","reported_seq":145,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.495082+0000","last_change":"2024-03-26T05:02:00.251677+0000","last_active":"2024-03-26T05:02:27.495082+0000","last_peered":"2024-03-26T05:02:27.495082+0000","last_clean":"2024-03-26T05:02:27.495082+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:02:27.495082+0000","last_undegraded":"2024-03-26T05:02:27.495082+0000","last_fullsized":"2024-03-26T05:02:27.495082+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'24","last_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:02:00.251579+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:12.295580+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.969052+0000","last_change":"2024-03-26T05:02:01.018716+0000","last_active":"2024-03-26T05:02:27.969052+0000","last_peered":"2024-03-26T05:02:27.969052+0000","last_clean":"2024-03-26T05:02:27.969052+0000","last_became_active":"2024-03-26T05:00:26.961724+0000","last_became_peered":"2024-03-26T05:00:26.961724+0000","last_unstale":"2024-03-26T05:02:27.969052+0000","last_undegraded":"2024-03-26T05:02:27.969052+0000","last_fullsized":"2024-03-26T05:02:27.969052+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:01.018637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:02:01.018637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:17.690879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.969172+0000","last_change":"2024-03-26T05:02:24.355241+0000","last_active":"2024-03-26T05:02:27.969172+0000","last_peered":"2024-03-26T05:02:27.969172+0000","last_clean":"2024-03-26T05:02:27.969172+0000","last_became_active":"2024-03-26T05:02:24.354784+0000","last_became_peered":"2024-03-26T05:02:24.354784+0000","last_unstale":"2024-03-26T05:02:27.969172+0000","last_undegraded":"2024-03-26T05:02:27.969172+0000","last_fullsized":"2024-03-26T05:02:27.969172+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:48.426837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653237+0000","last_change":"2024-03-26T05:01:38.822538+0000","last_active":"2024-03-26T05:02:27.653237+0000","last_peered":"2024-03-26T05:02:27.653237+0000","last_clean":"2024-03-26T05:02:27.653237+0000","last_became_active":"2024-03-26T05:00:28.983202+0000","last_became_peered":"2024-03-26T05:00:28.983202+0000","last_unstale":"2024-03-26T05:02:27.653237+0000","last_undegraded":"2024-03-26T05:02:27.653237+0000","last_fullsized":"2024-03-26T05:02:27.653237+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:38.822409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:38.822409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:41.940538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.966979+0000","last_change":"2024-03-26T05:01:25.629227+0000","last_active":"2024-03-26T05:02:27.966979+0000","last_peered":"2024-03-26T05:02:27.966979+0000","last_clean":"2024-03-26T05:02:27.966979+0000","last_became_active":"2024-03-26T05:00:22.773081+0000","last_became_peered":"2024-03-26T05:00:22.773081+0000","last_unstale":"2024-03-26T05:02:27.966979+0000","last_undegraded":"2024-03-26T05:02:27.966979+0000","last_fullsized":"2024-03-26T05:02:27.966979+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:25.629101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:25.629101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:43.255919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":228,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:28.301252+0000","last_change":"2024-03-26T05:01:36.780629+0000","last_active":"2024-03-26T05:02:28.301252+0000","last_peered":"2024-03-26T05:02:28.301252+0000","last_clean":"2024-03-26T05:02:28.301252+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:02:28.301252+0000","last_undegraded":"2024-03-26T05:02:28.301252+0000","last_fullsized":"2024-03-26T05:02:28.301252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:01:36.780518+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.077309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":121,"num_read_kb":116,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":129,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967222+0000","last_change":"2024-03-26T05:02:17.009773+0000","last_active":"2024-03-26T05:02:27.967222+0000","last_peered":"2024-03-26T05:02:27.967222+0000","last_clean":"2024-03-26T05:02:27.967222+0000","last_became_active":"2024-03-26T05:00:49.642276+0000","last_became_peered":"2024-03-26T05:00:49.642276+0000","last_unstale":"2024-03-26T05:02:27.967222+0000","last_undegraded":"2024-03-26T05:02:27.967222+0000","last_fullsized":"2024-03-26T05:02:27.967222+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:02:17.009685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:17.009685+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:43.580593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968455+0000","last_change":"2024-03-26T05:01:41.973895+0000","last_active":"2024-03-26T05:02:27.968455+0000","last_peered":"2024-03-26T05:02:27.968455+0000","last_clean":"2024-03-26T05:02:27.968455+0000","last_became_active":"2024-03-26T05:00:26.956677+0000","last_became_peered":"2024-03-26T05:00:26.956677+0000","last_unstale":"2024-03-26T05:02:27.968455+0000","last_undegraded":"2024-03-26T05:02:27.968455+0000","last_fullsized":"2024-03-26T05:02:27.968455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:41.973805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.973805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:54.658519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967191+0000","last_change":"2024-03-26T05:02:24.354486+0000","last_active":"2024-03-26T05:02:27.967191+0000","last_peered":"2024-03-26T05:02:27.967191+0000","last_clean":"2024-03-26T05:02:27.967191+0000","last_became_active":"2024-03-26T05:02:24.353979+0000","last_became_peered":"2024-03-26T05:02:24.353979+0000","last_unstale":"2024-03-26T05:02:27.967191+0000","last_undegraded":"2024-03-26T05:02:27.967191+0000","last_fullsized":"2024-03-26T05:02:27.967191+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:41.951495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.652987+0000","last_change":"2024-03-26T05:01:57.803229+0000","last_active":"2024-03-26T05:02:27.652987+0000","last_peered":"2024-03-26T05:02:27.652987+0000","last_clean":"2024-03-26T05:02:27.652987+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:02:27.652987+0000","last_undegraded":"2024-03-26T05:02:27.652987+0000","last_fullsized":"2024-03-26T05:02:27.652987+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:57.803148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:57.803148+0000","last_clean_scrub_stamp":"2024-03-26T05:01:57.803148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:11.528300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653395+0000","last_change":"2024-03-26T05:01:35.745231+0000","last_active":"2024-03-26T05:02:27.653395+0000","last_peered":"2024-03-26T05:02:27.653395+0000","last_clean":"2024-03-26T05:02:27.653395+0000","last_became_active":"2024-03-26T05:00:23.346665+0000","last_became_peered":"2024-03-26T05:00:23.346665+0000","last_unstale":"2024-03-26T05:02:27.653395+0000","last_undegraded":"2024-03-26T05:02:27.653395+0000","last_fullsized":"2024-03-26T05:02:27.653395+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:01:35.745100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.314539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'25","reported_seq":140,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967172+0000","last_change":"2024-03-26T05:01:44.739144+0000","last_active":"2024-03-26T05:02:27.967172+0000","last_peered":"2024-03-26T05:02:27.967172+0000","last_clean":"2024-03-26T05:02:27.967172+0000","last_became_active":"2024-03-26T05:00:25.648253+0000","last_became_peered":"2024-03-26T05:00:25.648253+0000","last_unstale":"2024-03-26T05:02:27.967172+0000","last_undegraded":"2024-03-26T05:02:27.967172+0000","last_fullsized":"2024-03-26T05:02:27.967172+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:01:44.739046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.739046+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:06.858172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.494729+0000","last_change":"2024-03-26T05:01:59.261306+0000","last_active":"2024-03-26T05:02:27.494729+0000","last_peered":"2024-03-26T05:02:27.494729+0000","last_clean":"2024-03-26T05:02:27.494729+0000","last_became_active":"2024-03-26T05:00:26.958689+0000","last_became_peered":"2024-03-26T05:00:26.958689+0000","last_unstale":"2024-03-26T05:02:27.494729+0000","last_undegraded":"2024-03-26T05:02:27.494729+0000","last_fullsized":"2024-03-26T05:02:27.494729+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:01:59.261180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:28.535218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358057+0000","last_change":"2024-03-26T05:02:24.353110+0000","last_active":"2024-03-26T05:02:27.358057+0000","last_peered":"2024-03-26T05:02:27.358057+0000","last_clean":"2024-03-26T05:02:27.358057+0000","last_became_active":"2024-03-26T05:02:24.352730+0000","last_became_peered":"2024-03-26T05:02:24.352730+0000","last_unstale":"2024-03-26T05:02:27.358057+0000","last_undegraded":"2024-03-26T05:02:27.358057+0000","last_fullsized":"2024-03-26T05:02:27.358057+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:29.578714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.966661+0000","last_change":"2024-03-26T05:01:46.735926+0000","last_active":"2024-03-26T05:02:27.966661+0000","last_peered":"2024-03-26T05:02:27.966661+0000","last_clean":"2024-03-26T05:02:27.966661+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:02:27.966661+0000","last_undegraded":"2024-03-26T05:02:27.966661+0000","last_fullsized":"2024-03-26T05:02:27.966661+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:46.735848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:46.735848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:09.041185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968145+0000","last_change":"2024-03-26T05:01:54.084913+0000","last_active":"2024-03-26T05:02:27.968145+0000","last_peered":"2024-03-26T05:02:27.968145+0000","last_clean":"2024-03-26T05:02:27.968145+0000","last_became_active":"2024-03-26T05:00:23.346876+0000","last_became_peered":"2024-03-26T05:00:23.346876+0000","last_unstale":"2024-03-26T05:02:27.968145+0000","last_undegraded":"2024-03-26T05:02:27.968145+0000","last_fullsized":"2024-03-26T05:02:27.968145+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:54.084818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:54.084818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:10.773277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'16","reported_seq":80,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.726887+0000","last_change":"2024-03-26T05:01:42.753638+0000","last_active":"2024-03-26T05:02:27.726887+0000","last_peered":"2024-03-26T05:02:27.726887+0000","last_clean":"2024-03-26T05:02:27.726887+0000","last_became_active":"2024-03-26T05:00:49.641444+0000","last_became_peered":"2024-03-26T05:00:49.641444+0000","last_unstale":"2024-03-26T05:02:27.726887+0000","last_undegraded":"2024-03-26T05:02:27.726887+0000","last_fullsized":"2024-03-26T05:02:27.726887+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:01:42.753523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:42.753523+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.891497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":122,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967432+0000","last_change":"2024-03-26T05:02:00.856271+0000","last_active":"2024-03-26T05:02:27.967432+0000","last_peered":"2024-03-26T05:02:27.967432+0000","last_clean":"2024-03-26T05:02:27.967432+0000","last_became_active":"2024-03-26T05:00:26.961907+0000","last_became_peered":"2024-03-26T05:00:26.961907+0000","last_unstale":"2024-03-26T05:02:27.967432+0000","last_undegraded":"2024-03-26T05:02:27.967432+0000","last_fullsized":"2024-03-26T05:02:27.967432+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T05:02:00.856193+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:02:00.856193+0000","last_clean_scrub_stamp":"2024-03-26T05:02:00.856193+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:16.336148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358019+0000","last_change":"2024-03-26T05:02:24.354811+0000","last_active":"2024-03-26T05:02:27.358019+0000","last_peered":"2024-03-26T05:02:27.358019+0000","last_clean":"2024-03-26T05:02:27.358019+0000","last_became_active":"2024-03-26T05:02:24.354286+0000","last_became_peered":"2024-03-26T05:02:24.354286+0000","last_unstale":"2024-03-26T05:02:27.358019+0000","last_undegraded":"2024-03-26T05:02:27.358019+0000","last_fullsized":"2024-03-26T05:02:27.358019+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:25.547697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968656+0000","last_change":"2024-03-26T05:02:12.058651+0000","last_active":"2024-03-26T05:02:27.968656+0000","last_peered":"2024-03-26T05:02:27.968656+0000","last_clean":"2024-03-26T05:02:27.968656+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:02:27.968656+0000","last_undegraded":"2024-03-26T05:02:27.968656+0000","last_fullsized":"2024-03-26T05:02:27.968656+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:12.058549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:12.058549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:32.520937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967350+0000","last_change":"2024-03-26T05:01:29.581397+0000","last_active":"2024-03-26T05:02:27.967350+0000","last_peered":"2024-03-26T05:02:27.967350+0000","last_clean":"2024-03-26T05:02:27.967350+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:02:27.967350+0000","last_undegraded":"2024-03-26T05:02:27.967350+0000","last_fullsized":"2024-03-26T05:02:27.967350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:01:29.581309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:29.581309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.525431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'20","reported_seq":124,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358435+0000","last_change":"2024-03-26T05:02:16.549820+0000","last_active":"2024-03-26T05:02:27.358435+0000","last_peered":"2024-03-26T05:02:27.358435+0000","last_clean":"2024-03-26T05:02:27.358435+0000","last_became_active":"2024-03-26T05:00:24.936315+0000","last_became_peered":"2024-03-26T05:00:24.936315+0000","last_unstale":"2024-03-26T05:02:27.358435+0000","last_undegraded":"2024-03-26T05:02:27.358435+0000","last_fullsized":"2024-03-26T05:02:27.358435+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:02:16.549700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:16.549700+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:23.360680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":120,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.357974+0000","last_change":"2024-03-26T05:02:19.057027+0000","last_active":"2024-03-26T05:02:27.357974+0000","last_peered":"2024-03-26T05:02:27.357974+0000","last_clean":"2024-03-26T05:02:27.357974+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:02:27.357974+0000","last_undegraded":"2024-03-26T05:02:27.357974+0000","last_fullsized":"2024-03-26T05:02:27.357974+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:02:19.056943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:02:19.056943+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:38.747940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358927+0000","last_change":"2024-03-26T05:02:24.353027+0000","last_active":"2024-03-26T05:02:27.358927+0000","last_peered":"2024-03-26T05:02:27.358927+0000","last_clean":"2024-03-26T05:02:27.358927+0000","last_became_active":"2024-03-26T05:02:24.352725+0000","last_became_peered":"2024-03-26T05:02:24.352725+0000","last_unstale":"2024-03-26T05:02:27.358927+0000","last_undegraded":"2024-03-26T05:02:27.358927+0000","last_fullsized":"2024-03-26T05:02:27.358927+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:38.519041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358249+0000","last_change":"2024-03-26T05:01:48.472310+0000","last_active":"2024-03-26T05:02:27.358249+0000","last_peered":"2024-03-26T05:02:27.358249+0000","last_clean":"2024-03-26T05:02:27.358249+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:02:27.358249+0000","last_undegraded":"2024-03-26T05:02:27.358249+0000","last_fullsized":"2024-03-26T05:02:27.358249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:48.472188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:48.472188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:03.679495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358460+0000","last_change":"2024-03-26T05:02:12.036166+0000","last_active":"2024-03-26T05:02:27.358460+0000","last_peered":"2024-03-26T05:02:27.358460+0000","last_clean":"2024-03-26T05:02:27.358460+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:02:27.358460+0000","last_undegraded":"2024-03-26T05:02:27.358460+0000","last_fullsized":"2024-03-26T05:02:27.358460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:12.036082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:12.036082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:39.441456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"96'40","reported_seq":157,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.357868+0000","last_change":"2024-03-26T05:01:42.152769+0000","last_active":"2024-03-26T05:02:27.357868+0000","last_peered":"2024-03-26T05:02:27.357868+0000","last_clean":"2024-03-26T05:02:27.357868+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:02:27.357868+0000","last_undegraded":"2024-03-26T05:02:27.357868+0000","last_fullsized":"2024-03-26T05:02:27.357868+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:01:42.152630+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:01:42.152630+0000","last_clean_scrub_stamp":"2024-03-26T05:01:42.152630+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:48.656601+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":79,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358213+0000","last_change":"2024-03-26T05:00:26.959009+0000","last_active":"2024-03-26T05:02:27.358213+0000","last_peered":"2024-03-26T05:02:27.358213+0000","last_clean":"2024-03-26T05:02:27.358213+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:02:27.358213+0000","last_undegraded":"2024-03-26T05:02:27.358213+0000","last_fullsized":"2024-03-26T05:02:27.358213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:00:25.933152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:29.538873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.494675+0000","last_change":"2024-03-26T05:02:24.352199+0000","last_active":"2024-03-26T05:02:27.494675+0000","last_peered":"2024-03-26T05:02:27.494675+0000","last_clean":"2024-03-26T05:02:27.494675+0000","last_became_active":"2024-03-26T05:02:24.351780+0000","last_became_peered":"2024-03-26T05:02:24.351780+0000","last_unstale":"2024-03-26T05:02:27.494675+0000","last_undegraded":"2024-03-26T05:02:27.494675+0000","last_fullsized":"2024-03-26T05:02:27.494675+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:51.693155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967711+0000","last_change":"2024-03-26T05:02:09.988265+0000","last_active":"2024-03-26T05:02:27.967711+0000","last_peered":"2024-03-26T05:02:27.967711+0000","last_clean":"2024-03-26T05:02:27.967711+0000","last_became_active":"2024-03-26T05:00:28.982825+0000","last_became_peered":"2024-03-26T05:00:28.982825+0000","last_unstale":"2024-03-26T05:02:27.967711+0000","last_undegraded":"2024-03-26T05:02:27.967711+0000","last_fullsized":"2024-03-26T05:02:27.967711+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:09.988184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:09.988184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:13.588269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":121,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.359267+0000","last_change":"2024-03-26T05:01:41.485196+0000","last_active":"2024-03-26T05:02:27.359267+0000","last_peered":"2024-03-26T05:02:27.359267+0000","last_clean":"2024-03-26T05:02:27.359267+0000","last_became_active":"2024-03-26T05:00:26.956596+0000","last_became_peered":"2024-03-26T05:00:26.956596+0000","last_unstale":"2024-03-26T05:02:27.359267+0000","last_undegraded":"2024-03-26T05:02:27.359267+0000","last_fullsized":"2024-03-26T05:02:27.359267+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:41.485086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:41.485086+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:59.948091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.357723+0000","last_change":"2024-03-26T05:02:07.066594+0000","last_active":"2024-03-26T05:02:27.357723+0000","last_peered":"2024-03-26T05:02:27.357723+0000","last_clean":"2024-03-26T05:02:27.357723+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:02:27.357723+0000","last_undegraded":"2024-03-26T05:02:27.357723+0000","last_fullsized":"2024-03-26T05:02:27.357723+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:07.066520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:07.066520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:13.754281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":115,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653730+0000","last_change":"2024-03-26T05:01:28.754168+0000","last_active":"2024-03-26T05:02:27.653730+0000","last_peered":"2024-03-26T05:02:27.653730+0000","last_clean":"2024-03-26T05:02:27.653730+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:02:27.653730+0000","last_undegraded":"2024-03-26T05:02:27.653730+0000","last_fullsized":"2024-03-26T05:02:27.653730+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:01:28.754068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:28.754068+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:40.695752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.494957+0000","last_change":"2024-03-26T05:02:24.352343+0000","last_active":"2024-03-26T05:02:27.494957+0000","last_peered":"2024-03-26T05:02:27.494957+0000","last_clean":"2024-03-26T05:02:27.494957+0000","last_became_active":"2024-03-26T05:02:24.352008+0000","last_became_peered":"2024-03-26T05:02:24.352008+0000","last_unstale":"2024-03-26T05:02:27.494957+0000","last_undegraded":"2024-03-26T05:02:27.494957+0000","last_fullsized":"2024-03-26T05:02:27.494957+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:36.401118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.357932+0000","last_change":"2024-03-26T05:02:14.036454+0000","last_active":"2024-03-26T05:02:27.357932+0000","last_peered":"2024-03-26T05:02:27.357932+0000","last_clean":"2024-03-26T05:02:27.357932+0000","last_became_active":"2024-03-26T05:00:29.461236+0000","last_became_peered":"2024-03-26T05:00:29.461236+0000","last_unstale":"2024-03-26T05:02:27.357932+0000","last_undegraded":"2024-03-26T05:02:27.357932+0000","last_fullsized":"2024-03-26T05:02:27.357932+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:14.036322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:14.036322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:38.455926+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.968404+0000","last_change":"2024-03-26T05:01:45.085244+0000","last_active":"2024-03-26T05:02:27.968404+0000","last_peered":"2024-03-26T05:02:27.968404+0000","last_clean":"2024-03-26T05:02:27.968404+0000","last_became_active":"2024-03-26T05:00:26.955805+0000","last_became_peered":"2024-03-26T05:00:26.955805+0000","last_unstale":"2024-03-26T05:02:27.968404+0000","last_undegraded":"2024-03-26T05:02:27.968404+0000","last_fullsized":"2024-03-26T05:02:27.968404+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:45.085111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:45.085111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:55.981772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358296+0000","last_change":"2024-03-26T05:01:26.442833+0000","last_active":"2024-03-26T05:02:27.358296+0000","last_peered":"2024-03-26T05:02:27.358296+0000","last_clean":"2024-03-26T05:02:27.358296+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:02:27.358296+0000","last_undegraded":"2024-03-26T05:02:27.358296+0000","last_fullsized":"2024-03-26T05:02:27.358296+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:26.442722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:26.442722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:39.204290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'17","reported_seq":122,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.359093+0000","last_change":"2024-03-26T05:01:54.489384+0000","last_active":"2024-03-26T05:02:27.359093+0000","last_peered":"2024-03-26T05:02:27.359093+0000","last_clean":"2024-03-26T05:02:27.359093+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:02:27.359093+0000","last_undegraded":"2024-03-26T05:02:27.359093+0000","last_fullsized":"2024-03-26T05:02:27.359093+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:01:54.489297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:54.489297+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:57.631248+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.727035+0000","last_change":"2024-03-26T05:02:24.358112+0000","last_active":"2024-03-26T05:02:27.727035+0000","last_peered":"2024-03-26T05:02:27.727035+0000","last_clean":"2024-03-26T05:02:27.727035+0000","last_became_active":"2024-03-26T05:02:24.357742+0000","last_became_peered":"2024-03-26T05:02:24.357742+0000","last_unstale":"2024-03-26T05:02:27.727035+0000","last_undegraded":"2024-03-26T05:02:27.727035+0000","last_fullsized":"2024-03-26T05:02:27.727035+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:35.509411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.727154+0000","last_change":"2024-03-26T05:01:53.801119+0000","last_active":"2024-03-26T05:02:27.727154+0000","last_peered":"2024-03-26T05:02:27.727154+0000","last_clean":"2024-03-26T05:02:27.727154+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:02:27.727154+0000","last_undegraded":"2024-03-26T05:02:27.727154+0000","last_fullsized":"2024-03-26T05:02:27.727154+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:53.801041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:01:53.801041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:03.065236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.653562+0000","last_change":"2024-03-26T05:01:37.774253+0000","last_active":"2024-03-26T05:02:27.653562+0000","last_peered":"2024-03-26T05:02:27.653562+0000","last_clean":"2024-03-26T05:02:27.653562+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:02:27.653562+0000","last_undegraded":"2024-03-26T05:02:27.653562+0000","last_fullsized":"2024-03-26T05:02:27.653562+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:37.774158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:37.774158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:01.227552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":93,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.494481+0000","last_change":"2024-03-26T05:01:28.386278+0000","last_active":"2024-03-26T05:02:27.494481+0000","last_peered":"2024-03-26T05:02:27.494481+0000","last_clean":"2024-03-26T05:02:27.494481+0000","last_became_active":"2024-03-26T05:00:22.768610+0000","last_became_peered":"2024-03-26T05:00:22.768610+0000","last_unstale":"2024-03-26T05:02:27.494481+0000","last_undegraded":"2024-03-26T05:02:27.494481+0000","last_fullsized":"2024-03-26T05:02:27.494481+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_clean_scrub_stamp":"2024-03-26T05:01:28.386172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:58.056496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":111,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.359258+0000","last_change":"2024-03-26T05:00:24.942796+0000","last_active":"2024-03-26T05:02:27.359258+0000","last_peered":"2024-03-26T05:02:27.359258+0000","last_clean":"2024-03-26T05:02:27.359258+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:02:27.359258+0000","last_undegraded":"2024-03-26T05:02:27.359258+0000","last_fullsized":"2024-03-26T05:02:27.359258+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:00:23.918327+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:29.538790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.967345+0000","last_change":"2024-03-26T05:02:24.352484+0000","last_active":"2024-03-26T05:02:27.967345+0000","last_peered":"2024-03-26T05:02:27.967345+0000","last_clean":"2024-03-26T05:02:27.967345+0000","last_became_active":"2024-03-26T05:02:24.352196+0000","last_became_peered":"2024-03-26T05:02:24.352196+0000","last_unstale":"2024-03-26T05:02:27.967345+0000","last_undegraded":"2024-03-26T05:02:27.967345+0000","last_fullsized":"2024-03-26T05:02:27.967345+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:48.052369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.359307+0000","last_change":"2024-03-26T05:02:02.591734+0000","last_active":"2024-03-26T05:02:27.359307+0000","last_peered":"2024-03-26T05:02:27.359307+0000","last_clean":"2024-03-26T05:02:27.359307+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:02:27.359307+0000","last_undegraded":"2024-03-26T05:02:27.359307+0000","last_fullsized":"2024-03-26T05:02:27.359307+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:02.591640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:02.591640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:23.287671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":120,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.726884+0000","last_change":"2024-03-26T05:01:43.782577+0000","last_active":"2024-03-26T05:02:27.726884+0000","last_peered":"2024-03-26T05:02:27.726884+0000","last_clean":"2024-03-26T05:02:27.726884+0000","last_became_active":"2024-03-26T05:00:26.957142+0000","last_became_peered":"2024-03-26T05:00:26.957142+0000","last_unstale":"2024-03-26T05:02:27.726884+0000","last_undegraded":"2024-03-26T05:02:27.726884+0000","last_fullsized":"2024-03-26T05:02:27.726884+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:01:43.782490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:01:43.782490+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.086882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358097+0000","last_change":"2024-03-26T05:01:39.142373+0000","last_active":"2024-03-26T05:02:27.358097+0000","last_peered":"2024-03-26T05:02:27.358097+0000","last_clean":"2024-03-26T05:02:27.358097+0000","last_became_active":"2024-03-26T05:00:22.778719+0000","last_became_peered":"2024-03-26T05:00:22.778719+0000","last_unstale":"2024-03-26T05:02:27.358097+0000","last_undegraded":"2024-03-26T05:02:27.358097+0000","last_fullsized":"2024-03-26T05:02:27.358097+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:01:39.142249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:01:39.142249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:46.076132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":106,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T05:02:27.358790+0000","last_change":"2024-03-26T05:01:44.403960+0000","last_active":"2024-03-26T05:02:27.358790+0000","last_peered":"2024-03-26T05:02:27.358790+0000","last_clean":"2024-03-26T05:02:27.358790+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:02:27.358790+0000","last_undegraded":"2024-03-26T05:02:27.358790+0000","last_fullsized":"2024-03-26T05:02:27.358790+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:01:44.403798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:01:44.403798+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:03:04.259798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":116,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":617,"ondisk_log_size":617,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005086,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28356,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716796,"statfs":{"total":95995035648,"available":95965999104,"internally_reserved":0,"allocated":1368064,"data_stored":993357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332131,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32500,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712652,"statfs":{"total":95995035648,"available":95961755648,"internally_reserved":0,"allocated":1351680,"data_stored":991841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":811008,"data_stored":402737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738434,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1380352,"data_stored":993510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411384,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":749568,"data_stored":401287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117042,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713204,"statfs":{"total":95995035648,"available":95962320896,"internally_reserved":0,"allocated":786432,"data_stored":401953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920814,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31948,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713204,"statfs":{"total":95995035648,"available":95962320896,"internally_reserved":0,"allocated":782336,"data_stored":402542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659176,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":761856,"data_stored":401799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:02:35.810 INFO:tasks.thrashosds.thrasher:Setting 3.16 to [1, 4, 5, 6, 3, 7] 2024-03-26T05:02:35.810 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.16', '1', '4', '5', '6', '3', '7'] 2024-03-26T05:02:35.810 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pg-upmap-items 3.16 1 4 5 6 3 7 2024-03-26T05:02:35.815 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:35.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:35.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:35.917 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:35.941 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:35.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:36.043 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:36.070 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:36.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:36.172 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:36.201 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:36.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:36.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:36.304 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:36.333 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:36.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:36.373 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-26T05:02:36.434 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:36.462 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:36.462 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:36 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/913783910' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:36.463 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:36 smithi080 ceph-mon[40404]: pgmap v69: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:36.463 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:36 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/913783910' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:36.464 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:36 smithi080 ceph-mon[29691]: pgmap v69: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:36.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:36.565 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:36.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:36.595 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:36.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:36.696 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:36.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:36.722 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:36.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:36.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:36 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/913783910' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:36.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:36 smithi132 ceph-mon[33500]: pgmap v69: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:36.823 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:36.851 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:36.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:36.953 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:36.983 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:36.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:37.085 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:37.113 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:37.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:37.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:37.215 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:37.239 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:37.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:37.321 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:02:37.341 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:37.345 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:02:37.368 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:37.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:37.373 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:02:37.381 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:37.382 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:37.400 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:02:37.422 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:02:37.442 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:02:37.463 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:02:37.470 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:37.483 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:02:37.484 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:02:37.484 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:02:37.484 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:02:37.484 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:02:37.484 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:02:37.484 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:02:37.484 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:02:37.484 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:02:37.496 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:37.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:37.597 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:37.599 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:37 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/913783910' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:37.599 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:37 smithi080 ceph-mon[40404]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T05:02:37.599 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:37 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3288890684' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:02:37.599 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:37 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/913783910' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:37.599 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:37 smithi080 ceph-mon[29691]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T05:02:37.599 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:37 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3288890684' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:02:37.623 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:37.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:37.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:37.750 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:37.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:37.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:37 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/913783910' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:37.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:37 smithi132 ceph-mon[33500]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T05:02:37.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:37 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3288890684' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:02:37.852 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:37.876 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:37.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:37.978 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:38.002 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:38.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:38.103 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:38.122 DEBUG:teuthology.orchestra.run.smithi080:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-26T05:02:38.125 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T05:02:38.125 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T05:02:38.131 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:38.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:38.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:38.193 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:hints = 1 2024-03-26T05:02:38.193 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T05:02:38.193 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Object prefix: benchmark_data_smithi080_80848 2024-03-26T05:02:38.194 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:02:38.194 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 0 0 0 0 0 0 - 0 2024-03-26T05:02:38.233 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:38.277 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:38.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:38.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:38.379 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:38.421 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:38.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:38.522 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:38.555 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:38.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:38.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:38.657 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:38.703 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:38.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:38.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:38 smithi132 ceph-mon[33500]: pgmap v71: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:38.804 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:38 smithi080 ceph-mon[40404]: pgmap v71: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:38.804 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:38 smithi080 ceph-mon[29691]: pgmap v71: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:38.804 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:38.834 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:38.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:38.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:38.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:38.968 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:38.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:38.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:39.070 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:39.102 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:39.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:39.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:39.194 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 1 16 3202 3186 199.1 199.125 0.0066706 0.00500294 2024-03-26T05:02:39.204 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:39.261 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:39.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:39.362 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:39.408 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:39.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:39.510 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:39.514 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:39 smithi080 ceph-mon[40404]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:02:39.514 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:39 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:39.514 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:39 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:39.515 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:39 smithi080 ceph-mon[29691]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:02:39.515 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:39 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:39.515 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:39 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:39.540 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:39.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:39.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:39.642 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:39.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:39.667 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:39.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:39.768 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:39 smithi132 ceph-mon[33500]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:02:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:39 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:39.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:39 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:39.809 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:39.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:39.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:39.911 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:39.955 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:39.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:40.057 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:40.084 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:02:40.089 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:40.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:40.191 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:40.194 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 2 16 5990 5974 186.668 174.25 0.0072615 0.00534871 2024-03-26T05:02:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:40.222 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:40.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:40.323 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:40.353 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:40.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:40.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:40.455 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:40.486 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:40.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:40.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:40.588 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:40.611 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:40.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:40.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:40.713 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:40.757 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:40.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:40.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:40 smithi132 ceph-mon[33500]: pgmap v72: 148 pgs: 148 active+clean; 287 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 39 MiB/s wr, 624 op/s 2024-03-26T05:02:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:40 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:02:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:40 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:02:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:40 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:40 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:02:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:40 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:02:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:40 smithi132 ceph-mon[33500]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T05:02:40.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:40 smithi132 ceph-mon[33500]: Reconfiguring daemon alertmanager.a on smithi080 2024-03-26T05:02:40.817 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[29691]: pgmap v72: 148 pgs: 148 active+clean; 287 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 39 MiB/s wr, 624 op/s 2024-03-26T05:02:40.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:02:40.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:02:40.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:40.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:02:40.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:02:40.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[29691]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T05:02:40.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[29691]: Reconfiguring daemon alertmanager.a on smithi080 2024-03-26T05:02:40.818 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[40404]: pgmap v72: 148 pgs: 148 active+clean; 287 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 39 MiB/s wr, 624 op/s 2024-03-26T05:02:40.818 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:02:40.818 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:02:40.818 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:40.819 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:02:40.819 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:02:40.819 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[40404]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T05:02:40.819 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:40 smithi080 ceph-mon[40404]: Reconfiguring daemon alertmanager.a on smithi080 2024-03-26T05:02:40.858 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:40.892 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:40.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:40.994 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:41.027 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:41.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:41.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:41.129 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:41.161 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:41.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:41.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:41.194 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 3 16 8627 8611 179.377 164.812 0.00444069 0.00556595 2024-03-26T05:02:41.260 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:02:41.263 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:41.292 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:41.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:41.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:41.394 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:41.422 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:41.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:41.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:41.524 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:41.569 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:41.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:41.671 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:41.713 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:41.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:41.815 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:41.860 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:41.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:41.962 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:41.993 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:41.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:42.038 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:41 smithi080 ceph-mon[29691]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T05:02:42.039 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:41 smithi080 ceph-mon[40404]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T05:02:42.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:41 smithi132 ceph-mon[33500]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T05:02:42.096 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:42.136 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:42.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:42.194 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 4 16 11303 11287 176.342 167.25 0.0095539 0.00566273 2024-03-26T05:02:42.238 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:42.273 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:42.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:42.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:42.375 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:42.409 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:42.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:42.458 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:42.512 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:42.558 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:42.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:42.659 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:42.704 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:42.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:42.806 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:42.857 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:42.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:42.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:42.959 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:43.000 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:43.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:43.101 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:43.145 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:43.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:43.194 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 5 16 13883 13867 173.312 161.25 0.00459478 0.00576205 2024-03-26T05:02:43.247 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:43.286 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:43.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:43.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:43.388 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:43.417 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:43.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:43.519 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:43.550 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:43.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:43.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:43.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:43 smithi080 ceph-mon[29691]: pgmap v74: 148 pgs: 148 active+clean; 287 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 39 MiB/s wr, 624 op/s 2024-03-26T05:02:43.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:43 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:02:43.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:43 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:02:43.585 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:43 smithi080 ceph-mon[40404]: pgmap v74: 148 pgs: 148 active+clean; 287 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 39 MiB/s wr, 624 op/s 2024-03-26T05:02:43.585 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:43 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:02:43.585 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:43 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:02:43.652 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:43.696 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:43.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:43.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:43 smithi132 ceph-mon[33500]: pgmap v74: 148 pgs: 148 active+clean; 287 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 39 MiB/s wr, 624 op/s 2024-03-26T05:02:43.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:43 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:02:43.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:43 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:02:43.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:43.829 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:43.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:43.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:43.931 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:43.939 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:4483 7.4 7:21a099b1:::benchmark_data_smithi080_80848_object4482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:39.631251+0000", 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.9809503080000002, 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.044412921000000001, 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:02:43.940 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:02:43.941 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 4483 2024-03-26T05:02:43.941 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.941 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:43.941 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.941 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:43.941 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631251+0000", 2024-03-26T05:02:43.941 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.941 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.941 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.941 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:43.941 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631251+0000", 2024-03-26T05:02:43.941 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631252+0000", 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.446e-06 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631276+0000", 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3819999999999999e-05 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:43.942 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631288+0000", 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1963e-05 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631296+0000", 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0190000000000006e-06 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631702+0000", 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00040575799999999999 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:43.943 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631820+0000", 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011801700000000001 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,4", 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631947+0000", 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012726 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.672640+0000", 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.040693247000000002 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.944 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.672664+0000", 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4133000000000001e-05 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.675545+0000", 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0028805369999999999 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.675597+0000", 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2745999999999999e-05 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.945 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.675663+0000", 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5975000000000004e-05 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:4486 7.4 7:29c712f5:::benchmark_data_smithi080_80848_object4485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:39.631571+0000", 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.9806301400000002, 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.045092992999999998, 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:43.946 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:43.947 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:02:43.947 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:02:43.947 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:02:43.947 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 4486 2024-03-26T05:02:43.947 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.947 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:43.947 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.947 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:43.947 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631571+0000", 2024-03-26T05:02:43.947 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.947 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.947 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631571+0000", 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631571+0000", 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9999999999999996e-07 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631594+0000", 2024-03-26T05:02:43.948 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2050000000000001e-05 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631613+0000", 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.948e-05 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.631621+0000", 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0050000000000004e-06 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632402+0000", 2024-03-26T05:02:43.949 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00078082200000000003 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632527+0000", 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012502800000000001 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,4", 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632652+0000", 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012491800000000001 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:02:43.950 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.675691+0000", 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.043039543 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.676515+0000", 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00082387400000000005 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.676550+0000", 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4798999999999997e-05 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.951 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.676618+0000", 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7735000000000001e-05 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.676664+0000", 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5939000000000002e-05 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:4490 7.f 7:fad9e23f:::benchmark_data_smithi080_80848_object4489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T05:02:43.952 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:39.632471+0000", 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.9797297230000002, 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.044891817000000001, 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 4490 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632471+0000", 2024-03-26T05:02:43.953 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632471+0000", 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632472+0000", 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1170000000000001e-06 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.954 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632494+0000", 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2067000000000001e-05 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632506+0000", 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1793e-05 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632514+0000", 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9189999999999999e-06 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.955 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632572+0000", 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7667000000000002e-05 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632658+0000", 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6237999999999997e-05 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.672811+0000", 2024-03-26T05:02:43.956 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.040153230999999998 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.674910+0000", 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0020988619999999999 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.677174+0000", 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022638649999999999 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:02:43.957 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.677270+0000", 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5657999999999998e-05 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.677324+0000", 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4503999999999999e-05 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.677363+0000", 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8896000000000002e-05 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.958 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.24701.0:4488 7.7 e109/103 7:eca76b1b:::benchmark_data_smithi080_80848_object4487:head v 109'278, mlcod=109'274)", 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:39.632726+0000", 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.9794748809999998, 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.044671321, 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632726+0000", 2024-03-26T05:02:43.959 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632726+0000", 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632727+0000", 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.99e-07 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.672402+0000", 2024-03-26T05:02:43.960 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.039675561999999998 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.672413+0000", 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0424999999999999e-05 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.672419+0000", 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0159999999999997e-06 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.673440+0000", 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010218099999999999 2024-03-26T05:02:43.961 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.673464+0000", 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3306999999999999e-05 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.677366+0000", 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0039022649999999998 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.677397+0000", 2024-03-26T05:02:43.962 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.1337000000000002e-05 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop_reply(client.24701.0:4483 7.4 e109/103 ondisk, result = 0)", 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:39.632755+0000", 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.9794457329999999, 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.039897565000000003, 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:43.963 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632755+0000", 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632755+0000", 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632755+0000", 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.1100000000000002e-07 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.964 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.632758+0000", 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.0290000000000001e-06 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.672225+0000", 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.039466551000000002 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.672241+0000", 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5625999999999999e-05 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.965 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.966 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:43.966 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.672623+0000", 2024-03-26T05:02:43.966 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000382684 2024-03-26T05:02:43.966 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.966 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.966 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:43.966 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.672633+0000", 2024-03-26T05:02:43.966 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5699999999999999e-06 2024-03-26T05:02:43.966 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.966 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.966 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.672653+0000", 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9794000000000001e-05 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop_reply(client.24701.0:4483 7.4 e109/103 ondisk, result = 0)", 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:39.633279+0000", 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.9789217350000001, 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.042390642999999999, 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:43.967 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.633279+0000", 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.633279+0000", 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.633280+0000", 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5199999999999997e-07 2024-03-26T05:02:43.968 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.633285+0000", 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8810000000000004e-06 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.674826+0000", 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.041541869000000002 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.675434+0000", 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00060723299999999999 2024-03-26T05:02:43.969 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.970 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.970 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:43.970 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.675531+0000", 2024-03-26T05:02:43.970 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7796999999999999e-05 2024-03-26T05:02:43.970 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.970 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.970 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:43.970 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.675539+0000", 2024-03-26T05:02:43.970 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5920000000000003e-06 2024-03-26T05:02:43.970 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.970 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.970 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:39.675670+0000", 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013071900000000001 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:6948 7.4 7:27a106b2:::benchmark_data_smithi080_80848_object6947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:40.594171+0000", 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.0180300760000001, 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.045130007, 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:43.971 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 6948 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594171+0000", 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:43.972 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594171+0000", 2024-03-26T05:02:43.973 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.973 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.973 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.973 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:43.973 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594172+0000", 2024-03-26T05:02:43.973 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7409999999999999e-06 2024-03-26T05:02:43.973 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.973 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.973 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:43.973 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594209+0000", 2024-03-26T05:02:43.973 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6238000000000001e-05 2024-03-26T05:02:43.973 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594221+0000", 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1844e-05 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594228+0000", 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.841e-06 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594313+0000", 2024-03-26T05:02:43.974 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4394000000000005e-05 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594476+0000", 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016327799999999999 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,4", 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594646+0000", 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017014299999999999 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:02:43.975 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637511+0000", 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.042864395 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.638482+0000", 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00097124699999999995 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639204+0000", 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00072217500000000003 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.976 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639259+0000", 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5396999999999998e-05 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639301+0000", 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1313999999999998e-05 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.977 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:6950 7.6 7:6937b730:::benchmark_data_smithi080_80848_object6949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:40.594249+0000", 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.0179519689999998, 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.044942579000000003, 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 6950 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.978 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594249+0000", 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594249+0000", 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594249+0000", 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2399999999999998e-07 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.979 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594271+0000", 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1582000000000001e-05 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594280+0000", 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0140000000000004e-06 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594288+0000", 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4800000000000004e-06 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.980 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594339+0000", 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1059e-05 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594432+0000", 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.3608000000000002e-05 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.594518+0000", 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5469000000000004e-05 2024-03-26T05:02:43.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.982 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.982 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:02:43.982 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595062+0000", 2024-03-26T05:02:43.982 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00054383399999999998 2024-03-26T05:02:43.982 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.982 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.982 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:43.982 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637774+0000", 2024-03-26T05:02:43.982 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.042712342 2024-03-26T05:02:43.982 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.982 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.982 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.638537+0000", 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00076306099999999999 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639142+0000", 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00060489500000000004 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639191+0000", 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9611000000000003e-05 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:43.983 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.24701.0:6953 7.7 e109/103 7:e073a04f:::benchmark_data_smithi080_80848_object6952:head v 109'432, mlcod=109'430)", 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:40.595127+0000", 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.0170735070000001, 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.043159836, 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595127+0000", 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595127+0000", 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595128+0000", 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8999999999999996e-07 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595228+0000", 2024-03-26T05:02:43.985 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9870000000000004e-05 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595238+0000", 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0579e-05 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595245+0000", 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9909999999999996e-06 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:43.986 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595312+0000", 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7027999999999996e-05 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595332+0000", 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9962999999999998e-05 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.638250+0000", 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.042917489000000003 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.638287+0000", 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7225999999999999e-05 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.24701.0:6954 7.5 e109/103 7:a74688ca:::benchmark_data_smithi080_80848_object6953:head v 109'436, mlcod=109'434)", 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:40.595378+0000", 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.0168224449999999, 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.043902025999999997, 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595378+0000", 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595378+0000", 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595379+0000", 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3799999999999996e-07 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595561+0000", 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000182017 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595580+0000", 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8879999999999999e-05 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595588+0000", 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7800000000000001e-06 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637656+0000", 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.042067689999999998 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637682+0000", 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6750999999999998e-05 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639232+0000", 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0015496209999999999 2024-03-26T05:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639280+0000", 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8449000000000003e-05 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop_reply(client.24701.0:6950 7.6 e109/103 ondisk, result = 0)", 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:40.595661+0000", 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.0165395159999999, 2024-03-26T05:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.042122078, 2024-03-26T05:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595661+0000", 2024-03-26T05:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595661+0000", 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595662+0000", 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5000000000000003e-07 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595665+0000", 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.0699999999999998e-06 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595674+0000", 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2350000000000005e-06 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595680+0000", 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2550000000000003e-06 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637758+0000", 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.042077333000000001 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637766+0000", 2024-03-26T05:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6419999999999996e-06 2024-03-26T05:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637783+0000", 2024-03-26T05:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6993000000000001e-05 2024-03-26T05:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:6955 7.4 7:27609bdc:::benchmark_data_smithi080_80848_object6954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:40.595732+0000", 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.0164684959999999, 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.044031597999999998, 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 6955 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595732+0000", 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595732+0000", 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595734+0000", 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.288e-06 2024-03-26T05:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.595778+0000", 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4570999999999997e-05 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637525+0000", 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.041747184 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637546+0000", 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0475999999999998e-05 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637616+0000", 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0549000000000005e-05 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637782+0000", 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000165799 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,4", 2024-03-26T05:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637975+0000", 2024-03-26T05:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019311600000000001 2024-03-26T05:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639521+0000", 2024-03-26T05:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0015455860000000001 2024-03-26T05:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639543+0000", 2024-03-26T05:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2552999999999999e-05 2024-03-26T05:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639639+0000", 2024-03-26T05:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5988000000000003e-05 2024-03-26T05:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639716+0000", 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6755000000000006e-05 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639764+0000", 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7732999999999999e-05 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop_reply(client.24701.0:6948 7.4 e109/103 ondisk, result = 0)", 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:40.596209+0000", 2024-03-26T05:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.015992174, 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.042279030000000002, 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.596209+0000", 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.596209+0000", 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.596209+0000", 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9100000000000003e-07 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.596214+0000", 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5050000000000001e-06 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637576+0000", 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.041362598 2024-03-26T05:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637587+0000", 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0222e-05 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.638474+0000", 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00088701600000000002 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.638479+0000", 2024-03-26T05:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8439999999999997e-06 2024-03-26T05:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.638488+0000", 2024-03-26T05:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1540000000000008e-06 2024-03-26T05:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.24701.0:6957 7.5 e109/103 7:ac3c6002:::benchmark_data_smithi080_80848_object6956:head v 109'437, mlcod=109'434)", 2024-03-26T05:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:40.597075+0000", 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.0151257579999999, 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.042371285000000002, 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.597075+0000", 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:44.008 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.597075+0000", 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.597076+0000", 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8700000000000005e-07 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.597229+0000", 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015268200000000001 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:44.009 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637698+0000", 2024-03-26T05:02:44.010 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.040469931000000001 2024-03-26T05:02:44.010 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.010 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.010 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:44.010 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.637708+0000", 2024-03-26T05:02:44.010 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9520000000000006e-06 2024-03-26T05:02:44.010 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.010 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.010 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:44.010 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.638237+0000", 2024-03-26T05:02:44.010 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00052863900000000002 2024-03-26T05:02:44.010 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.638257+0000", 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9987e-05 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639405+0000", 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001148163 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:40.639446+0000", 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1143999999999998e-05 2024-03-26T05:02:44.011 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:12762 7.f 7:fd5051dd:::benchmark_data_smithi080_80848_object12761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:42.737431+0000", 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.87476947900000002, 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.037469356000000002, 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:02:44.012 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 12762 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.737431+0000", 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.737431+0000", 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.737432+0000", 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1349999999999999e-06 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.737455+0000", 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2121999999999999e-05 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.737468+0000", 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3118e-05 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.737475+0000", 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4919999999999996e-06 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.737949+0000", 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00047368099999999999 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.738075+0000", 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000126155 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.738211+0000", 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000135549 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.739155+0000", 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00094426899999999997 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.742799+0000", 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036441440000000002 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.774808+0000", 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.032009178999999999 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.774859+0000", 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0482000000000002e-05 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.774901+0000", 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2030000000000002e-05 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:12771 7.f 7:fd4b8887:::benchmark_data_smithi080_80848_object12770:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:42.740539+0000", 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.87166230300000003, 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.034489138000000003, 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 12771 2024-03-26T05:02:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.740539+0000", 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.740539+0000", 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.740539+0000", 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7899999999999999e-07 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.740554+0000", 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5389000000000001e-05 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.740559+0000", 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4859999999999999e-06 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.740562+0000", 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3510000000000002e-06 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.741800+0000", 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0012374300000000001 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.741921+0000", 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000121643 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.742044+0000", 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000123125 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.742914+0000", 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00086962499999999996 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.743565+0000", 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00065093300000000002 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.774945+0000", 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.031379872000000003 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.774993+0000", 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8467999999999997e-05 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.775028+0000", 2024-03-26T05:02:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4437000000000002e-05 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:12778 7.6 7:607ee5b7:::benchmark_data_smithi080_80848_object12777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:42.743252+0000", 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.86894919000000004, 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.031819712999999999, 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:02:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 12778 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.743252+0000", 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.743252+0000", 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.743253+0000", 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.654e-06 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.743280+0000", 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.652e-05 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.743297+0000", 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7384000000000002e-05 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.026 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.743305+0000", 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8820000000000001e-06 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.743400+0000", 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4821999999999994e-05 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.743539+0000", 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000139431 2024-03-26T05:02:44.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.743674+0000", 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000134275 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.744958+0000", 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0012845809999999999 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.745245+0000", 2024-03-26T05:02:44.028 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00028642799999999998 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.774976+0000", 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.029731038000000001 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.775036+0000", 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0189000000000003e-05 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:44.029 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.775071+0000", 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5509000000000002e-05 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:12801 7.6 7:698815da:::benchmark_data_smithi080_80848_object12800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:42.763103+0000", 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.84909790299999999, 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.031211104999999999, 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:44.030 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 12801 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.763103+0000", 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.763103+0000", 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.763104+0000", 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1900000000000001e-07 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.763121+0000", 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7135000000000001e-05 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.763130+0000", 2024-03-26T05:02:44.032 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8939999999999999e-06 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.763137+0000", 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.306e-06 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.763187+0000", 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9713999999999999e-05 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:44.033 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.763268+0000", 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1550000000000004e-05 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.763357+0000", 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9017000000000005e-05 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.763991+0000", 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00063351300000000004 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.034 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.764187+0000", 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000196381 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.794220+0000", 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.030032532000000001 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.794273+0000", 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3449999999999998e-05 2024-03-26T05:02:44.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.794314+0000", 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0794000000000003e-05 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:14434 7.6 7:61044d6d:::benchmark_data_smithi080_80848_object14433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:43.428248+0000", 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.18395304300000001, 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.020062420000000001, 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:44.036 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 14434 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.428248+0000", 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:44.037 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.428248+0000", 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.428249+0000", 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9699999999999995e-07 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.428272+0000", 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3666999999999999e-05 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.428282+0000", 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7450000000000007e-06 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.428289+0000", 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9870000000000002e-06 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.431099+0000", 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0028105249999999999 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.431264+0000", 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016441399999999999 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.431416+0000", 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015181600000000001 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.433657+0000", 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022409819999999999 2024-03-26T05:02:44.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.434754+0000", 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010976180000000001 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.444956+0000", 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010202065 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.448265+0000", 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0033083259999999999 2024-03-26T05:02:44.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.448310+0000", 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5478000000000001e-05 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:14442 7.6 7:6ec1d0a6:::benchmark_data_smithi080_80848_object14441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:43.428645+0000", 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.183555936, 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.019821350000000001, 2024-03-26T05:02:44.042 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 14442 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.428645+0000", 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.428645+0000", 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.428645+0000", 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1099999999999996e-07 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.428663+0000", 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7711e-05 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.428671+0000", 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5580000000000003e-06 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.428676+0000", 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3519999999999997e-06 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.432601+0000", 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0039251590000000001 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.432728+0000", 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012638600000000001 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.432944+0000", 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000216632 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.433965+0000", 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001020883 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.438455+0000", 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0044900169999999998 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.448375+0000", 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0099195110000000006 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.448427+0000", 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2497999999999998e-05 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:02:44.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:02:44.048 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:02:44.048 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:43.448466+0000", 2024-03-26T05:02:44.048 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9131999999999997e-05 2024-03-26T05:02:44.048 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.048 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:44.048 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.048 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:02:44.048 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:02:44.048 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:02:44.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:44.049 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:44.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:44.150 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:44.183 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:44.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:44.194 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 6 16 16354 16338 170.164 154.438 0.00804322 0.00586903 2024-03-26T05:02:44.285 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:44.308 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:44.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:44.342 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:02:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:02:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:02:44.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:44.451 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:44.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:44.553 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:44.583 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:44.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:44.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:44 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:02:44.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:44 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:02:44.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:44 smithi080 ceph-mon[29691]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T05:02:44.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:44 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:02:44.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:44 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:02:44.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:44 smithi080 ceph-mon[40404]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T05:02:44.684 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:44.723 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:44.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:44.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:44 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:02:44.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:44 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:02:44.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:44 smithi132 ceph-mon[33500]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T05:02:44.826 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:44.864 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:44.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:44.966 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:44.998 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:44.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:45.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:02:44 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:02:45.100 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:45.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:45.129 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:45.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:45.195 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 7 16 18428 18412 164.371 129.625 0.00181919 0.00600082 2024-03-26T05:02:45.230 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:45.254 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:45.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:45.355 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:45.396 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:45.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:45.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:45.498 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:45.531 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:45.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:45.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:45 smithi080 ceph-mon[29691]: pgmap v76: 148 pgs: 148 active+clean; 287 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 696 B/s rd, 39 MiB/s wr, 624 op/s 2024-03-26T05:02:45.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:45 smithi080 ceph-mon[40404]: pgmap v76: 148 pgs: 148 active+clean; 287 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 696 B/s rd, 39 MiB/s wr, 624 op/s 2024-03-26T05:02:45.633 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:45.660 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:45.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:45.761 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:45.791 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:45.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:45.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:45 smithi132 ceph-mon[33500]: pgmap v76: 148 pgs: 148 active+clean; 287 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 696 B/s rd, 39 MiB/s wr, 624 op/s 2024-03-26T05:02:45.894 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:45.918 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:45.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:46.019 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:46.060 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:46.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:46.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:46.161 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:46.194 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:46.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:46.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:46.195 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 8 16 20409 20393 159.3 123.812 0.00362606 0.00627026 2024-03-26T05:02:46.295 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:46.324 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:46.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:46.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:46.425 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:46.465 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:46.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:46.566 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:46.590 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:46.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:46.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:46 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:46.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:46 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:46.691 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:46.734 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:46.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:46.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:46.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:46 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:46.836 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:46.875 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:46.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:46.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:46.977 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:47.017 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:47.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:47.119 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:47.147 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:47.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:47.194 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 9 15 22228 22213 154.238 113.75 0.00823395 0.00647717 2024-03-26T05:02:47.249 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:47.286 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:47.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:47.388 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:47.412 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:47.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:47.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:47.514 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:47.554 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:47.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:47.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:47 smithi080 ceph-mon[40404]: pgmap v77: 148 pgs: 148 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-26T05:02:47.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:47 smithi080 ceph-mon[29691]: pgmap v77: 148 pgs: 148 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-26T05:02:47.656 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:47.693 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:47.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:47.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:47 smithi132 ceph-mon[33500]: pgmap v77: 148 pgs: 148 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-26T05:02:47.795 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:47.836 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:47.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:47.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:47.938 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:47.982 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:47.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:48.083 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:48.113 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:48.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:48.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:48.195 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 10 16 23981 23965 149.764 109.5 0.00637263 0.00667053 2024-03-26T05:02:48.215 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:48.244 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:48.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:48.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:48.346 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:48.377 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:48.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:48.479 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:48.507 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:48.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:48.609 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:48.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:48 smithi080 ceph-mon[40404]: pgmap v78: 148 pgs: 148 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-26T05:02:48.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:48 smithi080 ceph-mon[29691]: pgmap v78: 148 pgs: 148 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-26T05:02:48.640 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:48.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:48.742 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:48.771 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:48.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:48.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:48 smithi132 ceph-mon[33500]: pgmap v78: 148 pgs: 148 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-26T05:02:48.873 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:48.915 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:48.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:49.016 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:49.047 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:49.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:49.148 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:49.188 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:49.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:49.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:49.195 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 11 16 25956 25940 147.37 123.438 0.00432319 0.00677909 2024-03-26T05:02:49.290 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:49.319 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:49.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:49.420 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:49.472 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:49.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:49.574 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:49.617 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:49.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:49.718 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:49.752 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:49.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:49.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:49.854 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:49.897 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:49.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:49.953 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:49.953 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:49.953 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:49.953 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:49.954 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T05:02:50.000 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:50.024 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:50.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:50.125 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:50.157 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:50.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:50.195 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 12 16 27803 27787 144.707 115.438 0.00294803 0.00689126 2024-03-26T05:02:50.259 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:50.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:02:50.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:50.301 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:50.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:50.403 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:50.437 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:50.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:50.538 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:50.570 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:50.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:50.672 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:50.714 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:50.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:50.815 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:50.856 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:50.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:50.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:50.959 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:51.018 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:51.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:51.048 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:50 smithi080 ceph-mon[40404]: pgmap v79: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 156 MiB/s wr, 2.49k op/s 2024-03-26T05:02:51.049 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:50 smithi080 ceph-mon[29691]: pgmap v79: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 156 MiB/s wr, 2.49k op/s 2024-03-26T05:02:51.120 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:51.167 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:51.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:51.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:51.195 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 13 16 29563 29547 142.036 110 0.0142436 0.00703342 2024-03-26T05:02:51.269 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:50 smithi132 ceph-mon[33500]: pgmap v79: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 156 MiB/s wr, 2.49k op/s 2024-03-26T05:02:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:51.317 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:51.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:51.418 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:51.458 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:51.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:51.560 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:51.600 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:51.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:51.702 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:51.734 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:51.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:51.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:51.835 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:51.876 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:51.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:51.978 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:52.003 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:52.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:52.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:52.105 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:52.135 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:52.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:52.195 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 14 16 31322 31306 139.743 109.938 0.0125292 0.00715073 2024-03-26T05:02:52.238 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:52.264 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:52.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:52.366 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:52.395 INFO:teuthology.orchestra.run.smithi080.stderr:set 3.16 pg_upmap_items mapping to [1->4,5->6,3->7] 2024-03-26T05:02:52.398 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:02:52.400 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:52.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:52.416 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:52 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/648806224' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.16", "id": [1, 4, 5, 6, 3, 7]}]: dispatch 2024-03-26T05:02:52.416 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:52 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.16", "id": [1, 4, 5, 6, 3, 7]}]: dispatch 2024-03-26T05:02:52.416 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:52 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3355891236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:02:52.416 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:52 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:02:52.416 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:52 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/648806224' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.16", "id": [1, 4, 5, 6, 3, 7]}]: dispatch 2024-03-26T05:02:52.416 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:52 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.16", "id": [1, 4, 5, 6, 3, 7]}]: dispatch 2024-03-26T05:02:52.416 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:52 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3355891236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:02:52.416 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:52 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:02:52.502 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:52.533 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:52.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:52.635 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:52.677 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:52.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:52.779 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:52.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:52 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/648806224' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.16", "id": [1, 4, 5, 6, 3, 7]}]: dispatch 2024-03-26T05:02:52.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:52 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.16", "id": [1, 4, 5, 6, 3, 7]}]: dispatch 2024-03-26T05:02:52.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:52 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3355891236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:02:52.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:52 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:02:52.803 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:52.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:52.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:52.905 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:52.951 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:52.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:53.053 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:53.085 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:53.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:53.187 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:53.195 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 15 16 32968 32952 137.285 102.875 0.00710634 0.00727862 2024-03-26T05:02:53.217 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:53.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:53.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:53.319 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:53.360 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:53.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:53.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:53.461 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:53.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:53.493 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:53.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:53.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:53.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:53 smithi080 ceph-mon[40404]: pgmap v80: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-26T05:02:53.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:53 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:02:53.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:53 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.16", "id": [1, 4, 5, 6, 3, 7]}]': finished 2024-03-26T05:02:53.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:53 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:02:53.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:53 smithi080 ceph-mon[40404]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T05:02:53.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:53 smithi080 ceph-mon[29691]: pgmap v80: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-26T05:02:53.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:53 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:02:53.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:53 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.16", "id": [1, 4, 5, 6, 3, 7]}]': finished 2024-03-26T05:02:53.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:53 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:02:53.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:53 smithi080 ceph-mon[29691]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T05:02:53.627 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:53.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:53.729 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:53.772 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:53.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:53.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:53 smithi132 ceph-mon[33500]: pgmap v80: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-26T05:02:53.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:53 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:02:53.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:53 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.16", "id": [1, 4, 5, 6, 3, 7]}]': finished 2024-03-26T05:02:53.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:53 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:02:53.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:53 smithi132 ceph-mon[33500]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T05:02:53.873 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:53.912 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:53.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:53.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:54.014 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:54.045 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:54.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:54.118 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:02:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:02:54.147 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:54.179 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:54.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:54.195 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 16 16 34928 34912 136.36 122.5 0.0118659 0.00732806 2024-03-26T05:02:54.281 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:54.312 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:54.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:54.414 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:54.457 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:54.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:54.558 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:54.575 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-mon[29691]: 5.1f scrub starts 2024-03-26T05:02:54.575 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-mon[29691]: 5.1f scrub ok 2024-03-26T05:02:54.575 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-mon[29691]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T05:02:54.575 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-mon[29691]: 6.1e scrub starts 2024-03-26T05:02:54.575 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-mon[29691]: 6.1e scrub ok 2024-03-26T05:02:54.575 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-mon[29691]: pgmap v83: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-26T05:02:54.576 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-mon[40404]: 5.1f scrub starts 2024-03-26T05:02:54.576 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-mon[40404]: 5.1f scrub ok 2024-03-26T05:02:54.576 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-mon[40404]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T05:02:54.576 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-mon[40404]: 6.1e scrub starts 2024-03-26T05:02:54.576 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-mon[40404]: 6.1e scrub ok 2024-03-26T05:02:54.576 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:54 smithi080 ceph-mon[40404]: pgmap v83: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-26T05:02:54.595 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:54.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:54.599 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:54.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:54.701 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:54.750 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:54.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:54.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:54 smithi132 ceph-mon[33500]: 5.1f scrub starts 2024-03-26T05:02:54.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:54 smithi132 ceph-mon[33500]: 5.1f scrub ok 2024-03-26T05:02:54.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:54 smithi132 ceph-mon[33500]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T05:02:54.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:54 smithi132 ceph-mon[33500]: 6.1e scrub starts 2024-03-26T05:02:54.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:54 smithi132 ceph-mon[33500]: 6.1e scrub ok 2024-03-26T05:02:54.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:54 smithi132 ceph-mon[33500]: pgmap v83: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-26T05:02:54.852 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:54.891 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:54.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:54.993 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:55.031 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:55.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:55.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:02:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:02:55.133 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:55.169 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:55.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:55.198 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 17 16 36817 36801 135.28 118.062 0.00929466 0.00738578 2024-03-26T05:02:55.270 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:55.289 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:55.331 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:55.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:55.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:55.428 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[29691]: 6.19 scrub starts 2024-03-26T05:02:55.428 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[29691]: 6.19 scrub ok 2024-03-26T05:02:55.428 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[29691]: 3.1a scrub starts 2024-03-26T05:02:55.428 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[29691]: 3.1a scrub ok 2024-03-26T05:02:55.428 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[29691]: 6.18 scrub starts 2024-03-26T05:02:55.428 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[29691]: 6.18 scrub ok 2024-03-26T05:02:55.428 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:55.429 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[40404]: 6.19 scrub starts 2024-03-26T05:02:55.429 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[40404]: 6.19 scrub ok 2024-03-26T05:02:55.429 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[40404]: 3.1a scrub starts 2024-03-26T05:02:55.429 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[40404]: 3.1a scrub ok 2024-03-26T05:02:55.429 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[40404]: 6.18 scrub starts 2024-03-26T05:02:55.429 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[40404]: 6.18 scrub ok 2024-03-26T05:02:55.429 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:55 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:55.433 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:55.482 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:55.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:55.583 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:55.623 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:55.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:55.724 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:55.763 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:55.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:55.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:55.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:55 smithi132 ceph-mon[33500]: 6.19 scrub starts 2024-03-26T05:02:55.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:55 smithi132 ceph-mon[33500]: 6.19 scrub ok 2024-03-26T05:02:55.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:55 smithi132 ceph-mon[33500]: 3.1a scrub starts 2024-03-26T05:02:55.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:55 smithi132 ceph-mon[33500]: 3.1a scrub ok 2024-03-26T05:02:55.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:55 smithi132 ceph-mon[33500]: 6.18 scrub starts 2024-03-26T05:02:55.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:55 smithi132 ceph-mon[33500]: 6.18 scrub ok 2024-03-26T05:02:55.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:55 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:55.865 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:55.911 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:55.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:56.013 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:56.045 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:56.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:56.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:56.147 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:56.179 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:56.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:56.196 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 18 15 38661 38646 134.17 115.312 0.00682649 0.00744828 2024-03-26T05:02:56.280 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:56.323 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:56.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:56.425 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:56.456 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:56.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:56.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:56.558 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:56.590 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:56.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:56.630 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:02:56.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:56 smithi080 ceph-mon[29691]: 3.1c scrub starts 2024-03-26T05:02:56.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:56 smithi080 ceph-mon[29691]: 3.1c scrub ok 2024-03-26T05:02:56.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:56 smithi080 ceph-mon[29691]: pgmap v84: 148 pgs: 148 active+clean; 2.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 144 MiB/s wr, 2.31k op/s 2024-03-26T05:02:56.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:56 smithi080 ceph-mon[40404]: 3.1c scrub starts 2024-03-26T05:02:56.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:56 smithi080 ceph-mon[40404]: 3.1c scrub ok 2024-03-26T05:02:56.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:56 smithi080 ceph-mon[40404]: pgmap v84: 148 pgs: 148 active+clean; 2.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 144 MiB/s wr, 2.31k op/s 2024-03-26T05:02:56.692 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:56.723 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:56.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:56.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:56.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:56 smithi132 ceph-mon[33500]: 3.1c scrub starts 2024-03-26T05:02:56.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:56 smithi132 ceph-mon[33500]: 3.1c scrub ok 2024-03-26T05:02:56.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:56 smithi132 ceph-mon[33500]: pgmap v84: 148 pgs: 148 active+clean; 2.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 144 MiB/s wr, 2.31k op/s 2024-03-26T05:02:56.825 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:56.854 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:56.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:56.956 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:02:57.004 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:57.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:57.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:02:57.106 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:02:57.136 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:57.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:02:57.196 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 19 16 40446 40430 132.975 111.5 0.0080677 0.00751425 2024-03-26T05:02:57.238 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:57.269 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:57.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:57.371 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:57.401 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:57.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:57.504 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:57.544 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:57.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:57.646 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:02:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:57.692 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:57.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:02:57.708 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:02:57 smithi080 systemd[1]: Stopping Ceph alertmanager.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:02:57.708 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:57 smithi080 ceph-mon[40404]: 3.13 scrub starts 2024-03-26T05:02:57.708 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:57 smithi080 ceph-mon[40404]: 3.13 scrub ok 2024-03-26T05:02:57.708 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:57 smithi080 ceph-mon[40404]: 4.d scrub starts 2024-03-26T05:02:57.709 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:57 smithi080 ceph-mon[29691]: 3.13 scrub starts 2024-03-26T05:02:57.709 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:57 smithi080 ceph-mon[29691]: 3.13 scrub ok 2024-03-26T05:02:57.709 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:57 smithi080 ceph-mon[29691]: 4.d scrub starts 2024-03-26T05:02:57.793 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:57.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:57 smithi132 ceph-mon[33500]: 3.13 scrub starts 2024-03-26T05:02:57.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:57 smithi132 ceph-mon[33500]: 3.13 scrub ok 2024-03-26T05:02:57.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:57 smithi132 ceph-mon[33500]: 4.d scrub starts 2024-03-26T05:02:57.824 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:57.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:57.926 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:02:57.965 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:57.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:57.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:02:58.067 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:58.096 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:58.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:58.196 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:02:58.204944+0000 min lat: 0.00124227 max lat: 0.239944 avg lat: 0.0075896 2024-03-26T05:02:58.196 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:02:58.196 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 20 16 42153 42137 131.66 106.688 0.00899843 0.0075896 2024-03-26T05:02:58.198 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:58.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:58.230 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:58.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:58.331 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:58.362 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:58.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:58.464 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:58.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:58.491 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:58.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:58.592 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:58.621 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:58.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:58.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:58.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[40404]: 4.4 scrub starts 2024-03-26T05:02:58.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[40404]: 4.4 scrub ok 2024-03-26T05:02:58.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[40404]: 3.f scrub starts 2024-03-26T05:02:58.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[40404]: 3.f scrub ok 2024-03-26T05:02:58.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[40404]: 4.d scrub ok 2024-03-26T05:02:58.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[40404]: pgmap v85: 148 pgs: 148 active+clean; 2.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-26T05:02:58.679 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[40404]: 3.1f scrub starts 2024-03-26T05:02:58.679 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[40404]: 3.1f scrub ok 2024-03-26T05:02:58.679 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[29691]: 4.4 scrub starts 2024-03-26T05:02:58.679 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[29691]: 4.4 scrub ok 2024-03-26T05:02:58.679 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[29691]: 3.f scrub starts 2024-03-26T05:02:58.679 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[29691]: 3.f scrub ok 2024-03-26T05:02:58.679 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[29691]: 4.d scrub ok 2024-03-26T05:02:58.679 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[29691]: pgmap v85: 148 pgs: 148 active+clean; 2.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-26T05:02:58.679 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[29691]: 3.1f scrub starts 2024-03-26T05:02:58.679 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:58 smithi080 ceph-mon[29691]: 3.1f scrub ok 2024-03-26T05:02:58.723 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:58.760 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:02:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:58 smithi132 ceph-mon[33500]: 4.4 scrub starts 2024-03-26T05:02:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:58 smithi132 ceph-mon[33500]: 4.4 scrub ok 2024-03-26T05:02:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:58 smithi132 ceph-mon[33500]: 3.f scrub starts 2024-03-26T05:02:58.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:58 smithi132 ceph-mon[33500]: 3.f scrub ok 2024-03-26T05:02:58.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:58 smithi132 ceph-mon[33500]: 4.d scrub ok 2024-03-26T05:02:58.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:58 smithi132 ceph-mon[33500]: pgmap v85: 148 pgs: 148 active+clean; 2.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-26T05:02:58.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:58 smithi132 ceph-mon[33500]: 3.1f scrub starts 2024-03-26T05:02:58.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:58 smithi132 ceph-mon[33500]: 3.1f scrub ok 2024-03-26T05:02:58.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:58.799 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:58.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:58.820 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:02:58.865 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:02:58.901 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:02:58.903 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:02:58.903 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 0 1 2024-03-26T05:02:58.907 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:02:58.927 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:58.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:02:58.954 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:02:58.993 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:02:59.029 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:59.035 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:02:59.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:59.059 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:59.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:59.090 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:02:59.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:59.175 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:02:59.188 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:02:59.188 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:02:59.188 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:02:59.188 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:02:59.188 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:02:59.189 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:02:59.189 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:02:59.189 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:02:59.198 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 21 16 43588 43572 129.65 89.6875 0.0120918 0.0077074 2024-03-26T05:02:59.227 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:59.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:59.329 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:59.388 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:59.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:59.490 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:59.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:59.528 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:59.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:02:59.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[40404]: 6.a scrub starts 2024-03-26T05:02:59.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[40404]: 6.a scrub ok 2024-03-26T05:02:59.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[40404]: 5.13 scrub starts 2024-03-26T05:02:59.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[40404]: 5.13 scrub ok 2024-03-26T05:02:59.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[40404]: 4.18 scrub starts 2024-03-26T05:02:59.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[40404]: 4.18 scrub ok 2024-03-26T05:02:59.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1399015095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:02:59.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[29691]: 6.a scrub starts 2024-03-26T05:02:59.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[29691]: 6.a scrub ok 2024-03-26T05:02:59.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[29691]: 5.13 scrub starts 2024-03-26T05:02:59.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[29691]: 5.13 scrub ok 2024-03-26T05:02:59.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[29691]: 4.18 scrub starts 2024-03-26T05:02:59.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[29691]: 4.18 scrub ok 2024-03-26T05:02:59.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:02:59 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1399015095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:02:59.629 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:02:59.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:59.659 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:59.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:02:59.760 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:02:59.790 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:02:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:59.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:02:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:59 smithi132 ceph-mon[33500]: 6.a scrub starts 2024-03-26T05:02:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:59 smithi132 ceph-mon[33500]: 6.a scrub ok 2024-03-26T05:02:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:59 smithi132 ceph-mon[33500]: 5.13 scrub starts 2024-03-26T05:02:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:59 smithi132 ceph-mon[33500]: 5.13 scrub ok 2024-03-26T05:02:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:59 smithi132 ceph-mon[33500]: 4.18 scrub starts 2024-03-26T05:02:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:59 smithi132 ceph-mon[33500]: 4.18 scrub ok 2024-03-26T05:02:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:02:59 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1399015095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:02:59.892 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:02:59.937 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:02:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:02:59.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:00.039 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:00.069 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:00.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:00.171 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:00.198 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 22 16 45226 45210 128.41 102.375 0.0128068 0.00778218 2024-03-26T05:03:00.202 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:00.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:00.275 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:03:00.275 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:03:00.303 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:00.334 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:00.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:00.437 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:00.466 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:00.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:00.568 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:00.600 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:00.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:00.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:00.701 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:00.747 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:00.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:00.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:00 smithi132 ceph-mon[33500]: 4.f scrub starts 2024-03-26T05:03:00.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:00 smithi132 ceph-mon[33500]: 4.f scrub ok 2024-03-26T05:03:00.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:00 smithi132 ceph-mon[33500]: 4.1c scrub starts 2024-03-26T05:03:00.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:00 smithi132 ceph-mon[33500]: 4.1c scrub ok 2024-03-26T05:03:00.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:00 smithi132 ceph-mon[33500]: 4.9 scrub starts 2024-03-26T05:03:00.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:00 smithi132 ceph-mon[33500]: 4.9 scrub ok 2024-03-26T05:03:00.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:00 smithi132 ceph-mon[33500]: pgmap v86: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-26T05:03:00.838 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[75215]: ts=2024-03-26T05:03:00.407Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T05:03:00.839 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:00 smithi080 podman[83036]: 2024-03-26 05:03:00.64229163 +0000 UTC m=+1.889421062 container died c399a78cc74f7df33cfa32216983fbd1b3eee0d901b66c3623d9aac545673d1e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:03:00.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[40404]: 4.f scrub starts 2024-03-26T05:03:00.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[40404]: 4.f scrub ok 2024-03-26T05:03:00.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[40404]: 4.1c scrub starts 2024-03-26T05:03:00.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[40404]: 4.1c scrub ok 2024-03-26T05:03:00.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[40404]: 4.9 scrub starts 2024-03-26T05:03:00.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[40404]: 4.9 scrub ok 2024-03-26T05:03:00.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[40404]: pgmap v86: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-26T05:03:00.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[29691]: 4.f scrub starts 2024-03-26T05:03:00.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[29691]: 4.f scrub ok 2024-03-26T05:03:00.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[29691]: 4.1c scrub starts 2024-03-26T05:03:00.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[29691]: 4.1c scrub ok 2024-03-26T05:03:00.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[29691]: 4.9 scrub starts 2024-03-26T05:03:00.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[29691]: 4.9 scrub ok 2024-03-26T05:03:00.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:00 smithi080 ceph-mon[29691]: pgmap v86: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-26T05:03:00.848 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:00.889 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:00.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:00.990 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:01.028 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:01.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:01.130 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:01.161 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:01.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:01.198 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 23 16 46826 46810 127.175 100 0.0043953 0.00784107 2024-03-26T05:03:01.264 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:01.303 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:01.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:01.405 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:01.445 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:01.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:01.547 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:01.575 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:01.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:01.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:01.677 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:01.695 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:01 smithi132 ceph-mon[33500]: 6.14 scrub starts 2024-03-26T05:03:01.695 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:01 smithi132 ceph-mon[33500]: 6.5 scrub starts 2024-03-26T05:03:01.695 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:01 smithi132 ceph-mon[33500]: 6.14 scrub ok 2024-03-26T05:03:01.695 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:01 smithi132 ceph-mon[33500]: 6.5 scrub ok 2024-03-26T05:03:01.695 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:01 smithi132 ceph-mon[33500]: 6.1b scrub starts 2024-03-26T05:03:01.695 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:01 smithi132 ceph-mon[33500]: 6.1b scrub ok 2024-03-26T05:03:01.709 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:01.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:01.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:01.811 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:01.843 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:01.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:01.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:01 smithi080 ceph-mon[29691]: 6.14 scrub starts 2024-03-26T05:03:01.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:01 smithi080 ceph-mon[29691]: 6.5 scrub starts 2024-03-26T05:03:01.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:01 smithi080 ceph-mon[29691]: 6.14 scrub ok 2024-03-26T05:03:01.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:01 smithi080 ceph-mon[29691]: 6.5 scrub ok 2024-03-26T05:03:01.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:01 smithi080 ceph-mon[29691]: 6.1b scrub starts 2024-03-26T05:03:01.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:01 smithi080 ceph-mon[29691]: 6.1b scrub ok 2024-03-26T05:03:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:01 smithi080 ceph-mon[40404]: 6.14 scrub starts 2024-03-26T05:03:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:01 smithi080 ceph-mon[40404]: 6.5 scrub starts 2024-03-26T05:03:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:01 smithi080 ceph-mon[40404]: 6.14 scrub ok 2024-03-26T05:03:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:01 smithi080 ceph-mon[40404]: 6.5 scrub ok 2024-03-26T05:03:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:01 smithi080 ceph-mon[40404]: 6.1b scrub starts 2024-03-26T05:03:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:01 smithi080 ceph-mon[40404]: 6.1b scrub ok 2024-03-26T05:03:01.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:01.976 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:01.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:01.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:02.077 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:02.109 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:02.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:02.198 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 24 16 48404 48388 125.985 98.625 0.00660263 0.00793191 2024-03-26T05:03:02.211 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:02.226 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:01 smithi080 podman[83036]: 2024-03-26 05:03:01.975817198 +0000 UTC m=+3.222946624 container cleanup c399a78cc74f7df33cfa32216983fbd1b3eee0d901b66c3623d9aac545673d1e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:03:02.226 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:01 smithi080 bash[83036]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T05:03:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:02.258 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:02.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:02.361 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:02.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:02.403 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:02.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:02.504 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:02.534 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:02.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:02.549 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:02 smithi080 ceph-mon[40404]: 2.0 scrub starts 2024-03-26T05:03:02.550 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:02 smithi080 ceph-mon[40404]: 2.0 scrub ok 2024-03-26T05:03:02.550 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:02 smithi080 ceph-mon[40404]: pgmap v87: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-26T05:03:02.550 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:02 smithi080 ceph-mon[29691]: 2.0 scrub starts 2024-03-26T05:03:02.551 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:02 smithi080 ceph-mon[29691]: 2.0 scrub ok 2024-03-26T05:03:02.551 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:02 smithi080 ceph-mon[29691]: pgmap v87: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-26T05:03:02.635 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:02.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:02.677 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:02.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:02.779 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:02.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:02 smithi132 ceph-mon[33500]: 2.0 scrub starts 2024-03-26T05:03:02.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:02 smithi132 ceph-mon[33500]: 2.0 scrub ok 2024-03-26T05:03:02.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:02 smithi132 ceph-mon[33500]: pgmap v87: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-26T05:03:02.817 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:02 smithi080 podman[83268]: 2024-03-26 05:03:02.589031712 +0000 UTC m=+2.120312913 container remove c399a78cc74f7df33cfa32216983fbd1b3eee0d901b66c3623d9aac545673d1e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:03:02.817 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:02 smithi080 podman[83268]: 2024-03-26 05:03:02.682229971 +0000 UTC m=+2.213511184 volume remove 1999c43d9e33008c07bd67e644a2c6e0ff54e4e840be89a07ed46518874d9e2a 2024-03-26T05:03:02.818 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:02.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:02.921 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:02.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:02.964 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:02.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:03.065 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:03.093 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:03.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:03.195 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:03.198 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 25 15 49751 49736 124.315 84.25 0.0132232 0.00803823 2024-03-26T05:03:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:03.224 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:03.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:03.325 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:03.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:03.358 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:03.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:03.459 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:03.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:03.501 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:03.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:03.601 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:03.628 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:03.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:03.730 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:03.756 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:03.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:03.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:03 smithi132 ceph-mon[33500]: 5.7 scrub starts 2024-03-26T05:03:03.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:03 smithi132 ceph-mon[33500]: 5.7 scrub ok 2024-03-26T05:03:03.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:03 smithi132 ceph-mon[33500]: 3.16 scrub starts 2024-03-26T05:03:03.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:03 smithi132 ceph-mon[33500]: 3.16 scrub ok 2024-03-26T05:03:03.858 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:03 smithi080 ceph-mon[29691]: 5.7 scrub starts 2024-03-26T05:03:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:03 smithi080 ceph-mon[29691]: 5.7 scrub ok 2024-03-26T05:03:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:03 smithi080 ceph-mon[29691]: 3.16 scrub starts 2024-03-26T05:03:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:03 smithi080 ceph-mon[29691]: 3.16 scrub ok 2024-03-26T05:03:03.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:03 smithi080 ceph-mon[40404]: 5.7 scrub starts 2024-03-26T05:03:03.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:03 smithi080 ceph-mon[40404]: 5.7 scrub ok 2024-03-26T05:03:03.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:03 smithi080 ceph-mon[40404]: 3.16 scrub starts 2024-03-26T05:03:03.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:03 smithi080 ceph-mon[40404]: 3.16 scrub ok 2024-03-26T05:03:03.886 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:03.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:03.988 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:04.030 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:04.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:04.131 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:04.172 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:04.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:04.198 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 26 16 50931 50915 122.368 73.6875 0.0104947 0.00816651 2024-03-26T05:03:04.274 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:04.309 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:04.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:04.368 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:03:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:03:04.411 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:04.456 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:04.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:04.510 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:03:04.558 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:04.589 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:04.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:04.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:04.691 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:04.732 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:04.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:04.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[40404]: 4.15 scrub starts 2024-03-26T05:03:04.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[40404]: 4.15 scrub ok 2024-03-26T05:03:04.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:03:04.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:03:04.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[40404]: 3.14 scrub starts 2024-03-26T05:03:04.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[40404]: 3.14 scrub ok 2024-03-26T05:03:04.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[40404]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T05:03:04.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[40404]: pgmap v89: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-03-26T05:03:04.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 6]}]: dispatch 2024-03-26T05:03:04.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:04.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1872817682' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:03:04.748 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:03:04.748 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[29691]: 4.15 scrub starts 2024-03-26T05:03:04.748 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[29691]: 4.15 scrub ok 2024-03-26T05:03:04.748 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:03:04.748 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:03:04.748 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[29691]: 3.14 scrub starts 2024-03-26T05:03:04.748 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[29691]: 3.14 scrub ok 2024-03-26T05:03:04.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[29691]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T05:03:04.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[29691]: pgmap v89: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-03-26T05:03:04.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 6]}]: dispatch 2024-03-26T05:03:04.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:04.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1872817682' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:03:04.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:04 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:03:04.792 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T05:03:04.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-mon[33500]: 4.15 scrub starts 2024-03-26T05:03:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-mon[33500]: 4.15 scrub ok 2024-03-26T05:03:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:03:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:03:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-mon[33500]: 3.14 scrub starts 2024-03-26T05:03:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-mon[33500]: 3.14 scrub ok 2024-03-26T05:03:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-mon[33500]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T05:03:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-mon[33500]: pgmap v89: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-03-26T05:03:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 6]}]: dispatch 2024-03-26T05:03:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1872817682' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:03:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:03:04.833 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:04.864 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:04.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:04.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:04.966 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:05.008 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:05.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:05.112 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:05.171 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:05.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:05.198 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 27 16 52417 52401 121.275 92.875 0.00517901 0.00824075 2024-03-26T05:03:05.271 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:05.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:03:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:03:05.312 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:05.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:05.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:05.414 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:05.455 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:05.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:05.550 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:03:05.557 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:05.618 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:05.664 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:05.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:05.724 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:05.765 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: 5.f scrub starts 2024-03-26T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: 5.f scrub ok 2024-03-26T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: 5.11 scrub starts 2024-03-26T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: 5.11 scrub ok 2024-03-26T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: 3.2 scrub starts 2024-03-26T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: 3.2 scrub ok 2024-03-26T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: Cluster is now healthy 2024-03-26T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 6]}]': finished 2024-03-26T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:05.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: 3.1 scrub starts 2024-03-26T05:03:05.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:05 smithi132 ceph-mon[33500]: 3.1 scrub ok 2024-03-26T05:03:05.795 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:05.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:05.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:05.804 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: 5.f scrub starts 2024-03-26T05:03:05.804 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: 5.f scrub ok 2024-03-26T05:03:05.804 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: 5.11 scrub starts 2024-03-26T05:03:05.804 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: 5.11 scrub ok 2024-03-26T05:03:05.804 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: 3.2 scrub starts 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: 3.2 scrub ok 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: Cluster is now healthy 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 6]}]': finished 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: 3.1 scrub starts 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[29691]: 3.1 scrub ok 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: 5.f scrub starts 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: 5.f scrub ok 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: 5.11 scrub starts 2024-03-26T05:03:05.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: 5.11 scrub ok 2024-03-26T05:03:05.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: 3.2 scrub starts 2024-03-26T05:03:05.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: 3.2 scrub ok 2024-03-26T05:03:05.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:03:05.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: Cluster is now healthy 2024-03-26T05:03:05.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 6]}]': finished 2024-03-26T05:03:05.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:03:05.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T05:03:05.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:05.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: 3.1 scrub starts 2024-03-26T05:03:05.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:05 smithi080 ceph-mon[40404]: 3.1 scrub ok 2024-03-26T05:03:05.897 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:05.941 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:05.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:06.042 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:06.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:06.091 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:06.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:06.094 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:06 smithi080 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T05:03:06.094 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:06 smithi080 systemd[1]: Stopped Ceph alertmanager.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:03:06.094 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:06 smithi080 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 1.185s CPU time. 2024-03-26T05:03:06.095 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:06 smithi080 systemd[1]: Starting Ceph alertmanager.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:03:06.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:06.199 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 28 16 53759 53743 119.939 83.875 0.0114215 0.00833229 2024-03-26T05:03:06.229 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:06.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:06.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:06.331 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:06.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:06.362 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:06.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:06.464 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:06.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:06.527 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:06.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:06.628 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:06.660 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:06.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:06.762 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:06.793 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:06.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:06 smithi132 ceph-mon[33500]: 6.1f deep-scrub starts 2024-03-26T05:03:06.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:06 smithi132 ceph-mon[33500]: 6.1f deep-scrub ok 2024-03-26T05:03:06.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:06 smithi132 ceph-mon[33500]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T05:03:06.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:06 smithi132 ceph-mon[33500]: pgmap v92: 148 pgs: 1 peering, 147 active+clean; 3.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-26T05:03:06.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:06 smithi132 ceph-mon[33500]: 3.1e scrub starts 2024-03-26T05:03:06.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:06 smithi132 ceph-mon[33500]: 3.1e scrub ok 2024-03-26T05:03:06.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:06 smithi132 ceph-mon[33500]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T05:03:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:06.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:06.819 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[40404]: 6.1f deep-scrub starts 2024-03-26T05:03:06.820 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[40404]: 6.1f deep-scrub ok 2024-03-26T05:03:06.820 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[40404]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T05:03:06.820 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[40404]: pgmap v92: 148 pgs: 1 peering, 147 active+clean; 3.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-26T05:03:06.820 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[40404]: 3.1e scrub starts 2024-03-26T05:03:06.820 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[40404]: 3.1e scrub ok 2024-03-26T05:03:06.820 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[40404]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T05:03:06.820 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[29691]: 6.1f deep-scrub starts 2024-03-26T05:03:06.820 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[29691]: 6.1f deep-scrub ok 2024-03-26T05:03:06.820 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[29691]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T05:03:06.820 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[29691]: pgmap v92: 148 pgs: 1 peering, 147 active+clean; 3.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-26T05:03:06.820 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[29691]: 3.1e scrub starts 2024-03-26T05:03:06.820 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[29691]: 3.1e scrub ok 2024-03-26T05:03:06.821 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:06 smithi080 ceph-mon[29691]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T05:03:06.896 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:06.935 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:06.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:06.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:07.037 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:07.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:07.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:07.170 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:07.199 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 29 16 55259 55243 119.035 93.75 0.01273 0.00839563 2024-03-26T05:03:07.201 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:07.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:07.303 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:07.306 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:07.369 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:07.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:07.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:07.471 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:07.500 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:07.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:07.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:07.543 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:03:07.602 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:07.687 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:07.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:07.788 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:07 smithi132 ceph-mon[33500]: 5.10 deep-scrub starts 2024-03-26T05:03:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:07 smithi132 ceph-mon[33500]: 5.10 deep-scrub ok 2024-03-26T05:03:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:07 smithi132 ceph-mon[33500]: 4.16 scrub starts 2024-03-26T05:03:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:07 smithi132 ceph-mon[33500]: 4.16 scrub ok 2024-03-26T05:03:07.833 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:07.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:07.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:07 smithi080 ceph-mon[29691]: 5.10 deep-scrub starts 2024-03-26T05:03:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:07 smithi080 ceph-mon[29691]: 5.10 deep-scrub ok 2024-03-26T05:03:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:07 smithi080 ceph-mon[29691]: 4.16 scrub starts 2024-03-26T05:03:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:07 smithi080 ceph-mon[29691]: 4.16 scrub ok 2024-03-26T05:03:07.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:07 smithi080 ceph-mon[40404]: 5.10 deep-scrub starts 2024-03-26T05:03:07.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:07 smithi080 ceph-mon[40404]: 5.10 deep-scrub ok 2024-03-26T05:03:07.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:07 smithi080 ceph-mon[40404]: 4.16 scrub starts 2024-03-26T05:03:07.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:07 smithi080 ceph-mon[40404]: 4.16 scrub ok 2024-03-26T05:03:07.935 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:07.965 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:07.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:08.066 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:08.105 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:08.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:08.199 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 30 16 56599 56583 117.859 83.75 0.00889685 0.00847936 2024-03-26T05:03:08.206 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:08.238 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:08.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:08.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:08.340 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:08.391 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:08.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:08.493 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:08.525 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:08.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[29691]: 6.4 scrub starts 2024-03-26T05:03:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[29691]: 6.4 scrub ok 2024-03-26T05:03:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[29691]: 4.8 scrub starts 2024-03-26T05:03:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[29691]: 4.8 scrub ok 2024-03-26T05:03:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[29691]: 5.8 scrub starts 2024-03-26T05:03:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[29691]: 5.8 scrub ok 2024-03-26T05:03:08.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[29691]: pgmap v94: 148 pgs: 1 peering, 147 active+clean; 3.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-26T05:03:08.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[40404]: 6.4 scrub starts 2024-03-26T05:03:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[40404]: 6.4 scrub ok 2024-03-26T05:03:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[40404]: 4.8 scrub starts 2024-03-26T05:03:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[40404]: 4.8 scrub ok 2024-03-26T05:03:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[40404]: 5.8 scrub starts 2024-03-26T05:03:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[40404]: 5.8 scrub ok 2024-03-26T05:03:08.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:08 smithi080 ceph-mon[40404]: pgmap v94: 148 pgs: 1 peering, 147 active+clean; 3.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-26T05:03:08.626 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:08.671 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:08.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:08.773 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:08.804 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:08.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:08.906 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:08.935 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:08.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:09.037 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:09.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:08 smithi132 ceph-mon[33500]: 6.4 scrub starts 2024-03-26T05:03:09.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:08 smithi132 ceph-mon[33500]: 6.4 scrub ok 2024-03-26T05:03:09.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:08 smithi132 ceph-mon[33500]: 4.8 scrub starts 2024-03-26T05:03:09.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:08 smithi132 ceph-mon[33500]: 4.8 scrub ok 2024-03-26T05:03:09.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:08 smithi132 ceph-mon[33500]: 5.8 scrub starts 2024-03-26T05:03:09.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:08 smithi132 ceph-mon[33500]: 5.8 scrub ok 2024-03-26T05:03:09.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:08 smithi132 ceph-mon[33500]: pgmap v94: 148 pgs: 1 peering, 147 active+clean; 3.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-26T05:03:09.093 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:09.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:09.195 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:09.200 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 31 16 57985 57969 116.852 86.625 0.00731165 0.00855233 2024-03-26T05:03:09.240 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:09.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:09.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:09.342 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:09.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:09.366 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:09.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:09.467 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:09.503 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:09.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:09.546 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:09 smithi080 podman[84231]: 2024-03-26 05:03:09.173463688 +0000 UTC m=+0.136526192 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T05:03:09.604 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:09.643 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:09.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:09.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:09.745 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:09.776 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:09.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:09.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:09.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:09 smithi080 ceph-mon[29691]: 2.1 scrub starts 2024-03-26T05:03:09.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:09 smithi080 ceph-mon[29691]: 2.1 scrub ok 2024-03-26T05:03:09.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:09 smithi080 ceph-mon[29691]: 3.b scrub starts 2024-03-26T05:03:09.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:09 smithi080 ceph-mon[29691]: 3.b scrub ok 2024-03-26T05:03:09.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:09 smithi080 ceph-mon[29691]: 5.1 scrub starts 2024-03-26T05:03:09.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:09 smithi080 ceph-mon[29691]: 5.1 scrub ok 2024-03-26T05:03:09.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:09 smithi080 ceph-mon[40404]: 2.1 scrub starts 2024-03-26T05:03:09.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:09 smithi080 ceph-mon[40404]: 2.1 scrub ok 2024-03-26T05:03:09.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:09 smithi080 ceph-mon[40404]: 3.b scrub starts 2024-03-26T05:03:09.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:09 smithi080 ceph-mon[40404]: 3.b scrub ok 2024-03-26T05:03:09.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:09 smithi080 ceph-mon[40404]: 5.1 scrub starts 2024-03-26T05:03:09.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:09 smithi080 ceph-mon[40404]: 5.1 scrub ok 2024-03-26T05:03:09.878 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:09.909 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:09.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:09.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:10.011 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:10.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:09 smithi132 ceph-mon[33500]: 2.1 scrub starts 2024-03-26T05:03:10.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:09 smithi132 ceph-mon[33500]: 2.1 scrub ok 2024-03-26T05:03:10.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:09 smithi132 ceph-mon[33500]: 3.b scrub starts 2024-03-26T05:03:10.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:09 smithi132 ceph-mon[33500]: 3.b scrub ok 2024-03-26T05:03:10.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:09 smithi132 ceph-mon[33500]: 5.1 scrub starts 2024-03-26T05:03:10.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:09 smithi132 ceph-mon[33500]: 5.1 scrub ok 2024-03-26T05:03:10.054 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:10.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:10.156 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:10.187 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:10.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:10.199 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 32 16 59354 59338 115.874 85.5625 0.00569015 0.00862368 2024-03-26T05:03:10.288 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:10.325 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:10.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:10.427 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:10.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:10.457 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:10.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:10.558 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:10.589 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:10.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:10.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:10.691 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:10.715 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:10.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:10.817 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:10.848 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:10.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:10.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 3.1d scrub starts 2024-03-26T05:03:10.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 3.1d scrub ok 2024-03-26T05:03:10.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 5.d deep-scrub starts 2024-03-26T05:03:10.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 5.d deep-scrub ok 2024-03-26T05:03:10.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 3.10 scrub starts 2024-03-26T05:03:10.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 3.10 scrub ok 2024-03-26T05:03:10.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: pgmap v95: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 135 MiB/s wr, 2.15k op/s 2024-03-26T05:03:10.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 5.1a scrub starts 2024-03-26T05:03:10.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 5.1a scrub ok 2024-03-26T05:03:10.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 3.1b deep-scrub starts 2024-03-26T05:03:10.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 3.1b deep-scrub ok 2024-03-26T05:03:10.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 4.14 scrub starts 2024-03-26T05:03:10.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 4.14 scrub ok 2024-03-26T05:03:10.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 4.1b scrub starts 2024-03-26T05:03:10.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[29691]: 4.1b scrub ok 2024-03-26T05:03:10.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 3.1d scrub starts 2024-03-26T05:03:10.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 3.1d scrub ok 2024-03-26T05:03:10.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 5.d deep-scrub starts 2024-03-26T05:03:10.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 5.d deep-scrub ok 2024-03-26T05:03:10.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 3.10 scrub starts 2024-03-26T05:03:10.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 3.10 scrub ok 2024-03-26T05:03:10.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: pgmap v95: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 135 MiB/s wr, 2.15k op/s 2024-03-26T05:03:10.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 5.1a scrub starts 2024-03-26T05:03:10.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 5.1a scrub ok 2024-03-26T05:03:10.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 3.1b deep-scrub starts 2024-03-26T05:03:10.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 3.1b deep-scrub ok 2024-03-26T05:03:10.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 4.14 scrub starts 2024-03-26T05:03:10.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 4.14 scrub ok 2024-03-26T05:03:10.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 4.1b scrub starts 2024-03-26T05:03:10.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:10 smithi080 ceph-mon[40404]: 4.1b scrub ok 2024-03-26T05:03:10.950 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:10.973 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:10.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:11.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 3.1d scrub starts 2024-03-26T05:03:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 3.1d scrub ok 2024-03-26T05:03:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 5.d deep-scrub starts 2024-03-26T05:03:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 5.d deep-scrub ok 2024-03-26T05:03:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 3.10 scrub starts 2024-03-26T05:03:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 3.10 scrub ok 2024-03-26T05:03:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: pgmap v95: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 135 MiB/s wr, 2.15k op/s 2024-03-26T05:03:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 5.1a scrub starts 2024-03-26T05:03:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 5.1a scrub ok 2024-03-26T05:03:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 3.1b deep-scrub starts 2024-03-26T05:03:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 3.1b deep-scrub ok 2024-03-26T05:03:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 4.14 scrub starts 2024-03-26T05:03:11.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 4.14 scrub ok 2024-03-26T05:03:11.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 4.1b scrub starts 2024-03-26T05:03:11.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:10 smithi132 ceph-mon[33500]: 4.1b scrub ok 2024-03-26T05:03:11.075 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:11.103 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:11.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:11.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:11.199 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 33 16 60519 60503 114.569 72.8125 0.00282234 0.00871436 2024-03-26T05:03:11.205 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:11.237 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:11.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:11.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:11.339 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:11.380 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:11.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:11.482 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:11.495 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:11.512 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:11.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:11.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:11.614 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:11.637 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:11.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:11.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:11.739 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:11.770 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:11.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:11.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:03:11.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:03:11.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[29691]: 5.9 scrub starts 2024-03-26T05:03:11.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[29691]: 5.9 scrub ok 2024-03-26T05:03:11.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[29691]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T05:03:11.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[29691]: 3.3 scrub starts 2024-03-26T05:03:11.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[29691]: 3.3 scrub ok 2024-03-26T05:03:11.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:03:11.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:03:11.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[40404]: 5.9 scrub starts 2024-03-26T05:03:11.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[40404]: 5.9 scrub ok 2024-03-26T05:03:11.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[40404]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T05:03:11.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[40404]: 3.3 scrub starts 2024-03-26T05:03:11.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:11 smithi080 ceph-mon[40404]: 3.3 scrub ok 2024-03-26T05:03:11.871 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:11.902 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:11.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:12.003 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:12.041 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:12.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:12.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:11 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:03:12.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:11 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:03:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:11 smithi132 ceph-mon[33500]: 5.9 scrub starts 2024-03-26T05:03:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:11 smithi132 ceph-mon[33500]: 5.9 scrub ok 2024-03-26T05:03:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:11 smithi132 ceph-mon[33500]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T05:03:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:11 smithi132 ceph-mon[33500]: 3.3 scrub starts 2024-03-26T05:03:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:11 smithi132 ceph-mon[33500]: 3.3 scrub ok 2024-03-26T05:03:12.142 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:12.174 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:12.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:12.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:12.199 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 34 16 61723 61707 113.412 75.25 0.00443913 0.00880668 2024-03-26T05:03:12.276 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:12.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:12.315 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:12.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:12.416 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:12.453 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:12.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:12.554 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:12.587 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:12.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:12.689 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:12.719 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:12.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:12.821 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:12.860 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:12.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:12.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[29691]: 2.2 scrub starts 2024-03-26T05:03:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[29691]: 2.2 scrub ok 2024-03-26T05:03:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[29691]: 4.2 scrub starts 2024-03-26T05:03:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[29691]: 4.2 scrub ok 2024-03-26T05:03:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[29691]: pgmap v97: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 822 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-26T05:03:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[29691]: 6.10 scrub starts 2024-03-26T05:03:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[29691]: 6.10 scrub ok 2024-03-26T05:03:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:03:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:03:12.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[40404]: 2.2 scrub starts 2024-03-26T05:03:12.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[40404]: 2.2 scrub ok 2024-03-26T05:03:12.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[40404]: 4.2 scrub starts 2024-03-26T05:03:12.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[40404]: 4.2 scrub ok 2024-03-26T05:03:12.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[40404]: pgmap v97: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 822 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-26T05:03:12.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[40404]: 6.10 scrub starts 2024-03-26T05:03:12.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[40404]: 6.10 scrub ok 2024-03-26T05:03:12.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:03:12.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:12 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:03:12.962 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:12.994 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:12.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:12.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:12 smithi132 ceph-mon[33500]: 2.2 scrub starts 2024-03-26T05:03:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:12 smithi132 ceph-mon[33500]: 2.2 scrub ok 2024-03-26T05:03:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:12 smithi132 ceph-mon[33500]: 4.2 scrub starts 2024-03-26T05:03:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:12 smithi132 ceph-mon[33500]: 4.2 scrub ok 2024-03-26T05:03:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:12 smithi132 ceph-mon[33500]: pgmap v97: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 822 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-26T05:03:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:12 smithi132 ceph-mon[33500]: 6.10 scrub starts 2024-03-26T05:03:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:12 smithi132 ceph-mon[33500]: 6.10 scrub ok 2024-03-26T05:03:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:12 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:03:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:12 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:03:13.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:13.138 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:13.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:13.200 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 35 16 63049 63033 112.54 82.875 0.0110302 0.0088804 2024-03-26T05:03:13.240 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:13.265 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:13.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:13.367 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:13.397 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:13.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:13.499 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:13.523 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:13.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:13.626 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:13.650 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:13.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:13.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:13.752 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:13.791 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:13.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[40404]: 5.12 scrub starts 2024-03-26T05:03:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[40404]: 5.12 scrub ok 2024-03-26T05:03:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[40404]: 5.6 scrub starts 2024-03-26T05:03:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[40404]: 5.6 scrub ok 2024-03-26T05:03:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[40404]: 5.18 scrub starts 2024-03-26T05:03:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[40404]: 5.18 scrub ok 2024-03-26T05:03:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[40404]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T05:03:13.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[29691]: 5.12 scrub starts 2024-03-26T05:03:13.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[29691]: 5.12 scrub ok 2024-03-26T05:03:13.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[29691]: 5.6 scrub starts 2024-03-26T05:03:13.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[29691]: 5.6 scrub ok 2024-03-26T05:03:13.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[29691]: 5.18 scrub starts 2024-03-26T05:03:13.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[29691]: 5.18 scrub ok 2024-03-26T05:03:13.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:13 smithi080 ceph-mon[29691]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T05:03:13.892 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:13.916 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:13.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:14.017 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:14.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:13 smithi132 ceph-mon[33500]: 5.12 scrub starts 2024-03-26T05:03:14.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:13 smithi132 ceph-mon[33500]: 5.12 scrub ok 2024-03-26T05:03:14.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:13 smithi132 ceph-mon[33500]: 5.6 scrub starts 2024-03-26T05:03:14.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:13 smithi132 ceph-mon[33500]: 5.6 scrub ok 2024-03-26T05:03:14.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:13 smithi132 ceph-mon[33500]: 5.18 scrub starts 2024-03-26T05:03:14.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:13 smithi132 ceph-mon[33500]: 5.18 scrub ok 2024-03-26T05:03:14.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:13 smithi132 ceph-mon[33500]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T05:03:14.048 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:14.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:14.150 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:14.194 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:14.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:14.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:14.199 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 36 16 64001 63985 111.066 59.5 0.0201552 0.00899877 2024-03-26T05:03:14.296 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:14.318 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:03:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:03:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:14.342 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:14.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:14.443 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:14.488 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:14.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:14.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:14.589 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:14.628 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:14.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:14.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:14.730 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:14.767 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:14.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:14.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[40404]: 5.16 scrub starts 2024-03-26T05:03:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[40404]: 5.16 scrub ok 2024-03-26T05:03:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[40404]: 4.5 scrub starts 2024-03-26T05:03:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[40404]: 4.5 scrub ok 2024-03-26T05:03:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[40404]: pgmap v99: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 691 B/s rd, 58 MiB/s wr, 930 op/s 2024-03-26T05:03:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[40404]: 3.6 scrub starts 2024-03-26T05:03:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[40404]: 3.6 scrub ok 2024-03-26T05:03:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[40404]: 4.0 deep-scrub starts 2024-03-26T05:03:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[40404]: 4.0 deep-scrub ok 2024-03-26T05:03:14.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[29691]: 5.16 scrub starts 2024-03-26T05:03:14.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[29691]: 5.16 scrub ok 2024-03-26T05:03:14.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[29691]: 4.5 scrub starts 2024-03-26T05:03:14.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[29691]: 4.5 scrub ok 2024-03-26T05:03:14.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[29691]: pgmap v99: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 691 B/s rd, 58 MiB/s wr, 930 op/s 2024-03-26T05:03:14.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[29691]: 3.6 scrub starts 2024-03-26T05:03:14.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[29691]: 3.6 scrub ok 2024-03-26T05:03:14.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[29691]: 4.0 deep-scrub starts 2024-03-26T05:03:14.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:14 smithi080 ceph-mon[29691]: 4.0 deep-scrub ok 2024-03-26T05:03:14.870 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:14.897 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:14.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:14.999 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:15.023 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:15.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:15.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:15.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:03:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:03:15.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:14 smithi132 ceph-mon[33500]: 5.16 scrub starts 2024-03-26T05:03:15.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:14 smithi132 ceph-mon[33500]: 5.16 scrub ok 2024-03-26T05:03:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:14 smithi132 ceph-mon[33500]: 4.5 scrub starts 2024-03-26T05:03:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:14 smithi132 ceph-mon[33500]: 4.5 scrub ok 2024-03-26T05:03:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:14 smithi132 ceph-mon[33500]: pgmap v99: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 691 B/s rd, 58 MiB/s wr, 930 op/s 2024-03-26T05:03:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:14 smithi132 ceph-mon[33500]: 3.6 scrub starts 2024-03-26T05:03:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:14 smithi132 ceph-mon[33500]: 3.6 scrub ok 2024-03-26T05:03:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:14 smithi132 ceph-mon[33500]: 4.0 deep-scrub starts 2024-03-26T05:03:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:14 smithi132 ceph-mon[33500]: 4.0 deep-scrub ok 2024-03-26T05:03:15.125 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:15.165 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:15.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:15.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:15.200 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 37 16 65236 65220 110.15 77.1875 0.00670818 0.00907377 2024-03-26T05:03:15.267 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:15.299 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:15.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:15.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:15.401 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:15.433 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:15.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:15.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:15.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:15.565 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:15.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:15.667 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:15.708 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:15.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:15.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:15.810 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:15.841 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:15.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:15.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:15.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[40404]: 6.1c scrub starts 2024-03-26T05:03:15.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[40404]: 6.1c scrub ok 2024-03-26T05:03:15.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[40404]: 5.19 scrub starts 2024-03-26T05:03:15.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[40404]: 5.19 scrub ok 2024-03-26T05:03:15.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[40404]: 3.c scrub starts 2024-03-26T05:03:15.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[40404]: 3.c scrub ok 2024-03-26T05:03:15.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[40404]: 6.17 scrub starts 2024-03-26T05:03:15.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[40404]: 6.17 scrub ok 2024-03-26T05:03:15.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[29691]: 6.1c scrub starts 2024-03-26T05:03:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[29691]: 6.1c scrub ok 2024-03-26T05:03:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[29691]: 5.19 scrub starts 2024-03-26T05:03:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[29691]: 5.19 scrub ok 2024-03-26T05:03:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[29691]: 3.c scrub starts 2024-03-26T05:03:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[29691]: 3.c scrub ok 2024-03-26T05:03:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[29691]: 6.17 scrub starts 2024-03-26T05:03:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[29691]: 6.17 scrub ok 2024-03-26T05:03:15.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:15 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:15.943 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:15.983 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:15.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:15 smithi132 ceph-mon[33500]: 6.1c scrub starts 2024-03-26T05:03:16.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:15 smithi132 ceph-mon[33500]: 6.1c scrub ok 2024-03-26T05:03:16.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:15 smithi132 ceph-mon[33500]: 5.19 scrub starts 2024-03-26T05:03:16.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:15 smithi132 ceph-mon[33500]: 5.19 scrub ok 2024-03-26T05:03:16.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:15 smithi132 ceph-mon[33500]: 3.c scrub starts 2024-03-26T05:03:16.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:15 smithi132 ceph-mon[33500]: 3.c scrub ok 2024-03-26T05:03:16.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:15 smithi132 ceph-mon[33500]: 6.17 scrub starts 2024-03-26T05:03:16.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:15 smithi132 ceph-mon[33500]: 6.17 scrub ok 2024-03-26T05:03:16.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:15 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:16.084 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:16.109 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:16.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:16.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:16.200 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 38 16 66417 66401 109.194 73.8125 0.0179632 0.00915335 2024-03-26T05:03:16.211 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:16.246 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:16.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:16.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:16.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:16.386 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:16.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:16.463 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:03:16.463 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:03:16.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:16.532 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:16.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:16.634 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:16.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:16.669 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:16.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:16.770 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:16.805 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:16.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:16.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:16.814 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[40404]: 3.4 scrub starts 2024-03-26T05:03:16.814 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[40404]: 3.4 scrub ok 2024-03-26T05:03:16.815 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[40404]: 5.b scrub starts 2024-03-26T05:03:16.815 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[40404]: 5.b scrub ok 2024-03-26T05:03:16.815 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[40404]: pgmap v100: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-26T05:03:16.815 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[40404]: 4.b scrub starts 2024-03-26T05:03:16.815 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[40404]: 4.b scrub ok 2024-03-26T05:03:16.815 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[40404]: 6.b scrub starts 2024-03-26T05:03:16.815 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[40404]: 6.b scrub ok 2024-03-26T05:03:16.815 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[40404]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T05:03:16.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[29691]: 3.4 scrub starts 2024-03-26T05:03:16.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[29691]: 3.4 scrub ok 2024-03-26T05:03:16.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[29691]: 5.b scrub starts 2024-03-26T05:03:16.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[29691]: 5.b scrub ok 2024-03-26T05:03:16.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[29691]: pgmap v100: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-26T05:03:16.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[29691]: 4.b scrub starts 2024-03-26T05:03:16.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[29691]: 4.b scrub ok 2024-03-26T05:03:16.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[29691]: 6.b scrub starts 2024-03-26T05:03:16.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[29691]: 6.b scrub ok 2024-03-26T05:03:16.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:16 smithi080 ceph-mon[29691]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T05:03:16.908 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:16.949 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:16.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:16 smithi132 ceph-mon[33500]: 3.4 scrub starts 2024-03-26T05:03:17.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:16 smithi132 ceph-mon[33500]: 3.4 scrub ok 2024-03-26T05:03:17.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:16 smithi132 ceph-mon[33500]: 5.b scrub starts 2024-03-26T05:03:17.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:16 smithi132 ceph-mon[33500]: 5.b scrub ok 2024-03-26T05:03:17.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:16 smithi132 ceph-mon[33500]: pgmap v100: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-26T05:03:17.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:16 smithi132 ceph-mon[33500]: 4.b scrub starts 2024-03-26T05:03:17.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:16 smithi132 ceph-mon[33500]: 4.b scrub ok 2024-03-26T05:03:17.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:16 smithi132 ceph-mon[33500]: 6.b scrub starts 2024-03-26T05:03:17.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:16 smithi132 ceph-mon[33500]: 6.b scrub ok 2024-03-26T05:03:17.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:16 smithi132 ceph-mon[33500]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T05:03:17.051 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:17.092 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:17.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:17.193 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:17.200 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 39 16 67683 67667 108.423 79.125 0.0143163 0.00921779 2024-03-26T05:03:17.223 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:17.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:17.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:17.256 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T05:03:17.325 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:17.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:17.376 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:17.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:17.469 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:17 smithi080 podman[84231]: 2024-03-26 05:03:17.340190111 +0000 UTC m=+8.303252617 volume create c0da353ee31bccddb1cefeadd5794e84b349c21397cc765455fb48b37749935c 2024-03-26T05:03:17.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:17.507 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:17.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:17.609 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:17.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:17.668 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:17.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:17.755 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:17 smithi080 ceph-mon[29691]: 5.1d scrub starts 2024-03-26T05:03:17.756 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:17 smithi080 ceph-mon[29691]: 5.1d scrub ok 2024-03-26T05:03:17.756 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:17 smithi080 ceph-mon[29691]: 4.3 scrub starts 2024-03-26T05:03:17.756 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:17 smithi080 ceph-mon[29691]: 4.3 scrub ok 2024-03-26T05:03:17.756 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:17 smithi080 ceph-mon[29691]: 3.18 scrub starts 2024-03-26T05:03:17.756 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:17 smithi080 ceph-mon[29691]: 3.18 scrub ok 2024-03-26T05:03:17.756 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:17 smithi080 ceph-mon[40404]: 5.1d scrub starts 2024-03-26T05:03:17.756 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:17 smithi080 ceph-mon[40404]: 5.1d scrub ok 2024-03-26T05:03:17.756 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:17 smithi080 ceph-mon[40404]: 4.3 scrub starts 2024-03-26T05:03:17.756 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:17 smithi080 ceph-mon[40404]: 4.3 scrub ok 2024-03-26T05:03:17.756 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:17 smithi080 ceph-mon[40404]: 3.18 scrub starts 2024-03-26T05:03:17.757 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:17 smithi080 ceph-mon[40404]: 3.18 scrub ok 2024-03-26T05:03:17.769 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:17.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:17.815 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:17.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:17.917 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:17.949 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:17.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:18.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:17 smithi132 ceph-mon[33500]: 5.1d scrub starts 2024-03-26T05:03:18.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:17 smithi132 ceph-mon[33500]: 5.1d scrub ok 2024-03-26T05:03:18.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:17 smithi132 ceph-mon[33500]: 4.3 scrub starts 2024-03-26T05:03:18.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:17 smithi132 ceph-mon[33500]: 4.3 scrub ok 2024-03-26T05:03:18.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:17 smithi132 ceph-mon[33500]: 3.18 scrub starts 2024-03-26T05:03:18.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:17 smithi132 ceph-mon[33500]: 3.18 scrub ok 2024-03-26T05:03:18.050 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:18.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:18.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:18.181 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:18.200 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:03:18.208713+0000 min lat: 0.00124227 max lat: 0.239944 avg lat: 0.00929343 2024-03-26T05:03:18.200 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:03:18.200 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 40 16 68855 68839 107.544 73.25 0.0122068 0.00929343 2024-03-26T05:03:18.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:18.212 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:18.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:18.314 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:18.358 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:18.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:18.460 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:18.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:18.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:18.600 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:18.648 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:18.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:18.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:18.750 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:18.781 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:18.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:18.819 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[40404]: 3.5 scrub starts 2024-03-26T05:03:18.820 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[40404]: 3.5 scrub ok 2024-03-26T05:03:18.820 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[40404]: pgmap v102: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 799 B/s rd, 59 MiB/s wr, 937 op/s 2024-03-26T05:03:18.820 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[40404]: 4.1e scrub starts 2024-03-26T05:03:18.820 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[40404]: 4.1e scrub ok 2024-03-26T05:03:18.820 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[40404]: 4.17 scrub starts 2024-03-26T05:03:18.820 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[40404]: 4.17 scrub ok 2024-03-26T05:03:18.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[29691]: 3.5 scrub starts 2024-03-26T05:03:18.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[29691]: 3.5 scrub ok 2024-03-26T05:03:18.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[29691]: pgmap v102: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 799 B/s rd, 59 MiB/s wr, 937 op/s 2024-03-26T05:03:18.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[29691]: 4.1e scrub starts 2024-03-26T05:03:18.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[29691]: 4.1e scrub ok 2024-03-26T05:03:18.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[29691]: 4.17 scrub starts 2024-03-26T05:03:18.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:18 smithi080 ceph-mon[29691]: 4.17 scrub ok 2024-03-26T05:03:18.883 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:18.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:18.936 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:18.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:19.037 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:19.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:18 smithi132 ceph-mon[33500]: 3.5 scrub starts 2024-03-26T05:03:19.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:18 smithi132 ceph-mon[33500]: 3.5 scrub ok 2024-03-26T05:03:19.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:18 smithi132 ceph-mon[33500]: pgmap v102: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 799 B/s rd, 59 MiB/s wr, 937 op/s 2024-03-26T05:03:19.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:18 smithi132 ceph-mon[33500]: 4.1e scrub starts 2024-03-26T05:03:19.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:18 smithi132 ceph-mon[33500]: 4.1e scrub ok 2024-03-26T05:03:19.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:18 smithi132 ceph-mon[33500]: 4.17 scrub starts 2024-03-26T05:03:19.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:18 smithi132 ceph-mon[33500]: 4.17 scrub ok 2024-03-26T05:03:19.081 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:19.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:19.183 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:19.202 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 41 16 70107 70091 106.827 78.25 0.0243893 0.0093551 2024-03-26T05:03:19.214 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:19.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:19.316 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:19.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:19.366 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:19.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:19.467 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:19.495 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:19.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:19.597 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:19.626 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:19.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:19.699 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:19 smithi080 podman[84231]: 2024-03-26 05:03:19.380865762 +0000 UTC m=+10.343928261 container create ee2c30ab9fb52eba29be1c968fe603e27bf98a5ef95e41d82323317d48fd263a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:03:19.700 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[40404]: 6.1d scrub starts 2024-03-26T05:03:19.700 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[29691]: 6.1d scrub starts 2024-03-26T05:03:19.727 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:19.757 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:19.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:19.858 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:19.886 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:19.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:19.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:19.988 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:20.031 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:20.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:20.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:19 smithi132 ceph-mon[33500]: 6.1d scrub starts 2024-03-26T05:03:20.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:19 smithi132 ceph-mon[33500]: 6.1d scrub ok 2024-03-26T05:03:20.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:19 smithi132 ceph-mon[33500]: 3.e scrub starts 2024-03-26T05:03:20.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:19 smithi132 ceph-mon[33500]: 3.e scrub ok 2024-03-26T05:03:20.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:19 smithi132 ceph-mon[33500]: 4.1a scrub starts 2024-03-26T05:03:20.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:19 smithi132 ceph-mon[33500]: 4.1a scrub ok 2024-03-26T05:03:20.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:20.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:20.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[40404]: 6.1d scrub ok 2024-03-26T05:03:20.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[40404]: 3.e scrub starts 2024-03-26T05:03:20.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[40404]: 3.e scrub ok 2024-03-26T05:03:20.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[40404]: 4.1a scrub starts 2024-03-26T05:03:20.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[40404]: 4.1a scrub ok 2024-03-26T05:03:20.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:20.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:20.045 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[29691]: 6.1d scrub ok 2024-03-26T05:03:20.045 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[29691]: 3.e scrub starts 2024-03-26T05:03:20.045 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[29691]: 3.e scrub ok 2024-03-26T05:03:20.045 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[29691]: 4.1a scrub starts 2024-03-26T05:03:20.045 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[29691]: 4.1a scrub ok 2024-03-26T05:03:20.045 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:20.045 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:20.133 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:20.175 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:20.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:20.201 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 42 16 71057 71041 105.698 59.375 0.0256316 0.0094559 2024-03-26T05:03:20.276 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:20.305 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:20.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:20.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:20.387 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:20 smithi080 podman[84231]: 2024-03-26 05:03:20.355747642 +0000 UTC m=+11.318810147 container init ee2c30ab9fb52eba29be1c968fe603e27bf98a5ef95e41d82323317d48fd263a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:03:20.388 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:20 smithi080 podman[84231]: 2024-03-26 05:03:20.368421547 +0000 UTC m=+11.331484052 container start ee2c30ab9fb52eba29be1c968fe603e27bf98a5ef95e41d82323317d48fd263a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:03:20.407 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:20.435 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:20.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:20.537 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:20.565 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:20.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:20.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:20.639 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: 5.c scrub starts 2024-03-26T05:03:20.640 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: 5.c scrub ok 2024-03-26T05:03:20.640 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: 4.a scrub starts 2024-03-26T05:03:20.640 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: 4.a scrub ok 2024-03-26T05:03:20.640 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: 3.9 scrub starts 2024-03-26T05:03:20.640 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: 3.9 scrub ok 2024-03-26T05:03:20.640 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/655235243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:03:20.640 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:03:20.640 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: pgmap v103: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-26T05:03:20.640 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: 6.0 scrub starts 2024-03-26T05:03:20.640 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: 6.0 scrub ok 2024-03-26T05:03:20.640 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: 5.14 deep-scrub starts 2024-03-26T05:03:20.640 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[29691]: 5.14 deep-scrub ok 2024-03-26T05:03:20.641 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[85141]: ts=2024-03-26T05:03:20.395Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T05:03:20.641 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[85141]: ts=2024-03-26T05:03:20.396Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T05:03:20.641 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[85141]: ts=2024-03-26T05:03:20.397Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.80 port=9094 2024-03-26T05:03:20.641 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[85141]: ts=2024-03-26T05:03:20.401Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T05:03:20.641 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[85141]: ts=2024-03-26T05:03:20.456Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T05:03:20.641 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[85141]: ts=2024-03-26T05:03:20.456Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T05:03:20.641 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[85141]: ts=2024-03-26T05:03:20.468Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T05:03:20.641 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[85141]: ts=2024-03-26T05:03:20.468Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T05:03:20.660 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.0 to 1 (10000) 2024-03-26T05:03:20.668 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:20.695 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:20.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:20.753 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:03:20.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:20.806 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:03:20.824 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:20.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:20.848 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:03:20.895 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: 5.c scrub starts 2024-03-26T05:03:20.895 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: 5.c scrub ok 2024-03-26T05:03:20.895 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: 4.a scrub starts 2024-03-26T05:03:20.896 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: 4.a scrub ok 2024-03-26T05:03:20.896 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: 3.9 scrub starts 2024-03-26T05:03:20.896 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: 3.9 scrub ok 2024-03-26T05:03:20.896 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/655235243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:03:20.896 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:03:20.896 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: pgmap v103: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-26T05:03:20.896 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: 6.0 scrub starts 2024-03-26T05:03:20.896 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: 6.0 scrub ok 2024-03-26T05:03:20.896 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: 5.14 deep-scrub starts 2024-03-26T05:03:20.896 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:20 smithi080 ceph-mon[40404]: 5.14 deep-scrub ok 2024-03-26T05:03:20.896 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:20 smithi080 bash[84231]: ee2c30ab9fb52eba29be1c968fe603e27bf98a5ef95e41d82323317d48fd263a 2024-03-26T05:03:20.897 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:20 smithi080 systemd[1]: Started Ceph alertmanager.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:03:20.897 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:03:20.925 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:20.934 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:03:20.951 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:20.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:20.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:20.982 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:03:21.027 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:03:21.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: 5.c scrub starts 2024-03-26T05:03:21.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: 5.c scrub ok 2024-03-26T05:03:21.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: 4.a scrub starts 2024-03-26T05:03:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: 4.a scrub ok 2024-03-26T05:03:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: 3.9 scrub starts 2024-03-26T05:03:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: 3.9 scrub ok 2024-03-26T05:03:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/655235243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:03:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:03:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: pgmap v103: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-26T05:03:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: 6.0 scrub starts 2024-03-26T05:03:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: 6.0 scrub ok 2024-03-26T05:03:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: 5.14 deep-scrub starts 2024-03-26T05:03:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:20 smithi132 ceph-mon[33500]: 5.14 deep-scrub ok 2024-03-26T05:03:21.053 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:21.071 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:03:21.078 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:03:21.079 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:03:21.079 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:03:21.079 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:03:21.079 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:03:21.079 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:03:21.079 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:03:21.079 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:03:21.100 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:21.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:21.201 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 43 16 71980 71964 104.581 57.6875 0.00871142 0.00955734 2024-03-26T05:03:21.201 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:21.240 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:21.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:21.341 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:21.372 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:21.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:21.474 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:21.515 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:21.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:21.540 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T05:03:21.617 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:21.665 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:21.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:21.766 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:21.818 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:21.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:21.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:21.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:03:21.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[29691]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T05:03:21.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2806054315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:03:21.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:21.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:21.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[29691]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T05:03:21.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[29691]: Reconfiguring daemon prometheus.a on smithi132 2024-03-26T05:03:21.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:03:21.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[40404]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T05:03:21.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2806054315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:03:21.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:21.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:21.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[40404]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T05:03:21.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:21 smithi080 ceph-mon[40404]: Reconfiguring daemon prometheus.a on smithi132 2024-03-26T05:03:21.919 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:21.965 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:21.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:21.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:21.990 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:21 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:03:21.990 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:21 smithi132 ceph-mon[33500]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T05:03:21.990 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:21 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2806054315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:03:21.990 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:21.990 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:21.990 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:21 smithi132 ceph-mon[33500]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T05:03:21.990 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:21 smithi132 ceph-mon[33500]: Reconfiguring daemon prometheus.a on smithi132 2024-03-26T05:03:22.068 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:22.109 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:22.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:22.201 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 44 15 73035 73020 103.704 66 0.0112462 0.00963796 2024-03-26T05:03:22.211 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:22.246 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:22.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:22.348 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:22.386 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:22.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:22.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:22.568 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:22.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:22.656 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[85141]: ts=2024-03-26T05:03:22.402Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001016945s 2024-03-26T05:03:22.669 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:22.700 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:22.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:22.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:22.801 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:22.830 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:22.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:22.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:22.932 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:22.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[29691]: 6.1 deep-scrub starts 2024-03-26T05:03:22.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[29691]: 6.1 deep-scrub ok 2024-03-26T05:03:22.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[29691]: 5.a scrub starts 2024-03-26T05:03:22.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[29691]: 5.a scrub ok 2024-03-26T05:03:22.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[29691]: 6.7 scrub starts 2024-03-26T05:03:22.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[29691]: 6.7 scrub ok 2024-03-26T05:03:22.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[29691]: pgmap v105: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-03-26T05:03:22.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[29691]: 3.12 scrub starts 2024-03-26T05:03:22.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[29691]: 3.12 scrub ok 2024-03-26T05:03:22.934 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[40404]: 6.1 deep-scrub starts 2024-03-26T05:03:22.934 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[40404]: 6.1 deep-scrub ok 2024-03-26T05:03:22.934 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[40404]: 5.a scrub starts 2024-03-26T05:03:22.934 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[40404]: 5.a scrub ok 2024-03-26T05:03:22.934 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[40404]: 6.7 scrub starts 2024-03-26T05:03:22.934 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[40404]: 6.7 scrub ok 2024-03-26T05:03:22.935 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[40404]: pgmap v105: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-03-26T05:03:22.935 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[40404]: 3.12 scrub starts 2024-03-26T05:03:22.935 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:22 smithi080 ceph-mon[40404]: 3.12 scrub ok 2024-03-26T05:03:22.964 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:22.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:22.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:22.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:22 smithi132 ceph-mon[33500]: 6.1 deep-scrub starts 2024-03-26T05:03:22.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:22 smithi132 ceph-mon[33500]: 6.1 deep-scrub ok 2024-03-26T05:03:22.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:22 smithi132 ceph-mon[33500]: 5.a scrub starts 2024-03-26T05:03:22.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:22 smithi132 ceph-mon[33500]: 5.a scrub ok 2024-03-26T05:03:22.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:22 smithi132 ceph-mon[33500]: 6.7 scrub starts 2024-03-26T05:03:22.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:22 smithi132 ceph-mon[33500]: 6.7 scrub ok 2024-03-26T05:03:22.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:22 smithi132 ceph-mon[33500]: pgmap v105: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-03-26T05:03:22.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:22 smithi132 ceph-mon[33500]: 3.12 scrub starts 2024-03-26T05:03:22.995 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:22 smithi132 ceph-mon[33500]: 3.12 scrub ok 2024-03-26T05:03:23.065 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:23.092 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:23.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:23.194 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:23.201 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 45 16 74267 74251 103.109 76.9375 0.0344598 0.00969331 2024-03-26T05:03:23.238 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:23.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:23.340 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:23.379 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:23.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:23.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:23.481 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:23.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:23.512 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:23.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:23.613 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:23.656 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:23.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:23.758 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:23.789 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:23.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:23.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:23.891 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:23.897 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:23 smithi132 ceph-mon[33500]: 6.f scrub starts 2024-03-26T05:03:23.897 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:23 smithi132 ceph-mon[33500]: 6.d scrub starts 2024-03-26T05:03:23.897 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:23 smithi132 ceph-mon[33500]: 6.d scrub ok 2024-03-26T05:03:23.898 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:23 smithi132 ceph-mon[33500]: 6.f scrub ok 2024-03-26T05:03:23.898 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:23 smithi132 ceph-mon[33500]: 4.13 scrub starts 2024-03-26T05:03:23.898 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:23 smithi132 ceph-mon[33500]: 4.13 scrub ok 2024-03-26T05:03:23.930 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:23.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:23.938 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:23 smithi080 ceph-mon[40404]: 6.f scrub starts 2024-03-26T05:03:23.938 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:23 smithi080 ceph-mon[40404]: 6.d scrub starts 2024-03-26T05:03:23.938 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:23 smithi080 ceph-mon[40404]: 6.d scrub ok 2024-03-26T05:03:23.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:23 smithi080 ceph-mon[40404]: 6.f scrub ok 2024-03-26T05:03:23.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:23 smithi080 ceph-mon[40404]: 4.13 scrub starts 2024-03-26T05:03:23.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:23 smithi080 ceph-mon[40404]: 4.13 scrub ok 2024-03-26T05:03:23.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:23 smithi080 ceph-mon[29691]: 6.f scrub starts 2024-03-26T05:03:23.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:23 smithi080 ceph-mon[29691]: 6.d scrub starts 2024-03-26T05:03:23.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:23 smithi080 ceph-mon[29691]: 6.d scrub ok 2024-03-26T05:03:23.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:23 smithi080 ceph-mon[29691]: 6.f scrub ok 2024-03-26T05:03:23.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:23 smithi080 ceph-mon[29691]: 4.13 scrub starts 2024-03-26T05:03:23.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:23 smithi080 ceph-mon[29691]: 4.13 scrub ok 2024-03-26T05:03:24.032 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:24.063 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:24.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:24.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:24.194 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:24.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:24.201 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 46 16 75434 75418 102.453 72.9375 0.00722719 0.00975573 2024-03-26T05:03:24.296 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:24.301 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:03:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:03:24.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:24.337 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:24.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:24.360 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:24.426 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:24.439 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:24.489 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:24.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:24.590 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:24.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:24.636 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:24.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:24.687 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:03:24.737 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:24.789 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:24.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:24.890 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:24.924 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:24.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:25.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:25.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-mon[33500]: 5.1b scrub starts 2024-03-26T05:03:25.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-mon[33500]: 5.1b scrub ok 2024-03-26T05:03:25.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:03:25.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:03:25.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-mon[33500]: 6.8 deep-scrub starts 2024-03-26T05:03:25.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-mon[33500]: 6.8 deep-scrub ok 2024-03-26T05:03:25.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-mon[33500]: 4.e scrub starts 2024-03-26T05:03:25.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-mon[33500]: 4.e scrub ok 2024-03-26T05:03:25.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-mon[33500]: pgmap v106: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 746 op/s 2024-03-26T05:03:25.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2252066539' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:03:25.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-mon[33500]: 5.3 deep-scrub starts 2024-03-26T05:03:25.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-mon[33500]: 5.3 deep-scrub ok 2024-03-26T05:03:25.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:03:24 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:03:25.045 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:24 smithi132 systemd[1]: Stopping Ceph prometheus.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:03:25.060 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:25.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:25.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[29691]: 5.1b scrub starts 2024-03-26T05:03:25.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[29691]: 5.1b scrub ok 2024-03-26T05:03:25.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:03:25.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:03:25.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[29691]: 6.8 deep-scrub starts 2024-03-26T05:03:25.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[29691]: 6.8 deep-scrub ok 2024-03-26T05:03:25.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[29691]: 4.e scrub starts 2024-03-26T05:03:25.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[29691]: 4.e scrub ok 2024-03-26T05:03:25.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[29691]: pgmap v106: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 746 op/s 2024-03-26T05:03:25.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2252066539' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:03:25.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[29691]: 5.3 deep-scrub starts 2024-03-26T05:03:25.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[29691]: 5.3 deep-scrub ok 2024-03-26T05:03:25.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[40404]: 5.1b scrub starts 2024-03-26T05:03:25.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[40404]: 5.1b scrub ok 2024-03-26T05:03:25.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:03:25.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:03:25.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[40404]: 6.8 deep-scrub starts 2024-03-26T05:03:25.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[40404]: 6.8 deep-scrub ok 2024-03-26T05:03:25.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[40404]: 4.e scrub starts 2024-03-26T05:03:25.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[40404]: 4.e scrub ok 2024-03-26T05:03:25.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[40404]: pgmap v106: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 746 op/s 2024-03-26T05:03:25.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2252066539' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:03:25.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[40404]: 5.3 deep-scrub starts 2024-03-26T05:03:25.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:24 smithi080 ceph-mon[40404]: 5.3 deep-scrub ok 2024-03-26T05:03:25.161 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:25.201 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:25.202 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 47 16 76601 76585 101.823 72.9375 0.0213638 0.00981623 2024-03-26T05:03:25.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:25.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:25.303 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:25.336 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:25.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:25.438 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:25.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:25.477 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:25.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:25.521 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[61930]: ts=2024-03-26T05:03:25.210Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T05:03:25.521 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[61930]: ts=2024-03-26T05:03:25.210Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T05:03:25.521 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[61930]: ts=2024-03-26T05:03:25.210Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T05:03:25.521 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[61930]: ts=2024-03-26T05:03:25.210Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T05:03:25.522 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[61930]: ts=2024-03-26T05:03:25.210Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T05:03:25.522 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[61930]: ts=2024-03-26T05:03:25.210Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T05:03:25.522 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[61930]: ts=2024-03-26T05:03:25.210Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T05:03:25.522 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[61930]: ts=2024-03-26T05:03:25.210Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T05:03:25.522 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[61930]: ts=2024-03-26T05:03:25.210Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T05:03:25.523 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[61930]: ts=2024-03-26T05:03:25.211Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T05:03:25.523 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[61930]: ts=2024-03-26T05:03:25.211Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T05:03:25.523 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[61930]: ts=2024-03-26T05:03:25.211Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T05:03:25.523 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 podman[69385]: 2024-03-26 05:03:25.220542798 +0000 UTC m=+0.142466016 container died ff3fac33065965f8293c0e05a2514174fa5e119722b0fa2a3ec4dca4db3b9444 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:03:25.578 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:25.617 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:25.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:25.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:25.720 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:25.764 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:25.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:25.793 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 podman[69385]: 2024-03-26 05:03:25.626977259 +0000 UTC m=+0.548900469 container cleanup ff3fac33065965f8293c0e05a2514174fa5e119722b0fa2a3ec4dca4db3b9444 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:03:25.794 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 bash[69385]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T05:03:25.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-mon[33500]: 4.12 scrub starts 2024-03-26T05:03:25.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-mon[33500]: 4.12 scrub ok 2024-03-26T05:03:25.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:03:25.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2252066539' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:03:25.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-mon[33500]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T05:03:25.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:25 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:25.866 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:25.897 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:25.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:25.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:25.999 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:26.028 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:26.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:26.040 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:03:26.064 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:25 smithi080 ceph-mon[40404]: 4.12 scrub starts 2024-03-26T05:03:26.064 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:25 smithi080 ceph-mon[40404]: 4.12 scrub ok 2024-03-26T05:03:26.064 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:25 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:03:26.064 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:25 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2252066539' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:03:26.064 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:25 smithi080 ceph-mon[40404]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T05:03:26.064 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:25 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:26.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:25 smithi080 ceph-mon[29691]: 4.12 scrub starts 2024-03-26T05:03:26.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:25 smithi080 ceph-mon[29691]: 4.12 scrub ok 2024-03-26T05:03:26.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:25 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:03:26.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:25 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2252066539' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:03:26.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:25 smithi080 ceph-mon[29691]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T05:03:26.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:25 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:26.129 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:26.175 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:26.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:26.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:26.202 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 48 16 77704 77688 101.138 68.9375 0.00960594 0.00988257 2024-03-26T05:03:26.277 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:26.293 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:25 smithi132 podman[69396]: 2024-03-26 05:03:25.895660624 +0000 UTC m=+0.675136709 container remove ff3fac33065965f8293c0e05a2514174fa5e119722b0fa2a3ec4dca4db3b9444 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:03:26.293 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:26 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T05:03:26.293 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:26 smithi132 systemd[1]: Stopped Ceph prometheus.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:03:26.293 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:26 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.319s CPU time. 2024-03-26T05:03:26.293 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:26 smithi132 systemd[1]: Starting Ceph prometheus.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:03:26.330 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:26.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:26.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:26.432 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:26.458 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:26.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:26.560 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:26.593 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:26.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:26.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:26.695 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:26.696 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:26 smithi132 podman[69533]: 2024-03-26 05:03:26.234629097 +0000 UTC m=+0.027681694 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T05:03:26.696 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:26 smithi132 podman[69533]: 2024-03-26 05:03:26.572134114 +0000 UTC m=+0.365186682 container create bcfb97148560743163156adcfac48e9ae3c4df8a3db0cfd86d2acd8cef5109c0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:03:26.729 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:26.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:26.832 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:26.874 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:26.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:26.975 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:27.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:27.010 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:27.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:27.111 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:27.147 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:27.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:27.202 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 49 16 78786 78770 100.454 67.625 0.0109699 0.00994956 2024-03-26T05:03:27.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:27.287 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:27.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:27.294 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 podman[69533]: 2024-03-26 05:03:27.108904191 +0000 UTC m=+0.901956762 container init bcfb97148560743163156adcfac48e9ae3c4df8a3db0cfd86d2acd8cef5109c0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:03:27.294 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 podman[69533]: 2024-03-26 05:03:27.114314048 +0000 UTC m=+0.907366617 container start bcfb97148560743163156adcfac48e9ae3c4df8a3db0cfd86d2acd8cef5109c0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:03:27.295 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.158Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T05:03:27.295 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.159Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T05:03:27.295 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.159Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi132 (none))" 2024-03-26T05:03:27.295 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.159Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T05:03:27.295 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.159Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T05:03:27.295 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.163Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T05:03:27.295 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.164Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.165Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.166Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.169Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.170Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.502µs 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.170Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.170Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.171Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.171Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=39.333µs wal_replay_duration=919.002µs wbl_replay_duration=147ns total_replay_duration=995.325µs 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.172Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.172Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.172Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.199Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=26.990878ms db_storage=1.589µs remote_storage=1.428µs web_handler=454ns query_engine=904ns scrape=587.403µs scrape_sd=392.056µs notify=891.092µs notify_sd=36.673µs rules=24.726687ms tracing=7.789µs 2024-03-26T05:03:27.296 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.199Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T05:03:27.297 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:03:27.199Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T05:03:27.389 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:27.436 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:27.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:27.537 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:27.571 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:27.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:27.610 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-mon[33500]: pgmap v108: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-26T05:03:27.610 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:27 smithi132 ceph-mon[33500]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T05:03:27.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-mon[40404]: pgmap v108: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-26T05:03:27.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-mon[40404]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T05:03:27.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-mon[29691]: pgmap v108: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-26T05:03:27.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-mon[29691]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T05:03:27.672 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:27.710 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:27.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:27.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:27.811 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:27.843 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:27.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:27.945 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:28.003 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:28.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:28.004 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:28.044 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 bash[69533]: bcfb97148560743163156adcfac48e9ae3c4df8a3db0cfd86d2acd8cef5109c0 2024-03-26T05:03:28.044 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:03:27 smithi132 systemd[1]: Started Ceph prometheus.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:03:28.066 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:27] ENGINE Bus STOPPING 2024-03-26T05:03:28.066 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:03:28.067 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:27] ENGINE Bus STOPPED 2024-03-26T05:03:28.067 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:27] ENGINE Bus STARTING 2024-03-26T05:03:28.067 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:27] ENGINE Serving on http://:::9283 2024-03-26T05:03:28.067 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:27] ENGINE Bus STARTED 2024-03-26T05:03:28.067 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:27] ENGINE Bus STOPPING 2024-03-26T05:03:28.067 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:03:28.067 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:27] ENGINE Bus STOPPED 2024-03-26T05:03:28.067 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:27 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:27] ENGINE Bus STARTING 2024-03-26T05:03:28.105 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:28.141 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:28.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:28.202 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 50 16 79839 79823 99.7608 65.8125 0.0161892 0.0100182 2024-03-26T05:03:28.243 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:28.303 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:28.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:28.361 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:28] ENGINE Serving on http://:::9283 2024-03-26T05:03:28.361 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:28] ENGINE Bus STARTED 2024-03-26T05:03:28.361 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:28] ENGINE Bus STOPPING 2024-03-26T05:03:28.361 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:28] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:03:28.361 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:28] ENGINE Bus STOPPED 2024-03-26T05:03:28.361 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:28] ENGINE Bus STARTING 2024-03-26T05:03:28.362 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:28] ENGINE Serving on http://:::9283 2024-03-26T05:03:28.362 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:03:28] ENGINE Bus STARTED 2024-03-26T05:03:28.405 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:28.451 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:28.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:28.552 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:28.599 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:28.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:28.700 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:28.735 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:28.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:28.836 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:28.864 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:28.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:28.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: pgmap v110: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 720 B/s rd, 48 MiB/s wr, 767 op/s 2024-03-26T05:03:28.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:28.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:28.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T05:03:28.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T05:03:28.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi080.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T05:03:28.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi080.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T05:03:28.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:28.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T05:03:28.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T05:03:28.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi132.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T05:03:28.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi132.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T05:03:28.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:28.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T05:03:28.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T05:03:28.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi132.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T05:03:28.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi132.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T05:03:28.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:28.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:03:28.872 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: pgmap v110: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 720 B/s rd, 48 MiB/s wr, 767 op/s 2024-03-26T05:03:28.872 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:28.872 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:28.872 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T05:03:28.873 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T05:03:28.873 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi080.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T05:03:28.873 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi080.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T05:03:28.873 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:28.873 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T05:03:28.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T05:03:28.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi132.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T05:03:28.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi132.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T05:03:28.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:28.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T05:03:28.875 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T05:03:28.875 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi132.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T05:03:28.875 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi132.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T05:03:28.875 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:28.875 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:28 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:03:28.933 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:03:28.933 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:03:28.933 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:03:28.933 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:03:28.933 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.934 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.14156.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-03-26T05:03:28.934 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T04:58:08.475648+0000", 2024-03-26T05:03:28.934 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 320.46212955800002, 2024-03-26T05:03:28.934 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.62116560099999996, 2024-03-26T05:03:28.934 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:28.934 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:28.934 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:28.934 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.14156", 2024-03-26T05:03:28.935 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1650870847", 2024-03-26T05:03:28.935 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 1 2024-03-26T05:03:28.935 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.935 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:28.935 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.935 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:28.935 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:08.475648+0000", 2024-03-26T05:03:28.935 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.936 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.936 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:28.936 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:08.475648+0000", 2024-03-26T05:03:28.936 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.936 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.936 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:28.937 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:08.475649+0000", 2024-03-26T05:03:28.937 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4099999999999997e-07 2024-03-26T05:03:28.937 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.937 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.937 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:28.937 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:08.475654+0000", 2024-03-26T05:03:28.937 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1390000000000003e-06 2024-03-26T05:03:28.937 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.938 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.938 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:28.938 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:08.475662+0000", 2024-03-26T05:03:28.938 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1610000000000003e-06 2024-03-26T05:03:28.938 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.938 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.939 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:28.939 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:08.475670+0000", 2024-03-26T05:03:28.939 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0080000000000002e-06 2024-03-26T05:03:28.939 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.939 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.939 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:28.940 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:08.475726+0000", 2024-03-26T05:03:28.940 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5577e-05 2024-03-26T05:03:28.940 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.940 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.940 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:03:28.940 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:08.475736+0000", 2024-03-26T05:03:28.940 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8099999999999992e-06 2024-03-26T05:03:28.941 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.941 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.941 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:28.941 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:09.095556+0000", 2024-03-26T05:03:28.941 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.61981991700000005 2024-03-26T05:03:28.942 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.942 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.942 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:28.942 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:09.095689+0000", 2024-03-26T05:03:28.942 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000133168 2024-03-26T05:03:28.942 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.942 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.943 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:03:28.943 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:09.095870+0000", 2024-03-26T05:03:28.943 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018131500000000001 2024-03-26T05:03:28.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.943 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.943 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:03:28.944 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:09.096489+0000", 2024-03-26T05:03:28.944 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00061915699999999995 2024-03-26T05:03:28.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.944 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.944 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:28.944 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:09.096677+0000", 2024-03-26T05:03:28.945 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018741100000000001 2024-03-26T05:03:28.945 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.945 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.945 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:28.945 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:09.096729+0000", 2024-03-26T05:03:28.945 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2707999999999997e-05 2024-03-26T05:03:28.945 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.946 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.946 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:28.946 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:09.096766+0000", 2024-03-26T05:03:28.946 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6996000000000003e-05 2024-03-26T05:03:28.946 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.946 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.947 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:28.947 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T04:58:09.096814+0000", 2024-03-26T05:03:28.947 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7392999999999998e-05 2024-03-26T05:03:28.947 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.947 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:28.947 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.947 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.947 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.947 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.14430.0:4 3.0 3:00000000::::head [pgnls start_epoch 68] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e68)", 2024-03-26T05:03:28.948 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:00:22.780388+0000", 2024-03-26T05:03:28.948 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 186.15738948500001, 2024-03-26T05:03:28.948 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.56714087899999999, 2024-03-26T05:03:28.948 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:28.948 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:03:28.948 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:28.948 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.14430", 2024-03-26T05:03:28.948 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/2244010176", 2024-03-26T05:03:28.948 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 4 2024-03-26T05:03:28.948 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.948 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:28.948 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.949 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:28.949 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.780388+0000", 2024-03-26T05:03:28.949 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.949 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.949 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.949 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:28.949 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.780388+0000", 2024-03-26T05:03:28.949 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.949 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.949 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.950 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:28.950 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.780390+0000", 2024-03-26T05:03:28.950 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3349999999999999e-06 2024-03-26T05:03:28.950 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.950 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.950 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:28.950 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.780395+0000", 2024-03-26T05:03:28.950 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8420000000000001e-06 2024-03-26T05:03:28.950 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.950 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.950 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:28.951 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.780406+0000", 2024-03-26T05:03:28.951 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0902e-05 2024-03-26T05:03:28.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.951 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.951 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:28.951 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.780419+0000", 2024-03-26T05:03:28.951 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2621e-05 2024-03-26T05:03:28.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.951 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.951 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:28.951 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.780462+0000", 2024-03-26T05:03:28.952 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3149999999999999e-05 2024-03-26T05:03:28.952 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.952 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.952 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:03:28.952 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.780469+0000", 2024-03-26T05:03:28.952 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1539999999999996e-06 2024-03-26T05:03:28.952 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.952 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:23.347072+0000", 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.56660283899999997 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:23.347112+0000", 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0184e-05 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:23.347529+0000", 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000416852 2024-03-26T05:03:28.953 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.954 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:28.954 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.954 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.954 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.954 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24332.0:2 3.0 3:00000000::::head [pgnls start_epoch 68] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e68)", 2024-03-26T05:03:28.954 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:00:22.903388+0000", 2024-03-26T05:03:28.954 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 186.03438987199999, 2024-03-26T05:03:28.954 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.44431402399999997, 2024-03-26T05:03:28.954 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:28.954 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:03:28.955 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:28.955 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24332", 2024-03-26T05:03:28.955 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3541271418", 2024-03-26T05:03:28.955 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 2 2024-03-26T05:03:28.955 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.955 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:28.955 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.955 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:28.955 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:03:28.957 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.903388+0000", 2024-03-26T05:03:28.957 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.957 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.957 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.957 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:28.957 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.903388+0000", 2024-03-26T05:03:28.957 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.957 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.957 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.957 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:28.957 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.903389+0000", 2024-03-26T05:03:28.957 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4839999999999999e-06 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.903394+0000", 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9189999999999998e-06 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.903404+0000", 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.448e-06 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.958 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.903412+0000", 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9400000000000002e-06 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.903463+0000", 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1730000000000001e-05 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:22.903471+0000", 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.61e-06 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.959 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:23.347559+0000", 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.44408747799999998 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:23.347587+0000", 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.8915999999999999e-05 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:00:23.347702+0000", 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000114499 2024-03-26T05:03:28.960 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:12790 7.b 7:d070bdbd:::benchmark_data_smithi080_80848_object12789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:42.757337+0000", 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 46.180441270000003, 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.038790287, 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 12790 2024-03-26T05:03:28.961 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.962 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:28.962 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.962 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:28.962 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.757337+0000", 2024-03-26T05:03:28.962 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.962 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.962 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.962 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.757337+0000", 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.757338+0000", 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3260000000000001e-06 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.757361+0000", 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2710000000000001e-05 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.963 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.757379+0000", 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8338999999999998e-05 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.757385+0000", 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1879999999999999e-06 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.757950+0000", 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00056499800000000004 2024-03-26T05:03:28.964 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.758065+0000", 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0001145 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.758390+0000", 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00032555399999999998 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.759475+0000", 2024-03-26T05:03:28.965 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010850440000000001 2024-03-26T05:03:28.966 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.966 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.966 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:28.966 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.794480+0000", 2024-03-26T05:03:28.966 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.035004652999999997 2024-03-26T05:03:28.966 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.966 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.966 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:28.966 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.796024+0000", 2024-03-26T05:03:28.966 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001543967 2024-03-26T05:03:28.966 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.966 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.796084+0000", 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0488000000000001e-05 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.796127+0000", 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2519999999999999e-05 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:12794 7.b 7:df6a3b95:::benchmark_data_smithi080_80848_object12793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T05:03:28.967 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:42.759379+0000", 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 46.178399216000003, 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.036979917000000001, 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 12794 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.759379+0000", 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.968 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.969 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.969 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:28.969 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.759379+0000", 2024-03-26T05:03:28.969 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.969 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.969 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.969 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:28.969 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.759380+0000", 2024-03-26T05:03:28.969 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2669999999999999e-06 2024-03-26T05:03:28.969 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.969 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.970 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:28.970 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.759401+0000", 2024-03-26T05:03:28.970 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1217e-05 2024-03-26T05:03:28.970 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.970 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.970 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:28.970 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.759412+0000", 2024-03-26T05:03:28.970 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0701e-05 2024-03-26T05:03:28.970 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.970 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.970 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:28.971 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.759529+0000", 2024-03-26T05:03:28.971 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117213 2024-03-26T05:03:28.971 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.971 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.971 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:28.971 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.759595+0000", 2024-03-26T05:03:28.971 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5982000000000006e-05 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.759714+0000", 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118893 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.759861+0000", 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014690800000000001 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.972 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:03:28.973 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.760562+0000", 2024-03-26T05:03:28.973 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00070144499999999998 2024-03-26T05:03:28.973 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.973 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.973 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:28.974 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.794526+0000", 2024-03-26T05:03:28.974 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.033963752999999999 2024-03-26T05:03:28.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.974 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.974 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:28.974 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.796253+0000", 2024-03-26T05:03:28.974 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001726902 2024-03-26T05:03:28.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.974 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.975 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:28.975 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.796314+0000", 2024-03-26T05:03:28.975 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1538999999999995e-05 2024-03-26T05:03:28.975 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.975 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.975 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:28.975 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:42.796358+0000", 2024-03-26T05:03:28.975 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4097000000000001e-05 2024-03-26T05:03:28.975 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.976 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:28.976 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.976 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.976 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:18343 7.b 7:d185c66a:::benchmark_data_smithi080_80848_object18342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:03:28.976 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:45.065440+0000", 2024-03-26T05:03:28.976 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 43.872337905000002, 2024-03-26T05:03:28.976 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.038059441999999999, 2024-03-26T05:03:28.976 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:28.977 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:28.977 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:28.977 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:03:28.977 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:03:28.977 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 18343 2024-03-26T05:03:28.977 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.977 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:28.977 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.978 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:28.978 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.065440+0000", 2024-03-26T05:03:28.978 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.978 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.978 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.978 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:28.979 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.065440+0000", 2024-03-26T05:03:28.979 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.979 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.979 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:28.979 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.065441+0000", 2024-03-26T05:03:28.979 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.37e-07 2024-03-26T05:03:28.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.980 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.980 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:28.980 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.065462+0000", 2024-03-26T05:03:28.980 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1109000000000001e-05 2024-03-26T05:03:28.980 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.980 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.980 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:28.981 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.065475+0000", 2024-03-26T05:03:28.981 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3709999999999999e-05 2024-03-26T05:03:28.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.981 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.981 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:28.981 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.065483+0000", 2024-03-26T05:03:28.981 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0569999999999993e-06 2024-03-26T05:03:28.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.981 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.982 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:28.982 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.066890+0000", 2024-03-26T05:03:28.982 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001406161 2024-03-26T05:03:28.982 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.982 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.982 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:28.982 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.067003+0000", 2024-03-26T05:03:28.982 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011349 2024-03-26T05:03:28.983 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.983 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.983 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:03:28.983 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.067138+0000", 2024-03-26T05:03:28.983 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000134752 2024-03-26T05:03:28.983 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.983 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.983 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:03:28.984 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.068559+0000", 2024-03-26T05:03:28.984 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00142154 2024-03-26T05:03:28.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.984 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.984 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:28.984 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.069669+0000", 2024-03-26T05:03:28.984 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00110932 2024-03-26T05:03:28.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.984 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.985 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:28.985 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.103418+0000", 2024-03-26T05:03:28.985 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.033748899999999998 2024-03-26T05:03:28.985 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.985 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.985 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:28.985 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.103463+0000", 2024-03-26T05:03:28.985 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4864999999999998e-05 2024-03-26T05:03:28.985 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.986 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.986 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:28.986 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:45.103499+0000", 2024-03-26T05:03:28.986 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6801000000000002e-05 2024-03-26T05:03:28.986 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.986 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:28.986 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.987 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.987 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:31392 7.b 7:d4d557cd:::benchmark_data_smithi080_80848_object31391:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:03:28.987 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:52.250478+0000", 2024-03-26T05:03:28.987 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 36.687299957999997, 2024-03-26T05:03:28.987 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13749654, 2024-03-26T05:03:28.987 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:28.987 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:28.987 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:28.987 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:03:28.988 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:03:28.988 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31392 2024-03-26T05:03:28.988 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.988 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:28.988 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.988 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:28.988 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.250478+0000", 2024-03-26T05:03:28.988 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.988 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.988 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.988 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.250478+0000", 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.250479+0000", 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3710000000000001e-06 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.250501+0000", 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2036999999999999e-05 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.989 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.250514+0000", 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.235e-05 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.250531+0000", 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7705000000000001e-05 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.250589+0000", 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8143999999999998e-05 2024-03-26T05:03:28.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.250791+0000", 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020200799999999999 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.251886+0000", 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010942009999999999 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.254501+0000", 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0026157289999999998 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.992 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:28.992 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.254750+0000", 2024-03-26T05:03:28.992 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000248161 2024-03-26T05:03:28.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.992 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.992 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:28.992 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.387855+0000", 2024-03-26T05:03:28.992 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13310535700000001 2024-03-26T05:03:28.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.992 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.387916+0000", 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0677000000000002e-05 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.387974+0000", 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8799999999999999e-05 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.993 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:31413 7.b 7:dbe430b8:::benchmark_data_smithi080_80848_object31412:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:52.262096+0000", 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 36.675682062999996, 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.12608419700000001, 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31413 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.994 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.262096+0000", 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.262096+0000", 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.262098+0000", 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9690000000000001e-06 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.995 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.262121+0000", 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3258e-05 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.262132+0000", 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1387e-05 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.262140+0000", 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5159999999999997e-06 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.997 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:28.997 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.262293+0000", 2024-03-26T05:03:28.997 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000153566 2024-03-26T05:03:28.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.997 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:28.997 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.262419+0000", 2024-03-26T05:03:28.997 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012549100000000001 2024-03-26T05:03:28.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.997 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:03:28.998 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.262548+0000", 2024-03-26T05:03:28.998 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000129078 2024-03-26T05:03:28.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.998 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.998 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:28.998 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.264089+0000", 2024-03-26T05:03:28.998 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0015409620000000001 2024-03-26T05:03:28.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:03:28.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.264134+0000", 2024-03-26T05:03:28.999 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5532999999999999e-05 2024-03-26T05:03:28.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:28.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.388033+0000", 2024-03-26T05:03:28.999 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.123898365 2024-03-26T05:03:28.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:28.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:28.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.388138+0000", 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000104665 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:52.388180+0000", 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2407000000000002e-05 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:42585 7.b 7:d9691296:::benchmark_data_smithi080_80848_object42584:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T05:03:29.000 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:58.464638+0000", 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 30.473140187999999, 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.041361637, 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 42585 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.464638+0000", 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.001 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.002 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.002 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:29.002 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.464638+0000", 2024-03-26T05:03:29.002 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.002 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.002 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.002 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:29.002 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.464639+0000", 2024-03-26T05:03:29.002 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.592e-06 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.464672+0000", 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2876000000000002e-05 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.464682+0000", 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7429999999999997e-06 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:29.003 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.464688+0000", 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5640000000000002e-06 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.465072+0000", 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00038375300000000002 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.465210+0000", 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013835200000000001 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.004 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.465352+0000", 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014126399999999999 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.467020+0000", 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0016686260000000001 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.473050+0000", 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0060295069999999999 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.505852+0000", 2024-03-26T05:03:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.032801645999999997 2024-03-26T05:03:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.505952+0000", 2024-03-26T05:03:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000100989 2024-03-26T05:03:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.006 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.007 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:29.008 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: pgmap v110: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 720 B/s rd, 48 MiB/s wr, 767 op/s 2024-03-26T05:03:29.008 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:29.008 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:29.008 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T05:03:29.008 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T05:03:29.008 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi080.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T05:03:29.008 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi080.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T05:03:29.008 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:29.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T05:03:29.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T05:03:29.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi132.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T05:03:29.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi132.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T05:03:29.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:29.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T05:03:29.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T05:03:29.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi132.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T05:03:29.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi132.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T05:03:29.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:29.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:28 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:03:29.010 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:58.505999+0000", 2024-03-26T05:03:29.010 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6724999999999997e-05 2024-03-26T05:03:29.010 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.010 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.24701.0:77556 7.9 e122/103 7:9fe3aa58:::benchmark_data_smithi080_80848_object77555:head v 122'4766, mlcod=122'4762)", 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:03:26.003616+0000", 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 2.9341614759999999, 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.100147232, 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:29.011 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.003616+0000", 2024-03-26T05:03:29.012 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.012 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.012 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.012 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:29.012 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.003616+0000", 2024-03-26T05:03:29.012 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.012 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.012 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.012 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:29.012 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.003617+0000", 2024-03-26T05:03:29.012 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7100000000000011e-07 2024-03-26T05:03:29.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.013 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:29.013 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.003644+0000", 2024-03-26T05:03:29.013 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6794000000000001e-05 2024-03-26T05:03:29.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.013 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:29.013 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.003656+0000", 2024-03-26T05:03:29.013 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2075e-05 2024-03-26T05:03:29.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.003664+0000", 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8990000000000001e-06 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.010845+0000", 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0071811440000000004 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.010864+0000", 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8689e-05 2024-03-26T05:03:29.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.103705+0000", 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.092841581000000006 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.103764+0000", 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8078999999999997e-05 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.015 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.24701.0:77560 7.9 e122/103 7:94da3b4a:::benchmark_data_smithi080_80848_object77559:head v 122'4767, mlcod=122'4762)", 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:03:26.009505+0000", 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 2.9282723320000001, 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.094289889000000002, 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.009505+0000", 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.016 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.009505+0000", 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.009506+0000", 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3799999999999996e-07 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.009528+0000", 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1336999999999999e-05 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.017 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.009538+0000", 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0543999999999999e-05 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.009546+0000", 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4449999999999999e-06 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.013283+0000", 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0037372740000000001 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.018 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.013301+0000", 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8352999999999999e-05 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.103769+0000", 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.090468152999999996 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.103795+0000", 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6044999999999999e-05 2024-03-26T05:03:29.019 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.020 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.020 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.020 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.020 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.020 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.24701.0:77568 7.9 e122/103 7:9361f97f:::benchmark_data_smithi080_80848_object77567:head v 122'4768, mlcod=122'4764)", 2024-03-26T05:03:29.020 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:03:26.014259+0000", 2024-03-26T05:03:29.020 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 2.923518445, 2024-03-26T05:03:29.020 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.089562329999999996, 2024-03-26T05:03:29.020 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:29.020 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:29.020 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.014259+0000", 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.014259+0000", 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.014260+0000", 2024-03-26T05:03:29.021 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8000000000000004e-07 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.014283+0000", 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2541999999999999e-05 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.014294+0000", 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1036000000000001e-05 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:29.022 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.014301+0000", 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1450000000000002e-06 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.015451+0000", 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0011504709999999999 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.015470+0000", 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8465e-05 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:29.023 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.103797+0000", 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.088327388000000007 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.103822+0000", 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4403e-05 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:77571 7.b 7:d24b9d76:::benchmark_data_smithi080_80848_object77570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-26T05:03:29.024 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:03:26.016370+0000", 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 2.9214073759999999, 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.088028844999999994, 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 77571 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.016370+0000", 2024-03-26T05:03:29.025 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.016370+0000", 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.016372+0000", 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.389e-06 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.026 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.016392+0000", 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0366000000000001e-05 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.016405+0000", 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2669e-05 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.016413+0000", 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2940000000000006e-06 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.027 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.028 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:29.028 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.016470+0000", 2024-03-26T05:03:29.028 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7188999999999997e-05 2024-03-26T05:03:29.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.028 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:29.028 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.016583+0000", 2024-03-26T05:03:29.028 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000112216 2024-03-26T05:03:29.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.029 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:03:29.029 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.016708+0000", 2024-03-26T05:03:29.029 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012521599999999999 2024-03-26T05:03:29.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.029 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:03:29.029 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.104190+0000", 2024-03-26T05:03:29.029 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.087482040999999997 2024-03-26T05:03:29.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.029 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:29.029 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.104230+0000", 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0522000000000003e-05 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.104288+0000", 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7478000000000001e-05 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.104347+0000", 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8995e-05 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.030 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.104399+0000", 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2469999999999997e-05 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.24701.0:77570 7.2 e122/103 7:4ba4430d:::benchmark_data_smithi080_80848_object77569:head v 122'4862, mlcod=122'4860)", 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:03:26.017081+0000", 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 2.9206970249999999, 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.092830847999999994, 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:29.031 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.017081+0000", 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.017081+0000", 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.017082+0000", 2024-03-26T05:03:29.032 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7600000000000006e-07 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.017135+0000", 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2916e-05 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.017145+0000", 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0721e-05 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:29.033 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.017152+0000", 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6950000000000003e-06 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.017267+0000", 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115302 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.017286+0000", 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8655999999999999e-05 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.035 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:29.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.109863+0000", 2024-03-26T05:03:29.035 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.092576627999999994 2024-03-26T05:03:29.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.035 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.109912+0000", 2024-03-26T05:03:29.035 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8953999999999999e-05 2024-03-26T05:03:29.035 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.035 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.035 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop_reply(client.24701.0:77571 7.b e122/103 ondisk, result = 0)", 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:03:26.019752+0000", 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 2.9180258920000002, 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.084653696, 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.019752+0000", 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.036 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.037 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.037 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:29.037 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.019752+0000", 2024-03-26T05:03:29.037 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.037 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.037 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:29.037 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.019753+0000", 2024-03-26T05:03:29.037 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2200000000000002e-07 2024-03-26T05:03:29.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.038 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:29.038 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.019758+0000", 2024-03-26T05:03:29.038 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1549999999999998e-06 2024-03-26T05:03:29.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.038 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:29.038 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.103670+0000", 2024-03-26T05:03:29.038 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.083911631 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.103675+0000", 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5119999999999999e-06 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.104276+0000", 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00060102799999999998 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.104282+0000", 2024-03-26T05:03:29.039 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.285e-06 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.104406+0000", 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000123163 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop_reply(client.24701.0:77571 7.b e122/103 ondisk, result = 0)", 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:03:26.020755+0000", 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 2.9170232110000001, 2024-03-26T05:03:29.040 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.083484716, 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.020755+0000", 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.020755+0000", 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.042 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:29.042 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.020755+0000", 2024-03-26T05:03:29.042 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1999999999999998e-07 2024-03-26T05:03:29.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.042 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.042 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:29.042 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.020761+0000", 2024-03-26T05:03:29.042 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3079999999999998e-06 2024-03-26T05:03:29.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.042 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.042 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.103643+0000", 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.082882540000000005 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.103653+0000", 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0011e-05 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.104216+0000", 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00056284 2024-03-26T05:03:29.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.104224+0000", 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1689999999999992e-06 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:26.104239+0000", 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5028000000000001e-05 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.044 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:78962 7.b 7:d4532509:::benchmark_data_smithi080_80848_object78961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:03:27.359366+0000", 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 1.5784122229999999, 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.041785911000000002, 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 78962 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:29.045 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359366+0000", 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359366+0000", 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359367+0000", 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5099999999999999e-06 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359393+0000", 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5907000000000001e-05 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359404+0000", 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0584000000000001e-05 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359411+0000", 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.114e-06 2024-03-26T05:03:29.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359753+0000", 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000342398 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359901+0000", 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014842699999999999 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.360053+0000", 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015189500000000001 2024-03-26T05:03:29.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.366434+0000", 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0063808160000000001 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.366590+0000", 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015537400000000001 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.401081+0000", 2024-03-26T05:03:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.034491755999999998 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.401121+0000", 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9517000000000001e-05 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.401151+0000", 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.0613e-05 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.050 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:78964 7.b 7:d8f00fe9:::benchmark_data_smithi080_80848_object78963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:03:27.359482+0000", 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 1.5782959170000002, 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.048186329999999999, 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 78964 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:29.051 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359482+0000", 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359482+0000", 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359482+0000", 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2099999999999996e-07 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.053 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:29.053 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359507+0000", 2024-03-26T05:03:29.053 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4147000000000001e-05 2024-03-26T05:03:29.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.053 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.053 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:29.053 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359518+0000", 2024-03-26T05:03:29.053 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1837e-05 2024-03-26T05:03:29.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.054 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.054 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:29.054 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.359525+0000", 2024-03-26T05:03:29.054 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8510000000000001e-06 2024-03-26T05:03:29.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.054 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.054 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:29.054 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.363196+0000", 2024-03-26T05:03:29.054 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036702150000000001 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.363315+0000", 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011968600000000001 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.363542+0000", 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000226937 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.055 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.366594+0000", 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0030519150000000001 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.366661+0000", 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7130000000000003e-05 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.407363+0000", 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.040701744999999998 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.056 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.407543+0000", 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018049599999999999 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.407668+0000", 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012475 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.057 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:78979 7.b 7:dfb04437:::benchmark_data_smithi080_80848_object78978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:03:27.372691+0000", 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 1.56508636, 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.035134457000000001, 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 78979 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:29.058 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.372691+0000", 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.372691+0000", 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.372693+0000", 2024-03-26T05:03:29.059 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.609e-06 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.372714+0000", 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1197000000000001e-05 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.372731+0000", 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6450999999999999e-05 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.060 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.372739+0000", 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0299999999999994e-06 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.372900+0000", 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016141899999999999 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.373207+0000", 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00030639399999999998 2024-03-26T05:03:29.061 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.062 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.062 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:03:29.062 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.373574+0000", 2024-03-26T05:03:29.062 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00036745300000000001 2024-03-26T05:03:29.062 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.062 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.062 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:29.062 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.376583+0000", 2024-03-26T05:03:29.062 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0030094760000000001 2024-03-26T05:03:29.062 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.062 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.376663+0000", 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9404000000000001e-05 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.407741+0000", 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.031077846999999999 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.407793+0000", 2024-03-26T05:03:29.063 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2561e-05 2024-03-26T05:03:29.064 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.064 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.064 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.064 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:27.407826+0000", 2024-03-26T05:03:29.064 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2616000000000001e-05 2024-03-26T05:03:29.064 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.064 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.064 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.064 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.064 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.065 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:79946 7.b 7:d6089425:::benchmark_data_smithi080_80848_object79945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T05:03:29.065 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:03:28.327908+0000", 2024-03-26T05:03:29.065 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.609869734, 2024-03-26T05:03:29.065 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.033283977999999999, 2024-03-26T05:03:29.065 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:03:29.065 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:03:29.065 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:03:29.065 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:03:29.065 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:03:29.065 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 79946 2024-03-26T05:03:29.066 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.066 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:03:29.066 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.066 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:03:29.066 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.327908+0000", 2024-03-26T05:03:29.066 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.066 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.066 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.067 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:03:29.067 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.327908+0000", 2024-03-26T05:03:29.067 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:03:29.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.067 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.067 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:03:29.067 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.327909+0000", 2024-03-26T05:03:29.067 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4600000000000005e-07 2024-03-26T05:03:29.068 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.068 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.068 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:03:29.068 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.327934+0000", 2024-03-26T05:03:29.068 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5222999999999999e-05 2024-03-26T05:03:29.068 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.068 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.068 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:03:29.068 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.327943+0000", 2024-03-26T05:03:29.069 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0399999999999998e-06 2024-03-26T05:03:29.069 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.069 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.069 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:03:29.069 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.327949+0000", 2024-03-26T05:03:29.069 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3489999999999997e-06 2024-03-26T05:03:29.069 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.070 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.070 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:03:29.070 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.331388+0000", 2024-03-26T05:03:29.070 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0034384279999999999 2024-03-26T05:03:29.070 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.070 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.071 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:03:29.071 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.331762+0000", 2024-03-26T05:03:29.071 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00037453999999999997 2024-03-26T05:03:29.071 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.071 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.071 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:03:29.071 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.332232+0000", 2024-03-26T05:03:29.072 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00046972300000000002 2024-03-26T05:03:29.072 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.072 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.072 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:03:29.072 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.337827+0000", 2024-03-26T05:03:29.072 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055952349999999996 2024-03-26T05:03:29.073 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.073 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.073 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:29.073 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.346106+0000", 2024-03-26T05:03:29.073 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0082792379999999995 2024-03-26T05:03:29.073 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.073 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.074 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:03:29.074 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.361027+0000", 2024-03-26T05:03:29.074 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014921131000000001 2024-03-26T05:03:29.074 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.074 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.074 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:03:29.074 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.361129+0000", 2024-03-26T05:03:29.074 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000101184 2024-03-26T05:03:29.075 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:03:29.075 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:03:29.075 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:03:29.075 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:03:28.361192+0000", 2024-03-26T05:03:29.075 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3341000000000002e-05 2024-03-26T05:03:29.075 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.076 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.076 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.076 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:03:29.076 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:03:29.076 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:03:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:29.122 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:29.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:29.202 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 51 16 80731 80715 98.8977 55.75 0.0163908 0.010106 2024-03-26T05:03:29.223 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:29.251 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:29.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:29.352 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:29.385 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:29.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:29.486 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:29.533 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:29.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:29.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:29.640 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:29.685 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:29.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:29.786 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:29.823 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:29.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:29.925 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:29.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:29.972 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:29.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:30.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:29 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3418725847' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:03:30.072 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:29 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3418725847' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:03:30.072 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:29 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3418725847' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:03:30.073 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:30.105 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:30.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:30.202 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 52 16 81872 81856 98.3672 71.3125 0.0118135 0.0101606 2024-03-26T05:03:30.205 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:30.245 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:30.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:30.347 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:30.386 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:30.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:30.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:30.532 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:30.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:30.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:30.634 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:30.662 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:30.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:30.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:30.741 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:03:30 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[85141]: ts=2024-03-26T05:03:30.405Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003819502s 2024-03-26T05:03:30.763 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:30.815 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:30.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:30.920 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:30.957 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:30.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:30.970 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T05:03:31.027 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:30 smithi132 ceph-mon[33500]: pgmap v111: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 920 op/s 2024-03-26T05:03:31.027 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:30 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3418725847' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:03:31.027 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:30 smithi132 ceph-mon[33500]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T05:03:31.060 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:31.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:30 smithi080 ceph-mon[29691]: pgmap v111: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 920 op/s 2024-03-26T05:03:31.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:30 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3418725847' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:03:31.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:30 smithi080 ceph-mon[29691]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T05:03:31.064 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:30 smithi080 ceph-mon[40404]: pgmap v111: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 920 op/s 2024-03-26T05:03:31.064 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:30 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3418725847' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:03:31.064 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:30 smithi080 ceph-mon[40404]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T05:03:31.109 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:31.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:31.203 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 53 16 82944 82928 97.7753 67 0.0250395 0.0102208 2024-03-26T05:03:31.211 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:31.258 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:31.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:31.319 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:03:31.360 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:31.367 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:03:31.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:31.411 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:31.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:31.413 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:03:31.457 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:03:31.496 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:03:31.512 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:31.536 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:03:31.556 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:31.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:31.575 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:03:31.615 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:03:31.625 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:03:31.625 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:03:31.625 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:03:31.625 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:03:31.625 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:03:31.626 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:03:31.626 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:03:31.626 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:03:31.659 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:31.689 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:31.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:31.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:31.791 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:31.819 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:31.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:31.835 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:31 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/35318987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:03:31.835 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:31 smithi080 ceph-mon[29691]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T05:03:31.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:31 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/35318987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:03:31.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:31 smithi080 ceph-mon[40404]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T05:03:31.883 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T05:03:31.921 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:31.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:31.955 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:31.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:31 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/35318987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:03:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:31 smithi132 ceph-mon[33500]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T05:03:32.057 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:32.103 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:32.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:32.203 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 54 16 83869 83853 97.0348 57.8125 0.0236735 0.0102999 2024-03-26T05:03:32.205 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:32.233 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:32.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:32.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:32.335 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:32.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:32.365 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:32.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:32.467 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:32.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:32.514 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:32.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:32.615 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:32.646 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:32.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:32.748 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:32.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:32.791 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:32.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:32.892 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:32.922 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:32.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:33.025 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:33.054 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:33.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:33.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:33.156 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:33.207 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T05:03:33.207 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 55 16 84990 84974 96.541 70.0625 0.0192718 0.0103531 2024-03-26T05:03:33.219 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:33.231 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:33.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:33.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:33.333 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:33.382 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:33.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:33.385 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:33 smithi080 ceph-mon[29691]: pgmap v114: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 885 op/s 2024-03-26T05:03:33.385 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:33 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:33.385 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:33 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:33.385 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:33 smithi080 ceph-mon[40404]: pgmap v114: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 885 op/s 2024-03-26T05:03:33.385 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:33 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:33.385 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:33 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:33.485 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:33.518 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:33.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:33.550 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:33.621 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:33.657 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:33.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:33.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:33.759 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:33.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:33 smithi132 ceph-mon[33500]: pgmap v114: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 885 op/s 2024-03-26T05:03:33.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:33 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:33.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:33 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:33.806 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:33.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:33.907 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:33.944 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:33.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:34.027 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:03:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:03:34.046 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:34.076 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:34.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:34.178 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:34.206 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:34.207 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 56 16 86001 85985 95.9418 63.1875 0.0141088 0.0104182 2024-03-26T05:03:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:34.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:34.309 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:34.336 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:34.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:34.438 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:34.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:34.482 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:34.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:34.532 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:03:34.584 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:34.613 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:34.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:34.715 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:34.744 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:34.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:34.846 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:34.900 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:34.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:35.001 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:35.052 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:35.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:35.153 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:35.194 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:35.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:35.207 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 57 16 87060 87044 95.4198 66.1875 0.00733773 0.0104742 2024-03-26T05:03:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:34 smithi132 ceph-mon[33500]: pgmap v115: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 691 B/s rd, 45 MiB/s wr, 717 op/s 2024-03-26T05:03:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:34 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:34 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:35.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:03:34 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:03:35.295 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:35.311 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:34 smithi080 ceph-mon[40404]: pgmap v115: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 691 B/s rd, 45 MiB/s wr, 717 op/s 2024-03-26T05:03:35.311 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:34 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:35.312 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:34 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:35.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:34 smithi080 ceph-mon[29691]: pgmap v115: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 691 B/s rd, 45 MiB/s wr, 717 op/s 2024-03-26T05:03:35.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:34 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:35.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:34 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:35.325 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:35.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:35.427 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:35.455 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:35.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:35.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:35.557 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:35.602 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:35.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:35.703 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:35.733 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:35.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:35.835 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:35.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:35.881 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:35.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:35.983 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:36.016 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:36.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:36.117 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:36.156 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:36.167 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:36.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:36.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:36.207 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 58 16 88106 88090 94.9017 65.375 0.0377172 0.0105317 2024-03-26T05:03:36.254 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:35 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:36.254 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:35 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:36.269 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:36.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:35 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:36.319 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:36.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:36.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:36.450 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:36.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:36.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:36.552 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:36.599 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:36.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:36.633 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:36.701 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:36.728 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:36.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:36.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:36.830 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:36.861 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:36.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:36.962 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:37.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:37.002 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:37.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:37.103 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:37.147 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:37.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:37.166 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:36 smithi080 ceph-mon[29691]: pgmap v116: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-26T05:03:37.166 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:36 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:03:37.166 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:36 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:03:37.166 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:36 smithi080 ceph-mon[29691]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T05:03:37.166 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:36 smithi080 ceph-mon[29691]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T05:03:37.166 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:36 smithi080 ceph-mon[40404]: pgmap v116: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-26T05:03:37.167 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:36 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:03:37.167 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:36 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:03:37.167 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:36 smithi080 ceph-mon[40404]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T05:03:37.167 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:36 smithi080 ceph-mon[40404]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T05:03:37.207 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 59 16 89179 89163 94.4298 67.0625 0.00670696 0.0105851 2024-03-26T05:03:37.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:36 smithi132 ceph-mon[33500]: pgmap v116: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-26T05:03:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:36 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:03:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:36 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:03:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:36 smithi132 ceph-mon[33500]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T05:03:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:36 smithi132 ceph-mon[33500]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T05:03:37.298 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:37.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:37.400 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:37.443 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:37.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:37.545 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:37.572 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:37.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:37.674 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:37.703 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:37.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:37.805 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:37.834 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:37.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:37.936 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:37.975 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:37.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:38.076 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:38.105 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:38.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:38.207 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:38.208 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:03:38.216413+0000 min lat: 0.00124227 max lat: 0.239944 avg lat: 0.0106421 2024-03-26T05:03:38.208 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:03:38.208 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 60 16 90206 90190 93.9258 64.1875 0.0171602 0.0106421 2024-03-26T05:03:38.234 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:38.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:38.334 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:03:38.334 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:03:38.336 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:38.372 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:38.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:38.474 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:38.504 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:38.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:38.606 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:38.635 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:38.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:38.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:38.737 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:38.775 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:38.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:38.876 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:38.905 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:38.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:38.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:39.007 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:39.039 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:39.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:39.140 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:39.170 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:39.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:39.208 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 61 16 91319 91303 93.5263 69.5625 0.0140458 0.0106877 2024-03-26T05:03:39.270 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:38 smithi132 ceph-mon[33500]: pgmap v119: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 799 B/s rd, 51 MiB/s wr, 822 op/s 2024-03-26T05:03:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:38 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:03:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:38 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:03:39.294 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:38 smithi080 ceph-mon[29691]: pgmap v119: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 799 B/s rd, 51 MiB/s wr, 822 op/s 2024-03-26T05:03:39.294 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:38 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:03:39.294 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:38 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:03:39.294 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:38 smithi080 ceph-mon[40404]: pgmap v119: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 799 B/s rd, 51 MiB/s wr, 822 op/s 2024-03-26T05:03:39.294 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:38 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:03:39.295 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:38 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:03:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:39.333 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:39.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:39.434 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:39.483 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:39.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:39.584 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:39.614 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:39.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:39.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:39.716 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:39.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:39.757 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:39.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:39.858 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:39.884 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:39.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:39.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:39.986 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:40.016 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:40.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:40.118 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:40.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:40.150 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:40.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:40.208 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 62 16 92365 92349 93.0722 65.375 0.00781799 0.0107397 2024-03-26T05:03:40.251 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:40.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:40.292 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:40.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:40.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:40 smithi132 ceph-mon[33500]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T05:03:40.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:40 smithi080 ceph-mon[40404]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T05:03:40.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:40 smithi080 ceph-mon[29691]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T05:03:40.393 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:40.433 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:40.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:40.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:40.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:40.563 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:40.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:40.665 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:40.709 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:40.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:40.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:40.811 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:40.842 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:40.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:40.944 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:40.985 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:40.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:40.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:41.087 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:41.115 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:41.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:41.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:41 smithi080 ceph-mon[29691]: pgmap v121: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T05:03:41.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:41 smithi080 ceph-mon[40404]: pgmap v121: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T05:03:41.208 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 63 16 93407 93391 92.6286 65.125 0.0161484 0.0107906 2024-03-26T05:03:41.217 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:41.252 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:41.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:41.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:41 smithi132 ceph-mon[33500]: pgmap v121: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T05:03:41.354 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:41.401 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:41.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:41.507 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:41.535 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:41.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:41.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:41.637 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:41.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:41.674 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:41.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:41.775 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:41.808 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:41.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:41.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:41.910 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:41.951 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:41.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:42.052 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:42.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:42.088 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:42.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:42.190 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:42.209 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 64 16 94268 94252 92.0209 53.8125 0.0315849 0.010862 2024-03-26T05:03:42.223 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:42.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:42.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:42.325 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:42.355 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:42.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:42.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:42.375 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:03:42.458 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:42.485 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:42.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:42.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:42.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:42.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:42 smithi080 ceph-mon[40404]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T05:03:42.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:42 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/391607023' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:03:42.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:42 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:03:42.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:42 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:03:42.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:42 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:03:42.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:42 smithi080 ceph-mon[29691]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T05:03:42.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:42 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/391607023' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:03:42.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:42 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:03:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:42 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:03:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:42 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:03:42.629 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:42.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:42.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:42.731 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:42.761 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:42.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:42.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:42.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:42 smithi132 ceph-mon[33500]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T05:03:42.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:42 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/391607023' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:03:42.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:42 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:03:42.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:42 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:03:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:42 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:03:42.863 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:42.894 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:42.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:42.995 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:43.026 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:43.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:43.128 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:43.169 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:43.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:43.209 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 65 16 95239 95223 91.5388 60.6875 0.012309 0.01092 2024-03-26T05:03:43.269 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:43.302 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:43.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:43.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:43.404 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:43.445 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:43.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:43.546 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:43.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:43.581 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:43.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:43.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:43 smithi080 ceph-mon[29691]: pgmap v123: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 889 B/s rd, 58 MiB/s wr, 922 op/s 2024-03-26T05:03:43.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:43 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:03:43.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:43 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:03:43.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:43 smithi080 ceph-mon[29691]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T05:03:43.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:43 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2412972898' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:03:43.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:43 smithi080 ceph-mon[40404]: pgmap v123: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 889 B/s rd, 58 MiB/s wr, 922 op/s 2024-03-26T05:03:43.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:43 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:03:43.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:43 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:03:43.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:43 smithi080 ceph-mon[40404]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T05:03:43.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:43 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2412972898' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:03:43.681 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:43.720 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:43.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:43.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:43 smithi132 ceph-mon[33500]: pgmap v123: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 889 B/s rd, 58 MiB/s wr, 922 op/s 2024-03-26T05:03:43.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:43 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:03:43.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:43 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:03:43.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:43 smithi132 ceph-mon[33500]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T05:03:43.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:43 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2412972898' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:03:43.821 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:43.865 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:43.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:43.967 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:44.006 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:44.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:44.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:44.107 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:44.138 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:44.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:44.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:44.209 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 66 16 96353 96337 91.2068 69.625 0.00735484 0.0109594 2024-03-26T05:03:44.239 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:44.277 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:44.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:44.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:44.367 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:03:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:03:44.379 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:44.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:44.425 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:44.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:44.526 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:44.556 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:44.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:44.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:44.658 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:44.688 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:44.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:44.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:44.790 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:44.829 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:44.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:44.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:44.932 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:44.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:45.065 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:45.094 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:45.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:45.196 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:45.209 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 67 16 97427 97411 90.8473 67.125 0.0196557 0.0110025 2024-03-26T05:03:45.224 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:45.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:45.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:45.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:03:44 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:03:45.327 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:45.355 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:45.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:45.458 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:45.486 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:45 smithi080 ceph-mon[29691]: pgmap v125: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 884 op/s 2024-03-26T05:03:45.487 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:45 smithi080 ceph-mon[40404]: pgmap v125: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 884 op/s 2024-03-26T05:03:45.488 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:45.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:45.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:45.535 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T05:03:45.591 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:45.620 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:03:45.620 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:03:45.620 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:03:45.620 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:03:45.621 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:03:45.621 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:03:45.621 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:03:45.621 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:03:45.621 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:03:45.621 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:03:45.621 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:03:45.621 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:03:45.621 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:03:45.621 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:03:45.621 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:03:45.621 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:03:45.624 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:45.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:45.726 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:45.755 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:45.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:45.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:45.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:45 smithi132 ceph-mon[33500]: pgmap v125: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 884 op/s 2024-03-26T05:03:45.859 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:45.913 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:45.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:46.015 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:46.065 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:46.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:46.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:46.209 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 68 16 98434 98418 90.4368 62.9375 0.00998084 0.0110518 2024-03-26T05:03:46.234 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:46.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:46.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:46.336 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:46.366 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:46.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:46.468 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:46.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:46.496 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:46.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:46.597 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:46.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:46 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:46.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:46 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:46.642 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:46.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:46.743 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:46.773 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:46.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:46.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:46 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:46.875 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:46.913 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:46.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:46.931 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:03:46.950 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:47.014 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:47.041 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:47.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:47.144 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:47.192 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:47.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:47.212 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 69 16 99360 99344 89.9613 57.875 0.0223405 0.0111106 2024-03-26T05:03:47.294 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:47.355 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:47.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:47.373 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T05:03:47.457 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:47.487 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:47.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:47.589 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:47.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:47 smithi080 ceph-mon[29691]: pgmap v126: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-26T05:03:47.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:47 smithi080 ceph-mon[29691]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T05:03:47.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:47 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1232731583' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:03:47.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:47 smithi080 ceph-mon[40404]: pgmap v126: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-26T05:03:47.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:47 smithi080 ceph-mon[40404]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T05:03:47.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:47 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1232731583' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:03:47.633 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:47.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:47.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:47.735 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:47.766 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:47.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:47 smithi132 ceph-mon[33500]: pgmap v126: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-26T05:03:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:47 smithi132 ceph-mon[33500]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T05:03:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:47 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1232731583' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:03:47.868 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:47.900 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:47.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:48.001 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:48.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:48.046 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:48.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:48.147 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:48.194 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:48.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:48.212 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 70 15 100380 100365 89.5877 63.8125 0.030524 0.0111573 2024-03-26T05:03:48.296 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:48.326 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:48.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:48.428 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:48.463 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:48.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:48.520 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:48 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1232731583' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:03:48.520 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:48 smithi080 ceph-mon[40404]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T05:03:48.526 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:48 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1232731583' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:03:48.526 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:48 smithi080 ceph-mon[29691]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T05:03:48.564 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:48.598 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:48.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:48.699 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:48.744 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:48.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:48.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:48 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1232731583' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:03:48.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:48 smithi132 ceph-mon[33500]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T05:03:48.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:48.873 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:48.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:48.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:48.974 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:49.004 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:03:49.006 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:49.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:49.108 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:49.168 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:49.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:49.213 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 71 16 101426 101410 89.2447 65.3125 0.0251685 0.0111992 2024-03-26T05:03:49.270 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:49.299 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:49.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:49.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 3 1 2024-03-26T05:03:49.400 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:49.428 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:49.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:49.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:49.530 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:49.557 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:49.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:49.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:49 smithi080 ceph-mon[29691]: pgmap v129: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 841 op/s 2024-03-26T05:03:49.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:49.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:49 smithi080 ceph-mon[40404]: pgmap v129: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 841 op/s 2024-03-26T05:03:49.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:49.658 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:49.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:49.715 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:49.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:49.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:49 smithi132 ceph-mon[33500]: pgmap v129: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 841 op/s 2024-03-26T05:03:49.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:49.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:49.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:49.847 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:49.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:49.948 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:49.992 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:49.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:49.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:50.094 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:50.122 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:50.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:50.215 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 72 16 102419 102403 88.8665 62.0625 0.0132404 0.0112476 2024-03-26T05:03:50.223 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:50.253 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:50.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:50.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:50.355 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:50.385 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:50.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:50.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:50.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:50.539 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:50.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:50.640 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:50.643 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:50.645 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:50.667 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:50.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:50.768 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:50.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:50.822 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:50.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:50.923 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:50.997 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:50.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:51.099 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:51.126 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:51.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:51.214 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 73 16 103309 103293 88.4111 55.625 0.00816657 0.0113051 2024-03-26T05:03:51.228 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:51.274 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:51.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:51.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:51.327 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:51 smithi080 ceph-mon[40404]: pgmap v130: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 50 MiB/s wr, 807 op/s 2024-03-26T05:03:51.327 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:51.327 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:51.327 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:51 smithi080 ceph-mon[29691]: pgmap v130: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 50 MiB/s wr, 807 op/s 2024-03-26T05:03:51.327 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:51.327 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:51.377 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:51.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:51.420 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:51.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:51.521 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:51.541 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:03:51.541 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:03:51.541 INFO:teuthology.orchestra.run.smithi080.stdout: "complaint_time": 30, 2024-03-26T05:03:51.541 INFO:teuthology.orchestra.run.smithi080.stdout: "num_blocked_ops": 0 2024-03-26T05:03:51.541 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:03:51.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:51 smithi132 ceph-mon[33500]: pgmap v130: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 50 MiB/s wr, 807 op/s 2024-03-26T05:03:51.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:51.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:51.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:51.551 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:51.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:51.652 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:51.691 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:51.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:51.793 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:51.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:51.832 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:51.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:51.933 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:51.961 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:51.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:51.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:52.063 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:52.092 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:52.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:52.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:52.194 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:52.214 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 74 16 104194 104178 87.9637 55.3125 0.0235757 0.0113637 2024-03-26T05:03:52.221 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:52.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:52.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:52.322 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:52.349 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:52.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:52.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:52.410 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:52.450 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:52.479 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:52.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:52.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:52.503 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:52 smithi080 ceph-mon[40404]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T05:03:52.503 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:52 smithi080 ceph-mon[29691]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T05:03:52.581 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:52.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:52.623 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:52.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:52.724 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:52.764 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:52.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:52.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:52 smithi132 ceph-mon[33500]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T05:03:52.867 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:52.895 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:52.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:52.997 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:53.040 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:53.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:53.142 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:53.183 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:53.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:53.214 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 75 16 105090 105074 87.5375 56 0.0144946 0.0114181 2024-03-26T05:03:53.285 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:53.311 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:53.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:53.314 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T05:03:53.412 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:53.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:53.441 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:53.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:53.542 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:53.569 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:53.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:53.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:53.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:53 smithi080 ceph-mon[40404]: pgmap v132: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 837 op/s 2024-03-26T05:03:53.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:53 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:03:53.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:53 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:03:53.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:53 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:53.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:53 smithi080 ceph-mon[29691]: pgmap v132: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 837 op/s 2024-03-26T05:03:53.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:53 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:03:53.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:53 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:03:53.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:53 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:53.671 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:53.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:53.699 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:53.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:53.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:53 smithi132 ceph-mon[33500]: pgmap v132: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 837 op/s 2024-03-26T05:03:53.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:53 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:03:53.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:53 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:03:53.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:53 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:03:53.800 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:53.828 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:53.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:53.930 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:53.960 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:53.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:53.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:54.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:54.093 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:54.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:54.195 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:54.214 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 76 16 106083 106067 87.2023 62.0625 0.017604 0.0114631 2024-03-26T05:03:54.261 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:54.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:54.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:54.363 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:54.367 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:03:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:03:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:03:54.389 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:54.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:54.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:54.491 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:54.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:54.541 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:54.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:54.643 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:54.672 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:54.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:54.774 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:54.803 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:54.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:54.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:54.905 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:54.947 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:54.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:55.049 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:55.112 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:55.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:55.213 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:55.214 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 77 15 107126 107111 86.9172 65.25 0.0234159 0.0115 2024-03-26T05:03:55.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:55.255 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:55.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:55.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:03:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:03:55.356 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:55.383 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:55.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:55.428 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:03:55.471 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:03:55.484 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:55.514 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:03:55.530 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:55.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:55.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:55.554 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:03:55.594 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:03:55.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:55 smithi080 ceph-mon[29691]: pgmap v133: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 691 B/s rd, 42 MiB/s wr, 679 op/s 2024-03-26T05:03:55.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:55 smithi080 ceph-mon[40404]: pgmap v133: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 691 B/s rd, 42 MiB/s wr, 679 op/s 2024-03-26T05:03:55.632 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:55.638 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:03:55.665 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:55.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:55.678 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:03:55.715 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:03:55.727 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:03:55.727 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:03:55.727 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:03:55.727 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:03:55.727 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:03:55.727 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:03:55.728 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:03:55.728 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:03:55.767 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:55.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:55 smithi132 ceph-mon[33500]: pgmap v133: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 691 B/s rd, 42 MiB/s wr, 679 op/s 2024-03-26T05:03:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:55.815 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:55.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:55.916 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:55.945 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:55.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:56.046 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:56.087 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:56.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:56.189 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:56.214 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 78 16 108115 108099 86.5945 61.75 0.0237981 0.0115437 2024-03-26T05:03:56.217 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:56.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:56.320 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:56.349 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:56.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:56.454 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:56.498 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:56.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:56.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:56.600 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:56.637 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:56.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:56.738 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:56.760 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:56 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:56.760 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:56 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/272807112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:03:56.760 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:56 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:03:56.760 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:56 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:03:56.760 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:56 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:56.760 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:56 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/272807112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:03:56.760 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:56 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:03:56.761 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:56 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:03:56.791 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:56.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:56.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:56 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:56.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:56 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/272807112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:03:56.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:56 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:03:56.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:56 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:03:56.894 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:56.933 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:56.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:56.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:57.035 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:57.097 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:57.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:57.198 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:57.216 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 79 16 109060 109044 86.2443 59.0625 0.0249484 0.0115899 2024-03-26T05:03:57.241 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:57.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:57.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:57.332 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:03:57.342 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:57.398 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:57.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:57.431 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.3 to 1 (10000) 2024-03-26T05:03:57.500 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:57.529 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:57.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:57.630 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:57.659 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:57 smithi080 ceph-mon[29691]: pgmap v134: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-26T05:03:57.659 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:57 smithi080 ceph-mon[29691]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T05:03:57.659 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:57 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3403899415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:03:57.659 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:57 smithi080 ceph-mon[40404]: pgmap v134: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-26T05:03:57.659 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:57 smithi080 ceph-mon[40404]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T05:03:57.660 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:57 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3403899415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:03:57.672 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:57.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:57.774 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:57.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:57 smithi132 ceph-mon[33500]: pgmap v134: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-26T05:03:57.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:57 smithi132 ceph-mon[33500]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T05:03:57.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:57 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3403899415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:03:57.803 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:57.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:57.905 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:57.947 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:57.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:58.048 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:58.081 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:58.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:58.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:58.183 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:03:58.213 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:58.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:58.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:03:58.216 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:03:58.224988+0000 min lat: 0.00124227 max lat: 0.239944 avg lat: 0.0116325 2024-03-26T05:03:58.216 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:03:58.216 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 80 16 110036 110020 85.9287 61 0.0136964 0.0116325 2024-03-26T05:03:58.314 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:58.358 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:58.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:58.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:58.444 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:03:58.460 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:58.489 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:58.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:58.591 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:58.622 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:58.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:58.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:58.682 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3403899415' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:03:58.682 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:58 smithi080 ceph-mon[29691]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T05:03:58.682 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:58 smithi080 ceph-mon[29691]: pgmap v137: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 798 B/s rd, 46 MiB/s wr, 733 op/s 2024-03-26T05:03:58.682 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1785143653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:03:58.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3403899415' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:03:58.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:58 smithi080 ceph-mon[40404]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T05:03:58.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:58 smithi080 ceph-mon[40404]: pgmap v137: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 798 B/s rd, 46 MiB/s wr, 733 op/s 2024-03-26T05:03:58.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1785143653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:03:58.724 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:03:58.768 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:58.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:03:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3403899415' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:03:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:58 smithi132 ceph-mon[33500]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T05:03:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:58 smithi132 ceph-mon[33500]: pgmap v137: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 798 B/s rd, 46 MiB/s wr, 733 op/s 2024-03-26T05:03:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1785143653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:03:58.809 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:03:58.870 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:03:58.914 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:58.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:58.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:03:59.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:59.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:59.072 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:59.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:03:59.173 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:59.217 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 81 15 111061 111046 85.6581 64.125 0.0246959 0.0116699 2024-03-26T05:03:59.221 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:59.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:59.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:59.323 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:03:59.352 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:59.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:03:59.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T05:03:59.453 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:03:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:59.505 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:59.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:03:59.606 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:03:59.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:59 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:03:59.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:59 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1785143653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:03:59.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:03:59 smithi080 ceph-mon[40404]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T05:03:59.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:59 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:03:59.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:59 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1785143653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:03:59.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:03:59 smithi080 ceph-mon[29691]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T05:03:59.633 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:59.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:03:59.735 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:03:59.763 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:03:59.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:59.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:03:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:59 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:03:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:59 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1785143653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:03:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:03:59 smithi132 ceph-mon[33500]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T05:03:59.865 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:03:59.916 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:03:59.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:03:59.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:00.018 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:00.069 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:00.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:00.170 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:00.213 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:00.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:00.218 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 82 16 112037 112021 85.3567 60.9375 0.0171742 0.0117097 2024-03-26T05:04:00.315 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:00.343 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:00.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:00.344 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:00.366 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:00.389 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:04:00.389 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:04:00.444 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:00.489 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:00.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:00.530 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:00 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:04:00.530 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:00 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:04:00.530 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:00 smithi080 ceph-mon[40404]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T05:04:00.530 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:00 smithi080 ceph-mon[40404]: pgmap v140: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 30 MiB/s wr, 477 op/s 2024-03-26T05:04:00.530 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:00 smithi080 ceph-mon[40404]: 5.1e scrub starts 2024-03-26T05:04:00.531 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:00 smithi080 ceph-mon[40404]: 5.1e scrub ok 2024-03-26T05:04:00.531 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:00 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:04:00.531 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:00 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:04:00.531 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:00 smithi080 ceph-mon[29691]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T05:04:00.531 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:00 smithi080 ceph-mon[29691]: pgmap v140: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 30 MiB/s wr, 477 op/s 2024-03-26T05:04:00.531 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:00 smithi080 ceph-mon[29691]: 5.1e scrub starts 2024-03-26T05:04:00.531 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:00 smithi080 ceph-mon[29691]: 5.1e scrub ok 2024-03-26T05:04:00.591 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:00.621 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:00.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:00.723 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:00.768 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:00.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:00.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:00 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:04:00.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:00 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:04:00.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:00 smithi132 ceph-mon[33500]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T05:04:00.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:00 smithi132 ceph-mon[33500]: pgmap v140: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 30 MiB/s wr, 477 op/s 2024-03-26T05:04:00.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:00 smithi132 ceph-mon[33500]: 5.1e scrub starts 2024-03-26T05:04:00.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:00 smithi132 ceph-mon[33500]: 5.1e scrub ok 2024-03-26T05:04:00.869 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:00.900 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:00.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:01.001 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:01.032 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:01.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:01.133 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:01.164 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:01.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:01.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:01.218 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 83 16 113002 112986 85.0549 60.3125 0.00434784 0.0117524 2024-03-26T05:04:01.265 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:01.302 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:01.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:01.404 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:01.449 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:01.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:01.551 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:01.591 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:01.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:01.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:01 smithi080 ceph-mon[40404]: 6.9 scrub starts 2024-03-26T05:04:01.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:01 smithi080 ceph-mon[40404]: 6.9 scrub ok 2024-03-26T05:04:01.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:01 smithi080 ceph-mon[40404]: 3.1a scrub starts 2024-03-26T05:04:01.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:01 smithi080 ceph-mon[40404]: 3.1a scrub ok 2024-03-26T05:04:01.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:01 smithi080 ceph-mon[29691]: 6.9 scrub starts 2024-03-26T05:04:01.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:01 smithi080 ceph-mon[29691]: 6.9 scrub ok 2024-03-26T05:04:01.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:01 smithi080 ceph-mon[29691]: 3.1a scrub starts 2024-03-26T05:04:01.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:01 smithi080 ceph-mon[29691]: 3.1a scrub ok 2024-03-26T05:04:01.693 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:01.728 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:01.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:01.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:01.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:01 smithi132 ceph-mon[33500]: 6.9 scrub starts 2024-03-26T05:04:01.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:01 smithi132 ceph-mon[33500]: 6.9 scrub ok 2024-03-26T05:04:01.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:01 smithi132 ceph-mon[33500]: 3.1a scrub starts 2024-03-26T05:04:01.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:01 smithi132 ceph-mon[33500]: 3.1a scrub ok 2024-03-26T05:04:01.804 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:04:01.830 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:01.898 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:01.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:02.002 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:02.051 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:02.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:02.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:02.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:02.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:02.222 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 84 16 113992 113976 84.7789 61.875 0.0128378 0.0117909 2024-03-26T05:04:02.286 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:02.336 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:02.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:02.438 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:02.473 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:02.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:02.559 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:02 smithi080 ceph-mon[29691]: 4.10 scrub starts 2024-03-26T05:04:02.559 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:02 smithi080 ceph-mon[29691]: 4.10 scrub ok 2024-03-26T05:04:02.559 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:02 smithi080 ceph-mon[29691]: 6.19 scrub starts 2024-03-26T05:04:02.559 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:02 smithi080 ceph-mon[29691]: 6.19 scrub ok 2024-03-26T05:04:02.559 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:02 smithi080 ceph-mon[29691]: pgmap v141: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 957 B/s rd, 58 MiB/s wr, 928 op/s 2024-03-26T05:04:02.560 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:02 smithi080 ceph-mon[40404]: 4.10 scrub starts 2024-03-26T05:04:02.560 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:02 smithi080 ceph-mon[40404]: 4.10 scrub ok 2024-03-26T05:04:02.560 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:02 smithi080 ceph-mon[40404]: 6.19 scrub starts 2024-03-26T05:04:02.560 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:02 smithi080 ceph-mon[40404]: 6.19 scrub ok 2024-03-26T05:04:02.560 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:02 smithi080 ceph-mon[40404]: pgmap v141: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 957 B/s rd, 58 MiB/s wr, 928 op/s 2024-03-26T05:04:02.574 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:02.621 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:02.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:02.722 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:02.755 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:02.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:02.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:02.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:02 smithi132 ceph-mon[33500]: 4.10 scrub starts 2024-03-26T05:04:02.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:02 smithi132 ceph-mon[33500]: 4.10 scrub ok 2024-03-26T05:04:02.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:02 smithi132 ceph-mon[33500]: 6.19 scrub starts 2024-03-26T05:04:02.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:02 smithi132 ceph-mon[33500]: 6.19 scrub ok 2024-03-26T05:04:02.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:02 smithi132 ceph-mon[33500]: pgmap v141: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 957 B/s rd, 58 MiB/s wr, 928 op/s 2024-03-26T05:04:02.857 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:02.893 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:02.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:02.995 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:03.036 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:03.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:03.137 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:03.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:03.166 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:03.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:03.195 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T05:04:03.219 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 85 16 115003 114987 84.5235 63.1875 0.0116039 0.0118267 2024-03-26T05:04:03.267 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:03.366 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:03.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:03.468 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:03.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:03.600 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:03.627 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:03.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:03.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:03.730 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:03.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:03.754 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:03.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:03.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:03 smithi132 ceph-mon[33500]: 7.0 scrub starts 2024-03-26T05:04:03.856 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:03.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:03 smithi080 ceph-mon[29691]: 7.0 scrub starts 2024-03-26T05:04:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:03 smithi080 ceph-mon[40404]: 7.0 scrub starts 2024-03-26T05:04:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:03.924 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:03.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:04.025 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:04.052 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:04.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:04.070 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:04.154 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:04.205 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:04.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:04.220 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 86 16 115785 115769 84.1081 48.875 0.00938234 0.0118844 2024-03-26T05:04:04.306 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:04.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:04.341 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:04.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:04.368 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:04:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:04:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:04:04.441 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:04.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:04.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:04.503 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T05:04:04.601 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:04.634 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:04.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:04.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:04.715 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:04 smithi080 ceph-mon[29691]: 5.4 scrub starts 2024-03-26T05:04:04.715 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:04 smithi080 ceph-mon[29691]: 5.4 scrub ok 2024-03-26T05:04:04.715 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:04 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1212535573' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:04:04.715 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:04 smithi080 ceph-mon[29691]: pgmap v142: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 807 B/s rd, 49 MiB/s wr, 783 op/s 2024-03-26T05:04:04.716 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:04.716 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:04 smithi080 ceph-mon[40404]: 5.4 scrub starts 2024-03-26T05:04:04.716 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:04 smithi080 ceph-mon[40404]: 5.4 scrub ok 2024-03-26T05:04:04.716 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:04 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1212535573' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:04:04.716 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:04 smithi080 ceph-mon[40404]: pgmap v142: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 807 B/s rd, 49 MiB/s wr, 783 op/s 2024-03-26T05:04:04.716 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:04.736 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:04.765 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:04.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:04.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:04 smithi132 ceph-mon[33500]: 5.4 scrub starts 2024-03-26T05:04:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:04 smithi132 ceph-mon[33500]: 5.4 scrub ok 2024-03-26T05:04:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:04 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1212535573' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:04:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:04 smithi132 ceph-mon[33500]: pgmap v142: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 807 B/s rd, 49 MiB/s wr, 783 op/s 2024-03-26T05:04:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:04.866 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:04.900 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:04.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:05.001 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:05.045 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:05.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:05.146 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:05.179 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:05.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:05.220 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 87 16 116792 116776 83.8648 62.9375 0.0148547 0.0119191 2024-03-26T05:04:05.280 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:05.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:04:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:04:05.320 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:05.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:05.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:05.422 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:05.455 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:05.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:05.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:05.557 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:05.587 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:05.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:05.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:05.688 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:05.721 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:05.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:05.728 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:05 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:04:05.728 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:05 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:04:05.728 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:05 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1212535573' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:04:05.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:05 smithi080 ceph-mon[29691]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T05:04:05.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:05 smithi080 ceph-mon[29691]: 7.0 scrub ok 2024-03-26T05:04:05.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:05 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:05.729 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:05 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:04:05.729 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:05 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:04:05.729 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:05 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1212535573' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:04:05.729 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:05 smithi080 ceph-mon[40404]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T05:04:05.729 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:05 smithi080 ceph-mon[40404]: 7.0 scrub ok 2024-03-26T05:04:05.729 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:05 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:05 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:04:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:05 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:04:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:05 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1212535573' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:04:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:05 smithi132 ceph-mon[33500]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T05:04:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:05 smithi132 ceph-mon[33500]: 7.0 scrub ok 2024-03-26T05:04:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:05 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:05.822 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:05.861 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:05.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:05.963 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:05.996 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:05.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:05.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:06.098 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:06.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:06.131 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:06.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:06.220 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 88 16 117854 117838 83.666 66.375 0.0157875 0.0119476 2024-03-26T05:04:06.231 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:06.276 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:06.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:06.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:06.411 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:06.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:06.512 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:06.542 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:06.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:06.644 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:06.675 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:06.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:06.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:06.777 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:06 smithi132 ceph-mon[33500]: 7.7 scrub starts 2024-03-26T05:04:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:06 smithi132 ceph-mon[33500]: 5.15 scrub starts 2024-03-26T05:04:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:06 smithi132 ceph-mon[33500]: 5.15 scrub ok 2024-03-26T05:04:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:06 smithi132 ceph-mon[33500]: pgmap v144: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-26T05:04:06.809 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:06.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:06.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:06 smithi080 ceph-mon[29691]: 7.7 scrub starts 2024-03-26T05:04:06.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:06 smithi080 ceph-mon[29691]: 5.15 scrub starts 2024-03-26T05:04:06.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:06 smithi080 ceph-mon[29691]: 5.15 scrub ok 2024-03-26T05:04:06.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:06 smithi080 ceph-mon[29691]: pgmap v144: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-26T05:04:06.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:06 smithi080 ceph-mon[40404]: 7.7 scrub starts 2024-03-26T05:04:06.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:06 smithi080 ceph-mon[40404]: 5.15 scrub starts 2024-03-26T05:04:06.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:06 smithi080 ceph-mon[40404]: 5.15 scrub ok 2024-03-26T05:04:06.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:06 smithi080 ceph-mon[40404]: pgmap v144: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-26T05:04:06.911 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:06.954 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:06.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:06.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:07.056 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:07.090 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:07.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:07.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:07.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:07.220 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 89 16 118768 118752 83.3678 57.125 0.0299402 0.0119904 2024-03-26T05:04:07.230 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:07.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:07.332 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:07.366 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:07.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:07.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:07.469 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:07.495 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 5 1 2024-03-26T05:04:07.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:07.519 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:07.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:07.621 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:07.674 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:07.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:07.776 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:07 smithi132 ceph-mon[33500]: 6.15 scrub starts 2024-03-26T05:04:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:07 smithi132 ceph-mon[33500]: 6.15 scrub ok 2024-03-26T05:04:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:07 smithi132 ceph-mon[33500]: 7.d scrub starts 2024-03-26T05:04:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:07 smithi132 ceph-mon[33500]: 7.e scrub starts 2024-03-26T05:04:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:07 smithi132 ceph-mon[33500]: 3.8 scrub starts 2024-03-26T05:04:07.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:07 smithi132 ceph-mon[33500]: 3.8 scrub ok 2024-03-26T05:04:07.830 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:07.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:07.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:07 smithi080 ceph-mon[40404]: 6.15 scrub starts 2024-03-26T05:04:07.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:07 smithi080 ceph-mon[40404]: 6.15 scrub ok 2024-03-26T05:04:07.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:07 smithi080 ceph-mon[40404]: 7.d scrub starts 2024-03-26T05:04:07.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:07 smithi080 ceph-mon[40404]: 7.e scrub starts 2024-03-26T05:04:07.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:07 smithi080 ceph-mon[40404]: 3.8 scrub starts 2024-03-26T05:04:07.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:07 smithi080 ceph-mon[40404]: 3.8 scrub ok 2024-03-26T05:04:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:07 smithi080 ceph-mon[29691]: 6.15 scrub starts 2024-03-26T05:04:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:07 smithi080 ceph-mon[29691]: 6.15 scrub ok 2024-03-26T05:04:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:07 smithi080 ceph-mon[29691]: 7.d scrub starts 2024-03-26T05:04:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:07 smithi080 ceph-mon[29691]: 7.e scrub starts 2024-03-26T05:04:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:07 smithi080 ceph-mon[29691]: 3.8 scrub starts 2024-03-26T05:04:07.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:07 smithi080 ceph-mon[29691]: 3.8 scrub ok 2024-03-26T05:04:07.932 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:07.966 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:07.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:08.067 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:08.094 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:08.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:08.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:08.113 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:04:08.147 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:04:08.194 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:04:08.196 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:08.220 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Total time run: 90.0105 2024-03-26T05:04:08.221 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Total writes made: 119496 2024-03-26T05:04:08.221 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Write size: 65536 2024-03-26T05:04:08.221 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Object size: 65536 2024-03-26T05:04:08.221 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Bandwidth (MB/sec): 82.9737 2024-03-26T05:04:08.221 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev Bandwidth: 30.9395 2024-03-26T05:04:08.221 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max bandwidth (MB/sec): 199.125 2024-03-26T05:04:08.221 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min bandwidth (MB/sec): 48.875 2024-03-26T05:04:08.221 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Average IOPS: 1327 2024-03-26T05:04:08.221 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev IOPS: 495.031 2024-03-26T05:04:08.221 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max IOPS: 3186 2024-03-26T05:04:08.221 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min IOPS: 782 2024-03-26T05:04:08.221 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Average Latency(s): 0.012048 2024-03-26T05:04:08.222 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev Latency(s): 0.00941437 2024-03-26T05:04:08.222 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max latency(s): 0.239944 2024-03-26T05:04:08.222 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min latency(s): 0.00124227 2024-03-26T05:04:08.223 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:04:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:08.223 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:08.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:08.229 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T05:04:08.257 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:04:08.294 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:04:08.324 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:08.333 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:04:08.352 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:08.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:08.402 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:04:08.404 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:04:08.404 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:04:08.404 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:04:08.404 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:04:08.404 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:04:08.404 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:04:08.404 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:04:08.404 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:04:08.454 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:08.505 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:08.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:08.607 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:08.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:08.639 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:08.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:08.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[29691]: 7.7 scrub ok 2024-03-26T05:04:08.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[29691]: 4.1d scrub starts 2024-03-26T05:04:08.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[29691]: 4.1d scrub ok 2024-03-26T05:04:08.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[29691]: 3.19 scrub starts 2024-03-26T05:04:08.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[29691]: 3.19 scrub ok 2024-03-26T05:04:08.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[29691]: pgmap v145: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-26T05:04:08.730 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2229321889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:04:08.730 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[40404]: 7.7 scrub ok 2024-03-26T05:04:08.730 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[40404]: 4.1d scrub starts 2024-03-26T05:04:08.730 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[40404]: 4.1d scrub ok 2024-03-26T05:04:08.730 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[40404]: 3.19 scrub starts 2024-03-26T05:04:08.730 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[40404]: 3.19 scrub ok 2024-03-26T05:04:08.730 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[40404]: pgmap v145: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-26T05:04:08.730 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:08 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2229321889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:04:08.740 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:08.767 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:08.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:08.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:08.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:08 smithi132 ceph-mon[33500]: 7.7 scrub ok 2024-03-26T05:04:08.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:08 smithi132 ceph-mon[33500]: 4.1d scrub starts 2024-03-26T05:04:08.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:08 smithi132 ceph-mon[33500]: 4.1d scrub ok 2024-03-26T05:04:08.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:08 smithi132 ceph-mon[33500]: 3.19 scrub starts 2024-03-26T05:04:08.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:08 smithi132 ceph-mon[33500]: 3.19 scrub ok 2024-03-26T05:04:08.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:08 smithi132 ceph-mon[33500]: pgmap v145: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-26T05:04:08.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:08 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2229321889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:04:08.869 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:08.874 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:08.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:08.896 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:08.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:08.997 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:09.027 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:09.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:09.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:09.129 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:09.171 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:09.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:09.273 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:09.313 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:09.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:09.415 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:09.442 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:09.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:09.544 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:09.553 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:04:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:09.574 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:09.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:09.675 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:09.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:09.705 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:09.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:09.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:09 smithi132 ceph-mon[33500]: 7.d scrub ok 2024-03-26T05:04:09.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:09 smithi132 ceph-mon[33500]: 7.8 scrub starts 2024-03-26T05:04:09.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:09 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:04:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:09 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:04:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:09 smithi132 ceph-mon[33500]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T05:04:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:09 smithi132 ceph-mon[33500]: 7.6 scrub starts 2024-03-26T05:04:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:09 smithi132 ceph-mon[33500]: 5.e scrub starts 2024-03-26T05:04:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:09 smithi132 ceph-mon[33500]: 5.e scrub ok 2024-03-26T05:04:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:09 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/815580527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:04:09.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:09 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:04:09.805 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:09.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:09.858 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:09.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:09.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[29691]: 7.d scrub ok 2024-03-26T05:04:09.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[29691]: 7.8 scrub starts 2024-03-26T05:04:09.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:04:09.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:04:09.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[29691]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T05:04:09.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[29691]: 7.6 scrub starts 2024-03-26T05:04:09.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[29691]: 5.e scrub starts 2024-03-26T05:04:09.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[29691]: 5.e scrub ok 2024-03-26T05:04:09.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/815580527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:04:09.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:04:09.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[40404]: 7.d scrub ok 2024-03-26T05:04:09.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[40404]: 7.8 scrub starts 2024-03-26T05:04:09.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:04:09.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:04:09.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[40404]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T05:04:09.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[40404]: 7.6 scrub starts 2024-03-26T05:04:09.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[40404]: 5.e scrub starts 2024-03-26T05:04:09.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[40404]: 5.e scrub ok 2024-03-26T05:04:09.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/815580527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:04:09.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:09 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:04:09.959 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:09.989 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:09.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:10.091 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:10.131 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:10.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:10.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:10.233 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:10.270 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:10.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:10.371 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:10.401 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:10.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:10.503 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:10.527 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:10.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:10.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:04:10.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[40404]: Cluster is now healthy 2024-03-26T05:04:10.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:04:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[40404]: 4.11 scrub starts 2024-03-26T05:04:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[40404]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T05:04:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[40404]: 4.11 scrub ok 2024-03-26T05:04:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[40404]: 7.e scrub ok 2024-03-26T05:04:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[40404]: pgmap v148: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 55 MiB/s wr, 928 op/s 2024-03-26T05:04:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[40404]: 6.2 scrub starts 2024-03-26T05:04:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[40404]: 6.2 scrub ok 2024-03-26T05:04:10.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:04:10.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[29691]: Cluster is now healthy 2024-03-26T05:04:10.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:04:10.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[29691]: 4.11 scrub starts 2024-03-26T05:04:10.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[29691]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T05:04:10.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[29691]: 4.11 scrub ok 2024-03-26T05:04:10.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[29691]: 7.e scrub ok 2024-03-26T05:04:10.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[29691]: pgmap v148: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 55 MiB/s wr, 928 op/s 2024-03-26T05:04:10.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[29691]: 6.2 scrub starts 2024-03-26T05:04:10.621 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:10 smithi080 ceph-mon[29691]: 6.2 scrub ok 2024-03-26T05:04:10.628 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:10.667 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:10.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:10.769 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:10.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:10 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:04:10.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:10 smithi132 ceph-mon[33500]: Cluster is now healthy 2024-03-26T05:04:10.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:10 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:04:10.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:10 smithi132 ceph-mon[33500]: 4.11 scrub starts 2024-03-26T05:04:10.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:10 smithi132 ceph-mon[33500]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T05:04:10.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:10 smithi132 ceph-mon[33500]: 4.11 scrub ok 2024-03-26T05:04:10.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:10 smithi132 ceph-mon[33500]: 7.e scrub ok 2024-03-26T05:04:10.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:10 smithi132 ceph-mon[33500]: pgmap v148: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 55 MiB/s wr, 928 op/s 2024-03-26T05:04:10.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:10 smithi132 ceph-mon[33500]: 6.2 scrub starts 2024-03-26T05:04:10.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:10 smithi132 ceph-mon[33500]: 6.2 scrub ok 2024-03-26T05:04:10.808 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:10.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:10.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:10.911 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:10.941 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:10.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:11.043 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:11.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:11.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:11.169 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:11.207 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:11.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:11.308 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:11.334 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:11.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:11.436 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:11.475 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:11.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:11.577 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:11.601 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:11.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:11.703 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:11.731 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:11.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:11.833 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:11.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:11 smithi080 ceph-mon[29691]: 6.13 scrub starts 2024-03-26T05:04:11.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:11 smithi080 ceph-mon[29691]: 6.13 scrub ok 2024-03-26T05:04:11.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:11 smithi080 ceph-mon[29691]: 3.1c scrub starts 2024-03-26T05:04:11.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:11 smithi080 ceph-mon[29691]: 3.1c scrub ok 2024-03-26T05:04:11.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:11 smithi080 ceph-mon[29691]: 6.16 scrub starts 2024-03-26T05:04:11.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:11 smithi080 ceph-mon[29691]: 6.16 scrub ok 2024-03-26T05:04:11.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:11 smithi080 ceph-mon[40404]: 6.13 scrub starts 2024-03-26T05:04:11.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:11 smithi080 ceph-mon[40404]: 6.13 scrub ok 2024-03-26T05:04:11.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:11 smithi080 ceph-mon[40404]: 3.1c scrub starts 2024-03-26T05:04:11.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:11 smithi080 ceph-mon[40404]: 3.1c scrub ok 2024-03-26T05:04:11.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:11 smithi080 ceph-mon[40404]: 6.16 scrub starts 2024-03-26T05:04:11.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:11 smithi080 ceph-mon[40404]: 6.16 scrub ok 2024-03-26T05:04:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:11.872 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:11.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:11.973 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:12.000 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:12.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:12.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:12.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:11 smithi132 ceph-mon[33500]: 6.13 scrub starts 2024-03-26T05:04:12.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:11 smithi132 ceph-mon[33500]: 6.13 scrub ok 2024-03-26T05:04:12.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:11 smithi132 ceph-mon[33500]: 3.1c scrub starts 2024-03-26T05:04:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:11 smithi132 ceph-mon[33500]: 3.1c scrub ok 2024-03-26T05:04:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:11 smithi132 ceph-mon[33500]: 6.16 scrub starts 2024-03-26T05:04:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:11 smithi132 ceph-mon[33500]: 6.16 scrub ok 2024-03-26T05:04:12.102 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:12.130 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:12.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:12.231 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:12.258 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:12.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:12.360 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:12.390 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:12.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:12.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:12.492 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:12.531 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:12.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:12.634 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:12.664 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:12.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:12.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:12.766 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:12.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:12.795 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:12.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:12.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[29691]: 7.8 scrub ok 2024-03-26T05:04:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[29691]: 6.6 scrub starts 2024-03-26T05:04:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[29691]: 6.6 scrub ok 2024-03-26T05:04:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[29691]: 4.7 scrub starts 2024-03-26T05:04:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[29691]: 4.7 scrub ok 2024-03-26T05:04:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[29691]: pgmap v149: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 68 MiB/s wr, 1.27k op/s 2024-03-26T05:04:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[29691]: 7.6 scrub ok 2024-03-26T05:04:12.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[40404]: 7.8 scrub ok 2024-03-26T05:04:12.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[40404]: 6.6 scrub starts 2024-03-26T05:04:12.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[40404]: 6.6 scrub ok 2024-03-26T05:04:12.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[40404]: 4.7 scrub starts 2024-03-26T05:04:12.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[40404]: 4.7 scrub ok 2024-03-26T05:04:12.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[40404]: pgmap v149: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 68 MiB/s wr, 1.27k op/s 2024-03-26T05:04:12.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:12 smithi080 ceph-mon[40404]: 7.6 scrub ok 2024-03-26T05:04:12.897 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:12.935 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:12.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:13.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:12 smithi132 ceph-mon[33500]: 7.8 scrub ok 2024-03-26T05:04:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:12 smithi132 ceph-mon[33500]: 6.6 scrub starts 2024-03-26T05:04:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:12 smithi132 ceph-mon[33500]: 6.6 scrub ok 2024-03-26T05:04:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:12 smithi132 ceph-mon[33500]: 4.7 scrub starts 2024-03-26T05:04:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:12 smithi132 ceph-mon[33500]: 4.7 scrub ok 2024-03-26T05:04:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:12 smithi132 ceph-mon[33500]: pgmap v149: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 68 MiB/s wr, 1.27k op/s 2024-03-26T05:04:13.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:12 smithi132 ceph-mon[33500]: 7.6 scrub ok 2024-03-26T05:04:13.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:13.071 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:13.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:13.172 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:13.202 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:13.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:13.304 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:13.345 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:13.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:13.447 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:13.479 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:13.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:13.581 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:13.611 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:13.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:13.713 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:13.743 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:13.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:13.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:13 smithi132 ceph-mon[33500]: 4.15 deep-scrub starts 2024-03-26T05:04:13.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:13 smithi132 ceph-mon[33500]: 4.15 deep-scrub ok 2024-03-26T05:04:13.799 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:13.845 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:13 smithi080 ceph-mon[40404]: 4.15 deep-scrub starts 2024-03-26T05:04:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:13 smithi080 ceph-mon[40404]: 4.15 deep-scrub ok 2024-03-26T05:04:13.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:13 smithi080 ceph-mon[29691]: 4.15 deep-scrub starts 2024-03-26T05:04:13.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:13 smithi080 ceph-mon[29691]: 4.15 deep-scrub ok 2024-03-26T05:04:13.906 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:13.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:13.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:14.008 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:14.049 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:14.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:14.149 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:14.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:14.173 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:14.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:14.274 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:14.311 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:14.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:14.367 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:04:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:04:14.413 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:14.437 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:14.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:14.539 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:14.570 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:14.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:14.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:14.703 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:14.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:14.804 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:14.842 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:14.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:14.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:14.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[29691]: 7.c scrub starts 2024-03-26T05:04:14.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[29691]: 6.14 scrub starts 2024-03-26T05:04:14.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[29691]: 6.14 scrub ok 2024-03-26T05:04:14.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[29691]: 7.a scrub starts 2024-03-26T05:04:14.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[29691]: 4.8 scrub starts 2024-03-26T05:04:14.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[29691]: 4.8 scrub ok 2024-03-26T05:04:14.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[29691]: pgmap v150: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 25 MiB/s wr, 558 op/s 2024-03-26T05:04:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[40404]: 7.c scrub starts 2024-03-26T05:04:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[40404]: 6.14 scrub starts 2024-03-26T05:04:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[40404]: 6.14 scrub ok 2024-03-26T05:04:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[40404]: 7.a scrub starts 2024-03-26T05:04:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[40404]: 4.8 scrub starts 2024-03-26T05:04:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[40404]: 4.8 scrub ok 2024-03-26T05:04:14.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:14 smithi080 ceph-mon[40404]: pgmap v150: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 25 MiB/s wr, 558 op/s 2024-03-26T05:04:14.874 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:14 smithi132 ceph-mon[33500]: 7.c scrub starts 2024-03-26T05:04:14.874 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:14 smithi132 ceph-mon[33500]: 6.14 scrub starts 2024-03-26T05:04:14.874 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:14 smithi132 ceph-mon[33500]: 6.14 scrub ok 2024-03-26T05:04:14.874 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:14 smithi132 ceph-mon[33500]: 7.a scrub starts 2024-03-26T05:04:14.874 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:14 smithi132 ceph-mon[33500]: 4.8 scrub starts 2024-03-26T05:04:14.874 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:14 smithi132 ceph-mon[33500]: 4.8 scrub ok 2024-03-26T05:04:14.874 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:14 smithi132 ceph-mon[33500]: pgmap v150: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 25 MiB/s wr, 558 op/s 2024-03-26T05:04:14.945 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:14.996 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:14.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:15.098 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:15.127 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:15.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:15.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:15.229 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:15.270 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:15.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:15.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:04:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:04:15.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:15.407 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:15.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:15.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:15.509 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:15.549 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:15.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:15.650 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:15.677 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:15.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:15.684 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:04:15.723 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:15 smithi080 ceph-mon[40404]: 3.17 scrub starts 2024-03-26T05:04:15.723 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:15 smithi080 ceph-mon[40404]: 3.17 scrub ok 2024-03-26T05:04:15.724 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:15 smithi080 ceph-mon[40404]: 7.c scrub ok 2024-03-26T05:04:15.724 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:15 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:15.724 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:15 smithi080 ceph-mon[29691]: 3.17 scrub starts 2024-03-26T05:04:15.724 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:15 smithi080 ceph-mon[29691]: 3.17 scrub ok 2024-03-26T05:04:15.724 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:15 smithi080 ceph-mon[29691]: 7.c scrub ok 2024-03-26T05:04:15.724 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:15 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:15.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:15.807 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:15.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:15.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:15.909 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:15.995 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:15.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:15 smithi132 ceph-mon[33500]: 3.17 scrub starts 2024-03-26T05:04:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:15 smithi132 ceph-mon[33500]: 3.17 scrub ok 2024-03-26T05:04:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:15 smithi132 ceph-mon[33500]: 7.c scrub ok 2024-03-26T05:04:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:15 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:16.096 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:16.137 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:16.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:16.239 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:16.268 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:16.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:16.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:16.369 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:16.398 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:16.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:16.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:16.499 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:16.530 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:16.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:16.631 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:16.668 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:16.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:16.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:16.691 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:16 smithi080 ceph-mon[29691]: 7.2 scrub starts 2024-03-26T05:04:16.691 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:16 smithi080 ceph-mon[29691]: pgmap v151: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 25 MiB/s wr, 1.11k op/s 2024-03-26T05:04:16.692 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:16 smithi080 ceph-mon[29691]: 3.11 scrub starts 2024-03-26T05:04:16.692 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:16 smithi080 ceph-mon[29691]: 3.11 scrub ok 2024-03-26T05:04:16.692 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:16 smithi080 ceph-mon[29691]: 4.c scrub starts 2024-03-26T05:04:16.692 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:16 smithi080 ceph-mon[29691]: 4.c scrub ok 2024-03-26T05:04:16.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:16 smithi080 ceph-mon[40404]: 7.2 scrub starts 2024-03-26T05:04:16.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:16 smithi080 ceph-mon[40404]: pgmap v151: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 25 MiB/s wr, 1.11k op/s 2024-03-26T05:04:16.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:16 smithi080 ceph-mon[40404]: 3.11 scrub starts 2024-03-26T05:04:16.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:16 smithi080 ceph-mon[40404]: 3.11 scrub ok 2024-03-26T05:04:16.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:16 smithi080 ceph-mon[40404]: 4.c scrub starts 2024-03-26T05:04:16.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:16 smithi080 ceph-mon[40404]: 4.c scrub ok 2024-03-26T05:04:16.770 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:16.836 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:16.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:16.937 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:16.966 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:04:16.968 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:16.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:16 smithi132 ceph-mon[33500]: 7.2 scrub starts 2024-03-26T05:04:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:16 smithi132 ceph-mon[33500]: pgmap v151: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 25 MiB/s wr, 1.11k op/s 2024-03-26T05:04:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:16 smithi132 ceph-mon[33500]: 3.11 scrub starts 2024-03-26T05:04:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:16 smithi132 ceph-mon[33500]: 3.11 scrub ok 2024-03-26T05:04:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:16 smithi132 ceph-mon[33500]: 4.c scrub starts 2024-03-26T05:04:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:16 smithi132 ceph-mon[33500]: 4.c scrub ok 2024-03-26T05:04:17.070 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:17.097 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:17.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:17.198 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:17.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:17.257 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:17.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:17.358 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:17.386 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:17.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:17.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:17.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:17.540 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:17.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:17.642 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:17.667 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:17 smithi080 ceph-mon[40404]: 7.a scrub ok 2024-03-26T05:04:17.668 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:17 smithi080 ceph-mon[29691]: 7.a scrub ok 2024-03-26T05:04:17.687 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:17.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:17.789 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:17.818 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:17.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:17.920 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:17.947 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:17.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:17.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:18.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:17 smithi132 ceph-mon[33500]: 7.a scrub ok 2024-03-26T05:04:18.049 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:18.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:18.078 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:18.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:18.179 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:18.207 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:18.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:18.309 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:18.334 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:18.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:18.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:18.436 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:18.482 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:18.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:18.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:18.613 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:18.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:18.715 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:18.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:18.755 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:18.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:18.856 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:18.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:18.896 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:18.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:18.997 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:19.027 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:19.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:19.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:18 smithi132 ceph-mon[33500]: 3.0 scrub starts 2024-03-26T05:04:19.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:18 smithi132 ceph-mon[33500]: 3.0 scrub ok 2024-03-26T05:04:19.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:18 smithi132 ceph-mon[33500]: pgmap v152: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 21 MiB/s wr, 960 op/s 2024-03-26T05:04:19.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:18 smithi132 ceph-mon[33500]: 1.0 scrub starts 2024-03-26T05:04:19.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:18 smithi132 ceph-mon[33500]: 1.0 scrub ok 2024-03-26T05:04:19.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:18 smithi132 ceph-mon[33500]: 6.1a scrub starts 2024-03-26T05:04:19.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:18 smithi132 ceph-mon[33500]: 6.1a scrub ok 2024-03-26T05:04:19.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:18 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3670863966' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:04:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[29691]: 3.0 scrub starts 2024-03-26T05:04:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[29691]: 3.0 scrub ok 2024-03-26T05:04:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[29691]: pgmap v152: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 21 MiB/s wr, 960 op/s 2024-03-26T05:04:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[29691]: 1.0 scrub starts 2024-03-26T05:04:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[29691]: 1.0 scrub ok 2024-03-26T05:04:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[29691]: 6.1a scrub starts 2024-03-26T05:04:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[29691]: 6.1a scrub ok 2024-03-26T05:04:19.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3670863966' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:04:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[40404]: 3.0 scrub starts 2024-03-26T05:04:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[40404]: 3.0 scrub ok 2024-03-26T05:04:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[40404]: pgmap v152: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 21 MiB/s wr, 960 op/s 2024-03-26T05:04:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[40404]: 1.0 scrub starts 2024-03-26T05:04:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[40404]: 1.0 scrub ok 2024-03-26T05:04:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[40404]: 6.1a scrub starts 2024-03-26T05:04:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[40404]: 6.1a scrub ok 2024-03-26T05:04:19.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:18 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3670863966' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:04:19.129 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:19.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:19.158 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:19.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:19.258 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:19.290 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:19.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:19.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:19.392 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:19.420 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:19.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:19.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:19.522 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:19.548 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.5 to 1 (10000) 2024-03-26T05:04:19.563 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:19.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:19.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:19.665 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:19.696 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:19.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:19.797 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:19.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3670863966' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T05:04:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[40404]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T05:04:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[40404]: 3.1d scrub starts 2024-03-26T05:04:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[40404]: 3.1d scrub ok 2024-03-26T05:04:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:04:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[40404]: 7.2 scrub ok 2024-03-26T05:04:19.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3670863966' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T05:04:19.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[29691]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T05:04:19.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[29691]: 3.1d scrub starts 2024-03-26T05:04:19.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[29691]: 3.1d scrub ok 2024-03-26T05:04:19.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:04:19.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:19.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:19 smithi080 ceph-mon[29691]: 7.2 scrub ok 2024-03-26T05:04:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:19.870 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:19.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:19.971 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:20.002 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:20.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:20.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:19 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3670863966' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T05:04:20.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:19 smithi132 ceph-mon[33500]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T05:04:20.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:19 smithi132 ceph-mon[33500]: 3.1d scrub starts 2024-03-26T05:04:20.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:19 smithi132 ceph-mon[33500]: 3.1d scrub ok 2024-03-26T05:04:20.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:04:20.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:20.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:19 smithi132 ceph-mon[33500]: 7.2 scrub ok 2024-03-26T05:04:20.104 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:20.149 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:20.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:20.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:20.251 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:20.296 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:20.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:20.397 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:20.430 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:20.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:20.531 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:20.570 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:20.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:20.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:20.702 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:20.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:20.804 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:20.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:20.849 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:20.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:20.877 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 6 1 2024-03-26T05:04:20.951 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:20.958 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:20 smithi080 ceph-mon[40404]: 5.1f scrub starts 2024-03-26T05:04:20.959 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:20 smithi080 ceph-mon[40404]: 5.1f scrub ok 2024-03-26T05:04:20.959 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:20 smithi080 ceph-mon[40404]: 7.b scrub starts 2024-03-26T05:04:20.959 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:20 smithi080 ceph-mon[40404]: pgmap v154: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 16 MiB/s wr, 889 op/s 2024-03-26T05:04:20.959 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:20 smithi080 ceph-mon[29691]: 5.1f scrub starts 2024-03-26T05:04:20.960 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:20 smithi080 ceph-mon[29691]: 5.1f scrub ok 2024-03-26T05:04:20.960 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:20 smithi080 ceph-mon[29691]: 7.b scrub starts 2024-03-26T05:04:20.960 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:20 smithi080 ceph-mon[29691]: pgmap v154: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 16 MiB/s wr, 889 op/s 2024-03-26T05:04:21.002 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:21.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:21.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:20 smithi132 ceph-mon[33500]: 5.1f scrub starts 2024-03-26T05:04:21.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:20 smithi132 ceph-mon[33500]: 5.1f scrub ok 2024-03-26T05:04:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:20 smithi132 ceph-mon[33500]: 7.b scrub starts 2024-03-26T05:04:21.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:20 smithi132 ceph-mon[33500]: pgmap v154: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 16 MiB/s wr, 889 op/s 2024-03-26T05:04:21.104 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:21.143 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:21.143 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:21.170 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:21.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:21.236 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:04:21.236 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:04:21.236 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.236 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:131132 7.a 7.37b4ecba (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T05:04:21.236 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:21.240163+0000", 2024-03-26T05:04:21.236 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.000133449, 2024-03-26T05:04:21.236 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016428699999999999, 2024-03-26T05:04:21.236 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 131132 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240163+0000", 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.238 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:04:21.238 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240163+0000", 2024-03-26T05:04:21.238 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:04:21.238 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.238 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.238 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:04:21.238 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240164+0000", 2024-03-26T05:04:21.238 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4559999999999999e-06 2024-03-26T05:04:21.238 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.238 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.239 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:04:21.239 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240170+0000", 2024-03-26T05:04:21.239 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8069999999999996e-06 2024-03-26T05:04:21.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.240 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.240 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:04:21.240 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240188+0000", 2024-03-26T05:04:21.240 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7496e-05 2024-03-26T05:04:21.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.240 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.240 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:04:21.240 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240197+0000", 2024-03-26T05:04:21.240 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4520000000000003e-06 2024-03-26T05:04:21.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.240 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240330+0000", 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000132533 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:131134 7.3 7.e9b45c73 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:21.240220+0000", 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 7.6662000000000006e-05, 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017592000000000001, 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:04:21.241 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "queued for pg", 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 131134 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240220+0000", 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:04:21.242 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240220+0000", 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240220+0000", 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3000000000000001e-07 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240223+0000", 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6910000000000002e-06 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.243 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240230+0000", 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6529999999999994e-06 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:21.240237+0000", 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5239999999999997e-06 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 2 2024-03-26T05:04:21.244 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:04:21.271 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:21.304 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:21.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:21.405 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:21.433 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:21.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:21.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:21.563 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:21.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:21.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:21.664 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:21.702 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:21.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:21.745 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:21 smithi080 ceph-mon[29691]: 4.19 scrub starts 2024-03-26T05:04:21.745 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:21 smithi080 ceph-mon[29691]: 4.19 scrub ok 2024-03-26T05:04:21.745 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:21 smithi080 ceph-mon[29691]: 5.1c scrub starts 2024-03-26T05:04:21.745 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:21 smithi080 ceph-mon[29691]: 5.1c scrub ok 2024-03-26T05:04:21.745 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:21 smithi080 ceph-mon[40404]: 4.19 scrub starts 2024-03-26T05:04:21.745 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:21 smithi080 ceph-mon[40404]: 4.19 scrub ok 2024-03-26T05:04:21.745 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:21 smithi080 ceph-mon[40404]: 5.1c scrub starts 2024-03-26T05:04:21.745 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:21 smithi080 ceph-mon[40404]: 5.1c scrub ok 2024-03-26T05:04:21.804 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:21.860 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:21.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:21.962 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:21.989 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:21.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:21.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:21 smithi132 ceph-mon[33500]: 4.19 scrub starts 2024-03-26T05:04:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:21 smithi132 ceph-mon[33500]: 4.19 scrub ok 2024-03-26T05:04:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:21 smithi132 ceph-mon[33500]: 5.1c scrub starts 2024-03-26T05:04:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:21 smithi132 ceph-mon[33500]: 5.1c scrub ok 2024-03-26T05:04:22.093 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:22.125 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:22.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:22.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:22.227 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:22.258 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:22.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:22.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:22.360 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:22.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:22.391 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:22.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:22.492 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:22.530 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:22.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:22.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:22.632 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:22.680 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:22.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:22.782 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:22.825 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:22.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: 3.16 scrub starts 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: 3.16 scrub ok 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: 5.b scrub starts 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: 5.b scrub ok 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: 7.b scrub ok 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: 5.12 scrub starts 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: 5.12 scrub ok 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: pgmap v155: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 901 op/s 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: 7.4 scrub starts 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: 6.e scrub starts 2024-03-26T05:04:22.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[40404]: 6.e scrub ok 2024-03-26T05:04:22.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: 3.16 scrub starts 2024-03-26T05:04:22.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: 3.16 scrub ok 2024-03-26T05:04:22.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: 5.b scrub starts 2024-03-26T05:04:22.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: 5.b scrub ok 2024-03-26T05:04:22.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: 7.b scrub ok 2024-03-26T05:04:22.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: 5.12 scrub starts 2024-03-26T05:04:22.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: 5.12 scrub ok 2024-03-26T05:04:22.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: pgmap v155: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 901 op/s 2024-03-26T05:04:22.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: 7.4 scrub starts 2024-03-26T05:04:22.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:04:22.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:04:22.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: 6.e scrub starts 2024-03-26T05:04:22.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:22 smithi080 ceph-mon[29691]: 6.e scrub ok 2024-03-26T05:04:22.927 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:22.957 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:22.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:22.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: 3.16 scrub starts 2024-03-26T05:04:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: 3.16 scrub ok 2024-03-26T05:04:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: 5.b scrub starts 2024-03-26T05:04:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: 5.b scrub ok 2024-03-26T05:04:23.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: 7.b scrub ok 2024-03-26T05:04:23.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: 5.12 scrub starts 2024-03-26T05:04:23.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: 5.12 scrub ok 2024-03-26T05:04:23.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: pgmap v155: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 901 op/s 2024-03-26T05:04:23.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: 7.4 scrub starts 2024-03-26T05:04:23.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:04:23.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:04:23.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: 6.e scrub starts 2024-03-26T05:04:23.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:22 smithi132 ceph-mon[33500]: 6.e scrub ok 2024-03-26T05:04:23.058 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:23.098 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:23.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:23.200 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:23.237 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:23.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:23.292 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T05:04:23.339 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:23.387 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:23.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:23.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:23.530 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:23.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:23.634 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:23.682 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:23.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:23.784 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:23.842 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:23.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:23.943 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:23.996 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:23.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:24.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:23 smithi080 ceph-mon[40404]: 4.4 scrub starts 2024-03-26T05:04:24.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:23 smithi080 ceph-mon[40404]: 4.4 scrub ok 2024-03-26T05:04:24.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:23 smithi080 ceph-mon[40404]: 5.7 scrub starts 2024-03-26T05:04:24.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:23 smithi080 ceph-mon[40404]: 5.7 scrub ok 2024-03-26T05:04:24.026 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:23 smithi080 ceph-mon[29691]: 4.4 scrub starts 2024-03-26T05:04:24.026 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:23 smithi080 ceph-mon[29691]: 4.4 scrub ok 2024-03-26T05:04:24.026 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:23 smithi080 ceph-mon[29691]: 5.7 scrub starts 2024-03-26T05:04:24.026 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:23 smithi080 ceph-mon[29691]: 5.7 scrub ok 2024-03-26T05:04:24.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:23 smithi132 ceph-mon[33500]: 4.4 scrub starts 2024-03-26T05:04:24.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:23 smithi132 ceph-mon[33500]: 4.4 scrub ok 2024-03-26T05:04:24.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:23 smithi132 ceph-mon[33500]: 5.7 scrub starts 2024-03-26T05:04:24.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:23 smithi132 ceph-mon[33500]: 5.7 scrub ok 2024-03-26T05:04:24.098 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:24.148 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:24.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:24.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:24.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:24.341 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:24.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:24.345 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:04:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:04:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:04:24.442 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:24.473 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:24.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:24.516 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:24.575 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:24.604 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:24.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:24.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:24.706 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:24.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:24.759 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:24.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:24.859 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:24.899 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:24.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:24.937 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:24 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:04:24.937 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:24 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:04:24.937 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:24 smithi080 ceph-mon[40404]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T05:04:24.937 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:24 smithi080 ceph-mon[40404]: pgmap v157: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 577 op/s 2024-03-26T05:04:24.937 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:24 smithi080 ceph-mon[40404]: 7.4 scrub ok 2024-03-26T05:04:24.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:24 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:04:24.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:24 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:04:24.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:24 smithi080 ceph-mon[29691]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T05:04:24.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:24 smithi080 ceph-mon[29691]: pgmap v157: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 577 op/s 2024-03-26T05:04:24.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:24 smithi080 ceph-mon[29691]: 7.4 scrub ok 2024-03-26T05:04:25.001 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:25.031 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:25.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:25.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:04:24 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:04:25.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:24 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:04:25.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:24 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:04:25.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:24 smithi132 ceph-mon[33500]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T05:04:25.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:24 smithi132 ceph-mon[33500]: pgmap v157: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 577 op/s 2024-03-26T05:04:25.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:24 smithi132 ceph-mon[33500]: 7.4 scrub ok 2024-03-26T05:04:25.132 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:25.162 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:25.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:25.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:25.266 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:25.294 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:25.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:25.396 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:25.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:25.425 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:25.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:25.526 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:25.552 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:25.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:25.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:25.654 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:25.681 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:25.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:25.785 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:25.811 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:25.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:25.816 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:25.913 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:25.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:25.940 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:25.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:25.962 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[29691]: 3.d scrub starts 2024-03-26T05:04:25.962 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[29691]: 3.d scrub ok 2024-03-26T05:04:25.962 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[29691]: 4.f scrub starts 2024-03-26T05:04:25.962 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[29691]: 4.f scrub ok 2024-03-26T05:04:25.962 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[29691]: 7.9 scrub starts 2024-03-26T05:04:25.962 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[29691]: 4.5 scrub starts 2024-03-26T05:04:25.962 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[29691]: 4.5 scrub ok 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[29691]: 4.1 scrub starts 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[29691]: 4.1 scrub ok 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[40404]: 3.d scrub starts 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[40404]: 3.d scrub ok 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[40404]: 4.f scrub starts 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[40404]: 4.f scrub ok 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[40404]: 7.9 scrub starts 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[40404]: 4.5 scrub starts 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[40404]: 4.5 scrub ok 2024-03-26T05:04:25.963 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:25.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[40404]: 4.1 scrub starts 2024-03-26T05:04:25.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[40404]: 4.1 scrub ok 2024-03-26T05:04:25.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:04:25.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:25 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:04:25.972 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:04:26.021 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:04:26.041 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:25 smithi132 ceph-mon[33500]: 3.d scrub starts 2024-03-26T05:04:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:25 smithi132 ceph-mon[33500]: 3.d scrub ok 2024-03-26T05:04:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:25 smithi132 ceph-mon[33500]: 4.f scrub starts 2024-03-26T05:04:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:25 smithi132 ceph-mon[33500]: 4.f scrub ok 2024-03-26T05:04:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:25 smithi132 ceph-mon[33500]: 7.9 scrub starts 2024-03-26T05:04:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:25 smithi132 ceph-mon[33500]: 4.5 scrub starts 2024-03-26T05:04:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:25 smithi132 ceph-mon[33500]: 4.5 scrub ok 2024-03-26T05:04:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:25 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:25 smithi132 ceph-mon[33500]: 4.1 scrub starts 2024-03-26T05:04:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:25 smithi132 ceph-mon[33500]: 4.1 scrub ok 2024-03-26T05:04:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:25 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:04:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:25 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:04:26.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:26.068 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:04:26.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:26.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:26.116 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:04:26.159 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:04:26.169 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:26.201 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:04:26.213 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:26.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:26.243 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:04:26.288 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:04:26.294 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:04:26.294 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:04:26.295 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:04:26.295 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:04:26.295 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:04:26.295 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:04:26.295 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:04:26.295 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:04:26.315 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:26.359 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:26.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:26.461 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:26.516 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:26.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:26.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:26.618 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:26.647 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:26.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:26.748 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:26.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:26.790 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:26.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:26.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[29691]: 6.3 scrub starts 2024-03-26T05:04:26.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[29691]: 6.3 scrub ok 2024-03-26T05:04:26.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[29691]: 6.c scrub starts 2024-03-26T05:04:26.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[29691]: 6.c scrub ok 2024-03-26T05:04:26.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[29691]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T05:04:26.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[29691]: pgmap v159: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-26T05:04:26.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2094873490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:04:26.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[40404]: 6.3 scrub starts 2024-03-26T05:04:26.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[40404]: 6.3 scrub ok 2024-03-26T05:04:26.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[40404]: 6.c scrub starts 2024-03-26T05:04:26.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[40404]: 6.c scrub ok 2024-03-26T05:04:26.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[40404]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T05:04:26.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[40404]: pgmap v159: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-26T05:04:26.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:26 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2094873490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:04:26.891 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:26.921 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:26.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:27.023 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:27.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:26 smithi132 ceph-mon[33500]: 6.3 scrub starts 2024-03-26T05:04:27.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:26 smithi132 ceph-mon[33500]: 6.3 scrub ok 2024-03-26T05:04:27.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:26 smithi132 ceph-mon[33500]: 6.c scrub starts 2024-03-26T05:04:27.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:26 smithi132 ceph-mon[33500]: 6.c scrub ok 2024-03-26T05:04:27.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:26 smithi132 ceph-mon[33500]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T05:04:27.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:26 smithi132 ceph-mon[33500]: pgmap v159: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-26T05:04:27.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:26 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2094873490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:04:27.055 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:27.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:27.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:27.158 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:27.194 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:27.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:27.296 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:27.339 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:27.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:27.440 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:27.489 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:27.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:27.591 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:27.622 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:27.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:27.723 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:27.763 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:27.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:27.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:27.865 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:27.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:27 smithi080 ceph-mon[29691]: 5.0 scrub starts 2024-03-26T05:04:27.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:27 smithi080 ceph-mon[29691]: 5.0 scrub ok 2024-03-26T05:04:27.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:27 smithi080 ceph-mon[29691]: 7.9 scrub ok 2024-03-26T05:04:27.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:27 smithi080 ceph-mon[40404]: 5.0 scrub starts 2024-03-26T05:04:27.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:27 smithi080 ceph-mon[40404]: 5.0 scrub ok 2024-03-26T05:04:27.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:27 smithi080 ceph-mon[40404]: 7.9 scrub ok 2024-03-26T05:04:27.901 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:27.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:28.003 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:28.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:27 smithi132 ceph-mon[33500]: 5.0 scrub starts 2024-03-26T05:04:28.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:27 smithi132 ceph-mon[33500]: 5.0 scrub ok 2024-03-26T05:04:28.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:27 smithi132 ceph-mon[33500]: 7.9 scrub ok 2024-03-26T05:04:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:28.046 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:28.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:28.147 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:28.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:28.186 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:28.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:28.287 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:28.320 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:28.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:28.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:28.422 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:28.461 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:28.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:28.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:28.563 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:28.592 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:28.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:28.694 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:28.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:28.732 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:28.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:28.747 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:04:28.833 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:28.875 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:28.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:28.977 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:28.995 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:28 smithi080 ceph-mon[40404]: 6.12 scrub starts 2024-03-26T05:04:28.995 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:28 smithi080 ceph-mon[40404]: 6.12 scrub ok 2024-03-26T05:04:28.996 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:28 smithi080 ceph-mon[40404]: pgmap v160: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T05:04:28.996 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:28 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1787116813' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:04:28.996 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:28 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:04:28.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:28 smithi080 ceph-mon[29691]: 6.12 scrub starts 2024-03-26T05:04:28.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:28 smithi080 ceph-mon[29691]: 6.12 scrub ok 2024-03-26T05:04:28.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:28 smithi080 ceph-mon[29691]: pgmap v160: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T05:04:28.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:28 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1787116813' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:04:28.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:28 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:04:29.023 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:29.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:29.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:29.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:28 smithi132 ceph-mon[33500]: 6.12 scrub starts 2024-03-26T05:04:29.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:28 smithi132 ceph-mon[33500]: 6.12 scrub ok 2024-03-26T05:04:29.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:28 smithi132 ceph-mon[33500]: pgmap v160: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T05:04:29.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:28 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1787116813' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:04:29.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:28 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:04:29.125 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:29.165 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:29.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:29.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:29.267 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:29.300 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:29.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:29.401 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:29.441 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:29.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:29.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:29.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:29.572 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:29.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:29.673 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:29.729 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:29.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:29.830 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:29.860 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:29.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:29.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:29 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:04:29.872 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:29 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:04:29.872 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:29 smithi080 ceph-mon[40404]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T05:04:29.872 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:29 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:04:29.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:29 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:04:29.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:29 smithi080 ceph-mon[29691]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T05:04:29.963 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:29.993 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:29.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:29.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:30.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:29 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:04:30.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:29 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:04:30.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:29 smithi132 ceph-mon[33500]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T05:04:30.095 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:30.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:30.126 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:30.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:30.228 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:30.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:30.269 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:30.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:30.371 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:30.396 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:30.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:30.498 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:30.539 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:30.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:30.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:30.641 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:30.671 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:30.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:30.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:30.773 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:30.804 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:30.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:30.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:30.906 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:30.942 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:30.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:30.991 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:30 smithi080 ceph-mon[29691]: pgmap v162: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-26T05:04:30.991 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:30 smithi080 ceph-mon[40404]: pgmap v162: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-26T05:04:31.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:30 smithi132 ceph-mon[33500]: pgmap v162: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-26T05:04:31.043 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:31.070 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:31.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:31.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:31.172 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:31.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:31.201 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:31.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:31.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:31.328 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:31.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:31.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:31.430 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:31.458 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:31.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:31.560 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:31.588 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:31.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:31.690 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:31.717 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:31.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:31.743 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:04:31.751 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.6 to 1 (10000) 2024-03-26T05:04:31.818 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:31.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:31 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/153902817' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:04:31.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:31 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/153902817' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:04:31.879 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:31.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:31.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:31.981 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:32.037 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:32.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:31 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/153902817' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:04:32.138 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:32.189 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:32.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:32.290 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:32.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:32.319 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:32.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:32.352 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:04:32.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:32.448 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:32.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:32.475 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 7 1 2024-03-26T05:04:32.550 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:32.577 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:32.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:32.679 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:32.708 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:32.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:32.809 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:32.855 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:32.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:32.946 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:04:32.946 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:04:32.956 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:32.970 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:33.026 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:33.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:33.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:32 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/153902817' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:04:33.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:32 smithi132 ceph-mon[33500]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T05:04:33.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:32 smithi132 ceph-mon[33500]: pgmap v164: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 843 B/s rd, 0 B/s wr, 809 op/s 2024-03-26T05:04:33.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:32 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/153902817' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:04:33.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:32 smithi080 ceph-mon[29691]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T05:04:33.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:32 smithi080 ceph-mon[29691]: pgmap v164: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 843 B/s rd, 0 B/s wr, 809 op/s 2024-03-26T05:04:33.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:32 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/153902817' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:04:33.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:32 smithi080 ceph-mon[40404]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T05:04:33.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:32 smithi080 ceph-mon[40404]: pgmap v164: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 843 B/s rd, 0 B/s wr, 809 op/s 2024-03-26T05:04:33.128 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:33.178 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:33.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:33.227 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:33.279 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:33.345 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:33.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:33.447 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:33.474 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:33.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:33.576 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:33.605 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:33.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:33.706 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:33.745 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:33.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:33.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:33.846 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:33.885 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:33.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:33.942 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T05:04:33.987 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:34.062 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:34.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:34.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:34.192 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:34.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:34.294 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:34.346 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:34.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:34.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:34.368 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:04:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:04:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:04:34.394 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:34.448 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:34.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:34.479 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:34.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:34.580 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:34.611 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:34.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:34.733 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:34.777 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:34.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:34.878 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:34.909 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:34.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:35.010 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:35.057 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:35.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:35.129 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:35.158 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:35.191 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:35.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:35.292 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:35 smithi132 ceph-mon[33500]: pgmap v165: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 613 op/s 2024-03-26T05:04:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:04:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:35.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:04:34 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:04:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:35.323 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:35.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:35.331 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:35 smithi080 ceph-mon[29691]: pgmap v165: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 613 op/s 2024-03-26T05:04:35.331 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:04:35.332 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:35.332 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:35 smithi080 ceph-mon[40404]: pgmap v165: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 613 op/s 2024-03-26T05:04:35.332 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:04:35.332 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:35.423 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:35.452 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:35.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:35.553 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:35.582 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:35.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:35.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:35.684 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:35.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:35.715 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:35.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:35.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:35.845 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:35.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:35.947 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:35.991 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:35.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:36.093 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:36.133 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:36.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:36.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:36.234 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:36.263 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:36.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:36.294 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:36.295 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:36.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:36.396 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:36.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:36.498 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:36.539 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:36.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:36.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:36.641 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:36.665 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:36.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:36.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:36.767 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:36.799 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:36.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:36.901 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:36.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:36.933 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:36.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:37.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:37.061 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:37.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:37.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:37.162 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:37.190 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:37.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:37.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:37.291 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:37.323 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:37.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:37.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:37 smithi080 ceph-mon[40404]: pgmap v166: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-26T05:04:37.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:37 smithi080 ceph-mon[29691]: pgmap v166: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-26T05:04:37.424 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:37.461 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:37.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:37.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:37 smithi132 ceph-mon[33500]: pgmap v166: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-26T05:04:37.561 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:37.604 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:37.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:37.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:37.706 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:37.738 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:37.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:37.840 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:37.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:37.864 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:37.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:37.965 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:37.995 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:37.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:38.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:38.133 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:38.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:38.235 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:38.281 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:38.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:38.382 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:38.421 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:38.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:38.522 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:38.571 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:38.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:38.673 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:38.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:38.725 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:38.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:38.826 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:38.854 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:38.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:38.956 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:38.984 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:38.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:38.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:39.086 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:39.112 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:39.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:39.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:39 smithi080 ceph-mon[40404]: pgmap v167: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-26T05:04:39.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:39 smithi080 ceph-mon[29691]: pgmap v167: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-26T05:04:39.213 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:39.255 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:39.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:39.356 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:39.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:39.386 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:39.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:39.487 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:39.527 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:39.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:39.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:39 smithi132 ceph-mon[33500]: pgmap v167: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-26T05:04:39.627 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:39.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:39.664 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:39.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:39.730 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:04:39.730 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:04:39.766 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:39.808 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:39.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:39.910 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:39.934 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:39.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:40.035 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:40.071 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:40.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:40.110 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:04:40.172 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:40.201 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:40.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:40.303 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:40.330 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:40.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:40.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:40 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3492606248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:04:40.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:40 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3492606248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:04:40.431 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:40.460 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:40.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:40.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:40.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:40 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3492606248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:04:40.561 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:40.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:40.605 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:40.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:40.706 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:40.749 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:40.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:40.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:40.851 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:40.897 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:40.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:40.999 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:41.036 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:41.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:41.138 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:41.172 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:41.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:41.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:41.274 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:41.301 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:41.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:41.357 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:41 smithi080 ceph-mon[40404]: pgmap v168: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 913 op/s 2024-03-26T05:04:41.358 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:41 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:04:41.358 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:41 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3492606248' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:04:41.358 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:41 smithi080 ceph-mon[40404]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T05:04:41.358 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:41 smithi080 ceph-mon[29691]: pgmap v168: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 913 op/s 2024-03-26T05:04:41.358 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:41 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:04:41.358 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:41 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3492606248' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:04:41.358 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:41 smithi080 ceph-mon[29691]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T05:04:41.403 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:41.447 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:41.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:41.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:41 smithi132 ceph-mon[33500]: pgmap v168: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 913 op/s 2024-03-26T05:04:41.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:41 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:04:41.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:41 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3492606248' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:04:41.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:41 smithi132 ceph-mon[33500]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T05:04:41.548 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:41.576 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:41.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:41.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:41.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:41.715 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:41.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:41.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:41.844 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:41.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:41.946 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:41.974 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:41.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:42.076 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:42.103 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:42.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:42.205 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:42.234 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:42.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:42.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:42.289 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T05:04:42.337 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:42.377 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:42.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:42.479 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:42.508 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:42.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:42.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:42.608 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:04:42.610 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:42.652 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:42.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:42.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:42.754 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:42.785 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:42.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:42.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:42.887 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:42.891 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:04:42.917 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:42.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:43.019 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:43.046 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:43.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:43.149 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:43.191 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:43.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:43.293 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:43.355 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:43.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:43.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:43.457 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:43.487 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:43.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:43.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:43.571 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:43 smithi080 ceph-mon[29691]: pgmap v170: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-26T05:04:43.572 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:43 smithi080 ceph-mon[40404]: pgmap v170: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-26T05:04:43.588 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:43.636 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:43.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:43.737 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:43.788 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:43.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:43.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:43 smithi132 ceph-mon[33500]: pgmap v170: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-26T05:04:43.860 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:04:43.889 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:43.915 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:04:43.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:43.934 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:43.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:43.964 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:04:44.025 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:04:44.035 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:44.069 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:44.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:44.080 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:04:44.082 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:44.118 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:04:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:04:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:04:44.126 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:04:44.171 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:44.189 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:04:44.200 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:44.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:44.242 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:04:44.251 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:04:44.251 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:04:44.252 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:04:44.252 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:04:44.252 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:04:44.252 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:04:44.252 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:04:44.252 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:04:44.301 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:44.328 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:44.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:44.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:44.430 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:44.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:44.461 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:44.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:44.562 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:44.594 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:44.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:44.665 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:44 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:04:44.665 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:44 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:04:44.665 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:44 smithi132 ceph-mon[33500]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T05:04:44.665 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:44 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2580071543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:04:44.696 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:44.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:44 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:04:44.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:44 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:04:44.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:44 smithi080 ceph-mon[29691]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T05:04:44.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:44 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2580071543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:04:44.711 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:44 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:04:44.711 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:44 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:04:44.711 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:44 smithi080 ceph-mon[40404]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T05:04:44.711 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:44 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2580071543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:04:44.733 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:44.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:44.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:44.835 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:44.868 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:44.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:44.969 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:44.998 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:04:44 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:04:44.998 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:04:44.998 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.24701.0:155440 7.4 e147/103 7:26b9d291:::benchmark_data_smithi080_80848_object35941:head v 147'9583, mlcod=147'9581)", 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:44.998837+0000", 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0064428970000000004, 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0064748640000000003, 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.998837+0000", 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.998837+0000", 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.998838+0000", 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.8599999999999999e-07 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.998841+0000", 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.016e-06 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.998845+0000", 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5739999999999997e-06 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.998848+0000", 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.7819999999999998e-06 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.998897+0000", 2024-03-26T05:04:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8621000000000001e-05 2024-03-26T05:04:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.998920+0000", 2024-03-26T05:04:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.3723999999999999e-05 2024-03-26T05:04:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:155441 7.1 7:8902bffe:::benchmark_data_smithi080_80848_object35942:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:44.999522+0000", 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0057587070000000001, 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0058606020000000003, 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 155441 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.999522+0000", 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.999522+0000", 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.999523+0000", 2024-03-26T05:04:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.615e-06 2024-03-26T05:04:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.999530+0000", 2024-03-26T05:04:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9179999999999997e-06 2024-03-26T05:04:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.999546+0000", 2024-03-26T05:04:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5889999999999999e-05 2024-03-26T05:04:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.999555+0000", 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0699999999999996e-06 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.999614+0000", 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8517000000000002e-05 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.999747+0000", 2024-03-26T05:04:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013326700000000001 2024-03-26T05:04:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:44.999878+0000", 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013100500000000001 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:155448 7.1 7:87bf379a:::benchmark_data_smithi080_80848_object35949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:45.001390+0000", 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0038906840000000002, 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040715919999999997, 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:04:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 155448 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001390+0000", 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001390+0000", 2024-03-26T05:04:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001391+0000", 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2190000000000001e-06 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001399+0000", 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3159999999999997e-06 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001418+0000", 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.8355999999999998e-05 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001427+0000", 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0839999999999998e-06 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.002300+0000", 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087330100000000005 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.002408+0000", 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010779400000000001 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.002514+0000", 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000105732 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.24701.0:155447 7.3 e147/103 7:c31a7a8e:::benchmark_data_smithi080_80848_object35948:head v 147'9729, mlcod=147'9727)", 2024-03-26T05:04:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:45.001715+0000", 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0035658830000000002, 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0038224470000000001, 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001715+0000", 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001715+0000", 2024-03-26T05:04:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001715+0000", 2024-03-26T05:04:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5199999999999995e-07 2024-03-26T05:04:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001720+0000", 2024-03-26T05:04:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4630000000000003e-06 2024-03-26T05:04:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001735+0000", 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5498e-05 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001743+0000", 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.17e-06 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001799+0000", 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6461e-05 2024-03-26T05:04:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:45.001819+0000", 2024-03-26T05:04:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.0397000000000001e-05 2024-03-26T05:04:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-03-26T05:04:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 4 2024-03-26T05:04:45.016 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:04:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:45.018 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:45.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:45.119 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:45.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:45.159 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:45.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:45.260 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:45.296 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:45.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:45.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:45.399 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:45.424 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:45.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:45.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:45.433 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.7 to 1 (10000) 2024-03-26T05:04:45.526 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:45.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:45.559 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:45.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:45.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:45 smithi080 ceph-mon[29691]: pgmap v172: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 656 op/s 2024-03-26T05:04:45.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:45 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3086085858' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:04:45.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:45 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:04:45.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:45 smithi080 ceph-mon[40404]: pgmap v172: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 656 op/s 2024-03-26T05:04:45.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:45 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3086085858' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:04:45.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:45 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:04:45.661 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:45.699 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:45.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:45.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:45.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:45 smithi132 ceph-mon[33500]: pgmap v172: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 656 op/s 2024-03-26T05:04:45.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:45 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3086085858' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:04:45.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:45 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:04:45.800 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:45.828 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:45.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:45.929 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:45.960 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:45.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:46.062 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:46.098 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:46.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:46.199 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:46.237 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:46.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:46.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:46.339 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:46.372 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:46.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:46.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:46.425 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T05:04:46.474 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:46.508 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:46.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:46.610 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:46.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:46.651 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:46.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:46.669 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:46 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:46.669 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:46 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:04:46.669 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:46 smithi132 ceph-mon[33500]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T05:04:46.753 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:46.773 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:04:46.779 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:46.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:46.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:46.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:46 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:46.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:46 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:04:46.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:46 smithi080 ceph-mon[29691]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T05:04:46.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:46 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:46.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:46 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:04:46.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:46 smithi080 ceph-mon[40404]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T05:04:46.881 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:46.915 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:46.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:47.017 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:47.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:47.061 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:47.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:47.162 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:47.187 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:47.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:47.289 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:47.329 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:47.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:47.430 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:47.455 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:47.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:47.538 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:04:47.556 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:47.635 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:47.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:47.737 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:47.762 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:47.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:47 smithi132 ceph-mon[33500]: pgmap v174: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.60k op/s 2024-03-26T05:04:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:47 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:04:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:47 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:04:47.863 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:47.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:47 smithi080 ceph-mon[29691]: pgmap v174: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.60k op/s 2024-03-26T05:04:47.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:47 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:04:47.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:47 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:04:47.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:47 smithi080 ceph-mon[40404]: pgmap v174: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.60k op/s 2024-03-26T05:04:47.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:47 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:04:47.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:47 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:04:47.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:47.896 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:47.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:47.997 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:48.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:48.035 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:48.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:48.136 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:48.163 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:48.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:48.264 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:48.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:48.302 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:48.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:48.403 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:48.439 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:48.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:48.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:48.541 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:48.581 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:48.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:48.596 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:04:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:04:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2024-03-26T05:04:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2024-03-26T05:04:48.597 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:04:48.682 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:48.719 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:48.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:48.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:48 smithi132 ceph-mon[33500]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T05:04:48.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:48 smithi132 ceph-mon[33500]: pgmap v176: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 877 op/s 2024-03-26T05:04:48.821 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:48.866 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:48.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:48.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:48.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:48 smithi080 ceph-mon[40404]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T05:04:48.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:48 smithi080 ceph-mon[40404]: pgmap v176: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 877 op/s 2024-03-26T05:04:48.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:48 smithi080 ceph-mon[29691]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T05:04:48.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:48 smithi080 ceph-mon[29691]: pgmap v176: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 877 op/s 2024-03-26T05:04:48.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:48.996 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:48.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:49.002 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T05:04:49.002 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:04:49.097 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:49.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:49.126 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:49.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:49.227 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:49.276 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:49.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:49.377 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:49.423 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T05:04:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:49.434 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:49.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:49.534 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:49.535 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:49.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:49.598 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:49.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:49.665 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:49.665 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:49.699 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:49.743 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:04:49.746 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:49.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:49.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:49.847 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:49.874 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:49.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:49.977 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:50.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:50.006 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:50.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:50.107 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:50.145 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:50.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:50.246 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:50.260 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:50.272 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:50.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:50.374 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:50.418 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:50.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:50.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:50.520 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:50.546 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:50.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:50.647 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:50.673 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:50.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:50.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:50.757 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:50 smithi132 ceph-mon[33500]: pgmap v177: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-26T05:04:50.757 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:50 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:04:50.757 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:50 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:04:50.776 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:50.807 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:50 smithi080 ceph-mon[40404]: pgmap v177: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-26T05:04:50.807 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:50 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:04:50.808 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:50 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:04:50.808 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:50 smithi080 ceph-mon[29691]: pgmap v177: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-26T05:04:50.808 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:50 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:04:50.808 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:50 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:04:50.810 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:50.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:50.912 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:50.937 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:50.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:51.039 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:51.070 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:51.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:51.172 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:51.226 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:51.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:51.327 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:51.359 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:51.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:51.461 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:51.490 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:51.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:51.524 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:04:51.592 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:51.623 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:51 smithi080 ceph-mon[40404]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T05:04:51.623 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:51 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2342020041' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:04:51.623 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:51 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:04:51.623 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:51 smithi080 ceph-mon[29691]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T05:04:51.624 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:51 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2342020041' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:04:51.624 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:51 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:04:51.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:51.640 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:51.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:51.741 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:51.763 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:51 smithi132 ceph-mon[33500]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T05:04:51.763 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:51 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2342020041' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:04:51.763 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:51 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:04:51.772 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:51.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:51.874 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:51.904 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:51.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:52.006 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:52.039 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:52.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:52.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:52.141 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:52.175 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:52.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:52.277 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:52.308 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:52.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:52.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:52.411 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:52.441 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:52.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:52.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:52 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:04:52.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:52 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:04:52.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:52 smithi132 ceph-mon[33500]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T05:04:52.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:52 smithi132 ceph-mon[33500]: pgmap v180: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 810 op/s 2024-03-26T05:04:52.544 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:52.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:52.583 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:52.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:52.685 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:52.717 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:52.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:52.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:52.785 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:52 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:04:52.786 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:52 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:04:52.786 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:52 smithi080 ceph-mon[29691]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T05:04:52.786 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:52 smithi080 ceph-mon[29691]: pgmap v180: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 810 op/s 2024-03-26T05:04:52.786 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:52 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:04:52.786 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:52 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:04:52.786 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:52 smithi080 ceph-mon[40404]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T05:04:52.786 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:52 smithi080 ceph-mon[40404]: pgmap v180: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 810 op/s 2024-03-26T05:04:52.819 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:52.849 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:52.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:52.951 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:52.985 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:52.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:53.087 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:53.156 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:53.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:53.257 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:53.297 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:53.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:53.337 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:04:53.346 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:04:53.399 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:53.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:53.439 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:53.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:53.541 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:53.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:53 smithi132 ceph-mon[33500]: 5.6 scrub starts 2024-03-26T05:04:53.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:53 smithi132 ceph-mon[33500]: 5.6 scrub ok 2024-03-26T05:04:53.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:53 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:04:53.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:53 smithi132 ceph-mon[33500]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:04:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:53.565 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:53.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:53.596 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:53 smithi080 ceph-mon[40404]: 5.6 scrub starts 2024-03-26T05:04:53.597 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:53 smithi080 ceph-mon[29691]: 5.6 scrub starts 2024-03-26T05:04:53.597 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:53 smithi080 ceph-mon[29691]: 5.6 scrub ok 2024-03-26T05:04:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:04:53.604 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:04:53.604 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:04:53.604 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:04:53.605 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.605 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18340 7.1 7:8b7e6025:::benchmark_data_smithi080_80848_object18339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:04:53.605 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.064655+0000", 2024-03-26T05:04:53.605 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 128.54355869099999, 2024-03-26T05:04:53.605 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.22265895099999999, 2024-03-26T05:04:53.605 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.605 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.605 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.606 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.606 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.606 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18340 2024-03-26T05:04:53.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.606 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.607 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064655+0000", 2024-03-26T05:04:53.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.607 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064655+0000", 2024-03-26T05:04:53.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064656+0000", 2024-03-26T05:04:53.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.424e-06 2024-03-26T05:04:53.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.608 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064739+0000", 2024-03-26T05:04:53.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2670999999999995e-05 2024-03-26T05:04:53.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.609 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064754+0000", 2024-03-26T05:04:53.609 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4780000000000001e-05 2024-03-26T05:04:53.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.609 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064763+0000", 2024-03-26T05:04:53.609 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8740000000000001e-06 2024-03-26T05:04:53.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.610 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064819+0000", 2024-03-26T05:04:53.610 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6740999999999998e-05 2024-03-26T05:04:53.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.610 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.610 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064924+0000", 2024-03-26T05:04:53.610 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104545 2024-03-26T05:04:53.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.610 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.610 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.065047+0000", 2024-03-26T05:04:53.610 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012353 2024-03-26T05:04:53.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.611 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.611 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.611 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.065997+0000", 2024-03-26T05:04:53.611 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00094926299999999999 2024-03-26T05:04:53.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.612 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.612 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.066103+0000", 2024-03-26T05:04:53.612 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000106575 2024-03-26T05:04:53.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.612 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.613 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287201+0000", 2024-03-26T05:04:53.613 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.22109801900000001 2024-03-26T05:04:53.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.613 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.613 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287266+0000", 2024-03-26T05:04:53.613 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4567999999999996e-05 2024-03-26T05:04:53.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.613 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.614 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287314+0000", 2024-03-26T05:04:53.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7960999999999997e-05 2024-03-26T05:04:53.614 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.614 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.614 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.614 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18358 7.1 7:8417ed37:::benchmark_data_smithi080_80848_object18357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:04:53.614 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.084602+0000", 2024-03-26T05:04:53.614 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 128.523611931, 2024-03-26T05:04:53.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20301822899999999, 2024-03-26T05:04:53.614 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18358 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.084602+0000", 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.084602+0000", 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.084603+0000", 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2330000000000001e-06 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.099940+0000", 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.015337491 2024-03-26T05:04:53.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.099960+0000", 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.9471999999999999e-05 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.099967+0000", 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2880000000000004e-06 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.100023+0000", 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6268999999999999e-05 2024-03-26T05:04:53.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.618 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.618 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.100143+0000", 2024-03-26T05:04:53.618 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011924700000000001 2024-03-26T05:04:53.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.618 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.618 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.100270+0000", 2024-03-26T05:04:53.619 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012700999999999999 2024-03-26T05:04:53.619 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.619 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.619 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.101482+0000", 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001212185 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.101961+0000", 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00047930800000000001 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287548+0000", 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.185586844 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287582+0000", 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.3637000000000003e-05 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287620+0000", 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8244999999999997e-05 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.621 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18365 7.1 7:86ff0e4b:::benchmark_data_smithi080_80848_object18364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.102039+0000", 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 128.50617421800001, 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18571283699999999, 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18365 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.622 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102039+0000", 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102039+0000", 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102040+0000", 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.139e-06 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102104+0000", 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3891000000000002e-05 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102114+0000", 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9869999999999994e-06 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102122+0000", 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4699999999999996e-06 2024-03-26T05:04:53.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102179+0000", 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7643999999999999e-05 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102319+0000", 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013995500000000001 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102422+0000", 2024-03-26T05:04:53.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000102944 2024-03-26T05:04:53.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.626 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.626 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.103247+0000", 2024-03-26T05:04:53.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082511399999999995 2024-03-26T05:04:53.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.626 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.626 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.103513+0000", 2024-03-26T05:04:53.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000265944 2024-03-26T05:04:53.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.627 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.627 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287662+0000", 2024-03-26T05:04:53.627 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18414857300000001 2024-03-26T05:04:53.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.627 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.627 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287715+0000", 2024-03-26T05:04:53.627 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2778999999999997e-05 2024-03-26T05:04:53.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287752+0000", 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7397000000000001e-05 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18384 7.1 7:85810267:::benchmark_data_smithi080_80848_object18383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.119204+0000", 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 128.48900968300001, 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16863742600000001, 2024-03-26T05:04:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.629 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.629 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.629 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.629 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.629 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18384 2024-03-26T05:04:53.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.629 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.629 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119204+0000", 2024-03-26T05:04:53.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119204+0000", 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119206+0000", 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.906e-06 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119341+0000", 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013563799999999999 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119355+0000", 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3217000000000001e-05 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119372+0000", 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.7275999999999999e-05 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119434+0000", 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1624000000000005e-05 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119579+0000", 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000145859 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119714+0000", 2024-03-26T05:04:53.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013468599999999999 2024-03-26T05:04:53.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.633 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.633 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.633 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.121523+0000", 2024-03-26T05:04:53.633 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0018088920000000001 2024-03-26T05:04:53.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.633 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.634 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.122062+0000", 2024-03-26T05:04:53.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053930099999999995 2024-03-26T05:04:53.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.634 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287780+0000", 2024-03-26T05:04:53.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16571784000000001 2024-03-26T05:04:53.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.634 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287810+0000", 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.9467000000000001e-05 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287841+0000", 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.1720000000000001e-05 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18416 7.1 7:83212609:::benchmark_data_smithi080_80848_object18415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:04:53.635 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.136847+0000", 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 128.47136681500001, 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.15127143200000001, 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18416 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136847+0000", 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136847+0000", 2024-03-26T05:04:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136848+0000", 2024-03-26T05:04:53.638 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1960000000000001e-06 2024-03-26T05:04:53.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.638 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.638 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136920+0000", 2024-03-26T05:04:53.638 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2155000000000003e-05 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136933+0000", 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2717999999999999e-05 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136939+0000", 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0950000000000001e-06 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.639 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136995+0000", 2024-03-26T05:04:53.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6252999999999998e-05 2024-03-26T05:04:53.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.137133+0000", 2024-03-26T05:04:53.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013836400000000001 2024-03-26T05:04:53.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.137300+0000", 2024-03-26T05:04:53.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000166074 2024-03-26T05:04:53.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.642 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.642 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.138066+0000", 2024-03-26T05:04:53.642 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00076619299999999997 2024-03-26T05:04:53.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.642 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.642 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.138417+0000", 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035126500000000002 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.288055+0000", 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14963824100000001 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.288090+0000", 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.4411e-05 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.643 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.288118+0000", 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.8467e-05 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18428 7.1 7:8c797d16:::benchmark_data_smithi080_80848_object18427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.154718+0000", 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 128.45349565800001, 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13353142200000001, 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.644 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18428 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154718+0000", 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154718+0000", 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154719+0000", 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1650000000000001e-06 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154775+0000", 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6200999999999999e-05 2024-03-26T05:04:53.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154787+0000", 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2072000000000001e-05 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154793+0000", 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8340000000000004e-06 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154834+0000", 2024-03-26T05:04:53.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1192999999999997e-05 2024-03-26T05:04:53.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154933+0000", 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9018000000000006e-05 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.155021+0000", 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7899000000000007e-05 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.155727+0000", 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070548900000000001 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.156041+0000", 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00031450100000000002 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.288158+0000", 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13211668100000001 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.288211+0000", 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2611000000000002e-05 2024-03-26T05:04:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.288249+0000", 2024-03-26T05:04:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8757999999999997e-05 2024-03-26T05:04:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22561 7.1 7:8ddb660b:::benchmark_data_smithi080_80848_object22560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:04:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.343328+0000", 2024-03-26T05:04:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 126.26488574299999, 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14323868000000001, 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22561 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343328+0000", 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343328+0000", 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343329+0000", 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5209999999999999e-06 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343406+0000", 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6990999999999995e-05 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343417+0000", 2024-03-26T05:04:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.101e-05 2024-03-26T05:04:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343426+0000", 2024-03-26T05:04:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4719999999999995e-06 2024-03-26T05:04:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343486+0000", 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0572999999999998e-05 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343618+0000", 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013210300000000001 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343789+0000", 2024-03-26T05:04:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017039099999999999 2024-03-26T05:04:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345355+0000", 2024-03-26T05:04:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001565821 2024-03-26T05:04:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.346123+0000", 2024-03-26T05:04:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00076833499999999998 2024-03-26T05:04:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.486489+0000", 2024-03-26T05:04:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14036646 2024-03-26T05:04:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.486534+0000", 2024-03-26T05:04:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4342000000000003e-05 2024-03-26T05:04:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.486566+0000", 2024-03-26T05:04:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2660999999999998e-05 2024-03-26T05:04:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:28176 7.1 7:843c9aac:::benchmark_data_smithi080_80848_object28175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:04:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:50.428842+0000", 2024-03-26T05:04:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 123.179371718, 2024-03-26T05:04:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.084973426000000005, 2024-03-26T05:04:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 28176 2024-03-26T05:04:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.428842+0000", 2024-03-26T05:04:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.428842+0000", 2024-03-26T05:04:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.428842+0000", 2024-03-26T05:04:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4899999999999995e-07 2024-03-26T05:04:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.428892+0000", 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9326000000000002e-05 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.428901+0000", 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.1050000000000001e-06 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.428933+0000", 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2363999999999997e-05 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.430186+0000", 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001252888 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.430283+0000", 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7157999999999994e-05 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.430424+0000", 2024-03-26T05:04:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014127599999999999 2024-03-26T05:04:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.433477+0000", 2024-03-26T05:04:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0030527409999999999 2024-03-26T05:04:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.433498+0000", 2024-03-26T05:04:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.1075999999999999e-05 2024-03-26T05:04:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.513738+0000", 2024-03-26T05:04:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.080239315000000005 2024-03-26T05:04:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.513785+0000", 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7089000000000003e-05 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:50.513815+0000", 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0539e-05 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31393 7.1 7:8ed06db3:::benchmark_data_smithi080_80848_object31392:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.251817+0000", 2024-03-26T05:04:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 121.356396828, 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13570653199999999, 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31393 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251817+0000", 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251817+0000", 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251818+0000", 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5200000000000001e-06 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251861+0000", 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3078000000000003e-05 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251874+0000", 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2662e-05 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251882+0000", 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9610000000000005e-06 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251937+0000", 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4648999999999998e-05 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252055+0000", 2024-03-26T05:04:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000118832 2024-03-26T05:04:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252183+0000", 2024-03-26T05:04:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012804099999999999 2024-03-26T05:04:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.253129+0000", 2024-03-26T05:04:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00094583600000000003 2024-03-26T05:04:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.255948+0000", 2024-03-26T05:04:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0028189959999999998 2024-03-26T05:04:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387405+0000", 2024-03-26T05:04:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13145715899999999 2024-03-26T05:04:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387469+0000", 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3329000000000003e-05 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387523+0000", 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4469000000000003e-05 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31398 7.1 7:8f429419:::benchmark_data_smithi080_80848_object31397:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.254661+0000", 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 121.353552659, 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.134356052, 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31398 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254661+0000", 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254661+0000", 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254662+0000", 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0380000000000001e-06 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254706+0000", 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4140000000000001e-05 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254716+0000", 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7580000000000004e-06 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254723+0000", 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1489999999999997e-06 2024-03-26T05:04:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254773+0000", 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.028e-05 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254868+0000", 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4629999999999994e-05 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.255006+0000", 2024-03-26T05:04:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000138434 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.256093+0000", 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010865339999999999 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.257095+0000", 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010021159999999999 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388900+0000", 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13180470799999999 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388970+0000", 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0507999999999996e-05 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.389017+0000", 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6757e-05 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31415 7.1 7:86e7a695:::benchmark_data_smithi080_80848_object31414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.263992+0000", 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 121.344221459, 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.125213354, 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31415 2024-03-26T05:04:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.263992+0000", 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.263992+0000", 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.263994+0000", 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.635e-06 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.264034+0000", 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9892999999999999e-05 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.264044+0000", 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0338e-05 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.264050+0000", 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9009999999999999e-06 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.264096+0000", 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6056000000000001e-05 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.264188+0000", 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2281999999999995e-05 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.264282+0000", 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4197999999999995e-05 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.265309+0000", 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010261479999999999 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.265787+0000", 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00047891299999999997 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.389074+0000", 2024-03-26T05:04:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.123286847 2024-03-26T05:04:53.684 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.389165+0000", 2024-03-26T05:04:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0341999999999997e-05 2024-03-26T05:04:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.389205+0000", 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0800999999999998e-05 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31421 7.1 7:8d6f8e09:::benchmark_data_smithi080_80848_object31420:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.266844+0000", 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 121.341369495, 2024-03-26T05:04:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.12681271899999999, 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31421 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266844+0000", 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266844+0000", 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266845+0000", 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0800000000000003e-07 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266891+0000", 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5985000000000001e-05 2024-03-26T05:04:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266901+0000", 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.004e-05 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266907+0000", 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7790000000000001e-06 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266958+0000", 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1307000000000002e-05 2024-03-26T05:04:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.267044+0000", 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6434000000000006e-05 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.267133+0000", 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8412999999999995e-05 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268018+0000", 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088498500000000003 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383541+0000", 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.115523535 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.393560+0000", 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010018681999999999 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.393614+0000", 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3761000000000002e-05 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.393657+0000", 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2889999999999998e-05 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(client.24701.0:31421 7.1 e111/103 ondisk, result = 0)", 2024-03-26T05:04:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.268204+0000", 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 121.340009075, 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.115345924, 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268204+0000", 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268204+0000", 2024-03-26T05:04:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268205+0000", 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8400000000000003e-07 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268212+0000", 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7229999999999998e-06 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268223+0000", 2024-03-26T05:04:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1517e-05 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268231+0000", 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.024e-06 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383525+0000", 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.115294747 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383534+0000", 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2310000000000002e-06 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383550+0000", 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5897999999999999e-05 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:72642 7.1 7:80736992:::benchmark_data_smithi080_80848_object72641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:03:21.835762+0000", 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.772451197999999, 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.089550371000000004, 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 72642 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.835762+0000", 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.835762+0000", 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.835764+0000", 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.728e-06 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.835841+0000", 2024-03-26T05:04:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6789e-05 2024-03-26T05:04:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.835853+0000", 2024-03-26T05:04:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1826e-05 2024-03-26T05:04:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.835861+0000", 2024-03-26T05:04:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7350000000000002e-06 2024-03-26T05:04:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.835908+0000", 2024-03-26T05:04:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6863999999999997e-05 2024-03-26T05:04:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.836043+0000", 2024-03-26T05:04:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013443400000000001 2024-03-26T05:04:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.836191+0000", 2024-03-26T05:04:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000148398 2024-03-26T05:04:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.838696+0000", 2024-03-26T05:04:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002505167 2024-03-26T05:04:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.839019+0000", 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00032282300000000002 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.925203+0000", 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.086184043000000002 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.925266+0000", 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2483999999999993e-05 2024-03-26T05:04:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:21.925313+0000", 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7080000000000003e-05 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:77844 7.1 7:8057a429:::benchmark_data_smithi080_80848_object77843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:03:26.347827+0000", 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 87.260387012999999, 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11528007, 2024-03-26T05:04:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 77844 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.347827+0000", 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.347827+0000", 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.347831+0000", 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2309999999999998e-06 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.347878+0000", 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6898999999999999e-05 2024-03-26T05:04:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350047+0000", 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021689029999999998 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350062+0000", 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5418999999999999e-05 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455338+0000", 2024-03-26T05:04:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.105275707 2024-03-26T05:04:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455486+0000", 2024-03-26T05:04:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000148455 2024-03-26T05:04:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455608+0000", 2024-03-26T05:04:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012162700000000001 2024-03-26T05:04:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.460540+0000", 2024-03-26T05:04:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0049318629999999999 2024-03-26T05:04:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.460581+0000", 2024-03-26T05:04:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0967000000000003e-05 2024-03-26T05:04:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.462999+0000", 2024-03-26T05:04:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0024186949999999998 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.463063+0000", 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3484000000000004e-05 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.463107+0000", 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3819999999999997e-05 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:77846 7.1 7:85e9d59f:::benchmark_data_smithi080_80848_object77845:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:03:26.350086+0000", 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 87.258127612999999, 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.115705316, 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 77846 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350086+0000", 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350086+0000", 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350087+0000", 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8899999999999998e-07 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350140+0000", 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2763000000000003e-05 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350151+0000", 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1508999999999999e-05 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350158+0000", 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3019999999999997e-06 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.458040+0000", 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10788157600000001 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.458169+0000", 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012913200000000001 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.458328+0000", 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000158426 2024-03-26T05:04:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.463157+0000", 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0048291560000000002 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.465562+0000", 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0024053759999999999 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.465639+0000", 2024-03-26T05:04:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7385999999999994e-05 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.465745+0000", 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000105134 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.465791+0000", 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6666999999999999e-05 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.24701.0:77840 7.4 e123/103 7:2ec32914:::benchmark_data_smithi080_80848_object77839:head v 123'4783, mlcod=122'4781)", 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:03:26.353585+0000", 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 87.254628319999995, 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10951504200000001, 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.353585+0000", 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.353585+0000", 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.353586+0000", 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.1e-07 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.353623+0000", 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7287999999999999e-05 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455155+0000", 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.101532487 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455164+0000", 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6819999999999992e-06 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455253+0000", 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8764000000000006e-05 2024-03-26T05:04:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455274+0000", 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.1390999999999999e-05 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.463065+0000", 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0077907289999999997 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.463100+0000", 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5391e-05 2024-03-26T05:04:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:77851 7.1 7:8769edb6:::benchmark_data_smithi080_80848_object77850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:03:26.357345+0000", 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 87.250868689000001, 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.108682501, 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 77851 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.357345+0000", 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.357345+0000", 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.357347+0000", 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6810000000000001e-06 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.357416+0000", 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9104999999999997e-05 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455123+0000", 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.097707710000000003 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455138+0000", 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5112e-05 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.460613+0000", 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0054743750000000001 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.460735+0000", 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000122352 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.460893+0000", 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015773600000000001 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.465804+0000", 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0049106699999999998 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.465867+0000", 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3629999999999999e-05 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.465938+0000", 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1093999999999994e-05 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.465991+0000", 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2519000000000003e-05 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.466027+0000", 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6517000000000003e-05 2024-03-26T05:04:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:123392 7.1 7:8d5fbf4d:::benchmark_data_smithi080_80848_object3893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-26T05:04:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:12.786298+0000", 2024-03-26T05:04:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 40.821915898, 2024-03-26T05:04:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.097955689999999998, 2024-03-26T05:04:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 123392 2024-03-26T05:04:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.786298+0000", 2024-03-26T05:04:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.786298+0000", 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.786299+0000", 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1209999999999999e-06 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.786304+0000", 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1730000000000003e-06 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.786317+0000", 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3558e-05 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.786324+0000", 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7329999999999997e-06 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.786379+0000", 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4376000000000003e-05 2024-03-26T05:04:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.786467+0000", 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7959000000000003e-05 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.786539+0000", 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2292999999999994e-05 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.789472+0000", 2024-03-26T05:04:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0029327139999999999 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.790784+0000", 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0013128149999999999 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.884177+0000", 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093392918000000005 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.884218+0000", 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0978e-05 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:12.884253+0000", 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5052000000000003e-05 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:158144 7.1 7:84525a55:::benchmark_data_smithi080_80848_object38645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:47.901942+0000", 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 5.7062713289999998, 2024-03-26T05:04:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.084522191999999996, 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158144 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.901942+0000", 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.901942+0000", 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.901944+0000", 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.669e-06 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.901949+0000", 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5210000000000002e-06 2024-03-26T05:04:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:04:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.901963+0000", 2024-03-26T05:04:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3151e-05 2024-03-26T05:04:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:04:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.901978+0000", 2024-03-26T05:04:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5481999999999998e-05 2024-03-26T05:04:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:04:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.902037+0000", 2024-03-26T05:04:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8536999999999999e-05 2024-03-26T05:04:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:04:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.902159+0000", 2024-03-26T05:04:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012292299999999999 2024-03-26T05:04:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.902271+0000", 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000111937 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.905631+0000", 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0033593210000000002 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.905727+0000", 2024-03-26T05:04:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6006000000000002e-05 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.986348+0000", 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.080620835000000002 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.986409+0000", 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1705000000000006e-05 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.986464+0000", 2024-03-26T05:04:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5105000000000002e-05 2024-03-26T05:04:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:04:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:04:53.740 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:04:53.779 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:53.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:53.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:53 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:04:53.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:53 smithi080 ceph-mon[29691]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:04:53.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:53 smithi080 ceph-mon[40404]: 5.6 scrub ok 2024-03-26T05:04:53.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:53 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:04:53.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:53 smithi080 ceph-mon[40404]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:04:53.881 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:53.912 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:53.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:53.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:54.014 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:54.055 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:54.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:54.156 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:54.180 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:54.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:54.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:54.284 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:04:54.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:54.327 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:54.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:54.328 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:04:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:04:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:04:54.428 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:54.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:54.479 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:54.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:54.580 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:54.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:54.614 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:54.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:54.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:54 smithi080 ceph-mon[40404]: 6.4 scrub starts 2024-03-26T05:04:54.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:54 smithi080 ceph-mon[40404]: 6.4 scrub ok 2024-03-26T05:04:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:54 smithi080 ceph-mon[40404]: pgmap v181: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 811 B/s rd, 0 B/s wr, 770 op/s 2024-03-26T05:04:54.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:54 smithi080 ceph-mon[29691]: 6.4 scrub starts 2024-03-26T05:04:54.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:54 smithi080 ceph-mon[29691]: 6.4 scrub ok 2024-03-26T05:04:54.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:54 smithi080 ceph-mon[29691]: pgmap v181: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 811 B/s rd, 0 B/s wr, 770 op/s 2024-03-26T05:04:54.715 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:54.750 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:54.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:54.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:54.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:54 smithi132 ceph-mon[33500]: 6.4 scrub starts 2024-03-26T05:04:54.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:54 smithi132 ceph-mon[33500]: 6.4 scrub ok 2024-03-26T05:04:54.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:54 smithi132 ceph-mon[33500]: pgmap v181: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 811 B/s rd, 0 B/s wr, 770 op/s 2024-03-26T05:04:54.852 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:54.890 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T05:04:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:54.894 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:54.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:54.995 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:55.036 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:55.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:55.124 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:04:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:04:55.137 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:55.184 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:55.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:55.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:55.287 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:55.322 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:55.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:55.422 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:55.484 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:55.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:55.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:55.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:55.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:55 smithi080 ceph-mon[29691]: 3.7 scrub starts 2024-03-26T05:04:55.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:55 smithi080 ceph-mon[29691]: 3.7 scrub ok 2024-03-26T05:04:55.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:55 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:55.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:55 smithi080 ceph-mon[29691]: 5.5 scrub starts 2024-03-26T05:04:55.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:55 smithi080 ceph-mon[29691]: 5.5 scrub ok 2024-03-26T05:04:55.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:55 smithi080 ceph-mon[29691]: 7.1 scrub starts 2024-03-26T05:04:55.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:55 smithi080 ceph-mon[40404]: 3.7 scrub starts 2024-03-26T05:04:55.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:55 smithi080 ceph-mon[40404]: 3.7 scrub ok 2024-03-26T05:04:55.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:55 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:55.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:55 smithi080 ceph-mon[40404]: 5.5 scrub starts 2024-03-26T05:04:55.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:55 smithi080 ceph-mon[40404]: 5.5 scrub ok 2024-03-26T05:04:55.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:55 smithi080 ceph-mon[40404]: 7.1 scrub starts 2024-03-26T05:04:55.621 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:55.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:55.695 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:04:55.723 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:55.777 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:55.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:55.787 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-03-26T05:04:51.780789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5167323510,"num_objects":79069,"num_object_clones":0,"num_object_copies":237207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79069,"num_whiteouts":0,"num_read":1484,"num_read_kb":1298,"num_write":160962,"num_write_kb":7650206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90584,"ondisk_log_size":90584,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15981392,"kb_used_data":15258120,"kb_used_omap":12,"kb_used_meta":722739,"kb_avail":733979824,"statfs":{"total":767960285184,"available":751595339776,"internally_reserved":0,"allocated":15624314880,"data_stored":15621785538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":740085296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,8,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-318373888,"num_objects":-4858,"num_object_clones":0,"num_object_copies":-14574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4858,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4858,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001530"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":178,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.471399+0000","last_change":"2024-03-26T05:04:18.210889+0000","last_active":"2024-03-26T05:04:47.471399+0000","last_peered":"2024-03-26T05:04:47.471399+0000","last_clean":"2024-03-26T05:04:47.471399+0000","last_became_active":"2024-03-26T05:00:28.972969+0000","last_became_peered":"2024-03-26T05:00:28.972969+0000","last_unstale":"2024-03-26T05:04:47.471399+0000","last_undegraded":"2024-03-26T05:04:47.471399+0000","last_fullsized":"2024-03-26T05:04:47.471399+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:18.210754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:18.210754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:32.582721+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":181,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.479276+0000","last_change":"2024-03-26T05:03:13.909344+0000","last_active":"2024-03-26T05:04:47.479276+0000","last_peered":"2024-03-26T05:04:47.479276+0000","last_clean":"2024-03-26T05:04:47.479276+0000","last_became_active":"2024-03-26T05:00:26.953154+0000","last_became_peered":"2024-03-26T05:00:26.953154+0000","last_unstale":"2024-03-26T05:04:47.479276+0000","last_undegraded":"2024-03-26T05:04:47.479276+0000","last_fullsized":"2024-03-26T05:04:47.479276+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:13.909270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:13.909270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":191,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.480289+0000","last_change":"2024-03-26T05:02:57.757086+0000","last_active":"2024-03-26T05:04:47.480289+0000","last_peered":"2024-03-26T05:04:47.480289+0000","last_clean":"2024-03-26T05:04:47.480289+0000","last_became_active":"2024-03-26T05:00:23.346607+0000","last_became_peered":"2024-03-26T05:00:23.346607+0000","last_unstale":"2024-03-26T05:04:47.480289+0000","last_undegraded":"2024-03-26T05:04:47.480289+0000","last_fullsized":"2024-03-26T05:04:47.480289+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:57.756968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:57.756968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'15","reported_seq":206,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.471547+0000","last_change":"2024-03-26T05:02:58.415674+0000","last_active":"2024-03-26T05:04:47.471547+0000","last_peered":"2024-03-26T05:04:47.471547+0000","last_clean":"2024-03-26T05:04:47.471547+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:04:47.471547+0000","last_undegraded":"2024-03-26T05:04:47.471547+0000","last_fullsized":"2024-03-26T05:04:47.471547+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:02:58.415577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:58.415577+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.099519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":169,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.484389+0000","last_change":"2024-03-26T05:03:00.192980+0000","last_active":"2024-03-26T05:04:47.484389+0000","last_peered":"2024-03-26T05:04:47.484389+0000","last_clean":"2024-03-26T05:04:47.484389+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:04:47.484389+0000","last_undegraded":"2024-03-26T05:04:47.484389+0000","last_fullsized":"2024-03-26T05:04:47.484389+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:00.192732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:00.192732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":174,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.471348+0000","last_change":"2024-03-26T05:03:12.394511+0000","last_active":"2024-03-26T05:04:47.471348+0000","last_peered":"2024-03-26T05:04:47.471348+0000","last_clean":"2024-03-26T05:04:47.471348+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:04:47.471348+0000","last_undegraded":"2024-03-26T05:04:47.471348+0000","last_fullsized":"2024-03-26T05:04:47.471348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:12.394429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:12.394429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.099545+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":190,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.481645+0000","last_change":"2024-03-26T05:03:06.237289+0000","last_active":"2024-03-26T05:04:47.481645+0000","last_peered":"2024-03-26T05:04:47.481645+0000","last_clean":"2024-03-26T05:04:47.481645+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:04:47.481645+0000","last_undegraded":"2024-03-26T05:04:47.481645+0000","last_fullsized":"2024-03-26T05:04:47.481645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:06.236963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:06.236963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'39","reported_seq":258,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.482815+0000","last_change":"2024-03-26T05:04:20.637111+0000","last_active":"2024-03-26T05:04:47.482815+0000","last_peered":"2024-03-26T05:04:47.482815+0000","last_clean":"2024-03-26T05:04:47.482815+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:04:47.482815+0000","last_undegraded":"2024-03-26T05:04:47.482815+0000","last_fullsized":"2024-03-26T05:04:47.482815+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'39","last_scrub_stamp":"2024-03-26T05:04:20.636687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:20.636687+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:31.492803+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":178,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.478726+0000","last_change":"2024-03-26T05:02:54.669047+0000","last_active":"2024-03-26T05:04:47.478726+0000","last_peered":"2024-03-26T05:04:47.478726+0000","last_clean":"2024-03-26T05:04:47.478726+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:04:47.478726+0000","last_undegraded":"2024-03-26T05:04:47.478726+0000","last_fullsized":"2024-03-26T05:04:47.478726+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:54.668919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:54.668919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":181,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.484791+0000","last_change":"2024-03-26T05:03:22.896270+0000","last_active":"2024-03-26T05:04:47.484791+0000","last_peered":"2024-03-26T05:04:47.484791+0000","last_clean":"2024-03-26T05:04:47.484791+0000","last_became_active":"2024-03-26T05:00:26.956013+0000","last_became_peered":"2024-03-26T05:00:26.956013+0000","last_unstale":"2024-03-26T05:04:47.484791+0000","last_undegraded":"2024-03-26T05:04:47.484791+0000","last_fullsized":"2024-03-26T05:04:47.484791+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:22.896182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:22.896182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:51.928139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":197,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485072+0000","last_change":"2024-03-26T05:04:18.640724+0000","last_active":"2024-03-26T05:04:47.485072+0000","last_peered":"2024-03-26T05:04:47.485072+0000","last_clean":"2024-03-26T05:04:47.485072+0000","last_became_active":"2024-03-26T05:00:22.768098+0000","last_became_peered":"2024-03-26T05:00:22.768098+0000","last_unstale":"2024-03-26T05:04:47.485072+0000","last_undegraded":"2024-03-26T05:04:47.485072+0000","last_fullsized":"2024-03-26T05:04:47.485072+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:18.640616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:18.640616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:32.003176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'21","reported_seq":226,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.471574+0000","last_change":"2024-03-26T05:03:18.506773+0000","last_active":"2024-03-26T05:04:47.471574+0000","last_peered":"2024-03-26T05:04:47.471574+0000","last_clean":"2024-03-26T05:04:47.471574+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:04:47.471574+0000","last_undegraded":"2024-03-26T05:04:47.471574+0000","last_fullsized":"2024-03-26T05:04:47.471574+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:03:18.506680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:18.506680+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.103980+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":185,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.484691+0000","last_change":"2024-03-26T05:04:01.704040+0000","last_active":"2024-03-26T05:04:47.484691+0000","last_peered":"2024-03-26T05:04:47.484691+0000","last_clean":"2024-03-26T05:04:47.484691+0000","last_became_active":"2024-03-26T05:00:28.982276+0000","last_became_peered":"2024-03-26T05:00:28.982276+0000","last_unstale":"2024-03-26T05:04:47.484691+0000","last_undegraded":"2024-03-26T05:04:47.484691+0000","last_fullsized":"2024-03-26T05:04:47.484691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:01.703895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:01.703895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:09.991523+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":180,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.492555+0000","last_change":"2024-03-26T05:03:09.812833+0000","last_active":"2024-03-26T05:04:47.492555+0000","last_peered":"2024-03-26T05:04:47.492555+0000","last_clean":"2024-03-26T05:04:47.492555+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:04:47.492555+0000","last_undegraded":"2024-03-26T05:04:47.492555+0000","last_fullsized":"2024-03-26T05:04:47.492555+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:09.812687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.812687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.851670+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":197,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485144+0000","last_change":"2024-03-26T05:04:10.613465+0000","last_active":"2024-03-26T05:04:47.485144+0000","last_peered":"2024-03-26T05:04:47.485144+0000","last_clean":"2024-03-26T05:04:47.485144+0000","last_became_active":"2024-03-26T05:00:22.766675+0000","last_became_peered":"2024-03-26T05:00:22.766675+0000","last_unstale":"2024-03-26T05:04:47.485144+0000","last_undegraded":"2024-03-26T05:04:47.485144+0000","last_fullsized":"2024-03-26T05:04:47.485144+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:10.613346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:10.613346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:19.750451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'11","reported_seq":200,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.472709+0000","last_change":"2024-03-26T05:03:10.376790+0000","last_active":"2024-03-26T05:04:47.472709+0000","last_peered":"2024-03-26T05:04:47.472709+0000","last_clean":"2024-03-26T05:04:47.472709+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:04:47.472709+0000","last_undegraded":"2024-03-26T05:04:47.472709+0000","last_fullsized":"2024-03-26T05:04:47.472709+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-26T05:03:10.376657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:10.376657+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.099629+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":179,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.472444+0000","last_change":"2024-03-26T05:02:53.505923+0000","last_active":"2024-03-26T05:04:47.472444+0000","last_peered":"2024-03-26T05:04:47.472444+0000","last_clean":"2024-03-26T05:04:47.472444+0000","last_became_active":"2024-03-26T05:00:29.461032+0000","last_became_peered":"2024-03-26T05:00:29.461032+0000","last_unstale":"2024-03-26T05:04:47.472444+0000","last_undegraded":"2024-03-26T05:04:47.472444+0000","last_fullsized":"2024-03-26T05:04:47.472444+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:53.505778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:02:53.505778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.099428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":181,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.481207+0000","last_change":"2024-03-26T05:03:15.886716+0000","last_active":"2024-03-26T05:04:47.481207+0000","last_peered":"2024-03-26T05:04:47.481207+0000","last_clean":"2024-03-26T05:04:47.481207+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:04:47.481207+0000","last_undegraded":"2024-03-26T05:04:47.481207+0000","last_fullsized":"2024-03-26T05:04:47.481207+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:15.885586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:15.885586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":190,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.479940+0000","last_change":"2024-03-26T05:03:09.914583+0000","last_active":"2024-03-26T05:04:47.479940+0000","last_peered":"2024-03-26T05:04:47.479940+0000","last_clean":"2024-03-26T05:04:47.479940+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:04:47.479940+0000","last_undegraded":"2024-03-26T05:04:47.479940+0000","last_fullsized":"2024-03-26T05:04:47.479940+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:09.914470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:09.914470+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.914470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":210,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.483923+0000","last_change":"2024-03-26T05:02:58.971442+0000","last_active":"2024-03-26T05:04:47.483923+0000","last_peered":"2024-03-26T05:04:47.483923+0000","last_clean":"2024-03-26T05:04:47.483923+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:04:47.483923+0000","last_undegraded":"2024-03-26T05:04:47.483923+0000","last_fullsized":"2024-03-26T05:04:47.483923+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:02:58.965020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:58.965020+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":178,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.489785+0000","last_change":"2024-03-26T05:03:05.268194+0000","last_active":"2024-03-26T05:04:47.489785+0000","last_peered":"2024-03-26T05:04:47.489785+0000","last_clean":"2024-03-26T05:04:47.489785+0000","last_became_active":"2024-03-26T05:00:29.463238+0000","last_became_peered":"2024-03-26T05:00:29.463238+0000","last_unstale":"2024-03-26T05:04:47.489785+0000","last_undegraded":"2024-03-26T05:04:47.489785+0000","last_fullsized":"2024-03-26T05:04:47.489785+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:05.268043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:05.268043+0000","last_clean_scrub_stamp":"2024-03-26T05:03:05.268043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":182,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.471933+0000","last_change":"2024-03-26T05:04:21.167015+0000","last_active":"2024-03-26T05:04:47.471933+0000","last_peered":"2024-03-26T05:04:47.471933+0000","last_clean":"2024-03-26T05:04:47.471933+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:04:47.471933+0000","last_undegraded":"2024-03-26T05:04:47.471933+0000","last_fullsized":"2024-03-26T05:04:47.471933+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:21.166893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.166893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:45.803222+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.471600+0000","last_change":"2024-03-26T05:04:01.071006+0000","last_active":"2024-03-26T05:04:47.471600+0000","last_peered":"2024-03-26T05:04:47.471600+0000","last_clean":"2024-03-26T05:04:47.471600+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:04:47.471600+0000","last_undegraded":"2024-03-26T05:04:47.471600+0000","last_fullsized":"2024-03-26T05:04:47.471600+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:01.070878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:04:01.070878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:14.920867+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'20","reported_seq":222,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.481423+0000","last_change":"2024-03-26T05:04:07.528781+0000","last_active":"2024-03-26T05:04:47.481423+0000","last_peered":"2024-03-26T05:04:47.481423+0000","last_clean":"2024-03-26T05:04:47.481423+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:04:47.481423+0000","last_undegraded":"2024-03-26T05:04:47.481423+0000","last_fullsized":"2024-03-26T05:04:47.481423+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:04:07.528619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:07.528619+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:36.984300+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":177,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.493885+0000","last_change":"2024-03-26T05:03:13.901836+0000","last_active":"2024-03-26T05:04:47.493885+0000","last_peered":"2024-03-26T05:04:47.493885+0000","last_clean":"2024-03-26T05:04:47.493885+0000","last_became_active":"2024-03-26T05:00:28.981505+0000","last_became_peered":"2024-03-26T05:00:28.981505+0000","last_unstale":"2024-03-26T05:04:47.493885+0000","last_undegraded":"2024-03-26T05:04:47.493885+0000","last_fullsized":"2024-03-26T05:04:47.493885+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:03:13.901714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:13.901714+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.851778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":254,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.475877+0000","last_change":"2024-03-26T05:04:18.963951+0000","last_active":"2024-03-26T05:04:47.475877+0000","last_peered":"2024-03-26T05:04:47.475877+0000","last_clean":"2024-03-26T05:04:47.475877+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:04:47.475877+0000","last_undegraded":"2024-03-26T05:04:47.475877+0000","last_fullsized":"2024-03-26T05:04:47.475877+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:18.963855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:18.963855+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:20.821553+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":190,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.479364+0000","last_change":"2024-03-26T05:04:07.578139+0000","last_active":"2024-03-26T05:04:47.479364+0000","last_peered":"2024-03-26T05:04:47.479364+0000","last_clean":"2024-03-26T05:04:47.479364+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:04:47.479364+0000","last_undegraded":"2024-03-26T05:04:47.479364+0000","last_fullsized":"2024-03-26T05:04:47.479364+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:04:07.578016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:07.578016+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:24.792226+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'16","reported_seq":214,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.480595+0000","last_change":"2024-03-26T05:03:17.868736+0000","last_active":"2024-03-26T05:04:47.480595+0000","last_peered":"2024-03-26T05:04:47.480595+0000","last_clean":"2024-03-26T05:04:47.480595+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:04:47.480595+0000","last_undegraded":"2024-03-26T05:04:47.480595+0000","last_fullsized":"2024-03-26T05:04:47.480595+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:03:17.867315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:17.867315+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":177,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.486073+0000","last_change":"2024-03-26T05:03:17.800276+0000","last_active":"2024-03-26T05:04:47.486073+0000","last_peered":"2024-03-26T05:04:47.486073+0000","last_clean":"2024-03-26T05:04:47.486073+0000","last_became_active":"2024-03-26T05:00:28.981049+0000","last_became_peered":"2024-03-26T05:00:28.981049+0000","last_unstale":"2024-03-26T05:04:47.486073+0000","last_undegraded":"2024-03-26T05:04:47.486073+0000","last_fullsized":"2024-03-26T05:04:47.486073+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:17.800166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:17.800166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":183,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.480952+0000","last_change":"2024-03-26T05:03:59.829911+0000","last_active":"2024-03-26T05:04:47.480952+0000","last_peered":"2024-03-26T05:04:47.480952+0000","last_clean":"2024-03-26T05:04:47.480952+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:04:47.480952+0000","last_undegraded":"2024-03-26T05:04:47.480952+0000","last_fullsized":"2024-03-26T05:04:47.480952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:59.829595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:59.829595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:14.748766+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":189,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.480917+0000","last_change":"2024-03-26T05:03:17.225355+0000","last_active":"2024-03-26T05:04:47.480917+0000","last_peered":"2024-03-26T05:04:47.480917+0000","last_clean":"2024-03-26T05:04:47.480917+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:04:47.480917+0000","last_undegraded":"2024-03-26T05:04:47.480917+0000","last_fullsized":"2024-03-26T05:04:47.480917+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:17.225243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:17.225243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'17","reported_seq":236,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.475810+0000","last_change":"2024-03-26T05:03:22.921461+0000","last_active":"2024-03-26T05:04:47.475810+0000","last_peered":"2024-03-26T05:04:47.475810+0000","last_clean":"2024-03-26T05:04:47.475810+0000","last_became_active":"2024-03-26T05:00:49.643308+0000","last_became_peered":"2024-03-26T05:00:49.643308+0000","last_unstale":"2024-03-26T05:04:47.475810+0000","last_undegraded":"2024-03-26T05:04:47.475810+0000","last_fullsized":"2024-03-26T05:04:47.475810+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:03:22.921353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:22.921353+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:49.646429+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":176,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.492356+0000","last_change":"2024-03-26T05:04:27.007041+0000","last_active":"2024-03-26T05:04:47.492356+0000","last_peered":"2024-03-26T05:04:47.492356+0000","last_clean":"2024-03-26T05:04:47.492356+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:04:47.492356+0000","last_undegraded":"2024-03-26T05:04:47.492356+0000","last_fullsized":"2024-03-26T05:04:47.492356+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:27.006904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:27.006904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:56.433967+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":180,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.475828+0000","last_change":"2024-03-26T05:03:04.024222+0000","last_active":"2024-03-26T05:04:47.475828+0000","last_peered":"2024-03-26T05:04:47.475828+0000","last_clean":"2024-03-26T05:04:47.475828+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:04:47.475828+0000","last_undegraded":"2024-03-26T05:04:47.475828+0000","last_fullsized":"2024-03-26T05:04:47.475828+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:04.024139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:04.024139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:49.646448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":190,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.480475+0000","last_change":"2024-03-26T05:04:14.715840+0000","last_active":"2024-03-26T05:04:47.480475+0000","last_peered":"2024-03-26T05:04:47.480475+0000","last_clean":"2024-03-26T05:04:47.480475+0000","last_became_active":"2024-03-26T05:00:22.774783+0000","last_became_peered":"2024-03-26T05:00:22.774783+0000","last_unstale":"2024-03-26T05:04:47.480475+0000","last_undegraded":"2024-03-26T05:04:47.480475+0000","last_fullsized":"2024-03-26T05:04:47.480475+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:14.715721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:14.715721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:39.237899+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"100'6","reported_seq":193,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.481001+0000","last_change":"2024-03-26T05:04:01.370962+0000","last_active":"2024-03-26T05:04:47.481001+0000","last_peered":"2024-03-26T05:04:47.481001+0000","last_clean":"2024-03-26T05:04:47.481001+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:04:47.481001+0000","last_undegraded":"2024-03-26T05:04:47.481001+0000","last_fullsized":"2024-03-26T05:04:47.481001+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'6","last_scrub_stamp":"2024-03-26T05:04:01.370804+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:04:01.370804+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:02.996344+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":177,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.491643+0000","last_change":"2024-03-26T05:04:10.585914+0000","last_active":"2024-03-26T05:04:47.491643+0000","last_peered":"2024-03-26T05:04:47.491643+0000","last_clean":"2024-03-26T05:04:47.491643+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:04:47.491643+0000","last_undegraded":"2024-03-26T05:04:47.491643+0000","last_fullsized":"2024-03-26T05:04:47.491643+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:10.585750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:10.585750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:23.988184+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":180,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.495365+0000","last_change":"2024-03-26T05:03:06.806992+0000","last_active":"2024-03-26T05:04:47.495365+0000","last_peered":"2024-03-26T05:04:47.495365+0000","last_clean":"2024-03-26T05:04:47.495365+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:04:47.495365+0000","last_undegraded":"2024-03-26T05:04:47.495365+0000","last_fullsized":"2024-03-26T05:04:47.495365+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:03:06.806863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.851646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":105,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.473954+0000","last_change":"2024-03-26T05:04:20.919136+0000","last_active":"2024-03-26T05:04:47.473954+0000","last_peered":"2024-03-26T05:04:47.473954+0000","last_clean":"2024-03-26T05:04:47.473954+0000","last_became_active":"2024-03-26T05:03:05.521331+0000","last_became_peered":"2024-03-26T05:03:05.521331+0000","last_unstale":"2024-03-26T05:04:47.473954+0000","last_undegraded":"2024-03-26T05:04:47.473954+0000","last_fullsized":"2024-03-26T05:04:47.473954+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:20.919020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:20.919020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:50.626775+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"75'17","reported_seq":216,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.482722+0000","last_change":"2024-03-26T05:04:09.559560+0000","last_active":"2024-03-26T05:04:47.482722+0000","last_peered":"2024-03-26T05:04:47.482722+0000","last_clean":"2024-03-26T05:04:47.482722+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:04:47.482722+0000","last_undegraded":"2024-03-26T05:04:47.482722+0000","last_fullsized":"2024-03-26T05:04:47.482722+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:04:09.559412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:09.559412+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:38.053118+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":178,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.477894+0000","last_change":"2024-03-26T05:03:11.864162+0000","last_active":"2024-03-26T05:04:47.477894+0000","last_peered":"2024-03-26T05:04:47.477894+0000","last_clean":"2024-03-26T05:04:47.477894+0000","last_became_active":"2024-03-26T05:00:28.985158+0000","last_became_peered":"2024-03-26T05:00:28.985158+0000","last_unstale":"2024-03-26T05:04:47.477894+0000","last_undegraded":"2024-03-26T05:04:47.477894+0000","last_fullsized":"2024-03-26T05:04:47.477894+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:11.864048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:11.864048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":173,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.484316+0000","last_change":"2024-03-26T05:02:58.192214+0000","last_active":"2024-03-26T05:04:47.484316+0000","last_peered":"2024-03-26T05:04:47.484316+0000","last_clean":"2024-03-26T05:04:47.484316+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:04:47.484316+0000","last_undegraded":"2024-03-26T05:04:47.484316+0000","last_fullsized":"2024-03-26T05:04:47.484316+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:58.191939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:02:58.191939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":180,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.493436+0000","last_change":"2024-03-26T05:02:07.036740+0000","last_active":"2024-03-26T05:04:47.493436+0000","last_peered":"2024-03-26T05:04:47.493436+0000","last_clean":"2024-03-26T05:04:47.493436+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:04:47.493436+0000","last_undegraded":"2024-03-26T05:04:47.493436+0000","last_fullsized":"2024-03-26T05:04:47.493436+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:07.036606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:07.036606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.851567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":201,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.478376+0000","last_change":"2024-03-26T05:03:23.778722+0000","last_active":"2024-03-26T05:04:47.478376+0000","last_peered":"2024-03-26T05:04:47.478376+0000","last_clean":"2024-03-26T05:04:47.478376+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:04:47.478376+0000","last_undegraded":"2024-03-26T05:04:47.478376+0000","last_fullsized":"2024-03-26T05:04:47.478376+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-26T05:03:23.778591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:23.778591+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":169,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.482343+0000","last_change":"2024-03-26T05:02:16.073043+0000","last_active":"2024-03-26T05:04:47.482343+0000","last_peered":"2024-03-26T05:04:47.482343+0000","last_clean":"2024-03-26T05:04:47.482343+0000","last_became_active":"2024-03-26T05:00:28.983417+0000","last_became_peered":"2024-03-26T05:00:28.983417+0000","last_unstale":"2024-03-26T05:04:47.482343+0000","last_undegraded":"2024-03-26T05:04:47.482343+0000","last_fullsized":"2024-03-26T05:04:47.482343+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:02:16.072965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":189,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.478227+0000","last_change":"2024-03-26T05:04:21.579387+0000","last_active":"2024-03-26T05:04:47.478227+0000","last_peered":"2024-03-26T05:04:47.478227+0000","last_clean":"2024-03-26T05:04:47.478227+0000","last_became_active":"2024-03-26T05:00:26.954653+0000","last_became_peered":"2024-03-26T05:00:26.954653+0000","last_unstale":"2024-03-26T05:04:47.478227+0000","last_undegraded":"2024-03-26T05:04:47.478227+0000","last_fullsized":"2024-03-26T05:04:47.478227+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:21.579155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.579155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:39.504032+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":182,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.471747+0000","last_change":"2024-03-26T05:03:03.408937+0000","last_active":"2024-03-26T05:04:47.471747+0000","last_peered":"2024-03-26T05:04:47.471747+0000","last_clean":"2024-03-26T05:04:47.471747+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:04:47.471747+0000","last_undegraded":"2024-03-26T05:04:47.471747+0000","last_fullsized":"2024-03-26T05:04:47.471747+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:03.408801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:03.408801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.099599+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"95'16","reported_seq":210,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.472553+0000","last_change":"2024-03-26T05:03:22.426319+0000","last_active":"2024-03-26T05:04:47.472553+0000","last_peered":"2024-03-26T05:04:47.472553+0000","last_clean":"2024-03-26T05:04:47.472553+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:04:47.472553+0000","last_undegraded":"2024-03-26T05:04:47.472553+0000","last_fullsized":"2024-03-26T05:04:47.472553+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:03:22.426170+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:03:22.426170+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.099574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":178,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.480124+0000","last_change":"2024-03-26T05:04:10.806723+0000","last_active":"2024-03-26T05:04:47.480124+0000","last_peered":"2024-03-26T05:04:47.480124+0000","last_clean":"2024-03-26T05:04:47.480124+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:04:47.480124+0000","last_undegraded":"2024-03-26T05:04:47.480124+0000","last_fullsized":"2024-03-26T05:04:47.480124+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:10.806588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:10.806588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:36.340744+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":243,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485176+0000","last_change":"2024-03-26T05:04:05.662804+0000","last_active":"2024-03-26T05:04:47.485176+0000","last_peered":"2024-03-26T05:04:47.485176+0000","last_clean":"2024-03-26T05:04:47.485176+0000","last_became_active":"2024-03-26T05:00:26.955836+0000","last_became_peered":"2024-03-26T05:00:26.955836+0000","last_unstale":"2024-03-26T05:04:47.485176+0000","last_undegraded":"2024-03-26T05:04:47.485176+0000","last_fullsized":"2024-03-26T05:04:47.485176+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:05.662699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:05.662699+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:21.460708+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":188,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.492763+0000","last_change":"2024-03-26T05:02:55.996225+0000","last_active":"2024-03-26T05:04:47.492763+0000","last_peered":"2024-03-26T05:04:47.492763+0000","last_clean":"2024-03-26T05:04:47.492763+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:04:47.492763+0000","last_undegraded":"2024-03-26T05:04:47.492763+0000","last_fullsized":"2024-03-26T05:04:47.492763+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:55.996111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:55.996111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.851622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'22","reported_seq":224,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.482481+0000","last_change":"2024-03-26T05:03:10.161219+0000","last_active":"2024-03-26T05:04:47.482481+0000","last_peered":"2024-03-26T05:04:47.482481+0000","last_clean":"2024-03-26T05:04:47.482481+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:04:47.482481+0000","last_undegraded":"2024-03-26T05:04:47.482481+0000","last_fullsized":"2024-03-26T05:04:47.482481+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:03:10.161091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:10.161091+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":170,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.472291+0000","last_change":"2024-03-26T05:03:14.456132+0000","last_active":"2024-03-26T05:04:47.472291+0000","last_peered":"2024-03-26T05:04:47.472291+0000","last_clean":"2024-03-26T05:04:47.472291+0000","last_became_active":"2024-03-26T05:00:28.979477+0000","last_became_peered":"2024-03-26T05:00:28.979477+0000","last_unstale":"2024-03-26T05:04:47.472291+0000","last_undegraded":"2024-03-26T05:04:47.472291+0000","last_fullsized":"2024-03-26T05:04:47.472291+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:14.456016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:14.456016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.103885+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"120'9","reported_seq":247,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.482929+0000","last_change":"2024-03-26T05:03:20.187175+0000","last_active":"2024-03-26T05:04:47.482929+0000","last_peered":"2024-03-26T05:04:47.482929+0000","last_clean":"2024-03-26T05:04:47.482929+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:04:47.482929+0000","last_undegraded":"2024-03-26T05:04:47.482929+0000","last_fullsized":"2024-03-26T05:04:47.482929+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'9","last_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:03:20.186818+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":190,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.478389+0000","last_change":"2024-03-26T05:03:21.927104+0000","last_active":"2024-03-26T05:04:47.478389+0000","last_peered":"2024-03-26T05:04:47.478389+0000","last_clean":"2024-03-26T05:04:47.478389+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:04:47.478389+0000","last_undegraded":"2024-03-26T05:04:47.478389+0000","last_fullsized":"2024-03-26T05:04:47.478389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:21.926965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:21.926965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"98'20","reported_seq":226,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485240+0000","last_change":"2024-03-26T05:04:12.659452+0000","last_active":"2024-03-26T05:04:47.485240+0000","last_peered":"2024-03-26T05:04:47.485240+0000","last_clean":"2024-03-26T05:04:47.485240+0000","last_became_active":"2024-03-26T05:00:25.648284+0000","last_became_peered":"2024-03-26T05:00:25.648284+0000","last_unstale":"2024-03-26T05:04:47.485240+0000","last_undegraded":"2024-03-26T05:04:47.485240+0000","last_fullsized":"2024-03-26T05:04:47.485240+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'20","last_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:04:12.659322+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:26.101052+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":185,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.484072+0000","last_change":"2024-03-26T05:04:13.174305+0000","last_active":"2024-03-26T05:04:47.484072+0000","last_peered":"2024-03-26T05:04:47.484072+0000","last_clean":"2024-03-26T05:04:47.484072+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:04:47.484072+0000","last_undegraded":"2024-03-26T05:04:47.484072+0000","last_fullsized":"2024-03-26T05:04:47.484072+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:13.174161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:13.174161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:29.585590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":173,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.489643+0000","last_change":"2024-03-26T05:02:18.010095+0000","last_active":"2024-03-26T05:04:47.489643+0000","last_peered":"2024-03-26T05:04:47.489643+0000","last_clean":"2024-03-26T05:04:47.489643+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:04:47.489643+0000","last_undegraded":"2024-03-26T05:04:47.489643+0000","last_fullsized":"2024-03-26T05:04:47.489643+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:02:18.010011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":188,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.492651+0000","last_change":"2024-03-26T05:04:15.786624+0000","last_active":"2024-03-26T05:04:47.492651+0000","last_peered":"2024-03-26T05:04:47.492651+0000","last_clean":"2024-03-26T05:04:47.492651+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:04:47.492651+0000","last_undegraded":"2024-03-26T05:04:47.492651+0000","last_fullsized":"2024-03-26T05:04:47.492651+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:15.786545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:15.786545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:45.427035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'21","reported_seq":226,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.480808+0000","last_change":"2024-03-26T05:03:06.871854+0000","last_active":"2024-03-26T05:04:47.480808+0000","last_peered":"2024-03-26T05:04:47.480808+0000","last_clean":"2024-03-26T05:04:47.480808+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:04:47.480808+0000","last_undegraded":"2024-03-26T05:04:47.480808+0000","last_fullsized":"2024-03-26T05:04:47.480808+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:03:06.871708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:06.871708+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":176,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.493342+0000","last_change":"2024-03-26T05:04:05.932455+0000","last_active":"2024-03-26T05:04:47.493342+0000","last_peered":"2024-03-26T05:04:47.493342+0000","last_clean":"2024-03-26T05:04:47.493342+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:04:47.493342+0000","last_undegraded":"2024-03-26T05:04:47.493342+0000","last_fullsized":"2024-03-26T05:04:47.493342+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:05.932270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:05.932270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:15.168921+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":181,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485410+0000","last_change":"2024-03-26T05:03:12.953956+0000","last_active":"2024-03-26T05:04:47.485410+0000","last_peered":"2024-03-26T05:04:47.485410+0000","last_clean":"2024-03-26T05:04:47.485410+0000","last_became_active":"2024-03-26T05:00:26.953416+0000","last_became_peered":"2024-03-26T05:00:26.953416+0000","last_unstale":"2024-03-26T05:04:47.485410+0000","last_undegraded":"2024-03-26T05:04:47.485410+0000","last_fullsized":"2024-03-26T05:04:47.485410+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:12.953811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:12.953811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:50.692612+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":194,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.473491+0000","last_change":"2024-03-26T05:03:09.078444+0000","last_active":"2024-03-26T05:04:47.473491+0000","last_peered":"2024-03-26T05:04:47.473491+0000","last_clean":"2024-03-26T05:04:47.473491+0000","last_became_active":"2024-03-26T05:00:22.771080+0000","last_became_peered":"2024-03-26T05:00:22.771080+0000","last_unstale":"2024-03-26T05:04:47.473491+0000","last_undegraded":"2024-03-26T05:04:47.473491+0000","last_fullsized":"2024-03-26T05:04:47.473491+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:03:09.078282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.078282+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:49.646466+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'12","reported_seq":206,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.489876+0000","last_change":"2024-03-26T05:03:18.221430+0000","last_active":"2024-03-26T05:04:47.489876+0000","last_peered":"2024-03-26T05:04:47.489876+0000","last_clean":"2024-03-26T05:04:47.489876+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:04:47.489876+0000","last_undegraded":"2024-03-26T05:04:47.489876+0000","last_fullsized":"2024-03-26T05:04:47.489876+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:03:18.221280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:18.221280+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"149'10037","reported_seq":10217,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.828846+0000","last_change":"2024-03-26T05:04:21.563737+0000","last_active":"2024-03-26T05:04:49.828846+0000","last_peered":"2024-03-26T05:04:49.828846+0000","last_clean":"2024-03-26T05:04:49.828846+0000","last_became_active":"2024-03-26T05:02:24.354658+0000","last_became_peered":"2024-03-26T05:02:24.354658+0000","last_unstale":"2024-03-26T05:04:49.828846+0000","last_undegraded":"2024-03-26T05:04:49.828846+0000","last_fullsized":"2024-03-26T05:04:49.828846+0000","mapping_epoch":103,"log_start":"123'5000","ondisk_log_start":"123'5000","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'8267","last_scrub_stamp":"2024-03-26T05:04:21.563226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.563226+0000","objects_scrubbed":6873,"log_size":5037,"log_dups_size":0,"ondisk_log_size":5037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:33.838683+0000","scrub_duration":2071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330760192,"num_objects":5047,"num_object_clones":0,"num_object_copies":15141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":482688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":179,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.486272+0000","last_change":"2024-03-26T05:02:57.960123+0000","last_active":"2024-03-26T05:04:47.486272+0000","last_peered":"2024-03-26T05:04:47.486272+0000","last_clean":"2024-03-26T05:04:47.486272+0000","last_became_active":"2024-03-26T05:00:29.461059+0000","last_became_peered":"2024-03-26T05:00:29.461059+0000","last_unstale":"2024-03-26T05:04:47.486272+0000","last_undegraded":"2024-03-26T05:04:47.486272+0000","last_fullsized":"2024-03-26T05:04:47.486272+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:02:57.960005+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:02:57.960005+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:50.692438+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'3","reported_seq":206,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.493451+0000","last_change":"2024-03-26T05:02:56.964577+0000","last_active":"2024-03-26T05:04:47.493451+0000","last_peered":"2024-03-26T05:04:47.493451+0000","last_clean":"2024-03-26T05:04:47.493451+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:04:47.493451+0000","last_undegraded":"2024-03-26T05:04:47.493451+0000","last_fullsized":"2024-03-26T05:04:47.493451+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-03-26T05:02:56.964448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:56.964448+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.851597+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'27","reported_seq":243,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.484667+0000","last_change":"2024-03-26T05:04:13.682488+0000","last_active":"2024-03-26T05:04:47.484667+0000","last_peered":"2024-03-26T05:04:47.484667+0000","last_clean":"2024-03-26T05:04:47.484667+0000","last_became_active":"2024-03-26T05:00:25.651328+0000","last_became_peered":"2024-03-26T05:00:25.651328+0000","last_unstale":"2024-03-26T05:04:47.484667+0000","last_undegraded":"2024-03-26T05:04:47.484667+0000","last_fullsized":"2024-03-26T05:04:47.484667+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'27","last_scrub_stamp":"2024-03-26T05:04:13.682349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:13.682349+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:18.855246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":243,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:48.442028+0000","last_change":"2024-03-26T05:03:10.843641+0000","last_active":"2024-03-26T05:04:48.442028+0000","last_peered":"2024-03-26T05:04:48.442028+0000","last_clean":"2024-03-26T05:04:48.442028+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:04:48.442028+0000","last_undegraded":"2024-03-26T05:04:48.442028+0000","last_fullsized":"2024-03-26T05:04:48.442028+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:03:10.843497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:10.843497+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"149'10187","reported_seq":10370,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.955659+0000","last_change":"2024-03-26T05:04:16.685339+0000","last_active":"2024-03-26T05:04:49.955659+0000","last_peered":"2024-03-26T05:04:49.955659+0000","last_clean":"2024-03-26T05:04:49.955659+0000","last_became_active":"2024-03-26T05:02:24.352844+0000","last_became_peered":"2024-03-26T05:02:24.352844+0000","last_unstale":"2024-03-26T05:04:49.955659+0000","last_undegraded":"2024-03-26T05:04:49.955659+0000","last_fullsized":"2024-03-26T05:04:49.955659+0000","mapping_epoch":103,"log_start":"125'5500","ondisk_log_start":"125'5500","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'8019","last_scrub_stamp":"2024-03-26T05:04:16.685114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:16.685114+0000","objects_scrubbed":7208,"log_size":4687,"log_dups_size":0,"ondisk_log_size":4687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:28.742298+0000","scrub_duration":3134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324075520,"num_objects":4945,"num_object_clones":0,"num_object_copies":14835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10187,"num_write_kb":484224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":177,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.484467+0000","last_change":"2024-03-26T05:03:16.177929+0000","last_active":"2024-03-26T05:04:47.484467+0000","last_peered":"2024-03-26T05:04:47.484467+0000","last_clean":"2024-03-26T05:04:47.484467+0000","last_became_active":"2024-03-26T05:00:28.985917+0000","last_became_peered":"2024-03-26T05:00:28.985917+0000","last_unstale":"2024-03-26T05:04:47.484467+0000","last_undegraded":"2024-03-26T05:04:47.484467+0000","last_fullsized":"2024-03-26T05:04:47.484467+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:16.177807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:16.177807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":188,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.494291+0000","last_change":"2024-03-26T05:03:17.962503+0000","last_active":"2024-03-26T05:04:47.494291+0000","last_peered":"2024-03-26T05:04:47.494291+0000","last_clean":"2024-03-26T05:04:47.494291+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:04:47.494291+0000","last_undegraded":"2024-03-26T05:04:47.494291+0000","last_fullsized":"2024-03-26T05:04:47.494291+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:17.962388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:17.962388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.851802+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'24","reported_seq":226,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.472216+0000","last_change":"2024-03-26T05:02:59.435952+0000","last_active":"2024-03-26T05:04:47.472216+0000","last_peered":"2024-03-26T05:04:47.472216+0000","last_clean":"2024-03-26T05:04:47.472216+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:04:47.472216+0000","last_undegraded":"2024-03-26T05:04:47.472216+0000","last_fullsized":"2024-03-26T05:04:47.472216+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'24","last_scrub_stamp":"2024-03-26T05:02:59.435799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:59.435799+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:54.146753+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":182,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.486044+0000","last_change":"2024-03-26T05:03:07.025015+0000","last_active":"2024-03-26T05:04:47.486044+0000","last_peered":"2024-03-26T05:04:47.486044+0000","last_clean":"2024-03-26T05:04:47.486044+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:04:47.486044+0000","last_undegraded":"2024-03-26T05:04:47.486044+0000","last_fullsized":"2024-03-26T05:04:47.486044+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:07.022207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:07.022207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"149'9886","reported_seq":10061,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.969339+0000","last_change":"2024-03-26T05:04:27.315840+0000","last_active":"2024-03-26T05:04:49.969339+0000","last_peered":"2024-03-26T05:04:49.969339+0000","last_clean":"2024-03-26T05:04:49.969339+0000","last_became_active":"2024-03-26T05:02:24.355930+0000","last_became_peered":"2024-03-26T05:02:24.355930+0000","last_unstale":"2024-03-26T05:04:49.969339+0000","last_undegraded":"2024-03-26T05:04:49.969339+0000","last_fullsized":"2024-03-26T05:04:49.969339+0000","mapping_epoch":103,"log_start":"120'4300","ondisk_log_start":"120'4300","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8447","last_scrub_stamp":"2024-03-26T05:04:27.315394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:27.315394+0000","objects_scrubbed":6429,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:56.899987+0000","scrub_duration":2530,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322699264,"num_objects":4924,"num_object_clones":0,"num_object_copies":14772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9886,"num_write_kb":473920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":177,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.494902+0000","last_change":"2024-03-26T05:03:23.071801+0000","last_active":"2024-03-26T05:04:47.494902+0000","last_peered":"2024-03-26T05:04:47.494902+0000","last_clean":"2024-03-26T05:04:47.494902+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:04:47.494902+0000","last_undegraded":"2024-03-26T05:04:47.494902+0000","last_fullsized":"2024-03-26T05:04:47.494902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:23.071667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:23.071667+0000","last_clean_scrub_stamp":"2024-03-26T05:03:23.071667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":189,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.490902+0000","last_change":"2024-03-26T05:04:21.231029+0000","last_active":"2024-03-26T05:04:47.490902+0000","last_peered":"2024-03-26T05:04:47.490902+0000","last_clean":"2024-03-26T05:04:47.490902+0000","last_became_active":"2024-03-26T05:00:26.961730+0000","last_became_peered":"2024-03-26T05:00:26.961730+0000","last_unstale":"2024-03-26T05:04:47.490902+0000","last_undegraded":"2024-03-26T05:04:47.490902+0000","last_fullsized":"2024-03-26T05:04:47.490902+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:21.230901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.230901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:36.141812+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":188,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.493709+0000","last_change":"2024-03-26T05:04:23.884517+0000","last_active":"2024-03-26T05:04:47.493709+0000","last_peered":"2024-03-26T05:04:47.493709+0000","last_clean":"2024-03-26T05:04:47.493709+0000","last_became_active":"2024-03-26T05:00:22.772590+0000","last_became_peered":"2024-03-26T05:00:22.772590+0000","last_unstale":"2024-03-26T05:04:47.493709+0000","last_undegraded":"2024-03-26T05:04:47.493709+0000","last_fullsized":"2024-03-26T05:04:47.493709+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:23.884406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:23.884406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:49.946181+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'25","reported_seq":228,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.473825+0000","last_change":"2024-03-26T05:03:19.037673+0000","last_active":"2024-03-26T05:04:47.473825+0000","last_peered":"2024-03-26T05:04:47.473825+0000","last_clean":"2024-03-26T05:04:47.473825+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:04:47.473825+0000","last_undegraded":"2024-03-26T05:04:47.473825+0000","last_fullsized":"2024-03-26T05:04:47.473825+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:03:19.037530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:19.037530+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"149'10103","reported_seq":10289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.910377+0000","last_change":"2024-03-26T05:04:11.124139+0000","last_active":"2024-03-26T05:04:49.910377+0000","last_peered":"2024-03-26T05:04:49.910377+0000","last_clean":"2024-03-26T05:04:49.910377+0000","last_became_active":"2024-03-26T05:02:24.353523+0000","last_became_peered":"2024-03-26T05:02:24.353523+0000","last_unstale":"2024-03-26T05:04:49.910377+0000","last_undegraded":"2024-03-26T05:04:49.910377+0000","last_fullsized":"2024-03-26T05:04:49.910377+0000","mapping_epoch":103,"log_start":"114'3200","ondisk_log_start":"114'3200","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7680","last_scrub_stamp":"2024-03-26T05:04:11.123962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:11.123962+0000","objects_scrubbed":7433,"log_size":6903,"log_dups_size":0,"ondisk_log_size":6903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:32.441783+0000","scrub_duration":2854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323289088,"num_objects":4933,"num_object_clones":0,"num_object_copies":14799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10103,"num_write_kb":481152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":179,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.478112+0000","last_change":"2024-03-26T05:04:00.814564+0000","last_active":"2024-03-26T05:04:47.478112+0000","last_peered":"2024-03-26T05:04:47.478112+0000","last_clean":"2024-03-26T05:04:47.478112+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:04:47.478112+0000","last_undegraded":"2024-03-26T05:04:47.478112+0000","last_fullsized":"2024-03-26T05:04:47.478112+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:00.814254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:00.814254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:21.361433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":181,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.484376+0000","last_change":"2024-03-26T05:03:20.891602+0000","last_active":"2024-03-26T05:04:47.484376+0000","last_peered":"2024-03-26T05:04:47.484376+0000","last_clean":"2024-03-26T05:04:47.484376+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:04:47.484376+0000","last_undegraded":"2024-03-26T05:04:47.484376+0000","last_fullsized":"2024-03-26T05:04:47.484376+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:20.891481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:20.891481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":189,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485034+0000","last_change":"2024-03-26T05:03:13.993194+0000","last_active":"2024-03-26T05:04:47.485034+0000","last_peered":"2024-03-26T05:04:47.485034+0000","last_clean":"2024-03-26T05:04:47.485034+0000","last_became_active":"2024-03-26T05:00:22.773770+0000","last_became_peered":"2024-03-26T05:00:22.773770+0000","last_unstale":"2024-03-26T05:04:47.485034+0000","last_undegraded":"2024-03-26T05:04:47.485034+0000","last_fullsized":"2024-03-26T05:04:47.485034+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:13.993045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:13.993045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:50.692464+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'15","reported_seq":214,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.480258+0000","last_change":"2024-03-26T05:03:15.852751+0000","last_active":"2024-03-26T05:04:47.480258+0000","last_peered":"2024-03-26T05:04:47.480258+0000","last_clean":"2024-03-26T05:04:47.480258+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:04:47.480258+0000","last_undegraded":"2024-03-26T05:04:47.480258+0000","last_fullsized":"2024-03-26T05:04:47.480258+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:03:15.852614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:15.852614+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"149'9988","reported_seq":10172,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.690217+0000","last_change":"2024-03-26T05:04:07.239552+0000","last_active":"2024-03-26T05:04:49.690217+0000","last_peered":"2024-03-26T05:04:49.690217+0000","last_clean":"2024-03-26T05:04:49.690217+0000","last_became_active":"2024-03-26T05:02:24.353248+0000","last_became_peered":"2024-03-26T05:02:24.353248+0000","last_unstale":"2024-03-26T05:04:49.690217+0000","last_undegraded":"2024-03-26T05:04:49.690217+0000","last_fullsized":"2024-03-26T05:04:49.690217+0000","mapping_epoch":103,"log_start":"117'3800","ondisk_log_start":"117'3800","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7371","last_scrub_stamp":"2024-03-26T05:04:07.239265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:07.239265+0000","objects_scrubbed":7298,"log_size":6188,"log_dups_size":0,"ondisk_log_size":6188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:31.785702+0000","scrub_duration":2188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319029248,"num_objects":4868,"num_object_clones":0,"num_object_copies":14604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9988,"num_write_kb":475392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":178,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.482136+0000","last_change":"2024-03-26T05:04:11.571539+0000","last_active":"2024-03-26T05:04:47.482136+0000","last_peered":"2024-03-26T05:04:47.482136+0000","last_clean":"2024-03-26T05:04:47.482136+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:04:47.482136+0000","last_undegraded":"2024-03-26T05:04:47.482136+0000","last_fullsized":"2024-03-26T05:04:47.482136+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:11.571425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:11.571425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:33.966929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":201,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.486119+0000","last_change":"2024-03-26T05:03:10.911325+0000","last_active":"2024-03-26T05:04:47.486119+0000","last_peered":"2024-03-26T05:04:47.486119+0000","last_clean":"2024-03-26T05:04:47.486119+0000","last_became_active":"2024-03-26T05:00:21.760521+0000","last_became_peered":"2024-03-26T05:00:21.760521+0000","last_unstale":"2024-03-26T05:04:47.486119+0000","last_undegraded":"2024-03-26T05:04:47.486119+0000","last_fullsized":"2024-03-26T05:04:47.486119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:03:10.911121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:03:10.911121+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:50.692406+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":183,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.472412+0000","last_change":"2024-03-26T05:03:11.352008+0000","last_active":"2024-03-26T05:04:47.472412+0000","last_peered":"2024-03-26T05:04:47.472412+0000","last_clean":"2024-03-26T05:04:47.472412+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:04:47.472412+0000","last_undegraded":"2024-03-26T05:04:47.472412+0000","last_fullsized":"2024-03-26T05:04:47.472412+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:11.351820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:11.351820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.099656+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'34","reported_seq":308,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.479401+0000","last_change":"2024-03-26T05:04:22.562805+0000","last_active":"2024-03-26T05:04:47.479401+0000","last_peered":"2024-03-26T05:04:47.479401+0000","last_clean":"2024-03-26T05:04:47.479401+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:04:47.479401+0000","last_undegraded":"2024-03-26T05:04:47.479401+0000","last_fullsized":"2024-03-26T05:04:47.479401+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'34","last_scrub_stamp":"2024-03-26T05:04:22.562552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:22.562552+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:46.860857+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":175,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.480014+0000","last_change":"2024-03-26T05:02:16.053713+0000","last_active":"2024-03-26T05:04:47.480014+0000","last_peered":"2024-03-26T05:04:47.480014+0000","last_clean":"2024-03-26T05:04:47.480014+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:04:47.480014+0000","last_undegraded":"2024-03-26T05:04:47.480014+0000","last_fullsized":"2024-03-26T05:04:47.480014+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:16.053635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:02:16.053635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"149'9876","reported_seq":10053,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.936314+0000","last_change":"2024-03-26T05:04:24.005788+0000","last_active":"2024-03-26T05:04:49.936314+0000","last_peered":"2024-03-26T05:04:49.936314+0000","last_clean":"2024-03-26T05:04:49.936314+0000","last_became_active":"2024-03-26T05:02:24.356658+0000","last_became_peered":"2024-03-26T05:02:24.356658+0000","last_unstale":"2024-03-26T05:04:49.936314+0000","last_undegraded":"2024-03-26T05:04:49.936314+0000","last_fullsized":"2024-03-26T05:04:49.936314+0000","mapping_epoch":103,"log_start":"120'4300","ondisk_log_start":"120'4300","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'8251","last_scrub_stamp":"2024-03-26T05:04:24.005629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:24.005629+0000","objects_scrubbed":6565,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:27.436033+0000","scrub_duration":2213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319422464,"num_objects":4874,"num_object_clones":0,"num_object_copies":14622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9876,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":177,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.493854+0000","last_change":"2024-03-26T05:02:59.958661+0000","last_active":"2024-03-26T05:04:47.493854+0000","last_peered":"2024-03-26T05:04:47.493854+0000","last_clean":"2024-03-26T05:04:47.493854+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:04:47.493854+0000","last_undegraded":"2024-03-26T05:04:47.493854+0000","last_fullsized":"2024-03-26T05:04:47.493854+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:59.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:59.958579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.851723+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":190,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.478036+0000","last_change":"2024-03-26T05:04:17.462393+0000","last_active":"2024-03-26T05:04:47.478036+0000","last_peered":"2024-03-26T05:04:47.478036+0000","last_clean":"2024-03-26T05:04:47.478036+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:04:47.478036+0000","last_undegraded":"2024-03-26T05:04:47.478036+0000","last_fullsized":"2024-03-26T05:04:47.478036+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:17.462281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:17.462281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:36.813984+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":199,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.483747+0000","last_change":"2024-03-26T05:03:08.050713+0000","last_active":"2024-03-26T05:04:47.483747+0000","last_peered":"2024-03-26T05:04:47.483747+0000","last_clean":"2024-03-26T05:04:47.483747+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:04:47.483747+0000","last_undegraded":"2024-03-26T05:04:47.483747+0000","last_fullsized":"2024-03-26T05:04:47.483747+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:08.050592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:03:08.050592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'19","reported_seq":225,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.478760+0000","last_change":"2024-03-26T05:04:11.649338+0000","last_active":"2024-03-26T05:04:47.478760+0000","last_peered":"2024-03-26T05:04:47.478760+0000","last_clean":"2024-03-26T05:04:47.478760+0000","last_became_active":"2024-03-26T05:00:24.937145+0000","last_became_peered":"2024-03-26T05:00:24.937145+0000","last_unstale":"2024-03-26T05:04:47.478760+0000","last_undegraded":"2024-03-26T05:04:47.478760+0000","last_fullsized":"2024-03-26T05:04:47.478760+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-26T05:04:11.649163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:11.649163+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:15.906395+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":182,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485430+0000","last_change":"2024-03-26T05:03:12.068847+0000","last_active":"2024-03-26T05:04:47.485430+0000","last_peered":"2024-03-26T05:04:47.485430+0000","last_clean":"2024-03-26T05:04:47.485430+0000","last_became_active":"2024-03-26T05:00:26.957936+0000","last_became_peered":"2024-03-26T05:00:26.957936+0000","last_unstale":"2024-03-26T05:04:47.485430+0000","last_undegraded":"2024-03-26T05:04:47.485430+0000","last_fullsized":"2024-03-26T05:04:47.485430+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:12.068728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:12.068728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"149'9901","reported_seq":10086,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.979295+0000","last_change":"2024-03-26T05:04:12.245145+0000","last_active":"2024-03-26T05:04:49.979295+0000","last_peered":"2024-03-26T05:04:49.979295+0000","last_clean":"2024-03-26T05:04:49.979295+0000","last_became_active":"2024-03-26T05:02:24.356129+0000","last_became_peered":"2024-03-26T05:02:24.356129+0000","last_unstale":"2024-03-26T05:04:49.979295+0000","last_undegraded":"2024-03-26T05:04:49.979295+0000","last_fullsized":"2024-03-26T05:04:49.979295+0000","mapping_epoch":103,"log_start":"120'4300","ondisk_log_start":"120'4300","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7587","last_scrub_stamp":"2024-03-26T05:04:12.244918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:12.244918+0000","objects_scrubbed":7304,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:29.947538+0000","scrub_duration":3431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320405504,"num_objects":4889,"num_object_clones":0,"num_object_copies":14667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9901,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":177,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485523+0000","last_change":"2024-03-26T05:03:20.949049+0000","last_active":"2024-03-26T05:04:47.485523+0000","last_peered":"2024-03-26T05:04:47.485523+0000","last_clean":"2024-03-26T05:04:47.485523+0000","last_became_active":"2024-03-26T05:00:28.979373+0000","last_became_peered":"2024-03-26T05:00:28.979373+0000","last_unstale":"2024-03-26T05:04:47.485523+0000","last_undegraded":"2024-03-26T05:04:47.485523+0000","last_fullsized":"2024-03-26T05:04:47.485523+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:20.948908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:20.948908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:50.692496+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":444,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.495074+0000","last_change":"2024-03-26T05:04:17.821154+0000","last_active":"2024-03-26T05:04:47.495074+0000","last_peered":"2024-03-26T05:04:47.495074+0000","last_clean":"2024-03-26T05:04:47.495074+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:04:47.495074+0000","last_undegraded":"2024-03-26T05:04:47.495074+0000","last_fullsized":"2024-03-26T05:04:47.495074+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:04:17.820870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:04:17.820870+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:31.192171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":190,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.482480+0000","last_change":"2024-03-26T05:03:04.289678+0000","last_active":"2024-03-26T05:04:47.482480+0000","last_peered":"2024-03-26T05:04:47.482480+0000","last_clean":"2024-03-26T05:04:47.482480+0000","last_became_active":"2024-03-26T05:00:22.774298+0000","last_became_peered":"2024-03-26T05:00:22.774298+0000","last_unstale":"2024-03-26T05:04:47.482480+0000","last_undegraded":"2024-03-26T05:04:47.482480+0000","last_fullsized":"2024-03-26T05:04:47.482480+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:04.289541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:04.289541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'24","reported_seq":277,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.474220+0000","last_change":"2024-03-26T05:04:24.867946+0000","last_active":"2024-03-26T05:04:47.474220+0000","last_peered":"2024-03-26T05:04:47.474220+0000","last_clean":"2024-03-26T05:04:47.474220+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:04:47.474220+0000","last_undegraded":"2024-03-26T05:04:47.474220+0000","last_fullsized":"2024-03-26T05:04:47.474220+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'24","last_scrub_stamp":"2024-03-26T05:04:24.867838+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:04:24.867838+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:48.459336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":181,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.493987+0000","last_change":"2024-03-26T05:04:02.971643+0000","last_active":"2024-03-26T05:04:47.493987+0000","last_peered":"2024-03-26T05:04:47.493987+0000","last_clean":"2024-03-26T05:04:47.493987+0000","last_became_active":"2024-03-26T05:00:26.961724+0000","last_became_peered":"2024-03-26T05:00:26.961724+0000","last_unstale":"2024-03-26T05:04:47.493987+0000","last_undegraded":"2024-03-26T05:04:47.493987+0000","last_fullsized":"2024-03-26T05:04:47.493987+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:02.971526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:02.971526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:16.073532+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"149'9880","reported_seq":9983,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.730337+0000","last_change":"2024-03-26T05:02:24.355241+0000","last_active":"2024-03-26T05:04:49.730337+0000","last_peered":"2024-03-26T05:04:49.730337+0000","last_clean":"2024-03-26T05:04:49.730337+0000","last_became_active":"2024-03-26T05:02:24.354784+0000","last_became_peered":"2024-03-26T05:02:24.354784+0000","last_unstale":"2024-03-26T05:04:49.730337+0000","last_undegraded":"2024-03-26T05:04:49.730337+0000","last_fullsized":"2024-03-26T05:04:49.730337+0000","mapping_epoch":103,"log_start":"121'4700","ondisk_log_start":"121'4700","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322174976,"num_objects":4916,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9880,"num_write_kb":473472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":178,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.479101+0000","last_change":"2024-03-26T05:03:06.809992+0000","last_active":"2024-03-26T05:04:47.479101+0000","last_peered":"2024-03-26T05:04:47.479101+0000","last_clean":"2024-03-26T05:04:47.479101+0000","last_became_active":"2024-03-26T05:00:28.983202+0000","last_became_peered":"2024-03-26T05:00:28.983202+0000","last_unstale":"2024-03-26T05:04:47.479101+0000","last_undegraded":"2024-03-26T05:04:47.479101+0000","last_fullsized":"2024-03-26T05:04:47.479101+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:06.809887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:06.809887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":191,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.478306+0000","last_change":"2024-03-26T05:03:04.821432+0000","last_active":"2024-03-26T05:04:47.478306+0000","last_peered":"2024-03-26T05:04:47.478306+0000","last_clean":"2024-03-26T05:04:47.478306+0000","last_became_active":"2024-03-26T05:00:22.773081+0000","last_became_peered":"2024-03-26T05:00:22.773081+0000","last_unstale":"2024-03-26T05:04:47.478306+0000","last_undegraded":"2024-03-26T05:04:47.478306+0000","last_fullsized":"2024-03-26T05:04:47.478306+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:04.821304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:04.821304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":463,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:48.466316+0000","last_change":"2024-03-26T05:03:00.940387+0000","last_active":"2024-03-26T05:04:48.466316+0000","last_peered":"2024-03-26T05:04:48.466316+0000","last_clean":"2024-03-26T05:04:48.466316+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:04:48.466316+0000","last_undegraded":"2024-03-26T05:04:48.466316+0000","last_fullsized":"2024-03-26T05:04:48.466316+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:03:00.940299+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:03:00.940299+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":261,"num_read_kb":256,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":217,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.480214+0000","last_change":"2024-03-26T05:02:17.009773+0000","last_active":"2024-03-26T05:04:47.480214+0000","last_peered":"2024-03-26T05:04:47.480214+0000","last_clean":"2024-03-26T05:04:47.480214+0000","last_became_active":"2024-03-26T05:00:49.642276+0000","last_became_peered":"2024-03-26T05:00:49.642276+0000","last_unstale":"2024-03-26T05:04:47.480214+0000","last_undegraded":"2024-03-26T05:04:47.480214+0000","last_fullsized":"2024-03-26T05:04:47.480214+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:02:17.009685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:17.009685+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":190,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485719+0000","last_change":"2024-03-26T05:04:22.633678+0000","last_active":"2024-03-26T05:04:47.485719+0000","last_peered":"2024-03-26T05:04:47.485719+0000","last_clean":"2024-03-26T05:04:47.485719+0000","last_became_active":"2024-03-26T05:00:26.956677+0000","last_became_peered":"2024-03-26T05:00:26.956677+0000","last_unstale":"2024-03-26T05:04:47.485719+0000","last_undegraded":"2024-03-26T05:04:47.485719+0000","last_fullsized":"2024-03-26T05:04:47.485719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:22.633581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:22.633581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:30.314139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"149'9967","reported_seq":10149,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.722473+0000","last_change":"2024-03-26T05:04:04.673613+0000","last_active":"2024-03-26T05:04:49.722473+0000","last_peered":"2024-03-26T05:04:49.722473+0000","last_clean":"2024-03-26T05:04:49.722473+0000","last_became_active":"2024-03-26T05:02:24.353979+0000","last_became_peered":"2024-03-26T05:02:24.353979+0000","last_unstale":"2024-03-26T05:04:49.722473+0000","last_undegraded":"2024-03-26T05:04:49.722473+0000","last_fullsized":"2024-03-26T05:04:49.722473+0000","mapping_epoch":103,"log_start":"119'4000","ondisk_log_start":"119'4000","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7253","last_scrub_stamp":"2024-03-26T05:04:04.673346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:04.673346+0000","objects_scrubbed":7208,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:08.953957+0000","scrub_duration":1959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321716224,"num_objects":4909,"num_object_clones":0,"num_object_copies":14727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":476032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":178,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.479180+0000","last_change":"2024-03-26T05:03:20.739630+0000","last_active":"2024-03-26T05:04:47.479180+0000","last_peered":"2024-03-26T05:04:47.479180+0000","last_clean":"2024-03-26T05:04:47.479180+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:04:47.479180+0000","last_undegraded":"2024-03-26T05:04:47.479180+0000","last_fullsized":"2024-03-26T05:04:47.479180+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:03:20.739242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":190,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.479284+0000","last_change":"2024-03-26T05:03:14.868503+0000","last_active":"2024-03-26T05:04:47.479284+0000","last_peered":"2024-03-26T05:04:47.479284+0000","last_clean":"2024-03-26T05:04:47.479284+0000","last_became_active":"2024-03-26T05:00:23.346665+0000","last_became_peered":"2024-03-26T05:00:23.346665+0000","last_unstale":"2024-03-26T05:04:47.479284+0000","last_undegraded":"2024-03-26T05:04:47.479284+0000","last_fullsized":"2024-03-26T05:04:47.479284+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:14.868378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:03:14.868378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'25","reported_seq":235,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.479848+0000","last_change":"2024-03-26T05:03:16.818394+0000","last_active":"2024-03-26T05:04:47.479848+0000","last_peered":"2024-03-26T05:04:47.479848+0000","last_clean":"2024-03-26T05:04:47.479848+0000","last_became_active":"2024-03-26T05:00:25.648253+0000","last_became_peered":"2024-03-26T05:00:25.648253+0000","last_unstale":"2024-03-26T05:04:47.479848+0000","last_undegraded":"2024-03-26T05:04:47.479848+0000","last_fullsized":"2024-03-26T05:04:47.479848+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:03:16.818031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:16.818031+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":181,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.473845+0000","last_change":"2024-03-26T05:04:04.071060+0000","last_active":"2024-03-26T05:04:47.473845+0000","last_peered":"2024-03-26T05:04:47.473845+0000","last_clean":"2024-03-26T05:04:47.473845+0000","last_became_active":"2024-03-26T05:00:26.958689+0000","last_became_peered":"2024-03-26T05:00:26.958689+0000","last_unstale":"2024-03-26T05:04:47.473845+0000","last_undegraded":"2024-03-26T05:04:47.473845+0000","last_fullsized":"2024-03-26T05:04:47.473845+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:04.070935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:04:04.070935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:18.040979+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"149'10043","reported_seq":10146,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.714446+0000","last_change":"2024-03-26T05:02:24.353110+0000","last_active":"2024-03-26T05:04:49.714446+0000","last_peered":"2024-03-26T05:04:49.714446+0000","last_clean":"2024-03-26T05:04:49.714446+0000","last_became_active":"2024-03-26T05:02:24.352730+0000","last_became_peered":"2024-03-26T05:02:24.352730+0000","last_unstale":"2024-03-26T05:04:49.714446+0000","last_undegraded":"2024-03-26T05:04:49.714446+0000","last_fullsized":"2024-03-26T05:04:49.714446+0000","mapping_epoch":103,"log_start":"124'5200","ondisk_log_start":"124'5200","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:54.146715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327352320,"num_objects":4995,"num_object_clones":0,"num_object_copies":14985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10043,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":179,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.478011+0000","last_change":"2024-03-26T05:03:19.935218+0000","last_active":"2024-03-26T05:04:47.478011+0000","last_peered":"2024-03-26T05:04:47.478011+0000","last_clean":"2024-03-26T05:04:47.478011+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:04:47.478011+0000","last_undegraded":"2024-03-26T05:04:47.478011+0000","last_fullsized":"2024-03-26T05:04:47.478011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:19.935094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:19.935094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":190,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485162+0000","last_change":"2024-03-26T05:03:16.983679+0000","last_active":"2024-03-26T05:04:47.485162+0000","last_peered":"2024-03-26T05:04:47.485162+0000","last_clean":"2024-03-26T05:04:47.485162+0000","last_became_active":"2024-03-26T05:00:23.346876+0000","last_became_peered":"2024-03-26T05:00:23.346876+0000","last_unstale":"2024-03-26T05:04:47.485162+0000","last_undegraded":"2024-03-26T05:04:47.485162+0000","last_fullsized":"2024-03-26T05:04:47.485162+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:16.983556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:16.983556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:50.692587+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'16","reported_seq":175,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.484640+0000","last_change":"2024-03-26T05:03:11.028000+0000","last_active":"2024-03-26T05:04:47.484640+0000","last_peered":"2024-03-26T05:04:47.484640+0000","last_clean":"2024-03-26T05:04:47.484640+0000","last_became_active":"2024-03-26T05:00:49.641444+0000","last_became_peered":"2024-03-26T05:00:49.641444+0000","last_unstale":"2024-03-26T05:04:47.484640+0000","last_undegraded":"2024-03-26T05:04:47.484640+0000","last_fullsized":"2024-03-26T05:04:47.484640+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:03:11.027857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:11.027857+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":246,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.480412+0000","last_change":"2024-03-26T05:03:23.981271+0000","last_active":"2024-03-26T05:04:47.480412+0000","last_peered":"2024-03-26T05:04:47.480412+0000","last_clean":"2024-03-26T05:04:47.480412+0000","last_became_active":"2024-03-26T05:00:26.961907+0000","last_became_peered":"2024-03-26T05:00:26.961907+0000","last_unstale":"2024-03-26T05:04:47.480412+0000","last_undegraded":"2024-03-26T05:04:47.480412+0000","last_fullsized":"2024-03-26T05:04:47.480412+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T05:03:23.981095+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:03:23.981095+0000","last_clean_scrub_stamp":"2024-03-26T05:03:23.981095+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.970126+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"149'10127","reported_seq":10309,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.978478+0000","last_change":"2024-03-26T05:04:19.068828+0000","last_active":"2024-03-26T05:04:49.978478+0000","last_peered":"2024-03-26T05:04:49.978478+0000","last_clean":"2024-03-26T05:04:49.978478+0000","last_became_active":"2024-03-26T05:02:24.354286+0000","last_became_peered":"2024-03-26T05:02:24.354286+0000","last_unstale":"2024-03-26T05:04:49.978478+0000","last_undegraded":"2024-03-26T05:04:49.978478+0000","last_fullsized":"2024-03-26T05:04:49.978478+0000","mapping_epoch":103,"log_start":"125'5500","ondisk_log_start":"125'5500","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'8121","last_scrub_stamp":"2024-03-26T05:04:19.068650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:19.068650+0000","objects_scrubbed":7009,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:25.300640+0000","scrub_duration":3488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322109440,"num_objects":4915,"num_object_clones":0,"num_object_copies":14745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10127,"num_write_kb":481344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":177,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.493035+0000","last_change":"2024-03-26T05:04:24.924461+0000","last_active":"2024-03-26T05:04:47.493035+0000","last_peered":"2024-03-26T05:04:47.493035+0000","last_clean":"2024-03-26T05:04:47.493035+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:04:47.493035+0000","last_undegraded":"2024-03-26T05:04:47.493035+0000","last_fullsized":"2024-03-26T05:04:47.493035+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:24.924377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:24.924377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:30.780658+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":192,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.480303+0000","last_change":"2024-03-26T05:03:13.867887+0000","last_active":"2024-03-26T05:04:47.480303+0000","last_peered":"2024-03-26T05:04:47.480303+0000","last_clean":"2024-03-26T05:04:47.480303+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:04:47.480303+0000","last_undegraded":"2024-03-26T05:04:47.480303+0000","last_fullsized":"2024-03-26T05:04:47.480303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:03:13.867756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:13.867756+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'20","reported_seq":220,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.471963+0000","last_change":"2024-03-26T05:04:25.126002+0000","last_active":"2024-03-26T05:04:47.471963+0000","last_peered":"2024-03-26T05:04:47.471963+0000","last_clean":"2024-03-26T05:04:47.471963+0000","last_became_active":"2024-03-26T05:00:24.936315+0000","last_became_peered":"2024-03-26T05:00:24.936315+0000","last_unstale":"2024-03-26T05:04:47.471963+0000","last_undegraded":"2024-03-26T05:04:47.471963+0000","last_fullsized":"2024-03-26T05:04:47.471963+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:04:25.125864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:25.125864+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:33.277048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":244,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485423+0000","last_change":"2024-03-26T05:04:26.670277+0000","last_active":"2024-03-26T05:04:47.485423+0000","last_peered":"2024-03-26T05:04:47.485423+0000","last_clean":"2024-03-26T05:04:47.485423+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:04:47.485423+0000","last_undegraded":"2024-03-26T05:04:47.485423+0000","last_fullsized":"2024-03-26T05:04:47.485423+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:26.669895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:26.669895+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:47.751228+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"149'10015","reported_seq":10118,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.949146+0000","last_change":"2024-03-26T05:02:24.353027+0000","last_active":"2024-03-26T05:04:49.949146+0000","last_peered":"2024-03-26T05:04:49.949146+0000","last_clean":"2024-03-26T05:04:49.949146+0000","last_became_active":"2024-03-26T05:02:24.352725+0000","last_became_peered":"2024-03-26T05:02:24.352725+0000","last_unstale":"2024-03-26T05:04:49.949146+0000","last_undegraded":"2024-03-26T05:04:49.949146+0000","last_fullsized":"2024-03-26T05:04:49.949146+0000","mapping_epoch":103,"log_start":"125'5300","ondisk_log_start":"125'5300","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":4715,"log_dups_size":0,"ondisk_log_size":4715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:53.748527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326172672,"num_objects":4977,"num_object_clones":0,"num_object_copies":14931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10015,"num_write_kb":479744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":179,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.471456+0000","last_change":"2024-03-26T05:04:10.112906+0000","last_active":"2024-03-26T05:04:47.471456+0000","last_peered":"2024-03-26T05:04:47.471456+0000","last_clean":"2024-03-26T05:04:47.471456+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:04:47.471456+0000","last_undegraded":"2024-03-26T05:04:47.471456+0000","last_fullsized":"2024-03-26T05:04:47.471456+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:10.112830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:10.112830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:12.840322+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":182,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.489422+0000","last_change":"2024-03-26T05:02:12.036166+0000","last_active":"2024-03-26T05:04:47.489422+0000","last_peered":"2024-03-26T05:04:47.489422+0000","last_clean":"2024-03-26T05:04:47.489422+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:04:47.489422+0000","last_undegraded":"2024-03-26T05:04:47.489422+0000","last_fullsized":"2024-03-26T05:04:47.489422+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:12.036082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:12.036082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"96'40","reported_seq":252,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.481162+0000","last_change":"2024-03-26T05:03:14.123866+0000","last_active":"2024-03-26T05:04:47.481162+0000","last_peered":"2024-03-26T05:04:47.481162+0000","last_clean":"2024-03-26T05:04:47.481162+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:04:47.481162+0000","last_undegraded":"2024-03-26T05:04:47.481162+0000","last_fullsized":"2024-03-26T05:04:47.481162+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:03:14.123706+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":175,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.471801+0000","last_change":"2024-03-26T05:03:09.332660+0000","last_active":"2024-03-26T05:04:47.471801+0000","last_peered":"2024-03-26T05:04:47.471801+0000","last_clean":"2024-03-26T05:04:47.471801+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:04:47.471801+0000","last_undegraded":"2024-03-26T05:04:47.471801+0000","last_fullsized":"2024-03-26T05:04:47.471801+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:09.332533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.332533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.099682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"149'9934","reported_seq":10117,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.741700+0000","last_change":"2024-03-26T05:04:14.777321+0000","last_active":"2024-03-26T05:04:49.741700+0000","last_peered":"2024-03-26T05:04:49.741700+0000","last_clean":"2024-03-26T05:04:49.741700+0000","last_became_active":"2024-03-26T05:02:24.351780+0000","last_became_peered":"2024-03-26T05:02:24.351780+0000","last_unstale":"2024-03-26T05:04:49.741700+0000","last_undegraded":"2024-03-26T05:04:49.741700+0000","last_fullsized":"2024-03-26T05:04:49.741700+0000","mapping_epoch":103,"log_start":"118'3800","ondisk_log_start":"118'3800","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7828","last_scrub_stamp":"2024-03-26T05:04:14.777151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:14.777151+0000","objects_scrubbed":7129,"log_size":6134,"log_dups_size":0,"ondisk_log_size":6134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:37.260876+0000","scrub_duration":1692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326238208,"num_objects":4978,"num_object_clones":0,"num_object_copies":14934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9934,"num_write_kb":477184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":177,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485043+0000","last_change":"2024-03-26T05:03:21.933021+0000","last_active":"2024-03-26T05:04:47.485043+0000","last_peered":"2024-03-26T05:04:47.485043+0000","last_clean":"2024-03-26T05:04:47.485043+0000","last_became_active":"2024-03-26T05:00:28.982825+0000","last_became_peered":"2024-03-26T05:00:28.982825+0000","last_unstale":"2024-03-26T05:04:47.485043+0000","last_undegraded":"2024-03-26T05:04:47.485043+0000","last_fullsized":"2024-03-26T05:04:47.485043+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:21.932883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:21.932883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:50.692521+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":244,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.472590+0000","last_change":"2024-03-26T05:04:09.071212+0000","last_active":"2024-03-26T05:04:47.472590+0000","last_peered":"2024-03-26T05:04:47.472590+0000","last_clean":"2024-03-26T05:04:47.472590+0000","last_became_active":"2024-03-26T05:00:26.956596+0000","last_became_peered":"2024-03-26T05:00:26.956596+0000","last_unstale":"2024-03-26T05:04:47.472590+0000","last_undegraded":"2024-03-26T05:04:47.472590+0000","last_fullsized":"2024-03-26T05:04:47.472590+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:09.071057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:09.071057+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:17.591868+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":190,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.481083+0000","last_change":"2024-03-26T05:04:06.525151+0000","last_active":"2024-03-26T05:04:47.481083+0000","last_peered":"2024-03-26T05:04:47.481083+0000","last_clean":"2024-03-26T05:04:47.481083+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:04:47.481083+0000","last_undegraded":"2024-03-26T05:04:47.481083+0000","last_fullsized":"2024-03-26T05:04:47.481083+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:06.525047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:06.525047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:21.294383+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":218,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.478917+0000","last_change":"2024-03-26T05:04:24.625537+0000","last_active":"2024-03-26T05:04:47.478917+0000","last_peered":"2024-03-26T05:04:47.478917+0000","last_clean":"2024-03-26T05:04:47.478917+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:04:47.478917+0000","last_undegraded":"2024-03-26T05:04:47.478917+0000","last_fullsized":"2024-03-26T05:04:47.478917+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:04:24.625406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:24.625406+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:47.792081+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"149'10185","reported_seq":10370,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.741666+0000","last_change":"2024-03-26T05:04:08.237214+0000","last_active":"2024-03-26T05:04:49.741666+0000","last_peered":"2024-03-26T05:04:49.741666+0000","last_clean":"2024-03-26T05:04:49.741666+0000","last_became_active":"2024-03-26T05:02:24.352008+0000","last_became_peered":"2024-03-26T05:02:24.352008+0000","last_unstale":"2024-03-26T05:04:49.741666+0000","last_undegraded":"2024-03-26T05:04:49.741666+0000","last_fullsized":"2024-03-26T05:04:49.741666+0000","mapping_epoch":103,"log_start":"118'4000","ondisk_log_start":"118'4000","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7544","last_scrub_stamp":"2024-03-26T05:04:08.237042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:08.237042+0000","objects_scrubbed":7490,"log_size":6185,"log_dups_size":0,"ondisk_log_size":6185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:12.281990+0000","scrub_duration":2206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321257496,"num_objects":4903,"num_object_clones":0,"num_object_copies":14709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10185,"num_write_kb":482753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":177,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.482385+0000","last_change":"2024-03-26T05:04:25.636127+0000","last_active":"2024-03-26T05:04:47.482385+0000","last_peered":"2024-03-26T05:04:47.482385+0000","last_clean":"2024-03-26T05:04:47.482385+0000","last_became_active":"2024-03-26T05:00:29.461236+0000","last_became_peered":"2024-03-26T05:00:29.461236+0000","last_unstale":"2024-03-26T05:04:47.482385+0000","last_undegraded":"2024-03-26T05:04:47.482385+0000","last_fullsized":"2024-03-26T05:04:47.482385+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:25.636012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:25.636012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:37.693140+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":181,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485661+0000","last_change":"2024-03-26T05:03:03.891855+0000","last_active":"2024-03-26T05:04:47.485661+0000","last_peered":"2024-03-26T05:04:47.485661+0000","last_clean":"2024-03-26T05:04:47.485661+0000","last_became_active":"2024-03-26T05:00:26.955805+0000","last_became_peered":"2024-03-26T05:00:26.955805+0000","last_unstale":"2024-03-26T05:04:47.485661+0000","last_undegraded":"2024-03-26T05:04:47.485661+0000","last_fullsized":"2024-03-26T05:04:47.485661+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:03.891766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:03.891766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:50.692548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":191,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.471478+0000","last_change":"2024-03-26T05:03:19.466684+0000","last_active":"2024-03-26T05:04:47.471478+0000","last_peered":"2024-03-26T05:04:47.471478+0000","last_clean":"2024-03-26T05:04:47.471478+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:04:47.471478+0000","last_undegraded":"2024-03-26T05:04:47.471478+0000","last_fullsized":"2024-03-26T05:04:47.471478+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:19.466546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:19.466546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.103943+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'17","reported_seq":217,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.472345+0000","last_change":"2024-03-26T05:03:23.417863+0000","last_active":"2024-03-26T05:04:47.472345+0000","last_peered":"2024-03-26T05:04:47.472345+0000","last_clean":"2024-03-26T05:04:47.472345+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:04:47.472345+0000","last_undegraded":"2024-03-26T05:04:47.472345+0000","last_fullsized":"2024-03-26T05:04:47.472345+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:03:23.417719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:23.417719+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:50.098029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"149'10103","reported_seq":10289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.847706+0000","last_change":"2024-03-26T05:04:09.590882+0000","last_active":"2024-03-26T05:04:49.847706+0000","last_peered":"2024-03-26T05:04:49.847706+0000","last_clean":"2024-03-26T05:04:49.847706+0000","last_became_active":"2024-03-26T05:02:24.357742+0000","last_became_peered":"2024-03-26T05:02:24.357742+0000","last_unstale":"2024-03-26T05:04:49.847706+0000","last_undegraded":"2024-03-26T05:04:49.847706+0000","last_fullsized":"2024-03-26T05:04:49.847706+0000","mapping_epoch":103,"log_start":"113'3200","ondisk_log_start":"113'3200","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7576","last_scrub_stamp":"2024-03-26T05:04:09.590405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:09.590405+0000","objects_scrubbed":7467,"log_size":6903,"log_dups_size":0,"ondisk_log_size":6903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:26.578806+0000","scrub_duration":3319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321847296,"num_objects":4911,"num_object_clones":0,"num_object_copies":14733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10103,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":177,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.483756+0000","last_change":"2024-03-26T05:03:21.940017+0000","last_active":"2024-03-26T05:04:47.483756+0000","last_peered":"2024-03-26T05:04:47.483756+0000","last_clean":"2024-03-26T05:04:47.483756+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:04:47.483756+0000","last_undegraded":"2024-03-26T05:04:47.483756+0000","last_fullsized":"2024-03-26T05:04:47.483756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:21.939782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:21.939782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":181,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.479497+0000","last_change":"2024-03-26T05:03:18.783710+0000","last_active":"2024-03-26T05:04:47.479497+0000","last_peered":"2024-03-26T05:04:47.479497+0000","last_clean":"2024-03-26T05:04:47.479497+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:04:47.479497+0000","last_undegraded":"2024-03-26T05:04:47.479497+0000","last_fullsized":"2024-03-26T05:04:47.479497+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:18.783471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:18.783471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":197,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.473595+0000","last_change":"2024-03-26T05:04:21.884967+0000","last_active":"2024-03-26T05:04:47.473595+0000","last_peered":"2024-03-26T05:04:47.473595+0000","last_clean":"2024-03-26T05:04:47.473595+0000","last_became_active":"2024-03-26T05:00:22.768610+0000","last_became_peered":"2024-03-26T05:00:22.768610+0000","last_unstale":"2024-03-26T05:04:47.473595+0000","last_undegraded":"2024-03-26T05:04:47.473595+0000","last_fullsized":"2024-03-26T05:04:47.473595+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:21.884876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.884876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:37.934924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":206,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.472190+0000","last_change":"2024-03-26T05:02:57.386713+0000","last_active":"2024-03-26T05:04:47.472190+0000","last_peered":"2024-03-26T05:04:47.472190+0000","last_clean":"2024-03-26T05:04:47.472190+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:04:47.472190+0000","last_undegraded":"2024-03-26T05:04:47.472190+0000","last_fullsized":"2024-03-26T05:04:47.472190+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:02:57.386555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:57.386555+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:48.099490+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"149'9923","reported_seq":10026,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:49.959553+0000","last_change":"2024-03-26T05:02:24.352484+0000","last_active":"2024-03-26T05:04:49.959553+0000","last_peered":"2024-03-26T05:04:49.959553+0000","last_clean":"2024-03-26T05:04:49.959553+0000","last_became_active":"2024-03-26T05:02:24.352196+0000","last_became_peered":"2024-03-26T05:02:24.352196+0000","last_unstale":"2024-03-26T05:04:49.959553+0000","last_undegraded":"2024-03-26T05:04:49.959553+0000","last_fullsized":"2024-03-26T05:04:49.959553+0000","mapping_epoch":103,"log_start":"120'4300","ondisk_log_start":"120'4300","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:02:23.329282+0000","objects_scrubbed":0,"log_size":5623,"log_dups_size":0,"ondisk_log_size":5623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.969863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318177280,"num_objects":4855,"num_object_clones":0,"num_object_copies":14565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9923,"num_write_kb":472896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":178,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.472602+0000","last_change":"2024-03-26T05:04:22.211543+0000","last_active":"2024-03-26T05:04:47.472602+0000","last_peered":"2024-03-26T05:04:47.472602+0000","last_clean":"2024-03-26T05:04:47.472602+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:04:47.472602+0000","last_undegraded":"2024-03-26T05:04:47.472602+0000","last_fullsized":"2024-03-26T05:04:47.472602+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:22.211431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:22.211431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:23.405861+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"117'9","reported_seq":244,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.485042+0000","last_change":"2024-03-26T05:03:09.035963+0000","last_active":"2024-03-26T05:04:47.485042+0000","last_peered":"2024-03-26T05:04:47.485042+0000","last_clean":"2024-03-26T05:04:47.485042+0000","last_became_active":"2024-03-26T05:00:26.957142+0000","last_became_peered":"2024-03-26T05:00:26.957142+0000","last_unstale":"2024-03-26T05:04:47.485042+0000","last_undegraded":"2024-03-26T05:04:47.485042+0000","last_fullsized":"2024-03-26T05:04:47.485042+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'9","last_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.035843+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":189,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.484238+0000","last_change":"2024-03-26T05:03:09.192328+0000","last_active":"2024-03-26T05:04:47.484238+0000","last_peered":"2024-03-26T05:04:47.484238+0000","last_clean":"2024-03-26T05:04:47.484238+0000","last_became_active":"2024-03-26T05:00:22.778719+0000","last_became_peered":"2024-03-26T05:00:22.778719+0000","last_unstale":"2024-03-26T05:04:47.484238+0000","last_undegraded":"2024-03-26T05:04:47.484238+0000","last_fullsized":"2024-03-26T05:04:47.484238+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:09.192105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.192105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":201,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-26T05:04:47.471833+0000","last_change":"2024-03-26T05:04:16.153198+0000","last_active":"2024-03-26T05:04:47.471833+0000","last_peered":"2024-03-26T05:04:47.471833+0000","last_clean":"2024-03-26T05:04:47.471833+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:04:47.471833+0000","last_undegraded":"2024-03-26T05:04:47.471833+0000","last_fullsized":"2024-03-26T05:04:47.471833+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:04:16.153054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:16.153054+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:39.530782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5166727192,"num_objects":78839,"num_object_clones":0,"num_object_copies":236517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78839,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160155,"num_write_kb":7647745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15500914688,"data_stored":15500902472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89755,"ondisk_log_size":89755,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":261,"num_read_kb":256,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":617,"ondisk_log_size":617,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005114,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1658560,"kb_used_data":1595384,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92086592,"statfs":{"total":95995035648,"available":94296670208,"internally_reserved":0,"allocated":1633673216,"data_stored":1633359152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332159,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1656820,"kb_used_data":1589548,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92088332,"statfs":{"total":95995035648,"available":94298451968,"internally_reserved":0,"allocated":1627697152,"data_stored":1627393884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2015268,"kb_used_data":1915804,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91729884,"statfs":{"total":95995035648,"available":93931401216,"internally_reserved":0,"allocated":1961783296,"data_stored":1961431596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2646352,"kb_used_data":2534664,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91098800,"statfs":{"total":95995035648,"available":93285171200,"internally_reserved":0,"allocated":2595495936,"data_stored":2595165665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2012900,"kb_used_data":1913692,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91732252,"statfs":{"total":95995035648,"available":93933826048,"internally_reserved":0,"allocated":1959620608,"data_stored":1959332970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117070,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2981964,"kb_used_data":2842432,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":90763188,"statfs":{"total":95995035648,"available":92941504512,"internally_reserved":0,"allocated":2910650368,"data_stored":2910326532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142801351},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920842,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1663304,"kb_used_data":1595964,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92081848,"statfs":{"total":95995035648,"available":94291812352,"internally_reserved":0,"allocated":1634267136,"data_stored":1633947985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659204,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1346224,"kb_used_data":1270632,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92398928,"statfs":{"total":95995035648,"available":94616502272,"internally_reserved":0,"allocated":1301127168,"data_stored":1300827754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2561150976,"data_stored":2561146904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1939214336,"data_stored":1939210264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1939668992,"data_stored":1939668992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2901082112,"data_stored":2901082112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1622016000,"data_stored":1622016000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1300037632,"data_stored":1300037632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1624641536,"data_stored":1624637464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1613103104,"data_stored":1613103104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:04:55.794 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:04:55.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:55 smithi132 ceph-mon[33500]: 3.7 scrub starts 2024-03-26T05:04:55.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:55 smithi132 ceph-mon[33500]: 3.7 scrub ok 2024-03-26T05:04:55.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:55 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:55.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:55 smithi132 ceph-mon[33500]: 5.5 scrub starts 2024-03-26T05:04:55.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:55 smithi132 ceph-mon[33500]: 5.5 scrub ok 2024-03-26T05:04:55.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:55 smithi132 ceph-mon[33500]: 7.1 scrub starts 2024-03-26T05:04:55.879 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:55.920 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:55.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:56.022 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:56.100 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:56.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:56.202 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:04:56.236 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:56.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:56.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:04:56.339 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:56.425 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:56.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:56.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:56.442 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:56.478 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:04:56.523 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:04:56.526 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:56.563 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:04:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:56.579 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:56.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:56.594 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:04:56.594 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:04:56.594 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[29691]: pgmap v182: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T05:04:56.594 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[29691]: 4.6 scrub starts 2024-03-26T05:04:56.594 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[29691]: 4.6 scrub ok 2024-03-26T05:04:56.594 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[29691]: 6.1e scrub starts 2024-03-26T05:04:56.594 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[29691]: 6.1e scrub ok 2024-03-26T05:04:56.594 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[29691]: 7.1 scrub ok 2024-03-26T05:04:56.594 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3430521926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:04:56.595 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:04:56.595 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:04:56.595 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[40404]: pgmap v182: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T05:04:56.595 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[40404]: 4.6 scrub starts 2024-03-26T05:04:56.595 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[40404]: 4.6 scrub ok 2024-03-26T05:04:56.595 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[40404]: 6.1e scrub starts 2024-03-26T05:04:56.595 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[40404]: 6.1e scrub ok 2024-03-26T05:04:56.595 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[40404]: 7.1 scrub ok 2024-03-26T05:04:56.596 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:56 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3430521926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:04:56.602 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:04:56.641 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:04:56.679 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:04:56.680 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:56.717 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:04:56.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:56.733 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:56.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:56.757 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:04:56.762 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:04:56.762 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:04:56.762 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:04:56.762 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:04:56.762 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:04:56.763 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:04:56.763 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:04:56.763 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:04:56.779 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:04:56.834 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:56.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:56.893 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:56.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:56.896 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:56 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:04:56.896 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:56 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:04:56.896 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:56 smithi132 ceph-mon[33500]: pgmap v182: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T05:04:56.896 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:56 smithi132 ceph-mon[33500]: 4.6 scrub starts 2024-03-26T05:04:56.896 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:56 smithi132 ceph-mon[33500]: 4.6 scrub ok 2024-03-26T05:04:56.896 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:56 smithi132 ceph-mon[33500]: 6.1e scrub starts 2024-03-26T05:04:56.896 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:56 smithi132 ceph-mon[33500]: 6.1e scrub ok 2024-03-26T05:04:56.896 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:56 smithi132 ceph-mon[33500]: 7.1 scrub ok 2024-03-26T05:04:56.896 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:56 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3430521926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:04:56.994 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:57.024 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:57.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:57.126 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:57.173 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:57.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:57.275 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:57.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:57.319 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:57.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:57.421 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:57.474 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:57.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:57.575 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:57.607 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:57.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:57.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:57 smithi080 ceph-mon[40404]: 6.1d scrub starts 2024-03-26T05:04:57.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:57 smithi080 ceph-mon[40404]: 6.1d scrub ok 2024-03-26T05:04:57.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:57 smithi080 ceph-mon[40404]: 7.3 scrub starts 2024-03-26T05:04:57.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:57 smithi080 ceph-mon[29691]: 6.1d scrub starts 2024-03-26T05:04:57.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:57 smithi080 ceph-mon[29691]: 6.1d scrub ok 2024-03-26T05:04:57.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:57 smithi080 ceph-mon[29691]: 7.3 scrub starts 2024-03-26T05:04:57.708 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:57.744 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:57.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:57.749 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:04:57.749 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:04:57.845 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:57.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:57.883 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:57.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:57.964 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:57 smithi132 ceph-mon[33500]: 6.1d scrub starts 2024-03-26T05:04:57.964 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:57 smithi132 ceph-mon[33500]: 6.1d scrub ok 2024-03-26T05:04:57.964 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:57 smithi132 ceph-mon[33500]: 7.3 scrub starts 2024-03-26T05:04:57.984 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:58.018 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:58.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:58.120 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:58.158 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:58.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:58.260 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:58.290 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:58.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:58.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:58.392 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:58.424 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:58.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:58.526 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:58.562 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:58.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:58.663 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:04:58.705 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:58.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:58.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:04:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:58 smithi132 ceph-mon[33500]: 2.1 scrub starts 2024-03-26T05:04:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:58 smithi132 ceph-mon[33500]: 2.1 scrub ok 2024-03-26T05:04:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:58 smithi132 ceph-mon[33500]: 7.5 scrub starts 2024-03-26T05:04:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:58 smithi132 ceph-mon[33500]: pgmap v183: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1k op/s 2024-03-26T05:04:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:58 smithi132 ceph-mon[33500]: 7.f scrub starts 2024-03-26T05:04:58.807 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:04:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:58.843 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:58.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:04:58.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:58 smithi080 ceph-mon[40404]: 2.1 scrub starts 2024-03-26T05:04:58.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:58 smithi080 ceph-mon[40404]: 2.1 scrub ok 2024-03-26T05:04:58.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:58 smithi080 ceph-mon[40404]: 7.5 scrub starts 2024-03-26T05:04:58.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:58 smithi080 ceph-mon[40404]: pgmap v183: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1k op/s 2024-03-26T05:04:58.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:58 smithi080 ceph-mon[40404]: 7.f scrub starts 2024-03-26T05:04:58.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:58 smithi080 ceph-mon[29691]: 2.1 scrub starts 2024-03-26T05:04:58.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:58 smithi080 ceph-mon[29691]: 2.1 scrub ok 2024-03-26T05:04:58.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:58 smithi080 ceph-mon[29691]: 7.5 scrub starts 2024-03-26T05:04:58.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:58 smithi080 ceph-mon[29691]: pgmap v183: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1k op/s 2024-03-26T05:04:58.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:58 smithi080 ceph-mon[29691]: 7.f scrub starts 2024-03-26T05:04:58.897 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T05:04:58.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:58.981 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:58.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:04:59.082 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:04:59.122 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:59.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:04:59.224 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:59.268 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:59.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:59.337 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:04:59.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:59.412 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:59.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:59.514 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:04:59.542 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:59.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:04:59.644 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:04:59.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:59.687 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:59.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:04:59.788 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:04:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:59.829 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:04:59.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:04:59.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:59 smithi080 ceph-mon[40404]: 7.3 scrub ok 2024-03-26T05:04:59.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:59 smithi080 ceph-mon[40404]: 6.11 scrub starts 2024-03-26T05:04:59.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:59 smithi080 ceph-mon[40404]: 6.11 scrub ok 2024-03-26T05:04:59.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:59 smithi080 ceph-mon[40404]: 6.18 scrub starts 2024-03-26T05:04:59.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:04:59 smithi080 ceph-mon[40404]: 6.18 scrub ok 2024-03-26T05:04:59.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:59 smithi080 ceph-mon[29691]: 7.3 scrub ok 2024-03-26T05:04:59.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:59 smithi080 ceph-mon[29691]: 6.11 scrub starts 2024-03-26T05:04:59.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:59 smithi080 ceph-mon[29691]: 6.11 scrub ok 2024-03-26T05:04:59.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:59 smithi080 ceph-mon[29691]: 6.18 scrub starts 2024-03-26T05:04:59.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:04:59 smithi080 ceph-mon[29691]: 6.18 scrub ok 2024-03-26T05:04:59.880 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:59 smithi132 ceph-mon[33500]: 7.3 scrub ok 2024-03-26T05:04:59.880 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:59 smithi132 ceph-mon[33500]: 6.11 scrub starts 2024-03-26T05:04:59.880 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:59 smithi132 ceph-mon[33500]: 6.11 scrub ok 2024-03-26T05:04:59.881 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:59 smithi132 ceph-mon[33500]: 6.18 scrub starts 2024-03-26T05:04:59.881 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:04:59 smithi132 ceph-mon[33500]: 6.18 scrub ok 2024-03-26T05:04:59.930 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:04:59.957 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:04:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:59.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:00.059 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:00.090 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:00.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:00.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:00.365 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:00.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:00.467 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:00.497 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:00.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:00.598 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:00.641 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:00.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:00.686 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:00 smithi132 ceph-mon[33500]: 2.2 scrub starts 2024-03-26T05:05:00.686 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:00 smithi132 ceph-mon[33500]: 2.2 scrub ok 2024-03-26T05:05:00.686 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:00 smithi132 ceph-mon[33500]: 7.5 scrub ok 2024-03-26T05:05:00.686 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:00 smithi132 ceph-mon[33500]: pgmap v184: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 734 op/s 2024-03-26T05:05:00.686 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:00 smithi132 ceph-mon[33500]: 5.17 scrub starts 2024-03-26T05:05:00.686 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:00 smithi132 ceph-mon[33500]: 5.17 scrub ok 2024-03-26T05:05:00.686 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:00 smithi132 ceph-mon[33500]: 3.1 scrub starts 2024-03-26T05:05:00.686 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:00 smithi132 ceph-mon[33500]: 3.1 scrub ok 2024-03-26T05:05:00.687 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:00 smithi132 ceph-mon[33500]: 7.f scrub ok 2024-03-26T05:05:00.742 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:00.787 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:00.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:00.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[29691]: 2.2 scrub starts 2024-03-26T05:05:00.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[29691]: 2.2 scrub ok 2024-03-26T05:05:00.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[29691]: 7.5 scrub ok 2024-03-26T05:05:00.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[29691]: pgmap v184: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 734 op/s 2024-03-26T05:05:00.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[29691]: 5.17 scrub starts 2024-03-26T05:05:00.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[29691]: 5.17 scrub ok 2024-03-26T05:05:00.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[29691]: 3.1 scrub starts 2024-03-26T05:05:00.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[29691]: 3.1 scrub ok 2024-03-26T05:05:00.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[29691]: 7.f scrub ok 2024-03-26T05:05:00.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[40404]: 2.2 scrub starts 2024-03-26T05:05:00.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[40404]: 2.2 scrub ok 2024-03-26T05:05:00.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[40404]: 7.5 scrub ok 2024-03-26T05:05:00.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[40404]: pgmap v184: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 734 op/s 2024-03-26T05:05:00.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[40404]: 5.17 scrub starts 2024-03-26T05:05:00.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[40404]: 5.17 scrub ok 2024-03-26T05:05:00.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[40404]: 3.1 scrub starts 2024-03-26T05:05:00.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[40404]: 3.1 scrub ok 2024-03-26T05:05:00.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:00 smithi080 ceph-mon[40404]: 7.f scrub ok 2024-03-26T05:05:00.888 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:00.917 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:00.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:01.019 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:01.067 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:01.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:01.169 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:01.195 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:01.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:01.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:01.297 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:01.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:01.329 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:01.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:01.430 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:01.460 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:01.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:01.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:01.562 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:01.589 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:01.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:01.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:01.691 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:01.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:01.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:01.779 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[40404]: 6.a scrub starts 2024-03-26T05:05:01.779 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[40404]: 6.a scrub ok 2024-03-26T05:05:01.779 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[40404]: 4.1c scrub starts 2024-03-26T05:05:01.780 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[40404]: 4.1c scrub ok 2024-03-26T05:05:01.780 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[40404]: 4.9 scrub starts 2024-03-26T05:05:01.780 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[40404]: 4.9 scrub ok 2024-03-26T05:05:01.780 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[40404]: 3.6 scrub starts 2024-03-26T05:05:01.780 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[40404]: 3.6 scrub ok 2024-03-26T05:05:01.780 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[29691]: 6.a scrub starts 2024-03-26T05:05:01.780 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[29691]: 6.a scrub ok 2024-03-26T05:05:01.780 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[29691]: 4.1c scrub starts 2024-03-26T05:05:01.780 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[29691]: 4.1c scrub ok 2024-03-26T05:05:01.780 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[29691]: 4.9 scrub starts 2024-03-26T05:05:01.780 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[29691]: 4.9 scrub ok 2024-03-26T05:05:01.780 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[29691]: 3.6 scrub starts 2024-03-26T05:05:01.781 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:01 smithi080 ceph-mon[29691]: 3.6 scrub ok 2024-03-26T05:05:01.819 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:01.848 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:01.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:01.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:01.950 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:01.957 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:05:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:02.026 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:02.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:02.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:01 smithi132 ceph-mon[33500]: 6.a scrub starts 2024-03-26T05:05:02.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:01 smithi132 ceph-mon[33500]: 6.a scrub ok 2024-03-26T05:05:02.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:01 smithi132 ceph-mon[33500]: 4.1c scrub starts 2024-03-26T05:05:02.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:01 smithi132 ceph-mon[33500]: 4.1c scrub ok 2024-03-26T05:05:02.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:01 smithi132 ceph-mon[33500]: 4.9 scrub starts 2024-03-26T05:05:02.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:01 smithi132 ceph-mon[33500]: 4.9 scrub ok 2024-03-26T05:05:02.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:01 smithi132 ceph-mon[33500]: 3.6 scrub starts 2024-03-26T05:05:02.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:01 smithi132 ceph-mon[33500]: 3.6 scrub ok 2024-03-26T05:05:02.126 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:02.157 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:02.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:02.259 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:02.319 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:05:02.323 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:05:02.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:02.345 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:02.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:02.447 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:02.495 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:02.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:02.596 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:02.635 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:05:02.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:02.647 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:02.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:02.748 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:02.783 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:02.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:02.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:02 smithi132 ceph-mon[33500]: pgmap v185: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 996 B/s rd, 0 B/s wr, 934 op/s 2024-03-26T05:05:02.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:02 smithi132 ceph-mon[33500]: 5.11 scrub starts 2024-03-26T05:05:02.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:02 smithi132 ceph-mon[33500]: 5.11 scrub ok 2024-03-26T05:05:02.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:02 smithi132 ceph-mon[33500]: 3.c scrub starts 2024-03-26T05:05:02.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:02 smithi132 ceph-mon[33500]: 3.c scrub ok 2024-03-26T05:05:02.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:02 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2276398023' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:05:02.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:02 smithi132 ceph-mon[33500]: 4.3 scrub starts 2024-03-26T05:05:02.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:02 smithi132 ceph-mon[33500]: 4.3 scrub ok 2024-03-26T05:05:02.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:02 smithi132 ceph-mon[33500]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:05:02.884 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:02.919 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:02.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:02.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[40404]: pgmap v185: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 996 B/s rd, 0 B/s wr, 934 op/s 2024-03-26T05:05:02.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[40404]: 5.11 scrub starts 2024-03-26T05:05:02.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[40404]: 5.11 scrub ok 2024-03-26T05:05:02.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[40404]: 3.c scrub starts 2024-03-26T05:05:02.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[40404]: 3.c scrub ok 2024-03-26T05:05:02.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2276398023' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:05:02.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[40404]: 4.3 scrub starts 2024-03-26T05:05:02.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[40404]: 4.3 scrub ok 2024-03-26T05:05:02.970 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[40404]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:05:02.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[29691]: pgmap v185: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 996 B/s rd, 0 B/s wr, 934 op/s 2024-03-26T05:05:02.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[29691]: 5.11 scrub starts 2024-03-26T05:05:02.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[29691]: 5.11 scrub ok 2024-03-26T05:05:02.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[29691]: 3.c scrub starts 2024-03-26T05:05:02.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[29691]: 3.c scrub ok 2024-03-26T05:05:02.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2276398023' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:05:02.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[29691]: 4.3 scrub starts 2024-03-26T05:05:02.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[29691]: 4.3 scrub ok 2024-03-26T05:05:02.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:02 smithi080 ceph-mon[29691]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:05:03.015 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-03-26T05:05:01.790510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4538964342,"num_objects":69481,"num_object_clones":0,"num_object_copies":208443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69481,"num_whiteouts":0,"num_read":1494,"num_read_kb":1308,"num_write":170550,"num_write_kb":7650206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90572,"ondisk_log_size":90572,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14170824,"kb_used_data":13378568,"kb_used_omap":12,"kb_used_meta":792115,"kb_avail":735790392,"statfs":{"total":767960285184,"available":753449361408,"internally_reserved":0,"allocated":13699653632,"data_stored":13697057714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":811126320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-628359168,"num_objects":-9588,"num_object_clones":0,"num_object_copies":-28764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9588,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9588,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.273680"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.519142+0000","last_change":"2024-03-26T05:04:18.210889+0000","last_active":"2024-03-26T05:04:51.519142+0000","last_peered":"2024-03-26T05:04:51.519142+0000","last_clean":"2024-03-26T05:04:51.519142+0000","last_became_active":"2024-03-26T05:00:28.972969+0000","last_became_peered":"2024-03-26T05:00:28.972969+0000","last_unstale":"2024-03-26T05:04:51.519142+0000","last_undegraded":"2024-03-26T05:04:51.519142+0000","last_fullsized":"2024-03-26T05:04:51.519142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:18.210754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:18.210754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:32.582721+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.538573+0000","last_change":"2024-03-26T05:03:13.909344+0000","last_active":"2024-03-26T05:04:51.538573+0000","last_peered":"2024-03-26T05:04:51.538573+0000","last_clean":"2024-03-26T05:04:51.538573+0000","last_became_active":"2024-03-26T05:00:26.953154+0000","last_became_peered":"2024-03-26T05:00:26.953154+0000","last_unstale":"2024-03-26T05:04:51.538573+0000","last_undegraded":"2024-03-26T05:04:51.538573+0000","last_fullsized":"2024-03-26T05:04:51.538573+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:13.909270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:13.909270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536382+0000","last_change":"2024-03-26T05:02:57.757086+0000","last_active":"2024-03-26T05:04:51.536382+0000","last_peered":"2024-03-26T05:04:51.536382+0000","last_clean":"2024-03-26T05:04:51.536382+0000","last_became_active":"2024-03-26T05:00:23.346607+0000","last_became_peered":"2024-03-26T05:00:23.346607+0000","last_unstale":"2024-03-26T05:04:51.536382+0000","last_undegraded":"2024-03-26T05:04:51.536382+0000","last_fullsized":"2024-03-26T05:04:51.536382+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:57.756968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:57.756968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.970047+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'15","reported_seq":210,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.518953+0000","last_change":"2024-03-26T05:02:58.415674+0000","last_active":"2024-03-26T05:04:51.518953+0000","last_peered":"2024-03-26T05:04:51.518953+0000","last_clean":"2024-03-26T05:04:51.518953+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:04:51.518953+0000","last_undegraded":"2024-03-26T05:04:51.518953+0000","last_fullsized":"2024-03-26T05:04:51.518953+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:02:58.415577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:58.415577+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:54.146788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":173,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.531940+0000","last_change":"2024-03-26T05:03:00.192980+0000","last_active":"2024-03-26T05:04:51.531940+0000","last_peered":"2024-03-26T05:04:51.531940+0000","last_clean":"2024-03-26T05:04:51.531940+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:04:51.531940+0000","last_undegraded":"2024-03-26T05:04:51.531940+0000","last_fullsized":"2024-03-26T05:04:51.531940+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:00.192732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:00.192732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:53.748814+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":178,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.518867+0000","last_change":"2024-03-26T05:03:12.394511+0000","last_active":"2024-03-26T05:04:51.518867+0000","last_peered":"2024-03-26T05:04:51.518867+0000","last_clean":"2024-03-26T05:04:51.518867+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:04:51.518867+0000","last_undegraded":"2024-03-26T05:04:51.518867+0000","last_fullsized":"2024-03-26T05:04:51.518867+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:12.394429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:12.394429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:54.146804+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.532983+0000","last_change":"2024-03-26T05:03:06.237289+0000","last_active":"2024-03-26T05:04:51.532983+0000","last_peered":"2024-03-26T05:04:51.532983+0000","last_clean":"2024-03-26T05:04:51.532983+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:04:51.532983+0000","last_undegraded":"2024-03-26T05:04:51.532983+0000","last_fullsized":"2024-03-26T05:04:51.532983+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:06.236963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:06.236963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:53.748680+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'39","reported_seq":262,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.534179+0000","last_change":"2024-03-26T05:04:20.637111+0000","last_active":"2024-03-26T05:04:51.534179+0000","last_peered":"2024-03-26T05:04:51.534179+0000","last_clean":"2024-03-26T05:04:51.534179+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:04:51.534179+0000","last_undegraded":"2024-03-26T05:04:51.534179+0000","last_fullsized":"2024-03-26T05:04:51.534179+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'39","last_scrub_stamp":"2024-03-26T05:04:20.636687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:20.636687+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:31.492803+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":190,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:58.964119+0000","last_change":"2024-03-26T05:04:58.964528+0000","last_active":"2024-03-26T05:04:58.964119+0000","last_peered":"2024-03-26T05:04:58.964119+0000","last_clean":"2024-03-26T05:04:58.964119+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:04:58.964119+0000","last_undegraded":"2024-03-26T05:04:58.964119+0000","last_fullsized":"2024-03-26T05:04:58.964119+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:58.963801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.963801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:21.663238+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536413+0000","last_change":"2024-03-26T05:03:22.896270+0000","last_active":"2024-03-26T05:04:51.536413+0000","last_peered":"2024-03-26T05:04:51.536413+0000","last_clean":"2024-03-26T05:04:51.536413+0000","last_became_active":"2024-03-26T05:00:26.956013+0000","last_became_peered":"2024-03-26T05:00:26.956013+0000","last_unstale":"2024-03-26T05:04:51.536413+0000","last_undegraded":"2024-03-26T05:04:51.536413+0000","last_fullsized":"2024-03-26T05:04:51.536413+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:22.896182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:22.896182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:51.928139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":201,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536718+0000","last_change":"2024-03-26T05:04:18.640724+0000","last_active":"2024-03-26T05:04:51.536718+0000","last_peered":"2024-03-26T05:04:51.536718+0000","last_clean":"2024-03-26T05:04:51.536718+0000","last_became_active":"2024-03-26T05:00:22.768098+0000","last_became_peered":"2024-03-26T05:00:22.768098+0000","last_unstale":"2024-03-26T05:04:51.536718+0000","last_undegraded":"2024-03-26T05:04:51.536718+0000","last_fullsized":"2024-03-26T05:04:51.536718+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:18.640616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:18.640616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:32.003176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'21","reported_seq":230,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.519438+0000","last_change":"2024-03-26T05:03:18.506773+0000","last_active":"2024-03-26T05:04:51.519438+0000","last_peered":"2024-03-26T05:04:51.519438+0000","last_clean":"2024-03-26T05:04:51.519438+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:04:51.519438+0000","last_undegraded":"2024-03-26T05:04:51.519438+0000","last_fullsized":"2024-03-26T05:04:51.519438+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:03:18.506680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:18.506680+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.103980+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":189,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536344+0000","last_change":"2024-03-26T05:04:01.704040+0000","last_active":"2024-03-26T05:04:51.536344+0000","last_peered":"2024-03-26T05:04:51.536344+0000","last_clean":"2024-03-26T05:04:51.536344+0000","last_became_active":"2024-03-26T05:00:28.982276+0000","last_became_peered":"2024-03-26T05:00:28.982276+0000","last_unstale":"2024-03-26T05:04:51.536344+0000","last_undegraded":"2024-03-26T05:04:51.536344+0000","last_fullsized":"2024-03-26T05:04:51.536344+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:01.703895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:01.703895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:09.991523+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530179+0000","last_change":"2024-03-26T05:03:09.812833+0000","last_active":"2024-03-26T05:04:51.530179+0000","last_peered":"2024-03-26T05:04:51.530179+0000","last_clean":"2024-03-26T05:04:51.530179+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:04:51.530179+0000","last_undegraded":"2024-03-26T05:04:51.530179+0000","last_fullsized":"2024-03-26T05:04:51.530179+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:09.812687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.812687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":201,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536699+0000","last_change":"2024-03-26T05:04:10.613465+0000","last_active":"2024-03-26T05:04:51.536699+0000","last_peered":"2024-03-26T05:04:51.536699+0000","last_clean":"2024-03-26T05:04:51.536699+0000","last_became_active":"2024-03-26T05:00:22.766675+0000","last_became_peered":"2024-03-26T05:00:22.766675+0000","last_unstale":"2024-03-26T05:04:51.536699+0000","last_undegraded":"2024-03-26T05:04:51.536699+0000","last_fullsized":"2024-03-26T05:04:51.536699+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:10.613346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:10.613346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:19.750451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'11","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.519138+0000","last_change":"2024-03-26T05:03:10.376790+0000","last_active":"2024-03-26T05:04:51.519138+0000","last_peered":"2024-03-26T05:04:51.519138+0000","last_clean":"2024-03-26T05:04:51.519138+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:04:51.519138+0000","last_undegraded":"2024-03-26T05:04:51.519138+0000","last_fullsized":"2024-03-26T05:04:51.519138+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-26T05:03:10.376657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:10.376657+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:54.146855+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:56.073855+0000","last_change":"2024-03-26T05:04:56.073891+0000","last_active":"2024-03-26T05:04:56.073855+0000","last_peered":"2024-03-26T05:04:56.073855+0000","last_clean":"2024-03-26T05:04:56.073855+0000","last_became_active":"2024-03-26T05:00:29.461032+0000","last_became_peered":"2024-03-26T05:00:29.461032+0000","last_unstale":"2024-03-26T05:04:56.073855+0000","last_undegraded":"2024-03-26T05:04:56.073855+0000","last_fullsized":"2024-03-26T05:04:56.073855+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:56.073763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:04:56.073763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:25.502690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.537916+0000","last_change":"2024-03-26T05:03:15.886716+0000","last_active":"2024-03-26T05:04:51.537916+0000","last_peered":"2024-03-26T05:04:51.537916+0000","last_clean":"2024-03-26T05:04:51.537916+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:04:51.537916+0000","last_undegraded":"2024-03-26T05:04:51.537916+0000","last_fullsized":"2024-03-26T05:04:51.537916+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:15.885586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:15.885586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529496+0000","last_change":"2024-03-26T05:03:09.914583+0000","last_active":"2024-03-26T05:04:51.529496+0000","last_peered":"2024-03-26T05:04:51.529496+0000","last_clean":"2024-03-26T05:04:51.529496+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:04:51.529496+0000","last_undegraded":"2024-03-26T05:04:51.529496+0000","last_fullsized":"2024-03-26T05:04:51.529496+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:09.914470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:09.914470+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.914470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.970072+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":214,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.538635+0000","last_change":"2024-03-26T05:02:58.971442+0000","last_active":"2024-03-26T05:04:51.538635+0000","last_peered":"2024-03-26T05:04:51.538635+0000","last_clean":"2024-03-26T05:04:51.538635+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:04:51.538635+0000","last_undegraded":"2024-03-26T05:04:51.538635+0000","last_fullsized":"2024-03-26T05:04:51.538635+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:02:58.965020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:58.965020+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536141+0000","last_change":"2024-03-26T05:03:05.268194+0000","last_active":"2024-03-26T05:04:51.536141+0000","last_peered":"2024-03-26T05:04:51.536141+0000","last_clean":"2024-03-26T05:04:51.536141+0000","last_became_active":"2024-03-26T05:00:29.463238+0000","last_became_peered":"2024-03-26T05:00:29.463238+0000","last_unstale":"2024-03-26T05:04:51.536141+0000","last_undegraded":"2024-03-26T05:04:51.536141+0000","last_fullsized":"2024-03-26T05:04:51.536141+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:05.268043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:05.268043+0000","last_clean_scrub_stamp":"2024-03-26T05:03:05.268043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:53.748763+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":186,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.520286+0000","last_change":"2024-03-26T05:04:21.167015+0000","last_active":"2024-03-26T05:04:51.520286+0000","last_peered":"2024-03-26T05:04:51.520286+0000","last_clean":"2024-03-26T05:04:51.520286+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:04:51.520286+0000","last_undegraded":"2024-03-26T05:04:51.520286+0000","last_fullsized":"2024-03-26T05:04:51.520286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:21.166893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.166893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:45.803222+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":202,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.519539+0000","last_change":"2024-03-26T05:04:01.071006+0000","last_active":"2024-03-26T05:04:51.519539+0000","last_peered":"2024-03-26T05:04:51.519539+0000","last_clean":"2024-03-26T05:04:51.519539+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:04:51.519539+0000","last_undegraded":"2024-03-26T05:04:51.519539+0000","last_fullsized":"2024-03-26T05:04:51.519539+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:01.070878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:04:01.070878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:14.920867+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'20","reported_seq":226,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536535+0000","last_change":"2024-03-26T05:04:07.528781+0000","last_active":"2024-03-26T05:04:51.536535+0000","last_peered":"2024-03-26T05:04:51.536535+0000","last_clean":"2024-03-26T05:04:51.536535+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:04:51.536535+0000","last_undegraded":"2024-03-26T05:04:51.536535+0000","last_fullsized":"2024-03-26T05:04:51.536535+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:04:07.528619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:07.528619+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:36.984300+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":181,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530368+0000","last_change":"2024-03-26T05:03:13.901836+0000","last_active":"2024-03-26T05:04:51.530368+0000","last_peered":"2024-03-26T05:04:51.530368+0000","last_clean":"2024-03-26T05:04:51.530368+0000","last_became_active":"2024-03-26T05:00:28.981505+0000","last_became_peered":"2024-03-26T05:00:28.981505+0000","last_unstale":"2024-03-26T05:04:51.530368+0000","last_undegraded":"2024-03-26T05:04:51.530368+0000","last_fullsized":"2024-03-26T05:04:51.530368+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:03:13.901714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:13.901714+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":260,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:53.442416+0000","last_change":"2024-03-26T05:04:18.963951+0000","last_active":"2024-03-26T05:04:53.442416+0000","last_peered":"2024-03-26T05:04:53.442416+0000","last_clean":"2024-03-26T05:04:53.442416+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:04:53.442416+0000","last_undegraded":"2024-03-26T05:04:53.442416+0000","last_fullsized":"2024-03-26T05:04:53.442416+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:18.963855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:18.963855+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:20.821553+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.538646+0000","last_change":"2024-03-26T05:04:07.578139+0000","last_active":"2024-03-26T05:04:51.538646+0000","last_peered":"2024-03-26T05:04:51.538646+0000","last_clean":"2024-03-26T05:04:51.538646+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:04:51.538646+0000","last_undegraded":"2024-03-26T05:04:51.538646+0000","last_fullsized":"2024-03-26T05:04:51.538646+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:04:07.578016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:07.578016+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:24.792226+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'16","reported_seq":218,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530673+0000","last_change":"2024-03-26T05:03:17.868736+0000","last_active":"2024-03-26T05:04:51.530673+0000","last_peered":"2024-03-26T05:04:51.530673+0000","last_clean":"2024-03-26T05:04:51.530673+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:04:51.530673+0000","last_undegraded":"2024-03-26T05:04:51.530673+0000","last_fullsized":"2024-03-26T05:04:51.530673+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:03:17.867315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:17.867315+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.970183+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":189,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:55.814412+0000","last_change":"2024-03-26T05:04:55.814469+0000","last_active":"2024-03-26T05:04:55.814412+0000","last_peered":"2024-03-26T05:04:55.814412+0000","last_clean":"2024-03-26T05:04:55.814412+0000","last_became_active":"2024-03-26T05:00:28.981049+0000","last_became_peered":"2024-03-26T05:00:28.981049+0000","last_unstale":"2024-03-26T05:04:55.814412+0000","last_undegraded":"2024-03-26T05:04:55.814412+0000","last_fullsized":"2024-03-26T05:04:55.814412+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:55.814330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:55.814330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:11.096973+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.535475+0000","last_change":"2024-03-26T05:03:59.829911+0000","last_active":"2024-03-26T05:04:51.535475+0000","last_peered":"2024-03-26T05:04:51.535475+0000","last_clean":"2024-03-26T05:04:51.535475+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:04:51.535475+0000","last_undegraded":"2024-03-26T05:04:51.535475+0000","last_fullsized":"2024-03-26T05:04:51.535475+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:59.829595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:59.829595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:14.748766+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.535049+0000","last_change":"2024-03-26T05:03:17.225355+0000","last_active":"2024-03-26T05:04:51.535049+0000","last_peered":"2024-03-26T05:04:51.535049+0000","last_clean":"2024-03-26T05:04:51.535049+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:04:51.535049+0000","last_undegraded":"2024-03-26T05:04:51.535049+0000","last_fullsized":"2024-03-26T05:04:51.535049+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:17.225243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:17.225243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:53.748737+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'17","reported_seq":248,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:55.688426+0000","last_change":"2024-03-26T05:04:55.688482+0000","last_active":"2024-03-26T05:04:55.688426+0000","last_peered":"2024-03-26T05:04:55.688426+0000","last_clean":"2024-03-26T05:04:55.688426+0000","last_became_active":"2024-03-26T05:00:49.643308+0000","last_became_peered":"2024-03-26T05:00:49.643308+0000","last_unstale":"2024-03-26T05:04:55.688426+0000","last_undegraded":"2024-03-26T05:04:55.688426+0000","last_fullsized":"2024-03-26T05:04:55.688426+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:04:55.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:55.688341+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:04.276553+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529493+0000","last_change":"2024-03-26T05:04:27.007041+0000","last_active":"2024-03-26T05:04:51.529493+0000","last_peered":"2024-03-26T05:04:51.529493+0000","last_clean":"2024-03-26T05:04:51.529493+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:04:51.529493+0000","last_undegraded":"2024-03-26T05:04:51.529493+0000","last_fullsized":"2024-03-26T05:04:51.529493+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:27.006904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:27.006904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:56.433967+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.528946+0000","last_change":"2024-03-26T05:03:04.024222+0000","last_active":"2024-03-26T05:04:51.528946+0000","last_peered":"2024-03-26T05:04:51.528946+0000","last_clean":"2024-03-26T05:04:51.528946+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:04:51.528946+0000","last_undegraded":"2024-03-26T05:04:51.528946+0000","last_fullsized":"2024-03-26T05:04:51.528946+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:04.024139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:04.024139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:55.659053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536594+0000","last_change":"2024-03-26T05:04:14.715840+0000","last_active":"2024-03-26T05:04:51.536594+0000","last_peered":"2024-03-26T05:04:51.536594+0000","last_clean":"2024-03-26T05:04:51.536594+0000","last_became_active":"2024-03-26T05:00:22.774783+0000","last_became_peered":"2024-03-26T05:00:22.774783+0000","last_unstale":"2024-03-26T05:04:51.536594+0000","last_undegraded":"2024-03-26T05:04:51.536594+0000","last_fullsized":"2024-03-26T05:04:51.536594+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:14.715721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:14.715721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:39.237899+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"100'6","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.531634+0000","last_change":"2024-03-26T05:04:01.370962+0000","last_active":"2024-03-26T05:04:51.531634+0000","last_peered":"2024-03-26T05:04:51.531634+0000","last_clean":"2024-03-26T05:04:51.531634+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:04:51.531634+0000","last_undegraded":"2024-03-26T05:04:51.531634+0000","last_fullsized":"2024-03-26T05:04:51.531634+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'6","last_scrub_stamp":"2024-03-26T05:04:01.370804+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:04:01.370804+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:02.996344+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":181,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.532188+0000","last_change":"2024-03-26T05:04:10.585914+0000","last_active":"2024-03-26T05:04:51.532188+0000","last_peered":"2024-03-26T05:04:51.532188+0000","last_clean":"2024-03-26T05:04:51.532188+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:04:51.532188+0000","last_undegraded":"2024-03-26T05:04:51.532188+0000","last_fullsized":"2024-03-26T05:04:51.532188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:10.585750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:10.585750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:23.988184+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529899+0000","last_change":"2024-03-26T05:03:06.806992+0000","last_active":"2024-03-26T05:04:51.529899+0000","last_peered":"2024-03-26T05:04:51.529899+0000","last_clean":"2024-03-26T05:04:51.529899+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:04:51.529899+0000","last_undegraded":"2024-03-26T05:04:51.529899+0000","last_fullsized":"2024-03-26T05:04:51.529899+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:03:06.806863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":109,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529505+0000","last_change":"2024-03-26T05:04:20.919136+0000","last_active":"2024-03-26T05:04:51.529505+0000","last_peered":"2024-03-26T05:04:51.529505+0000","last_clean":"2024-03-26T05:04:51.529505+0000","last_became_active":"2024-03-26T05:03:05.521331+0000","last_became_peered":"2024-03-26T05:03:05.521331+0000","last_unstale":"2024-03-26T05:04:51.529505+0000","last_undegraded":"2024-03-26T05:04:51.529505+0000","last_fullsized":"2024-03-26T05:04:51.529505+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:20.919020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:20.919020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:50.626775+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"75'17","reported_seq":220,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.534094+0000","last_change":"2024-03-26T05:04:09.559560+0000","last_active":"2024-03-26T05:04:51.534094+0000","last_peered":"2024-03-26T05:04:51.534094+0000","last_clean":"2024-03-26T05:04:51.534094+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:04:51.534094+0000","last_undegraded":"2024-03-26T05:04:51.534094+0000","last_fullsized":"2024-03-26T05:04:51.534094+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:04:09.559412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:09.559412+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:38.053118+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.528526+0000","last_change":"2024-03-26T05:03:11.864162+0000","last_active":"2024-03-26T05:04:51.528526+0000","last_peered":"2024-03-26T05:04:51.528526+0000","last_clean":"2024-03-26T05:04:51.528526+0000","last_became_active":"2024-03-26T05:00:28.985158+0000","last_became_peered":"2024-03-26T05:00:28.985158+0000","last_unstale":"2024-03-26T05:04:51.528526+0000","last_undegraded":"2024-03-26T05:04:51.528526+0000","last_fullsized":"2024-03-26T05:04:51.528526+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:11.864048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:11.864048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.970022+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.531844+0000","last_change":"2024-03-26T05:02:58.192214+0000","last_active":"2024-03-26T05:04:51.531844+0000","last_peered":"2024-03-26T05:04:51.531844+0000","last_clean":"2024-03-26T05:04:51.531844+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:04:51.531844+0000","last_undegraded":"2024-03-26T05:04:51.531844+0000","last_fullsized":"2024-03-26T05:04:51.531844+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:58.191939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:02:58.191939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:53.748602+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530561+0000","last_change":"2024-03-26T05:02:07.036740+0000","last_active":"2024-03-26T05:04:51.530561+0000","last_peered":"2024-03-26T05:04:51.530561+0000","last_clean":"2024-03-26T05:04:51.530561+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:04:51.530561+0000","last_undegraded":"2024-03-26T05:04:51.530561+0000","last_fullsized":"2024-03-26T05:04:51.530561+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:07.036606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:07.036606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":205,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.537052+0000","last_change":"2024-03-26T05:03:23.778722+0000","last_active":"2024-03-26T05:04:51.537052+0000","last_peered":"2024-03-26T05:04:51.537052+0000","last_clean":"2024-03-26T05:04:51.537052+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:04:51.537052+0000","last_undegraded":"2024-03-26T05:04:51.537052+0000","last_fullsized":"2024-03-26T05:04:51.537052+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-26T05:03:23.778591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:23.778591+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":181,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:58.834220+0000","last_change":"2024-03-26T05:04:58.834254+0000","last_active":"2024-03-26T05:04:58.834220+0000","last_peered":"2024-03-26T05:04:58.834220+0000","last_clean":"2024-03-26T05:04:58.834220+0000","last_became_active":"2024-03-26T05:00:28.983417+0000","last_became_peered":"2024-03-26T05:00:28.983417+0000","last_unstale":"2024-03-26T05:04:58.834220+0000","last_undegraded":"2024-03-26T05:04:58.834220+0000","last_fullsized":"2024-03-26T05:04:58.834220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:58.834141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.834141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:12.740874+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536873+0000","last_change":"2024-03-26T05:04:21.579387+0000","last_active":"2024-03-26T05:04:51.536873+0000","last_peered":"2024-03-26T05:04:51.536873+0000","last_clean":"2024-03-26T05:04:51.536873+0000","last_became_active":"2024-03-26T05:00:26.954653+0000","last_became_peered":"2024-03-26T05:00:26.954653+0000","last_unstale":"2024-03-26T05:04:51.536873+0000","last_undegraded":"2024-03-26T05:04:51.536873+0000","last_fullsized":"2024-03-26T05:04:51.536873+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:21.579155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.579155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:39.504032+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":186,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.520150+0000","last_change":"2024-03-26T05:03:03.408937+0000","last_active":"2024-03-26T05:04:51.520150+0000","last_peered":"2024-03-26T05:04:51.520150+0000","last_clean":"2024-03-26T05:04:51.520150+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:04:51.520150+0000","last_undegraded":"2024-03-26T05:04:51.520150+0000","last_fullsized":"2024-03-26T05:04:51.520150+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:03.408801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:03.408801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:54.146837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"95'16","reported_seq":214,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.520002+0000","last_change":"2024-03-26T05:03:22.426319+0000","last_active":"2024-03-26T05:04:51.520002+0000","last_peered":"2024-03-26T05:04:51.520002+0000","last_clean":"2024-03-26T05:04:51.520002+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:04:51.520002+0000","last_undegraded":"2024-03-26T05:04:51.520002+0000","last_fullsized":"2024-03-26T05:04:51.520002+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:03:22.426170+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:03:22.426170+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:54.146822+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530326+0000","last_change":"2024-03-26T05:04:10.806723+0000","last_active":"2024-03-26T05:04:51.530326+0000","last_peered":"2024-03-26T05:04:51.530326+0000","last_clean":"2024-03-26T05:04:51.530326+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:04:51.530326+0000","last_undegraded":"2024-03-26T05:04:51.530326+0000","last_fullsized":"2024-03-26T05:04:51.530326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:10.806588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:10.806588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:36.340744+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":249,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:53.442711+0000","last_change":"2024-03-26T05:04:05.662804+0000","last_active":"2024-03-26T05:04:53.442711+0000","last_peered":"2024-03-26T05:04:53.442711+0000","last_clean":"2024-03-26T05:04:53.442711+0000","last_became_active":"2024-03-26T05:00:26.955836+0000","last_became_peered":"2024-03-26T05:00:26.955836+0000","last_unstale":"2024-03-26T05:04:53.442711+0000","last_undegraded":"2024-03-26T05:04:53.442711+0000","last_fullsized":"2024-03-26T05:04:53.442711+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:05.662699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:05.662699+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:21.460708+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529743+0000","last_change":"2024-03-26T05:02:55.996225+0000","last_active":"2024-03-26T05:04:51.529743+0000","last_peered":"2024-03-26T05:04:51.529743+0000","last_clean":"2024-03-26T05:04:51.529743+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:04:51.529743+0000","last_undegraded":"2024-03-26T05:04:51.529743+0000","last_fullsized":"2024-03-26T05:04:51.529743+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:55.996111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:55.996111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'22","reported_seq":228,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.531867+0000","last_change":"2024-03-26T05:03:10.161219+0000","last_active":"2024-03-26T05:04:51.531867+0000","last_peered":"2024-03-26T05:04:51.531867+0000","last_clean":"2024-03-26T05:04:51.531867+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:04:51.531867+0000","last_undegraded":"2024-03-26T05:04:51.531867+0000","last_fullsized":"2024-03-26T05:04:51.531867+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:03:10.161091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:10.161091+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:53.748655+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":174,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.521032+0000","last_change":"2024-03-26T05:03:14.456132+0000","last_active":"2024-03-26T05:04:51.521032+0000","last_peered":"2024-03-26T05:04:51.521032+0000","last_clean":"2024-03-26T05:04:51.521032+0000","last_became_active":"2024-03-26T05:00:28.979477+0000","last_became_peered":"2024-03-26T05:00:28.979477+0000","last_unstale":"2024-03-26T05:04:51.521032+0000","last_undegraded":"2024-03-26T05:04:51.521032+0000","last_fullsized":"2024-03-26T05:04:51.521032+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:14.456016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:14.456016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.103885+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"120'9","reported_seq":253,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:53.442583+0000","last_change":"2024-03-26T05:03:20.187175+0000","last_active":"2024-03-26T05:04:53.442583+0000","last_peered":"2024-03-26T05:04:53.442583+0000","last_clean":"2024-03-26T05:04:53.442583+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:04:53.442583+0000","last_undegraded":"2024-03-26T05:04:53.442583+0000","last_fullsized":"2024-03-26T05:04:53.442583+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'9","last_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:03:20.186818+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529013+0000","last_change":"2024-03-26T05:03:21.927104+0000","last_active":"2024-03-26T05:04:51.529013+0000","last_peered":"2024-03-26T05:04:51.529013+0000","last_clean":"2024-03-26T05:04:51.529013+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:04:51.529013+0000","last_undegraded":"2024-03-26T05:04:51.529013+0000","last_fullsized":"2024-03-26T05:04:51.529013+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:21.926965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:21.926965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.970097+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"98'20","reported_seq":230,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.537085+0000","last_change":"2024-03-26T05:04:12.659452+0000","last_active":"2024-03-26T05:04:51.537085+0000","last_peered":"2024-03-26T05:04:51.537085+0000","last_clean":"2024-03-26T05:04:51.537085+0000","last_became_active":"2024-03-26T05:00:25.648284+0000","last_became_peered":"2024-03-26T05:00:25.648284+0000","last_unstale":"2024-03-26T05:04:51.537085+0000","last_undegraded":"2024-03-26T05:04:51.537085+0000","last_fullsized":"2024-03-26T05:04:51.537085+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'20","last_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:04:12.659322+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:26.101052+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":189,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.538955+0000","last_change":"2024-03-26T05:04:13.174305+0000","last_active":"2024-03-26T05:04:51.538955+0000","last_peered":"2024-03-26T05:04:51.538955+0000","last_clean":"2024-03-26T05:04:51.538955+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:04:51.538955+0000","last_undegraded":"2024-03-26T05:04:51.538955+0000","last_fullsized":"2024-03-26T05:04:51.538955+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:13.174161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:13.174161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:29.585590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.846206+0000","last_change":"2024-03-26T05:04:59.846243+0000","last_active":"2024-03-26T05:04:59.846206+0000","last_peered":"2024-03-26T05:04:59.846206+0000","last_clean":"2024-03-26T05:04:59.846206+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:04:59.846206+0000","last_undegraded":"2024-03-26T05:04:59.846206+0000","last_fullsized":"2024-03-26T05:04:59.846206+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:59.846123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:04:59.846123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:00.016038+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529606+0000","last_change":"2024-03-26T05:04:15.786624+0000","last_active":"2024-03-26T05:04:51.529606+0000","last_peered":"2024-03-26T05:04:51.529606+0000","last_clean":"2024-03-26T05:04:51.529606+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:04:51.529606+0000","last_undegraded":"2024-03-26T05:04:51.529606+0000","last_fullsized":"2024-03-26T05:04:51.529606+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:15.786545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:15.786545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:45.427035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'21","reported_seq":230,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.535315+0000","last_change":"2024-03-26T05:03:06.871854+0000","last_active":"2024-03-26T05:04:51.535315+0000","last_peered":"2024-03-26T05:04:51.535315+0000","last_clean":"2024-03-26T05:04:51.535315+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:04:51.535315+0000","last_undegraded":"2024-03-26T05:04:51.535315+0000","last_fullsized":"2024-03-26T05:04:51.535315+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:03:06.871708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:06.871708+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.970154+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530364+0000","last_change":"2024-03-26T05:04:05.932455+0000","last_active":"2024-03-26T05:04:51.530364+0000","last_peered":"2024-03-26T05:04:51.530364+0000","last_clean":"2024-03-26T05:04:51.530364+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:04:51.530364+0000","last_undegraded":"2024-03-26T05:04:51.530364+0000","last_fullsized":"2024-03-26T05:04:51.530364+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:05.932270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:05.932270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:15.168921+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536880+0000","last_change":"2024-03-26T05:03:12.953956+0000","last_active":"2024-03-26T05:04:51.536880+0000","last_peered":"2024-03-26T05:04:51.536880+0000","last_clean":"2024-03-26T05:04:51.536880+0000","last_became_active":"2024-03-26T05:00:26.953416+0000","last_became_peered":"2024-03-26T05:00:26.953416+0000","last_unstale":"2024-03-26T05:04:51.536880+0000","last_undegraded":"2024-03-26T05:04:51.536880+0000","last_fullsized":"2024-03-26T05:04:51.536880+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:12.953811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:12.953811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:55.713321+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":198,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.528600+0000","last_change":"2024-03-26T05:03:09.078444+0000","last_active":"2024-03-26T05:04:51.528600+0000","last_peered":"2024-03-26T05:04:51.528600+0000","last_clean":"2024-03-26T05:04:51.528600+0000","last_became_active":"2024-03-26T05:00:22.771080+0000","last_became_peered":"2024-03-26T05:00:22.771080+0000","last_unstale":"2024-03-26T05:04:51.528600+0000","last_undegraded":"2024-03-26T05:04:51.528600+0000","last_fullsized":"2024-03-26T05:04:51.528600+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:03:09.078282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.078282+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:55.659107+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'12","reported_seq":210,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536379+0000","last_change":"2024-03-26T05:03:18.221430+0000","last_active":"2024-03-26T05:04:51.536379+0000","last_peered":"2024-03-26T05:04:51.536379+0000","last_clean":"2024-03-26T05:04:51.536379+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:04:51.536379+0000","last_undegraded":"2024-03-26T05:04:51.536379+0000","last_fullsized":"2024-03-26T05:04:51.536379+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:03:18.221280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:18.221280+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:53.748904+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"151'10620","reported_seq":10804,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.861376+0000","last_change":"2024-03-26T05:04:21.563737+0000","last_active":"2024-03-26T05:04:59.861376+0000","last_peered":"2024-03-26T05:04:59.861376+0000","last_clean":"2024-03-26T05:04:59.861376+0000","last_became_active":"2024-03-26T05:02:24.354658+0000","last_became_peered":"2024-03-26T05:02:24.354658+0000","last_unstale":"2024-03-26T05:04:59.861376+0000","last_undegraded":"2024-03-26T05:04:59.861376+0000","last_fullsized":"2024-03-26T05:04:59.861376+0000","mapping_epoch":103,"log_start":"127'5600","ondisk_log_start":"127'5600","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'8267","last_scrub_stamp":"2024-03-26T05:04:21.563226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.563226+0000","objects_scrubbed":6873,"log_size":5020,"log_dups_size":0,"ondisk_log_size":5020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:33.838683+0000","scrub_duration":2071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292552704,"num_objects":4464,"num_object_clones":0,"num_object_copies":13392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10620,"num_write_kb":482688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.538149+0000","last_change":"2024-03-26T05:02:57.960123+0000","last_active":"2024-03-26T05:04:51.538149+0000","last_peered":"2024-03-26T05:04:51.538149+0000","last_clean":"2024-03-26T05:04:51.538149+0000","last_became_active":"2024-03-26T05:00:29.461059+0000","last_became_peered":"2024-03-26T05:00:29.461059+0000","last_unstale":"2024-03-26T05:04:51.538149+0000","last_undegraded":"2024-03-26T05:04:51.538149+0000","last_fullsized":"2024-03-26T05:04:51.538149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:02:57.960005+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:02:57.960005+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:55.713224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'3","reported_seq":211,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530776+0000","last_change":"2024-03-26T05:02:56.964577+0000","last_active":"2024-03-26T05:04:51.530776+0000","last_peered":"2024-03-26T05:04:51.530776+0000","last_clean":"2024-03-26T05:04:51.530776+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:04:51.530776+0000","last_undegraded":"2024-03-26T05:04:51.530776+0000","last_fullsized":"2024-03-26T05:04:51.530776+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-03-26T05:02:56.964448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:02:56.964448+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'27","reported_seq":247,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536290+0000","last_change":"2024-03-26T05:04:13.682488+0000","last_active":"2024-03-26T05:04:51.536290+0000","last_peered":"2024-03-26T05:04:51.536290+0000","last_clean":"2024-03-26T05:04:51.536290+0000","last_became_active":"2024-03-26T05:00:25.651328+0000","last_became_peered":"2024-03-26T05:00:25.651328+0000","last_unstale":"2024-03-26T05:04:51.536290+0000","last_undegraded":"2024-03-26T05:04:51.536290+0000","last_fullsized":"2024-03-26T05:04:51.536290+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'27","last_scrub_stamp":"2024-03-26T05:04:13.682349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:13.682349+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:18.855246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":249,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:53.442744+0000","last_change":"2024-03-26T05:03:10.843641+0000","last_active":"2024-03-26T05:04:53.442744+0000","last_peered":"2024-03-26T05:04:53.442744+0000","last_clean":"2024-03-26T05:04:53.442744+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:04:53.442744+0000","last_undegraded":"2024-03-26T05:04:53.442744+0000","last_fullsized":"2024-03-26T05:04:53.442744+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:03:10.843497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:10.843497+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"151'10795","reported_seq":10981,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.975593+0000","last_change":"2024-03-26T05:04:16.685339+0000","last_active":"2024-03-26T05:04:59.975593+0000","last_peered":"2024-03-26T05:04:59.975593+0000","last_clean":"2024-03-26T05:04:59.975593+0000","last_became_active":"2024-03-26T05:02:24.352844+0000","last_became_peered":"2024-03-26T05:02:24.352844+0000","last_unstale":"2024-03-26T05:04:59.975593+0000","last_undegraded":"2024-03-26T05:04:59.975593+0000","last_fullsized":"2024-03-26T05:04:59.975593+0000","mapping_epoch":103,"log_start":"130'6100","ondisk_log_start":"130'6100","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'8019","last_scrub_stamp":"2024-03-26T05:04:16.685114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:16.685114+0000","objects_scrubbed":7208,"log_size":4695,"log_dups_size":0,"ondisk_log_size":4695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:28.742298+0000","scrub_duration":3134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284229632,"num_objects":4337,"num_object_clones":0,"num_object_copies":13011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10795,"num_write_kb":484224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.532074+0000","last_change":"2024-03-26T05:03:16.177929+0000","last_active":"2024-03-26T05:04:51.532074+0000","last_peered":"2024-03-26T05:04:51.532074+0000","last_clean":"2024-03-26T05:04:51.532074+0000","last_became_active":"2024-03-26T05:00:28.985917+0000","last_became_peered":"2024-03-26T05:00:28.985917+0000","last_unstale":"2024-03-26T05:04:51.532074+0000","last_undegraded":"2024-03-26T05:04:51.532074+0000","last_fullsized":"2024-03-26T05:04:51.532074+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:16.177807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:16.177807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530697+0000","last_change":"2024-03-26T05:03:17.962503+0000","last_active":"2024-03-26T05:04:51.530697+0000","last_peered":"2024-03-26T05:04:51.530697+0000","last_clean":"2024-03-26T05:04:51.530697+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:04:51.530697+0000","last_undegraded":"2024-03-26T05:04:51.530697+0000","last_fullsized":"2024-03-26T05:04:51.530697+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:17.962388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:17.962388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'24","reported_seq":230,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.519536+0000","last_change":"2024-03-26T05:02:59.435952+0000","last_active":"2024-03-26T05:04:51.519536+0000","last_peered":"2024-03-26T05:04:51.519536+0000","last_clean":"2024-03-26T05:04:51.519536+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:04:51.519536+0000","last_undegraded":"2024-03-26T05:04:51.519536+0000","last_fullsized":"2024-03-26T05:04:51.519536+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'24","last_scrub_stamp":"2024-03-26T05:02:59.435799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:59.435799+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:54.146753+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":186,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.538730+0000","last_change":"2024-03-26T05:03:07.025015+0000","last_active":"2024-03-26T05:04:51.538730+0000","last_peered":"2024-03-26T05:04:51.538730+0000","last_clean":"2024-03-26T05:04:51.538730+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:04:51.538730+0000","last_undegraded":"2024-03-26T05:04:51.538730+0000","last_fullsized":"2024-03-26T05:04:51.538730+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:07.022207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:07.022207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"151'10502","reported_seq":10681,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.995630+0000","last_change":"2024-03-26T05:04:27.315840+0000","last_active":"2024-03-26T05:04:59.995630+0000","last_peered":"2024-03-26T05:04:59.995630+0000","last_clean":"2024-03-26T05:04:59.995630+0000","last_became_active":"2024-03-26T05:02:24.355930+0000","last_became_peered":"2024-03-26T05:02:24.355930+0000","last_unstale":"2024-03-26T05:04:59.995630+0000","last_undegraded":"2024-03-26T05:04:59.995630+0000","last_fullsized":"2024-03-26T05:04:59.995630+0000","mapping_epoch":103,"log_start":"123'4900","ondisk_log_start":"123'4900","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8447","last_scrub_stamp":"2024-03-26T05:04:27.315394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:27.315394+0000","objects_scrubbed":6429,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:56.899987+0000","scrub_duration":2530,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282329088,"num_objects":4308,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10502,"num_write_kb":473920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":181,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529417+0000","last_change":"2024-03-26T05:03:23.071801+0000","last_active":"2024-03-26T05:04:51.529417+0000","last_peered":"2024-03-26T05:04:51.529417+0000","last_clean":"2024-03-26T05:04:51.529417+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:04:51.529417+0000","last_undegraded":"2024-03-26T05:04:51.529417+0000","last_fullsized":"2024-03-26T05:04:51.529417+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:23.071667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:23.071667+0000","last_clean_scrub_stamp":"2024-03-26T05:03:23.071667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.539269+0000","last_change":"2024-03-26T05:04:21.231029+0000","last_active":"2024-03-26T05:04:51.539269+0000","last_peered":"2024-03-26T05:04:51.539269+0000","last_clean":"2024-03-26T05:04:51.539269+0000","last_became_active":"2024-03-26T05:00:26.961730+0000","last_became_peered":"2024-03-26T05:00:26.961730+0000","last_unstale":"2024-03-26T05:04:51.539269+0000","last_undegraded":"2024-03-26T05:04:51.539269+0000","last_fullsized":"2024-03-26T05:04:51.539269+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:21.230901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.230901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:36.141812+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530848+0000","last_change":"2024-03-26T05:04:23.884517+0000","last_active":"2024-03-26T05:04:51.530848+0000","last_peered":"2024-03-26T05:04:51.530848+0000","last_clean":"2024-03-26T05:04:51.530848+0000","last_became_active":"2024-03-26T05:00:22.772590+0000","last_became_peered":"2024-03-26T05:00:22.772590+0000","last_unstale":"2024-03-26T05:04:51.530848+0000","last_undegraded":"2024-03-26T05:04:51.530848+0000","last_fullsized":"2024-03-26T05:04:51.530848+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:23.884406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:23.884406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:49.946181+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'25","reported_seq":232,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.528786+0000","last_change":"2024-03-26T05:03:19.037673+0000","last_active":"2024-03-26T05:04:51.528786+0000","last_peered":"2024-03-26T05:04:51.528786+0000","last_clean":"2024-03-26T05:04:51.528786+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:04:51.528786+0000","last_undegraded":"2024-03-26T05:04:51.528786+0000","last_fullsized":"2024-03-26T05:04:51.528786+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:03:19.037530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:19.037530+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:55.659127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"151'10682","reported_seq":10870,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.815314+0000","last_change":"2024-03-26T05:04:11.124139+0000","last_active":"2024-03-26T05:04:59.815314+0000","last_peered":"2024-03-26T05:04:59.815314+0000","last_clean":"2024-03-26T05:04:59.815314+0000","last_became_active":"2024-03-26T05:02:24.353523+0000","last_became_peered":"2024-03-26T05:02:24.353523+0000","last_unstale":"2024-03-26T05:04:59.815314+0000","last_undegraded":"2024-03-26T05:04:59.815314+0000","last_fullsized":"2024-03-26T05:04:59.815314+0000","mapping_epoch":103,"log_start":"117'3800","ondisk_log_start":"117'3800","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7680","last_scrub_stamp":"2024-03-26T05:04:11.123962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:11.123962+0000","objects_scrubbed":7433,"log_size":6882,"log_dups_size":0,"ondisk_log_size":6882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:32.441783+0000","scrub_duration":2854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285343744,"num_objects":4354,"num_object_clones":0,"num_object_copies":13062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10682,"num_write_kb":481152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529189+0000","last_change":"2024-03-26T05:04:00.814564+0000","last_active":"2024-03-26T05:04:51.529189+0000","last_peered":"2024-03-26T05:04:51.529189+0000","last_clean":"2024-03-26T05:04:51.529189+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:04:51.529189+0000","last_undegraded":"2024-03-26T05:04:51.529189+0000","last_fullsized":"2024-03-26T05:04:51.529189+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:00.814254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:00.814254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:21.361433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":186,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.542588+0000","last_change":"2024-03-26T05:03:20.891602+0000","last_active":"2024-03-26T05:04:51.542588+0000","last_peered":"2024-03-26T05:04:51.542588+0000","last_clean":"2024-03-26T05:04:51.542588+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:04:51.542588+0000","last_undegraded":"2024-03-26T05:04:51.542588+0000","last_fullsized":"2024-03-26T05:04:51.542588+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:20.891481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:20.891481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536624+0000","last_change":"2024-03-26T05:03:13.993194+0000","last_active":"2024-03-26T05:04:51.536624+0000","last_peered":"2024-03-26T05:04:51.536624+0000","last_clean":"2024-03-26T05:04:51.536624+0000","last_became_active":"2024-03-26T05:00:22.773770+0000","last_became_peered":"2024-03-26T05:00:22.773770+0000","last_unstale":"2024-03-26T05:04:51.536624+0000","last_undegraded":"2024-03-26T05:04:51.536624+0000","last_fullsized":"2024-03-26T05:04:51.536624+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:13.993045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:13.993045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:55.713239+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'15","reported_seq":219,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529620+0000","last_change":"2024-03-26T05:03:15.852751+0000","last_active":"2024-03-26T05:04:51.529620+0000","last_peered":"2024-03-26T05:04:51.529620+0000","last_clean":"2024-03-26T05:04:51.529620+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:04:51.529620+0000","last_undegraded":"2024-03-26T05:04:51.529620+0000","last_fullsized":"2024-03-26T05:04:51.529620+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:03:15.852614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:15.852614+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"151'10582","reported_seq":10770,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.756560+0000","last_change":"2024-03-26T05:04:07.239552+0000","last_active":"2024-03-26T05:04:59.756560+0000","last_peered":"2024-03-26T05:04:59.756560+0000","last_clean":"2024-03-26T05:04:59.756560+0000","last_became_active":"2024-03-26T05:02:24.353248+0000","last_became_peered":"2024-03-26T05:02:24.353248+0000","last_unstale":"2024-03-26T05:04:59.756560+0000","last_undegraded":"2024-03-26T05:04:59.756560+0000","last_fullsized":"2024-03-26T05:04:59.756560+0000","mapping_epoch":103,"log_start":"120'4400","ondisk_log_start":"120'4400","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7371","last_scrub_stamp":"2024-03-26T05:04:07.239265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:07.239265+0000","objects_scrubbed":7298,"log_size":6182,"log_dups_size":0,"ondisk_log_size":6182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:31.785702+0000","scrub_duration":2188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280100864,"num_objects":4274,"num_object_clones":0,"num_object_copies":12822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10582,"num_write_kb":475392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.537782+0000","last_change":"2024-03-26T05:04:11.571539+0000","last_active":"2024-03-26T05:04:51.537782+0000","last_peered":"2024-03-26T05:04:51.537782+0000","last_clean":"2024-03-26T05:04:51.537782+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:04:51.537782+0000","last_undegraded":"2024-03-26T05:04:51.537782+0000","last_fullsized":"2024-03-26T05:04:51.537782+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:11.571425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:11.571425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:33.966929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":213,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:58.935653+0000","last_change":"2024-03-26T05:04:58.935696+0000","last_active":"2024-03-26T05:04:58.935653+0000","last_peered":"2024-03-26T05:04:58.935653+0000","last_clean":"2024-03-26T05:04:58.935653+0000","last_became_active":"2024-03-26T05:00:21.760521+0000","last_became_peered":"2024-03-26T05:00:21.760521+0000","last_unstale":"2024-03-26T05:04:58.935653+0000","last_undegraded":"2024-03-26T05:04:58.935653+0000","last_fullsized":"2024-03-26T05:04:58.935653+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:04:58.935568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.935568+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:28.155396+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.519241+0000","last_change":"2024-03-26T05:03:11.352008+0000","last_active":"2024-03-26T05:04:51.519241+0000","last_peered":"2024-03-26T05:04:51.519241+0000","last_clean":"2024-03-26T05:04:51.519241+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:04:51.519241+0000","last_undegraded":"2024-03-26T05:04:51.519241+0000","last_fullsized":"2024-03-26T05:04:51.519241+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:11.351820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:11.351820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:54.146870+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'34","reported_seq":314,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:53.448052+0000","last_change":"2024-03-26T05:04:22.562805+0000","last_active":"2024-03-26T05:04:53.448052+0000","last_peered":"2024-03-26T05:04:53.448052+0000","last_clean":"2024-03-26T05:04:53.448052+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:04:53.448052+0000","last_undegraded":"2024-03-26T05:04:53.448052+0000","last_fullsized":"2024-03-26T05:04:53.448052+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'34","last_scrub_stamp":"2024-03-26T05:04:22.562552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:22.562552+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:46.860857+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:54.973382+0000","last_change":"2024-03-26T05:04:54.973421+0000","last_active":"2024-03-26T05:04:54.973382+0000","last_peered":"2024-03-26T05:04:54.973382+0000","last_clean":"2024-03-26T05:04:54.973382+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:04:54.973382+0000","last_undegraded":"2024-03-26T05:04:54.973382+0000","last_fullsized":"2024-03-26T05:04:54.973382+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:54.973306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:54.973306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:56.082970+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"151'10497","reported_seq":10678,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.979788+0000","last_change":"2024-03-26T05:04:24.005788+0000","last_active":"2024-03-26T05:04:59.979788+0000","last_peered":"2024-03-26T05:04:59.979788+0000","last_clean":"2024-03-26T05:04:59.979788+0000","last_became_active":"2024-03-26T05:02:24.356658+0000","last_became_peered":"2024-03-26T05:02:24.356658+0000","last_unstale":"2024-03-26T05:04:59.979788+0000","last_undegraded":"2024-03-26T05:04:59.979788+0000","last_fullsized":"2024-03-26T05:04:59.979788+0000","mapping_epoch":103,"log_start":"123'4900","ondisk_log_start":"123'4900","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'8251","last_scrub_stamp":"2024-03-26T05:04:24.005629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:24.005629+0000","objects_scrubbed":6565,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:27.436033+0000","scrub_duration":2213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278724608,"num_objects":4253,"num_object_clones":0,"num_object_copies":12759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10497,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":181,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530893+0000","last_change":"2024-03-26T05:02:59.958661+0000","last_active":"2024-03-26T05:04:51.530893+0000","last_peered":"2024-03-26T05:04:51.530893+0000","last_clean":"2024-03-26T05:04:51.530893+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:04:51.530893+0000","last_undegraded":"2024-03-26T05:04:51.530893+0000","last_fullsized":"2024-03-26T05:04:51.530893+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:59.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:59.958579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.538273+0000","last_change":"2024-03-26T05:04:17.462393+0000","last_active":"2024-03-26T05:04:51.538273+0000","last_peered":"2024-03-26T05:04:51.538273+0000","last_clean":"2024-03-26T05:04:51.538273+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:04:51.538273+0000","last_undegraded":"2024-03-26T05:04:51.538273+0000","last_fullsized":"2024-03-26T05:04:51.538273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:17.462281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:17.462281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:36.813984+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":211,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:57.410637+0000","last_change":"2024-03-26T05:04:57.410678+0000","last_active":"2024-03-26T05:04:57.410637+0000","last_peered":"2024-03-26T05:04:57.410637+0000","last_clean":"2024-03-26T05:04:57.410637+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:04:57.410637+0000","last_undegraded":"2024-03-26T05:04:57.410637+0000","last_fullsized":"2024-03-26T05:04:57.410637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:57.410553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:04:57.410553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:06.632271+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'19","reported_seq":229,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.537488+0000","last_change":"2024-03-26T05:04:11.649338+0000","last_active":"2024-03-26T05:04:51.537488+0000","last_peered":"2024-03-26T05:04:51.537488+0000","last_clean":"2024-03-26T05:04:51.537488+0000","last_became_active":"2024-03-26T05:00:24.937145+0000","last_became_peered":"2024-03-26T05:00:24.937145+0000","last_unstale":"2024-03-26T05:04:51.537488+0000","last_undegraded":"2024-03-26T05:04:51.537488+0000","last_fullsized":"2024-03-26T05:04:51.537488+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-26T05:04:11.649163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:11.649163+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:15.906395+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:52.294131+0000","last_change":"2024-03-26T05:04:52.294165+0000","last_active":"2024-03-26T05:04:52.294131+0000","last_peered":"2024-03-26T05:04:52.294131+0000","last_clean":"2024-03-26T05:04:52.294131+0000","last_became_active":"2024-03-26T05:00:26.957936+0000","last_became_peered":"2024-03-26T05:00:26.957936+0000","last_unstale":"2024-03-26T05:04:52.294131+0000","last_undegraded":"2024-03-26T05:04:52.294131+0000","last_fullsized":"2024-03-26T05:04:52.294131+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:52.294056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:52.294056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:55.374354+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"151'10448","reported_seq":10637,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:05:00.008775+0000","last_change":"2024-03-26T05:04:12.245145+0000","last_active":"2024-03-26T05:05:00.008775+0000","last_peered":"2024-03-26T05:05:00.008775+0000","last_clean":"2024-03-26T05:05:00.008775+0000","last_became_active":"2024-03-26T05:02:24.356129+0000","last_became_peered":"2024-03-26T05:02:24.356129+0000","last_unstale":"2024-03-26T05:05:00.008775+0000","last_undegraded":"2024-03-26T05:05:00.008775+0000","last_fullsized":"2024-03-26T05:05:00.008775+0000","mapping_epoch":103,"log_start":"123'4800","ondisk_log_start":"123'4800","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7587","last_scrub_stamp":"2024-03-26T05:04:12.244918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:12.244918+0000","objects_scrubbed":7304,"log_size":5648,"log_dups_size":0,"ondisk_log_size":5648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:29.947538+0000","scrub_duration":3431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284557312,"num_objects":4342,"num_object_clones":0,"num_object_copies":13026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536817+0000","last_change":"2024-03-26T05:03:20.949049+0000","last_active":"2024-03-26T05:04:51.536817+0000","last_peered":"2024-03-26T05:04:51.536817+0000","last_clean":"2024-03-26T05:04:51.536817+0000","last_became_active":"2024-03-26T05:00:28.979373+0000","last_became_peered":"2024-03-26T05:00:28.979373+0000","last_unstale":"2024-03-26T05:04:51.536817+0000","last_undegraded":"2024-03-26T05:04:51.536817+0000","last_fullsized":"2024-03-26T05:04:51.536817+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:20.948908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:20.948908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:55.713254+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":448,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529512+0000","last_change":"2024-03-26T05:04:17.821154+0000","last_active":"2024-03-26T05:04:51.529512+0000","last_peered":"2024-03-26T05:04:51.529512+0000","last_clean":"2024-03-26T05:04:51.529512+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:04:51.529512+0000","last_undegraded":"2024-03-26T05:04:51.529512+0000","last_fullsized":"2024-03-26T05:04:51.529512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:04:17.820870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:04:17.820870+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:31.192171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.537980+0000","last_change":"2024-03-26T05:03:04.289678+0000","last_active":"2024-03-26T05:04:51.537980+0000","last_peered":"2024-03-26T05:04:51.537980+0000","last_clean":"2024-03-26T05:04:51.537980+0000","last_became_active":"2024-03-26T05:00:22.774298+0000","last_became_peered":"2024-03-26T05:00:22.774298+0000","last_unstale":"2024-03-26T05:04:51.537980+0000","last_undegraded":"2024-03-26T05:04:51.537980+0000","last_fullsized":"2024-03-26T05:04:51.537980+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:04.289541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:04.289541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:53.748626+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'24","reported_seq":283,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:53.442504+0000","last_change":"2024-03-26T05:04:24.867946+0000","last_active":"2024-03-26T05:04:53.442504+0000","last_peered":"2024-03-26T05:04:53.442504+0000","last_clean":"2024-03-26T05:04:53.442504+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:04:53.442504+0000","last_undegraded":"2024-03-26T05:04:53.442504+0000","last_fullsized":"2024-03-26T05:04:53.442504+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'24","last_scrub_stamp":"2024-03-26T05:04:24.867838+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:04:24.867838+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:48.459336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530384+0000","last_change":"2024-03-26T05:04:02.971643+0000","last_active":"2024-03-26T05:04:51.530384+0000","last_peered":"2024-03-26T05:04:51.530384+0000","last_clean":"2024-03-26T05:04:51.530384+0000","last_became_active":"2024-03-26T05:00:26.961724+0000","last_became_peered":"2024-03-26T05:00:26.961724+0000","last_unstale":"2024-03-26T05:04:51.530384+0000","last_undegraded":"2024-03-26T05:04:51.530384+0000","last_fullsized":"2024-03-26T05:04:51.530384+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:02.971526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:02.971526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:16.073532+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"151'10503","reported_seq":10662,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.727433+0000","last_change":"2024-03-26T05:04:59.106118+0000","last_active":"2024-03-26T05:04:59.727433+0000","last_peered":"2024-03-26T05:04:59.727433+0000","last_clean":"2024-03-26T05:04:59.727433+0000","last_became_active":"2024-03-26T05:02:24.354784+0000","last_became_peered":"2024-03-26T05:02:24.354784+0000","last_unstale":"2024-03-26T05:04:59.727433+0000","last_undegraded":"2024-03-26T05:04:59.727433+0000","last_fullsized":"2024-03-26T05:04:59.727433+0000","mapping_epoch":103,"log_start":"125'5300","ondisk_log_start":"125'5300","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'10450","last_scrub_stamp":"2024-03-26T05:04:59.105579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:59.105579+0000","objects_scrubbed":4395,"log_size":5203,"log_dups_size":0,"ondisk_log_size":5203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:22.819768+0000","scrub_duration":1358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281346048,"num_objects":4293,"num_object_clones":0,"num_object_copies":12879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10503,"num_write_kb":473472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":190,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:53.760245+0000","last_change":"2024-03-26T05:04:53.760276+0000","last_active":"2024-03-26T05:04:53.760245+0000","last_peered":"2024-03-26T05:04:53.760245+0000","last_clean":"2024-03-26T05:04:53.760245+0000","last_became_active":"2024-03-26T05:00:28.983202+0000","last_became_peered":"2024-03-26T05:00:28.983202+0000","last_unstale":"2024-03-26T05:04:53.760245+0000","last_undegraded":"2024-03-26T05:04:53.760245+0000","last_fullsized":"2024-03-26T05:04:53.760245+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:53.760171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:53.760171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:07.693633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.994968+0000","last_change":"2024-03-26T05:04:59.995006+0000","last_active":"2024-03-26T05:04:59.994968+0000","last_peered":"2024-03-26T05:04:59.994968+0000","last_clean":"2024-03-26T05:04:59.994968+0000","last_became_active":"2024-03-26T05:00:22.773081+0000","last_became_peered":"2024-03-26T05:00:22.773081+0000","last_unstale":"2024-03-26T05:04:59.994968+0000","last_undegraded":"2024-03-26T05:04:59.994968+0000","last_fullsized":"2024-03-26T05:04:59.994968+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:59.994887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:59.994887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:06.597851+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":477,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:58.476195+0000","last_change":"2024-03-26T05:03:00.940387+0000","last_active":"2024-03-26T05:04:58.476195+0000","last_peered":"2024-03-26T05:04:58.476195+0000","last_clean":"2024-03-26T05:04:58.476195+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:04:58.476195+0000","last_undegraded":"2024-03-26T05:04:58.476195+0000","last_fullsized":"2024-03-26T05:04:58.476195+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:03:00.940299+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:03:00.940299+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":271,"num_read_kb":266,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":229,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:56.020221+0000","last_change":"2024-03-26T05:04:56.020273+0000","last_active":"2024-03-26T05:04:56.020221+0000","last_peered":"2024-03-26T05:04:56.020221+0000","last_clean":"2024-03-26T05:04:56.020221+0000","last_became_active":"2024-03-26T05:00:49.642276+0000","last_became_peered":"2024-03-26T05:00:49.642276+0000","last_unstale":"2024-03-26T05:04:56.020221+0000","last_undegraded":"2024-03-26T05:04:56.020221+0000","last_fullsized":"2024-03-26T05:04:56.020221+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:04:56.020141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:56.020141+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:08.790686+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.537048+0000","last_change":"2024-03-26T05:04:22.633678+0000","last_active":"2024-03-26T05:04:51.537048+0000","last_peered":"2024-03-26T05:04:51.537048+0000","last_clean":"2024-03-26T05:04:51.537048+0000","last_became_active":"2024-03-26T05:00:26.956677+0000","last_became_peered":"2024-03-26T05:00:26.956677+0000","last_unstale":"2024-03-26T05:04:51.537048+0000","last_undegraded":"2024-03-26T05:04:51.537048+0000","last_fullsized":"2024-03-26T05:04:51.537048+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:22.633581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:22.633581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:30.314139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"151'10575","reported_seq":10762,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.723214+0000","last_change":"2024-03-26T05:04:04.673613+0000","last_active":"2024-03-26T05:04:59.723214+0000","last_peered":"2024-03-26T05:04:59.723214+0000","last_clean":"2024-03-26T05:04:59.723214+0000","last_became_active":"2024-03-26T05:02:24.353979+0000","last_became_peered":"2024-03-26T05:02:24.353979+0000","last_unstale":"2024-03-26T05:04:59.723214+0000","last_undegraded":"2024-03-26T05:04:59.723214+0000","last_fullsized":"2024-03-26T05:04:59.723214+0000","mapping_epoch":103,"log_start":"121'4600","ondisk_log_start":"121'4600","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7253","last_scrub_stamp":"2024-03-26T05:04:04.673346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:04.673346+0000","objects_scrubbed":7208,"log_size":5975,"log_dups_size":0,"ondisk_log_size":5975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:08.953957+0000","scrub_duration":1959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281870336,"num_objects":4301,"num_object_clones":0,"num_object_copies":12903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10575,"num_write_kb":476032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.538492+0000","last_change":"2024-03-26T05:03:20.739630+0000","last_active":"2024-03-26T05:04:51.538492+0000","last_peered":"2024-03-26T05:04:51.538492+0000","last_clean":"2024-03-26T05:04:51.538492+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:04:51.538492+0000","last_undegraded":"2024-03-26T05:04:51.538492+0000","last_fullsized":"2024-03-26T05:04:51.538492+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:03:20.739242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.537615+0000","last_change":"2024-03-26T05:03:14.868503+0000","last_active":"2024-03-26T05:04:51.537615+0000","last_peered":"2024-03-26T05:04:51.537615+0000","last_clean":"2024-03-26T05:04:51.537615+0000","last_became_active":"2024-03-26T05:00:23.346665+0000","last_became_peered":"2024-03-26T05:00:23.346665+0000","last_unstale":"2024-03-26T05:04:51.537615+0000","last_undegraded":"2024-03-26T05:04:51.537615+0000","last_fullsized":"2024-03-26T05:04:51.537615+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:14.868378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:03:14.868378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'25","reported_seq":239,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529411+0000","last_change":"2024-03-26T05:03:16.818394+0000","last_active":"2024-03-26T05:04:51.529411+0000","last_peered":"2024-03-26T05:04:51.529411+0000","last_clean":"2024-03-26T05:04:51.529411+0000","last_became_active":"2024-03-26T05:00:25.648253+0000","last_became_peered":"2024-03-26T05:00:25.648253+0000","last_unstale":"2024-03-26T05:04:51.529411+0000","last_undegraded":"2024-03-26T05:04:51.529411+0000","last_fullsized":"2024-03-26T05:04:51.529411+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:03:16.818031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:16.818031+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.969995+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.528791+0000","last_change":"2024-03-26T05:04:04.071060+0000","last_active":"2024-03-26T05:04:51.528791+0000","last_peered":"2024-03-26T05:04:51.528791+0000","last_clean":"2024-03-26T05:04:51.528791+0000","last_became_active":"2024-03-26T05:00:26.958689+0000","last_became_peered":"2024-03-26T05:00:26.958689+0000","last_unstale":"2024-03-26T05:04:51.528791+0000","last_undegraded":"2024-03-26T05:04:51.528791+0000","last_fullsized":"2024-03-26T05:04:51.528791+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:04.070935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:04:04.070935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:18.040979+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"151'10631","reported_seq":10792,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.725548+0000","last_change":"2024-03-26T05:04:56.303089+0000","last_active":"2024-03-26T05:04:59.725548+0000","last_peered":"2024-03-26T05:04:59.725548+0000","last_clean":"2024-03-26T05:04:59.725548+0000","last_became_active":"2024-03-26T05:02:24.352730+0000","last_became_peered":"2024-03-26T05:02:24.352730+0000","last_unstale":"2024-03-26T05:04:59.725548+0000","last_undegraded":"2024-03-26T05:04:59.725548+0000","last_fullsized":"2024-03-26T05:04:59.725548+0000","mapping_epoch":103,"log_start":"128'5800","ondisk_log_start":"128'5800","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'10463","last_scrub_stamp":"2024-03-26T05:04:56.302865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:56.302865+0000","objects_scrubbed":4618,"log_size":4831,"log_dups_size":0,"ondisk_log_size":4831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:00.980986+0000","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288817152,"num_objects":4407,"num_object_clones":0,"num_object_copies":13221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10631,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.529696+0000","last_change":"2024-03-26T05:03:19.935218+0000","last_active":"2024-03-26T05:04:51.529696+0000","last_peered":"2024-03-26T05:04:51.529696+0000","last_clean":"2024-03-26T05:04:51.529696+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:04:51.529696+0000","last_undegraded":"2024-03-26T05:04:51.529696+0000","last_fullsized":"2024-03-26T05:04:51.529696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:19.935094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:19.935094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.970237+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536843+0000","last_change":"2024-03-26T05:03:16.983679+0000","last_active":"2024-03-26T05:04:51.536843+0000","last_peered":"2024-03-26T05:04:51.536843+0000","last_clean":"2024-03-26T05:04:51.536843+0000","last_became_active":"2024-03-26T05:00:23.346876+0000","last_became_peered":"2024-03-26T05:00:23.346876+0000","last_unstale":"2024-03-26T05:04:51.536843+0000","last_undegraded":"2024-03-26T05:04:51.536843+0000","last_fullsized":"2024-03-26T05:04:51.536843+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:16.983556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:16.983556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:55.713300+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'16","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.543016+0000","last_change":"2024-03-26T05:03:11.028000+0000","last_active":"2024-03-26T05:04:51.543016+0000","last_peered":"2024-03-26T05:04:51.543016+0000","last_clean":"2024-03-26T05:04:51.543016+0000","last_became_active":"2024-03-26T05:00:49.641444+0000","last_became_peered":"2024-03-26T05:00:49.641444+0000","last_unstale":"2024-03-26T05:04:51.543016+0000","last_undegraded":"2024-03-26T05:04:51.543016+0000","last_fullsized":"2024-03-26T05:04:51.543016+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:03:11.027857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:11.027857+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":252,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:53.443914+0000","last_change":"2024-03-26T05:03:23.981271+0000","last_active":"2024-03-26T05:04:53.443914+0000","last_peered":"2024-03-26T05:04:53.443914+0000","last_clean":"2024-03-26T05:04:53.443914+0000","last_became_active":"2024-03-26T05:00:26.961907+0000","last_became_peered":"2024-03-26T05:00:26.961907+0000","last_unstale":"2024-03-26T05:04:53.443914+0000","last_undegraded":"2024-03-26T05:04:53.443914+0000","last_fullsized":"2024-03-26T05:04:53.443914+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T05:03:23.981095+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:03:23.981095+0000","last_clean_scrub_stamp":"2024-03-26T05:03:23.981095+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"151'10704","reported_seq":10890,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.995571+0000","last_change":"2024-03-26T05:04:19.068828+0000","last_active":"2024-03-26T05:04:59.995571+0000","last_peered":"2024-03-26T05:04:59.995571+0000","last_clean":"2024-03-26T05:04:59.995571+0000","last_became_active":"2024-03-26T05:02:24.354286+0000","last_became_peered":"2024-03-26T05:02:24.354286+0000","last_unstale":"2024-03-26T05:04:59.995571+0000","last_undegraded":"2024-03-26T05:04:59.995571+0000","last_fullsized":"2024-03-26T05:04:59.995571+0000","mapping_epoch":103,"log_start":"130'6000","ondisk_log_start":"130'6000","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'8121","last_scrub_stamp":"2024-03-26T05:04:19.068650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:19.068650+0000","objects_scrubbed":7009,"log_size":4704,"log_dups_size":0,"ondisk_log_size":4704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:25.300640+0000","scrub_duration":3488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284295168,"num_objects":4338,"num_object_clones":0,"num_object_copies":13014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10704,"num_write_kb":481344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":181,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530077+0000","last_change":"2024-03-26T05:04:24.924461+0000","last_active":"2024-03-26T05:04:51.530077+0000","last_peered":"2024-03-26T05:04:51.530077+0000","last_clean":"2024-03-26T05:04:51.530077+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:04:51.530077+0000","last_undegraded":"2024-03-26T05:04:51.530077+0000","last_fullsized":"2024-03-26T05:04:51.530077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:24.924377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:24.924377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:30.780658+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.530520+0000","last_change":"2024-03-26T05:03:13.867887+0000","last_active":"2024-03-26T05:04:51.530520+0000","last_peered":"2024-03-26T05:04:51.530520+0000","last_clean":"2024-03-26T05:04:51.530520+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:04:51.530520+0000","last_undegraded":"2024-03-26T05:04:51.530520+0000","last_fullsized":"2024-03-26T05:04:51.530520+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:03:13.867756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:13.867756+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.969949+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'20","reported_seq":224,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.519425+0000","last_change":"2024-03-26T05:04:25.126002+0000","last_active":"2024-03-26T05:04:51.519425+0000","last_peered":"2024-03-26T05:04:51.519425+0000","last_clean":"2024-03-26T05:04:51.519425+0000","last_became_active":"2024-03-26T05:00:24.936315+0000","last_became_peered":"2024-03-26T05:00:24.936315+0000","last_unstale":"2024-03-26T05:04:51.519425+0000","last_undegraded":"2024-03-26T05:04:51.519425+0000","last_fullsized":"2024-03-26T05:04:51.519425+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:04:25.125864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:25.125864+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:33.277048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":250,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:53.448913+0000","last_change":"2024-03-26T05:04:26.670277+0000","last_active":"2024-03-26T05:04:53.448913+0000","last_peered":"2024-03-26T05:04:53.448913+0000","last_clean":"2024-03-26T05:04:53.448913+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:04:53.448913+0000","last_undegraded":"2024-03-26T05:04:53.448913+0000","last_fullsized":"2024-03-26T05:04:53.448913+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:26.669895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:26.669895+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:47.751228+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"151'10599","reported_seq":10760,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:05:00.010663+0000","last_change":"2024-03-26T05:04:58.534809+0000","last_active":"2024-03-26T05:05:00.010663+0000","last_peered":"2024-03-26T05:05:00.010663+0000","last_clean":"2024-03-26T05:05:00.010663+0000","last_became_active":"2024-03-26T05:02:24.352725+0000","last_became_peered":"2024-03-26T05:02:24.352725+0000","last_unstale":"2024-03-26T05:05:00.010663+0000","last_undegraded":"2024-03-26T05:05:00.010663+0000","last_fullsized":"2024-03-26T05:05:00.010663+0000","mapping_epoch":103,"log_start":"128'5900","ondisk_log_start":"128'5900","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'10511","last_scrub_stamp":"2024-03-26T05:04:58.534644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.534644+0000","objects_scrubbed":4512,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:22.836586+0000","scrub_duration":1678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287899648,"num_objects":4393,"num_object_clones":0,"num_object_copies":13179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10599,"num_write_kb":479744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.519255+0000","last_change":"2024-03-26T05:04:10.112906+0000","last_active":"2024-03-26T05:04:51.519255+0000","last_peered":"2024-03-26T05:04:51.519255+0000","last_clean":"2024-03-26T05:04:51.519255+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:04:51.519255+0000","last_undegraded":"2024-03-26T05:04:51.519255+0000","last_fullsized":"2024-03-26T05:04:51.519255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:10.112830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:10.112830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:12.840322+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:54.805745+0000","last_change":"2024-03-26T05:04:54.805782+0000","last_active":"2024-03-26T05:04:54.805745+0000","last_peered":"2024-03-26T05:04:54.805745+0000","last_clean":"2024-03-26T05:04:54.805745+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:04:54.805745+0000","last_undegraded":"2024-03-26T05:04:54.805745+0000","last_fullsized":"2024-03-26T05:04:54.805745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:54.805655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:54.805655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:16.316884+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"96'40","reported_seq":256,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.531856+0000","last_change":"2024-03-26T05:03:14.123866+0000","last_active":"2024-03-26T05:04:51.531856+0000","last_peered":"2024-03-26T05:04:51.531856+0000","last_clean":"2024-03-26T05:04:51.531856+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:04:51.531856+0000","last_undegraded":"2024-03-26T05:04:51.531856+0000","last_fullsized":"2024-03-26T05:04:51.531856+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:03:14.123706+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:53.748711+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":179,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.519185+0000","last_change":"2024-03-26T05:03:09.332660+0000","last_active":"2024-03-26T05:04:51.519185+0000","last_peered":"2024-03-26T05:04:51.519185+0000","last_clean":"2024-03-26T05:04:51.519185+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:04:51.519185+0000","last_undegraded":"2024-03-26T05:04:51.519185+0000","last_fullsized":"2024-03-26T05:04:51.519185+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:09.332533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.332533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:54.146886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"151'10523","reported_seq":10710,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.780009+0000","last_change":"2024-03-26T05:04:14.777321+0000","last_active":"2024-03-26T05:04:59.780009+0000","last_peered":"2024-03-26T05:04:59.780009+0000","last_clean":"2024-03-26T05:04:59.780009+0000","last_became_active":"2024-03-26T05:02:24.351780+0000","last_became_peered":"2024-03-26T05:02:24.351780+0000","last_unstale":"2024-03-26T05:04:59.780009+0000","last_undegraded":"2024-03-26T05:04:59.780009+0000","last_fullsized":"2024-03-26T05:04:59.780009+0000","mapping_epoch":103,"log_start":"120'4400","ondisk_log_start":"120'4400","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7828","last_scrub_stamp":"2024-03-26T05:04:14.777151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:14.777151+0000","objects_scrubbed":7129,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:37.260876+0000","scrub_duration":1692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287637504,"num_objects":4389,"num_object_clones":0,"num_object_copies":13167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":477184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":181,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.536695+0000","last_change":"2024-03-26T05:03:21.933021+0000","last_active":"2024-03-26T05:04:51.536695+0000","last_peered":"2024-03-26T05:04:51.536695+0000","last_clean":"2024-03-26T05:04:51.536695+0000","last_became_active":"2024-03-26T05:00:28.982825+0000","last_became_peered":"2024-03-26T05:00:28.982825+0000","last_unstale":"2024-03-26T05:04:51.536695+0000","last_undegraded":"2024-03-26T05:04:51.536695+0000","last_fullsized":"2024-03-26T05:04:51.536695+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:21.932883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:21.932883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:55.713270+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":251,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:58.443892+0000","last_change":"2024-03-26T05:04:09.071212+0000","last_active":"2024-03-26T05:04:58.443892+0000","last_peered":"2024-03-26T05:04:58.443892+0000","last_clean":"2024-03-26T05:04:58.443892+0000","last_became_active":"2024-03-26T05:00:26.956596+0000","last_became_peered":"2024-03-26T05:00:26.956596+0000","last_unstale":"2024-03-26T05:04:58.443892+0000","last_undegraded":"2024-03-26T05:04:58.443892+0000","last_fullsized":"2024-03-26T05:04:58.443892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:09.071057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:09.071057+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:17.591868+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.531745+0000","last_change":"2024-03-26T05:04:06.525151+0000","last_active":"2024-03-26T05:04:51.531745+0000","last_peered":"2024-03-26T05:04:51.531745+0000","last_clean":"2024-03-26T05:04:51.531745+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:04:51.531745+0000","last_undegraded":"2024-03-26T05:04:51.531745+0000","last_fullsized":"2024-03-26T05:04:51.531745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:06.525047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:06.525047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:21.294383+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":223,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.537743+0000","last_change":"2024-03-26T05:04:24.625537+0000","last_active":"2024-03-26T05:04:51.537743+0000","last_peered":"2024-03-26T05:04:51.537743+0000","last_clean":"2024-03-26T05:04:51.537743+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:04:51.537743+0000","last_undegraded":"2024-03-26T05:04:51.537743+0000","last_fullsized":"2024-03-26T05:04:51.537743+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:04:24.625406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:24.625406+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:47.792081+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"151'10835","reported_seq":11025,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.752353+0000","last_change":"2024-03-26T05:04:08.237214+0000","last_active":"2024-03-26T05:04:59.752353+0000","last_peered":"2024-03-26T05:04:59.752353+0000","last_clean":"2024-03-26T05:04:59.752353+0000","last_became_active":"2024-03-26T05:02:24.352008+0000","last_became_peered":"2024-03-26T05:02:24.352008+0000","last_unstale":"2024-03-26T05:04:59.752353+0000","last_undegraded":"2024-03-26T05:04:59.752353+0000","last_fullsized":"2024-03-26T05:04:59.752353+0000","mapping_epoch":103,"log_start":"121'4700","ondisk_log_start":"121'4700","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7544","last_scrub_stamp":"2024-03-26T05:04:08.237042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:08.237042+0000","objects_scrubbed":7490,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:12.281990+0000","scrub_duration":2206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278659096,"num_objects":4253,"num_object_clones":0,"num_object_copies":12759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4253,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10835,"num_write_kb":482753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.531811+0000","last_change":"2024-03-26T05:04:25.636127+0000","last_active":"2024-03-26T05:04:51.531811+0000","last_peered":"2024-03-26T05:04:51.531811+0000","last_clean":"2024-03-26T05:04:51.531811+0000","last_became_active":"2024-03-26T05:00:29.461236+0000","last_became_peered":"2024-03-26T05:00:29.461236+0000","last_unstale":"2024-03-26T05:04:51.531811+0000","last_undegraded":"2024-03-26T05:04:51.531811+0000","last_fullsized":"2024-03-26T05:04:51.531811+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:25.636012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:25.636012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:37.693140+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":186,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.537042+0000","last_change":"2024-03-26T05:03:03.891855+0000","last_active":"2024-03-26T05:04:51.537042+0000","last_peered":"2024-03-26T05:04:51.537042+0000","last_clean":"2024-03-26T05:04:51.537042+0000","last_became_active":"2024-03-26T05:00:26.955805+0000","last_became_peered":"2024-03-26T05:00:26.955805+0000","last_unstale":"2024-03-26T05:04:51.537042+0000","last_undegraded":"2024-03-26T05:04:51.537042+0000","last_fullsized":"2024-03-26T05:04:51.537042+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:03.891766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:03.891766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.519277+0000","last_change":"2024-03-26T05:03:19.466684+0000","last_active":"2024-03-26T05:04:51.519277+0000","last_peered":"2024-03-26T05:04:51.519277+0000","last_clean":"2024-03-26T05:04:51.519277+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:04:51.519277+0000","last_undegraded":"2024-03-26T05:04:51.519277+0000","last_fullsized":"2024-03-26T05:04:51.519277+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:19.466546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:19.466546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:04:52.103943+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'17","reported_seq":222,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.520893+0000","last_change":"2024-03-26T05:03:23.417863+0000","last_active":"2024-03-26T05:04:51.520893+0000","last_peered":"2024-03-26T05:04:51.520893+0000","last_clean":"2024-03-26T05:04:51.520893+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:04:51.520893+0000","last_undegraded":"2024-03-26T05:04:51.520893+0000","last_fullsized":"2024-03-26T05:04:51.520893+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:03:23.417719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:23.417719+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"151'10728","reported_seq":10918,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:59.890740+0000","last_change":"2024-03-26T05:04:09.590882+0000","last_active":"2024-03-26T05:04:59.890740+0000","last_peered":"2024-03-26T05:04:59.890740+0000","last_clean":"2024-03-26T05:04:59.890740+0000","last_became_active":"2024-03-26T05:02:24.357742+0000","last_became_peered":"2024-03-26T05:02:24.357742+0000","last_unstale":"2024-03-26T05:04:59.890740+0000","last_undegraded":"2024-03-26T05:04:59.890740+0000","last_fullsized":"2024-03-26T05:04:59.890740+0000","mapping_epoch":103,"log_start":"118'3900","ondisk_log_start":"118'3900","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7576","last_scrub_stamp":"2024-03-26T05:04:09.590405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:09.590405+0000","objects_scrubbed":7467,"log_size":6828,"log_dups_size":0,"ondisk_log_size":6828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:26.578806+0000","scrub_duration":3319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280887296,"num_objects":4286,"num_object_clones":0,"num_object_copies":12858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10728,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":181,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.538994+0000","last_change":"2024-03-26T05:03:21.940017+0000","last_active":"2024-03-26T05:04:51.538994+0000","last_peered":"2024-03-26T05:04:51.538994+0000","last_clean":"2024-03-26T05:04:51.538994+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:04:51.538994+0000","last_undegraded":"2024-03-26T05:04:51.538994+0000","last_fullsized":"2024-03-26T05:04:51.538994+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:21.939782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:21.939782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":186,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.537892+0000","last_change":"2024-03-26T05:03:18.783710+0000","last_active":"2024-03-26T05:04:51.537892+0000","last_peered":"2024-03-26T05:04:51.537892+0000","last_clean":"2024-03-26T05:04:51.537892+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:04:51.537892+0000","last_undegraded":"2024-03-26T05:04:51.537892+0000","last_fullsized":"2024-03-26T05:04:51.537892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:18.783471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:18.783471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":201,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.528688+0000","last_change":"2024-03-26T05:04:21.884967+0000","last_active":"2024-03-26T05:04:51.528688+0000","last_peered":"2024-03-26T05:04:51.528688+0000","last_clean":"2024-03-26T05:04:51.528688+0000","last_became_active":"2024-03-26T05:00:22.768610+0000","last_became_peered":"2024-03-26T05:00:22.768610+0000","last_unstale":"2024-03-26T05:04:51.528688+0000","last_undegraded":"2024-03-26T05:04:51.528688+0000","last_fullsized":"2024-03-26T05:04:51.528688+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:21.884876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.884876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:37.934924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":211,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.520674+0000","last_change":"2024-03-26T05:02:57.386713+0000","last_active":"2024-03-26T05:04:51.520674+0000","last_peered":"2024-03-26T05:04:51.520674+0000","last_clean":"2024-03-26T05:04:51.520674+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:04:51.520674+0000","last_undegraded":"2024-03-26T05:04:51.520674+0000","last_fullsized":"2024-03-26T05:04:51.520674+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:02:57.386555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:02:57.386555+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"151'10519","reported_seq":10677,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:05:00.025380+0000","last_change":"2024-03-26T05:05:00.025500+0000","last_active":"2024-03-26T05:05:00.025380+0000","last_peered":"2024-03-26T05:05:00.025380+0000","last_clean":"2024-03-26T05:05:00.025380+0000","last_became_active":"2024-03-26T05:02:24.352196+0000","last_became_peered":"2024-03-26T05:02:24.352196+0000","last_unstale":"2024-03-26T05:05:00.025380+0000","last_undegraded":"2024-03-26T05:05:00.025380+0000","last_fullsized":"2024-03-26T05:05:00.025380+0000","mapping_epoch":103,"log_start":"123'4900","ondisk_log_start":"123'4900","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'10519","last_scrub_stamp":"2024-03-26T05:05:00.025304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:05:00.025304+0000","objects_scrubbed":4328,"log_size":5619,"log_dups_size":0,"ondisk_log_size":5619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:29.127404+0000","scrub_duration":2074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279117824,"num_objects":4259,"num_object_clones":0,"num_object_copies":12777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10519,"num_write_kb":472896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.521076+0000","last_change":"2024-03-26T05:04:22.211543+0000","last_active":"2024-03-26T05:04:51.521076+0000","last_peered":"2024-03-26T05:04:51.521076+0000","last_clean":"2024-03-26T05:04:51.521076+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:04:51.521076+0000","last_undegraded":"2024-03-26T05:04:51.521076+0000","last_fullsized":"2024-03-26T05:04:51.521076+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:22.211431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:22.211431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:23.405861+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"117'9","reported_seq":251,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:58.445709+0000","last_change":"2024-03-26T05:03:09.035963+0000","last_active":"2024-03-26T05:04:58.445709+0000","last_peered":"2024-03-26T05:04:58.445709+0000","last_clean":"2024-03-26T05:04:58.445709+0000","last_became_active":"2024-03-26T05:00:26.957142+0000","last_became_peered":"2024-03-26T05:00:26.957142+0000","last_unstale":"2024-03-26T05:04:58.445709+0000","last_undegraded":"2024-03-26T05:04:58.445709+0000","last_fullsized":"2024-03-26T05:04:58.445709+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'9","last_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.035843+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.531778+0000","last_change":"2024-03-26T05:03:09.192328+0000","last_active":"2024-03-26T05:04:51.531778+0000","last_peered":"2024-03-26T05:04:51.531778+0000","last_clean":"2024-03-26T05:04:51.531778+0000","last_became_active":"2024-03-26T05:00:22.778719+0000","last_became_peered":"2024-03-26T05:00:22.778719+0000","last_unstale":"2024-03-26T05:04:51.531778+0000","last_undegraded":"2024-03-26T05:04:51.531778+0000","last_fullsized":"2024-03-26T05:04:51.531778+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:09.192105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.192105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":206,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-26T05:04:51.520300+0000","last_change":"2024-03-26T05:04:16.153198+0000","last_active":"2024-03-26T05:04:51.520300+0000","last_peered":"2024-03-26T05:04:51.520300+0000","last_clean":"2024-03-26T05:04:51.520300+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:04:51.520300+0000","last_undegraded":"2024-03-26T05:04:51.520300+0000","last_fullsized":"2024-03-26T05:04:51.520300+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:04:16.153054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:16.153054+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:39.530782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4538368024,"num_objects":69251,"num_object_clones":0,"num_object_copies":207753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69251,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169743,"num_write_kb":7647745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13617737728,"data_stored":13617725512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89743,"ondisk_log_size":89743,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":271,"num_read_kb":266,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":617,"ondisk_log_size":617,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005116,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1485016,"kb_used_data":1409424,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92260136,"statfs":{"total":95995035648,"available":94474379264,"internally_reserved":0,"allocated":1443250176,"data_stored":1442927790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332161,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1494092,"kb_used_data":1396292,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":92251060,"statfs":{"total":95995035648,"available":94465085440,"internally_reserved":0,"allocated":1429803008,"data_stored":1429491418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":100137414},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1760060,"kb_used_data":1660788,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91985092,"statfs":{"total":95995035648,"available":94192734208,"internally_reserved":0,"allocated":1700646912,"data_stored":1700286890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2291368,"kb_used_data":2192160,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91453784,"statfs":{"total":95995035648,"available":93648674816,"internally_reserved":0,"allocated":2244771840,"data_stored":2244433247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1790460,"kb_used_data":1678772,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91954692,"statfs":{"total":95995035648,"available":94161604608,"internally_reserved":0,"allocated":1719062528,"data_stored":1718766568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117072,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2641696,"kb_used_data":2502232,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91103456,"statfs":{"total":95995035648,"available":93289938944,"internally_reserved":0,"allocated":2562285568,"data_stored":2561953410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142801351},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1507420,"kb_used_data":1409620,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":92237732,"statfs":{"total":95995035648,"available":94451437568,"internally_reserved":0,"allocated":1443450880,"data_stored":1443123407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":100137413},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659206,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1200712,"kb_used_data":1129280,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92544440,"statfs":{"total":95995035648,"available":94765506560,"internally_reserved":0,"allocated":1156382720,"data_stored":1156074984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2241728512,"data_stored":2241724440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1699352576,"data_stored":1699348504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1704591360,"data_stored":1704591360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2553937920,"data_stored":2553937920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1427832832,"data_stored":1427832832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1141506048,"data_stored":1141506048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1427705856,"data_stored":1427701784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1421082624,"data_stored":1421082624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:05:03.021 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T05:05:03.022 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:05:03.022 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-26T05:05:03.022 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T05:05:03.023 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:03.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:03.059 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:03.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:03.160 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:03.210 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:03.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:03.311 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:03.356 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:03.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:03.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:03.458 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:03.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:03.490 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:03.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:03.591 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:03.621 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:03.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:03.723 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:03.769 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:03.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:03.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:05:03.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[40404]: Cluster is now healthy 2024-03-26T05:05:03.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2276398023' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[40404]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[40404]: 3.10 scrub starts 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[40404]: 3.10 scrub ok 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[40404]: 6.10 scrub starts 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[40404]: 6.10 scrub ok 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[29691]: Cluster is now healthy 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2276398023' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[29691]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[29691]: 3.10 scrub starts 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[29691]: 3.10 scrub ok 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[29691]: 6.10 scrub starts 2024-03-26T05:05:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:03 smithi080 ceph-mon[29691]: 6.10 scrub ok 2024-03-26T05:05:03.871 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:03.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:03.906 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:03.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:04.007 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:04.043 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:04.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:03 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:05:04.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:03 smithi132 ceph-mon[33500]: Cluster is now healthy 2024-03-26T05:05:04.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:03 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2276398023' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:05:04.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:03 smithi132 ceph-mon[33500]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T05:05:04.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:03 smithi132 ceph-mon[33500]: 3.10 scrub starts 2024-03-26T05:05:04.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:03 smithi132 ceph-mon[33500]: 3.10 scrub ok 2024-03-26T05:05:04.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:03 smithi132 ceph-mon[33500]: 6.10 scrub starts 2024-03-26T05:05:04.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:03 smithi132 ceph-mon[33500]: 6.10 scrub ok 2024-03-26T05:05:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:04.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:04.145 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:04.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:04.195 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:04.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:04.294 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:05:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:05:04.296 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:04.326 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:04.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:04.428 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:04.460 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:04.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:04.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:04.561 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:04.595 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:04.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:04.696 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:04.724 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:04.756 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:04.789 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:04.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:04.891 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:04.892 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: 5.19 scrub starts 2024-03-26T05:05:04.892 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: 5.19 scrub ok 2024-03-26T05:05:04.892 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: 4.2 scrub starts 2024-03-26T05:05:04.892 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: 4.2 scrub ok 2024-03-26T05:05:04.892 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: pgmap v187: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 958 op/s 2024-03-26T05:05:04.892 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: 6.7 deep-scrub starts 2024-03-26T05:05:04.892 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: 6.7 deep-scrub ok 2024-03-26T05:05:04.892 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: 3.1f scrub starts 2024-03-26T05:05:04.893 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: 3.1f scrub ok 2024-03-26T05:05:04.893 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: 4.d scrub starts 2024-03-26T05:05:04.893 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: 4.d scrub ok 2024-03-26T05:05:04.893 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:04.893 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:04.893 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: 5.19 scrub starts 2024-03-26T05:05:04.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: 5.19 scrub ok 2024-03-26T05:05:04.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: 4.2 scrub starts 2024-03-26T05:05:04.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: 4.2 scrub ok 2024-03-26T05:05:04.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: pgmap v187: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 958 op/s 2024-03-26T05:05:04.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: 6.7 deep-scrub starts 2024-03-26T05:05:04.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: 6.7 deep-scrub ok 2024-03-26T05:05:04.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: 3.1f scrub starts 2024-03-26T05:05:04.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: 3.1f scrub ok 2024-03-26T05:05:04.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: 4.d scrub starts 2024-03-26T05:05:04.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: 4.d scrub ok 2024-03-26T05:05:04.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:04.894 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:04.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:04.926 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:04.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:04.931 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: 5.19 scrub starts 2024-03-26T05:05:04.931 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: 5.19 scrub ok 2024-03-26T05:05:04.931 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: 4.2 scrub starts 2024-03-26T05:05:04.931 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: 4.2 scrub ok 2024-03-26T05:05:04.931 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: pgmap v187: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 958 op/s 2024-03-26T05:05:04.931 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: 6.7 deep-scrub starts 2024-03-26T05:05:04.931 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: 6.7 deep-scrub ok 2024-03-26T05:05:04.931 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: 3.1f scrub starts 2024-03-26T05:05:04.931 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: 3.1f scrub ok 2024-03-26T05:05:04.931 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: 4.d scrub starts 2024-03-26T05:05:04.931 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: 4.d scrub ok 2024-03-26T05:05:04.931 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:04.932 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:05.027 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:05.068 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:05.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:05.169 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:05.173 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:05:05.173 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:05:05.189 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:05:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:05:05.214 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:05.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:05.316 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:05.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:05.375 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:05.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:05.476 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:05.517 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:05.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:05.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:05.619 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:05.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:05.657 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:05.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:05.758 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:05.788 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:05.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:05.890 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:05.942 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:05.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:05.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T05:05:06.044 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:06.099 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:06.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:06.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: 5.13 deep-scrub starts 2024-03-26T05:05:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: 3.15 deep-scrub starts 2024-03-26T05:05:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: 3.15 deep-scrub ok 2024-03-26T05:05:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: 5.13 deep-scrub ok 2024-03-26T05:05:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: 4.a scrub starts 2024-03-26T05:05:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: 4.a scrub ok 2024-03-26T05:05:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: 6.d scrub starts 2024-03-26T05:05:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: 6.d scrub ok 2024-03-26T05:05:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:06.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: 5.13 deep-scrub starts 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: 3.15 deep-scrub starts 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: 3.15 deep-scrub ok 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: 5.13 deep-scrub ok 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: 4.a scrub starts 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: 4.a scrub ok 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: 6.d scrub starts 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: 6.d scrub ok 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:06.200 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:06.252 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:06.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:06.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:06.268 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:05:06.282 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:05:06.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: 5.13 deep-scrub starts 2024-03-26T05:05:06.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: 3.15 deep-scrub starts 2024-03-26T05:05:06.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: 3.15 deep-scrub ok 2024-03-26T05:05:06.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: 5.13 deep-scrub ok 2024-03-26T05:05:06.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: 4.a scrub starts 2024-03-26T05:05:06.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: 4.a scrub ok 2024-03-26T05:05:06.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: 6.d scrub starts 2024-03-26T05:05:06.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:06.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: 6.d scrub ok 2024-03-26T05:05:06.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:06.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:06.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:06.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:06.354 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:06.382 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:06.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:06.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:06.484 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:06.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:06.515 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:06.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:06.616 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:06.669 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:06.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:06.771 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:06.827 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:06.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:06.929 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:06.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:06.977 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:06.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:06 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:05:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:06 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:05:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:06 smithi132 ceph-mon[33500]: pgmap v188: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 968 op/s 2024-03-26T05:05:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:06 smithi132 ceph-mon[33500]: 3.4 scrub starts 2024-03-26T05:05:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:06 smithi132 ceph-mon[33500]: 3.4 scrub ok 2024-03-26T05:05:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:06 smithi132 ceph-mon[33500]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T05:05:07.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:06 smithi132 ceph-mon[33500]: 4.18 scrub starts 2024-03-26T05:05:07.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:06 smithi132 ceph-mon[33500]: 4.18 scrub ok 2024-03-26T05:05:07.078 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:07.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:07.117 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:07.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:07.218 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:07.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:07.244 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:07.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:07.258 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:05:07.258 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:05:07.258 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[40404]: pgmap v188: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 968 op/s 2024-03-26T05:05:07.259 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[40404]: 3.4 scrub starts 2024-03-26T05:05:07.259 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[40404]: 3.4 scrub ok 2024-03-26T05:05:07.259 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[40404]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T05:05:07.259 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[40404]: 4.18 scrub starts 2024-03-26T05:05:07.259 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[40404]: 4.18 scrub ok 2024-03-26T05:05:07.259 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:05:07.259 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:05:07.259 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[29691]: pgmap v188: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 968 op/s 2024-03-26T05:05:07.259 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[29691]: 3.4 scrub starts 2024-03-26T05:05:07.259 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[29691]: 3.4 scrub ok 2024-03-26T05:05:07.259 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[29691]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T05:05:07.260 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[29691]: 4.18 scrub starts 2024-03-26T05:05:07.260 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:06 smithi080 ceph-mon[29691]: 4.18 scrub ok 2024-03-26T05:05:07.345 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:07.374 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:07.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:07.476 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:07.506 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:07.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:07.607 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:07.639 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:07.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:07.741 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:07.769 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:07.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:07.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:07.875 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:07.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:07.921 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:07.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:07.945 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[40404]: 5.f scrub starts 2024-03-26T05:05:07.945 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[40404]: 5.f scrub ok 2024-03-26T05:05:07.945 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[40404]: 3.f deep-scrub starts 2024-03-26T05:05:07.946 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[40404]: 3.f deep-scrub ok 2024-03-26T05:05:07.946 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:05:07.946 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:05:07.946 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:07.946 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[29691]: 5.f scrub starts 2024-03-26T05:05:07.946 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[29691]: 5.f scrub ok 2024-03-26T05:05:07.946 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[29691]: 3.f deep-scrub starts 2024-03-26T05:05:07.946 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[29691]: 3.f deep-scrub ok 2024-03-26T05:05:07.946 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:05:07.946 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:05:07.947 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:07 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:07.978 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:08.021 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:08.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:07 smithi132 ceph-mon[33500]: 5.f scrub starts 2024-03-26T05:05:08.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:07 smithi132 ceph-mon[33500]: 5.f scrub ok 2024-03-26T05:05:08.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:07 smithi132 ceph-mon[33500]: 3.f deep-scrub starts 2024-03-26T05:05:08.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:07 smithi132 ceph-mon[33500]: 3.f deep-scrub ok 2024-03-26T05:05:08.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:07 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:05:08.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:07 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:05:08.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:07 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:08.052 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:08.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:08.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:08.187 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:08.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:08.288 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:08.333 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:08.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:08.435 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:08.470 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:08.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:08.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:08.505 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:05:08.506 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:05:08.572 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:08.624 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:08.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:08.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:08.751 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:08.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:08.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:08.775 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:05:08.785 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:05:08.785 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:05:08.785 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 153, 2024-03-26T05:05:08.786 INFO:teuthology.orchestra.run.smithi080.stdout: "fsid": "dd09e9cc-eb2c-11ee-95d0-87774f69a715", 2024-03-26T05:05:08.786 INFO:teuthology.orchestra.run.smithi080.stdout: "created": "2024-03-26T04:54:51.106954+0000", 2024-03-26T05:05:08.786 INFO:teuthology.orchestra.run.smithi080.stdout: "modified": "2024-03-26T05:05:05.893008+0000", 2024-03-26T05:05:08.786 INFO:teuthology.orchestra.run.smithi080.stdout: "last_up_change": "2024-03-26T05:00:08.801055+0000", 2024-03-26T05:05:08.786 INFO:teuthology.orchestra.run.smithi080.stdout: "last_in_change": "2024-03-26T04:59:50.693588+0000", 2024-03-26T05:05:08.786 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T05:05:08.786 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_num": 5799936, 2024-03-26T05:05:08.786 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_set": [ 2024-03-26T05:05:08.786 INFO:teuthology.orchestra.run.smithi080.stdout: "pglog_hardlimit", 2024-03-26T05:05:08.786 INFO:teuthology.orchestra.run.smithi080.stdout: "purged_snapdirs", 2024-03-26T05:05:08.786 INFO:teuthology.orchestra.run.smithi080.stdout: "recovery_deletes", 2024-03-26T05:05:08.786 INFO:teuthology.orchestra.run.smithi080.stdout: "sortbitwise" 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_version": 18, 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_snapshot": "", 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_max": 7, 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: "max_osd": 8, 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: "require_min_compat_client": "luminous", 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: "min_compat_client": "luminous", 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: "require_osd_release": "squid", 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: "allow_crimson": false, 2024-03-26T05:05:08.787 INFO:teuthology.orchestra.run.smithi080.stdout: "pools": [ 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 1, 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": ".mgr", 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T04:58:06.876819+0000", 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:05:08.788 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 1, 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 1, 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 1, 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 1, 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 1, 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:05:08.789 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:05:08.790 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.790 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "22", 2024-03-26T05:05:08.790 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:05:08.790 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:05:08.790 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:05:08.790 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:05:08.790 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:05:08.790 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:05:08.790 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:05:08.790 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:05:08.790 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:05:08.793 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:05:08.793 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:05:08.793 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:05:08.793 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:05:08.793 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:05:08.794 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:05:08.794 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:05:08.794 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:05:08.794 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:05:08.794 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:05:08.794 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:05:08.794 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:05:08.794 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:05:08.794 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:05:08.794 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:05:08.795 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: "options": { 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_max": 32, 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_min": 1 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": {} 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 7.8899998664855957, 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 7.8899998664855957, 2024-03-26T05:05:08.796 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.37999999523162842, 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 3, 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 3, 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 2, 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "datapool", 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:20.705417+0000", 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 8193, 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T05:05:08.797 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 3, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 3, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 3, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 3, 2024-03-26T05:05:08.798 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 3, 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "71", 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:05:08.799 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 2, 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 71, 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:05:08.800 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:05:08.801 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:05:08.802 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:05:08.802 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:05:08.802 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:05:08.802 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:05:08.802 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:05:08.802 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:05:08.802 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:05:08.802 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd": {} 2024-03-26T05:05:08.802 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.802 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:05:08.802 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:05:08.802 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.6500000953674316, 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.6500000953674316, 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 3, 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": ".rgw.root", 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:21.150368+0000", 2024-03-26T05:05:08.803 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:05:08.804 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:05:08.805 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:05:08.805 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:05:08.805 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:05:08.805 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:05:08.805 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:05:08.805 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:05:08.805 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:05:08.805 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:05:08.805 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:05:08.805 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.805 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "69", 2024-03-26T05:05:08.806 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:05:08.806 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:05:08.806 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:05:08.806 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:05:08.806 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:05:08.806 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:05:08.806 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:05:08.806 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:05:08.807 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:05:08.808 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.5, 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.5, 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.5, 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.5, 2024-03-26T05:05:08.809 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 4, 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.log", 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:23.638065+0000", 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:05:08.810 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:05:08.811 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "71", 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:05:08.812 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:05:08.813 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:05:08.814 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.25, 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.25, 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2.25, 2024-03-26T05:05:08.815 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2.25, 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 5, 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.control", 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:25.181000+0000", 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:05:08.816 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:05:08.817 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "73", 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:05:08.818 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:05:08.819 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:05:08.820 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.25, 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.25, 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:05:08.821 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.25, 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.25, 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 6, 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.meta", 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:27.249998+0000", 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:05:08.822 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:05:08.823 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "75", 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:05:08.824 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:05:08.825 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:05:08.826 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:05:08.826 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:05:08.826 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:05:08.826 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:05:08.826 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:05:08.826 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:05:08.826 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.826 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:05:08.826 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:05:08.826 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:05:08.827 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:05:08.827 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:05:08.827 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:05:08.827 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:05:08.827 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:05:08.827 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:05:08.827 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:05:08.828 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:05:08.828 INFO:teuthology.orchestra.run.smithi080.stdout: "options": { 2024-03-26T05:05:08.828 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_bias": 4 2024-03-26T05:05:08.828 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.828 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:05:08.828 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:05:08.828 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.828 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:05:08.828 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:05:08.828 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.75, 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.75, 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.75, 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.75, 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 7, 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "unique_pool_0", 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:02:23.280375+0000", 2024-03-26T05:05:08.829 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 16, 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 16, 2024-03-26T05:05:08.830 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 16, 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 16, 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 16, 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "109", 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:05:08.831 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:05:08.832 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:05:08.833 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "rados": {} 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:05:08.834 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2, 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2, 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2, 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2, 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.835 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 0, 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366", 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 8, 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 103, 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.836 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6802", 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6803", 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.837 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6804", 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6805", 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:05:08.838 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6808", 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6809", 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.839 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6806", 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6807", 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.840 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.841 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6803/1840686622", 2024-03-26T05:05:08.841 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6805/1840686622", 2024-03-26T05:05:08.841 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6809/1840686622", 2024-03-26T05:05:08.841 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6807/1840686622", 2024-03-26T05:05:08.841 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:05:08.842 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:05:08.842 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:05:08.842 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.842 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.842 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.842 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 1, 2024-03-26T05:05:08.842 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23", 2024-03-26T05:05:08.842 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:05:08.842 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:05:08.842 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:05:08.842 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 13, 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 103, 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6810", 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:05:08.843 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.844 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.844 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.844 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6811", 2024-03-26T05:05:08.844 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:05:08.844 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.844 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.844 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.844 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:05:08.844 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.844 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.844 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.844 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6812", 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6813", 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.845 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6816", 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6817", 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.846 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6814", 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6815", 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6811/1726833996", 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6813/1726833996", 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6817/1726833996", 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6815/1726833996", 2024-03-26T05:05:08.847 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 2, 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "536bec9b-9636-4e67-b452-ed1a55c22ab6", 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 18, 2024-03-26T05:05:08.848 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 103, 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6818", 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6819", 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:05:08.849 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6820", 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6821", 2024-03-26T05:05:08.850 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6824", 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6825", 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:05:08.851 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6822", 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6823", 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:05:08.852 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6819/128713757", 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6821/128713757", 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6825/128713757", 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6823/128713757", 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 3, 2024-03-26T05:05:08.853 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "640cfdb6-b9c5-446d-8312-504ae12c7818", 2024-03-26T05:05:08.854 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:05:08.854 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:05:08.854 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:05:08.854 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:05:08.854 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:05:08.854 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:05:08.854 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 27, 2024-03-26T05:05:08.854 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 103, 2024-03-26T05:05:08.854 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:05:08.854 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:05:08.854 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:05:08.854 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6826", 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6827", 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.855 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6828", 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6829", 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.856 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6832", 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6833", 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.857 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6830", 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6831", 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.858 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6827/418771441", 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6829/418771441", 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6833/418771441", 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6831/418771441", 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 4, 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639", 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:05:08.859 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 33, 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 103, 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.860 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6800", 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6801", 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.861 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6802", 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6803", 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.862 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6806", 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6807", 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.863 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6804", 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6805", 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6801/1441071423", 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6803/1441071423", 2024-03-26T05:05:08.864 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6807/1441071423", 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6805/1441071423", 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 5, 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc", 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:05:08.865 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 41, 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 103, 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6808", 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.866 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6809", 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6810", 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.867 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6811", 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6814", 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.868 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6815", 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6812", 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.869 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6813", 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6809/794504625", 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6811/794504625", 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6815/794504625", 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6813/794504625", 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:05:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 6, 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5", 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 51, 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 115, 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:05:08.871 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6816", 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6817", 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.872 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6818", 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6819", 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.873 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.874 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.874 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:05:08.874 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.874 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.874 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.874 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6822", 2024-03-26T05:05:08.874 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:05:08.874 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.874 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.874 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.874 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6823", 2024-03-26T05:05:08.874 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:05:08.875 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.875 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.875 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.875 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:05:08.875 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.875 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.875 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.875 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6820", 2024-03-26T05:05:08.875 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:05:08.875 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.875 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.875 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6821", 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6817/2209078972", 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6819/2209078972", 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6823/2209078972", 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6821/2209078972", 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:05:08.876 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.877 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.877 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.877 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 7, 2024-03-26T05:05:08.877 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d", 2024-03-26T05:05:08.877 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:05:08.877 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:05:08.877 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:05:08.877 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:05:08.877 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:05:08.877 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:05:08.877 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 61, 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 103, 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6824", 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6825", 2024-03-26T05:05:08.878 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:05:08.879 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.879 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.879 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.879 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:05:08.879 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.879 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.879 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:05:08.880 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.880 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6826", 2024-03-26T05:05:08.880 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:05:08.880 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.880 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.880 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.880 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6827", 2024-03-26T05:05:08.880 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:05:08.880 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.880 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6830", 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6831", 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:05:08.881 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.882 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.882 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.882 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:05:08.882 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:05:08.882 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.882 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:05:08.882 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6828", 2024-03-26T05:05:08.882 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:05:08.882 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.882 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.882 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:05:08.882 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6829", 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6825/3655388282", 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6827/3655388282", 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6831/3655388282", 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6829/3655388282", 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.883 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:05:08.884 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_xinfo": [ 2024-03-26T05:05:08.884 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.884 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 0, 2024-03-26T05:05:08.884 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:05:08.884 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:05:08.884 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:05:08.884 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:05:08.884 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:05:08.884 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:04:25.384118+0000", 2024-03-26T05:05:08.884 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:05:08.884 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.884 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.885 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 1, 2024-03-26T05:05:08.885 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:05:08.885 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:05:08.885 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:05:08.885 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:05:08.885 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:05:08.885 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:04:42.933605+0000", 2024-03-26T05:05:08.885 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:05:08.885 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.885 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.885 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 2, 2024-03-26T05:05:08.885 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:04:23.556968+0000", 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 3, 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:05:08.886 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:04:50.016444+0000", 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 4, 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:03:58.633037+0000", 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 5, 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:04:08.450522+0000", 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 6, 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:05:08.888 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:04:46.632559+0000", 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 7, 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:05:08.889 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:05:05.645006+0000", 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap": [], 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap_items": [ 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.16", 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 5, 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 6 2024-03-26T05:05:08.890 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.2", 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.891 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.6", 2024-03-26T05:05:08.892 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:05:08.892 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.892 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:05:08.892 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 5 2024-03-26T05:05:08.892 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.892 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.892 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.892 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.892 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.1f", 2024-03-26T05:05:08.892 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:05:08.892 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:05:08.892 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap_primaries": [], 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_temp": [], 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_temp": [], 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: "blocklist": { 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3992539385": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/2770889532": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/2466284312": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/490056571": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:05:08.893 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1986135300": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3796611170": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1993138634": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1832498181": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/2466284312": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1650870847": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/1418683843": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3147301987": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/2770889532": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/571905941": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1451012560": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1235125152": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/1418683843": "2024-03-27T05:01:03.689739+0000" 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.894 INFO:teuthology.orchestra.run.smithi080.stdout: "range_blocklist": {}, 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profiles": { 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: "default": { 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: "crush-failure-domain": "osd", 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: "k": "2", 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: "m": "1", 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: "plugin": "jerasure", 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: "technique": "reed_sol_van" 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps_queue": [], 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: "new_removed_snaps": [], 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: "new_purged_snaps": [], 2024-03-26T05:05:08.895 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_node_flags": {}, 2024-03-26T05:05:08.896 INFO:teuthology.orchestra.run.smithi080.stdout: "device_class_flags": {}, 2024-03-26T05:05:08.896 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode": { 2024-03-26T05:05:08.896 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode_enabled": false, 2024-03-26T05:05:08.896 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_bucket_count": 0, 2024-03-26T05:05:08.896 INFO:teuthology.orchestra.run.smithi080.stdout: "degraded_stretch_mode": 0, 2024-03-26T05:05:08.896 INFO:teuthology.orchestra.run.smithi080.stdout: "recovering_stretch_mode": 0, 2024-03-26T05:05:08.896 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode_bucket": 0 2024-03-26T05:05:08.896 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:05:08.896 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:05:08.896 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:05:08.897 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:08.920 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:05:08.923 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:08.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:08.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:08.963 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:05:09.012 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:05:09.025 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:09.054 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[29691]: 3.2 scrub starts 2024-03-26T05:05:09.055 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[29691]: 3.2 scrub ok 2024-03-26T05:05:09.055 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[29691]: pgmap v190: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 987 op/s 2024-03-26T05:05:09.055 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[29691]: 3.5 scrub starts 2024-03-26T05:05:09.055 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[29691]: 3.5 scrub ok 2024-03-26T05:05:09.055 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[29691]: 3.1b scrub starts 2024-03-26T05:05:09.055 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[29691]: 3.1b scrub ok 2024-03-26T05:05:09.055 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3936695732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:05:09.055 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/167080726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:05:09.055 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[29691]: 3.13 scrub starts 2024-03-26T05:05:09.055 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[29691]: 3.13 scrub ok 2024-03-26T05:05:09.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[40404]: 3.2 scrub starts 2024-03-26T05:05:09.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[40404]: 3.2 scrub ok 2024-03-26T05:05:09.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[40404]: pgmap v190: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 987 op/s 2024-03-26T05:05:09.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[40404]: 3.5 scrub starts 2024-03-26T05:05:09.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[40404]: 3.5 scrub ok 2024-03-26T05:05:09.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[40404]: 3.1b scrub starts 2024-03-26T05:05:09.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[40404]: 3.1b scrub ok 2024-03-26T05:05:09.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3936695732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:05:09.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/167080726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:05:09.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[40404]: 3.13 scrub starts 2024-03-26T05:05:09.056 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:08 smithi080 ceph-mon[40404]: 3.13 scrub ok 2024-03-26T05:05:09.057 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:05:09.064 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:09.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:09.096 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:05:09.104 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:05:09.104 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:05:09.104 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:05:09.104 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:05:09.104 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:05:09.104 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:05:09.104 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:05:09.105 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:05:09.166 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:09.196 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:09.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:09.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:09.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:08 smithi132 ceph-mon[33500]: 3.2 scrub starts 2024-03-26T05:05:09.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:08 smithi132 ceph-mon[33500]: 3.2 scrub ok 2024-03-26T05:05:09.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:08 smithi132 ceph-mon[33500]: pgmap v190: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 987 op/s 2024-03-26T05:05:09.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:08 smithi132 ceph-mon[33500]: 3.5 scrub starts 2024-03-26T05:05:09.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:08 smithi132 ceph-mon[33500]: 3.5 scrub ok 2024-03-26T05:05:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:08 smithi132 ceph-mon[33500]: 3.1b scrub starts 2024-03-26T05:05:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:08 smithi132 ceph-mon[33500]: 3.1b scrub ok 2024-03-26T05:05:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:08 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3936695732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:05:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:08 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/167080726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:05:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:08 smithi132 ceph-mon[33500]: 3.13 scrub starts 2024-03-26T05:05:09.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:08 smithi132 ceph-mon[33500]: 3.13 scrub ok 2024-03-26T05:05:09.297 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:09.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:09.329 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:09.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:09.431 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:09.465 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:09.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:09.566 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:09.604 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:09.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:09.705 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:09.733 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:09.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:09.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:09.835 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:09.862 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T05:05:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:09.863 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:09.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:09.964 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:10.000 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:10.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:10.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:10.102 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:10.135 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:10.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:10.198 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:05:10.237 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:10.270 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:10.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:10.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:10 smithi132 ceph-mon[33500]: 4.14 scrub starts 2024-03-26T05:05:10.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:10 smithi132 ceph-mon[33500]: 4.14 scrub ok 2024-03-26T05:05:10.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:10 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:05:10.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:10 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:05:10.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:10 smithi132 ceph-mon[33500]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T05:05:10.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:10 smithi132 ceph-mon[33500]: 5.18 scrub starts 2024-03-26T05:05:10.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:10 smithi132 ceph-mon[33500]: 5.18 scrub ok 2024-03-26T05:05:10.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:10 smithi132 ceph-mon[33500]: 5.a scrub starts 2024-03-26T05:05:10.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:10 smithi132 ceph-mon[33500]: 5.a scrub ok 2024-03-26T05:05:10.317 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[40404]: 4.14 scrub starts 2024-03-26T05:05:10.317 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[40404]: 4.14 scrub ok 2024-03-26T05:05:10.317 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:05:10.317 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:05:10.317 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[40404]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T05:05:10.318 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[40404]: 5.18 scrub starts 2024-03-26T05:05:10.318 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[40404]: 5.18 scrub ok 2024-03-26T05:05:10.318 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[40404]: 5.a scrub starts 2024-03-26T05:05:10.318 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[40404]: 5.a scrub ok 2024-03-26T05:05:10.318 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[29691]: 4.14 scrub starts 2024-03-26T05:05:10.318 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[29691]: 4.14 scrub ok 2024-03-26T05:05:10.318 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:05:10.318 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:05:10.318 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[29691]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T05:05:10.318 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[29691]: 5.18 scrub starts 2024-03-26T05:05:10.318 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[29691]: 5.18 scrub ok 2024-03-26T05:05:10.318 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[29691]: 5.a scrub starts 2024-03-26T05:05:10.319 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:10 smithi080 ceph-mon[29691]: 5.a scrub ok 2024-03-26T05:05:10.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:10.415 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:10.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:10.516 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:10.549 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:10.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:10.650 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:10.693 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:10.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:10.796 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:10.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:10.842 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:10.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:10.943 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:10.977 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:10.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:10.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:11.079 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:11.111 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:11.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:11.213 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:11.247 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:11.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:11.349 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:11.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:11 smithi080 ceph-mon[40404]: pgmap v192: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 976 op/s 2024-03-26T05:05:11.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:11 smithi080 ceph-mon[40404]: 5.16 scrub starts 2024-03-26T05:05:11.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:11 smithi080 ceph-mon[40404]: 5.16 scrub ok 2024-03-26T05:05:11.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:11 smithi080 ceph-mon[40404]: 3.12 scrub starts 2024-03-26T05:05:11.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:11 smithi080 ceph-mon[40404]: 3.12 scrub ok 2024-03-26T05:05:11.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:11 smithi080 ceph-mon[29691]: pgmap v192: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 976 op/s 2024-03-26T05:05:11.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:11 smithi080 ceph-mon[29691]: 5.16 scrub starts 2024-03-26T05:05:11.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:11 smithi080 ceph-mon[29691]: 5.16 scrub ok 2024-03-26T05:05:11.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:11 smithi080 ceph-mon[29691]: 3.12 scrub starts 2024-03-26T05:05:11.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:11 smithi080 ceph-mon[29691]: 3.12 scrub ok 2024-03-26T05:05:11.382 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:11 smithi132 ceph-mon[33500]: pgmap v192: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 976 op/s 2024-03-26T05:05:11.382 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:11 smithi132 ceph-mon[33500]: 5.16 scrub starts 2024-03-26T05:05:11.382 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:11 smithi132 ceph-mon[33500]: 5.16 scrub ok 2024-03-26T05:05:11.382 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:11 smithi132 ceph-mon[33500]: 3.12 scrub starts 2024-03-26T05:05:11.382 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:11 smithi132 ceph-mon[33500]: 3.12 scrub ok 2024-03-26T05:05:11.390 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:11.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:11.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:11.492 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:11.535 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:11.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:11.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:11.637 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:11.688 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:11.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:11.790 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:11.825 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:11.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:11.927 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:11.964 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:11.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:12.066 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:12.109 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:12.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:12.136 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:182272 7.7 7:e7fd31c1:::benchmark_data_smithi080_80848_object62773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:12.127138+0000", 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.016915289999999999, 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016948158000000001, 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 182272 2024-03-26T05:05:12.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.127138+0000", 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.127138+0000", 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:12.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.127139+0000", 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.009e-06 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.127144+0000", 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4750000000000004e-06 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.127159+0000", 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4996e-05 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.127167+0000", 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7710000000000006e-06 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.127734+0000", 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00056738199999999996 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.127857+0000", 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000123342 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.127988+0000", 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013068300000000001 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.132166+0000", 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.004177905 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:182285 7.d 7:b8e68b23:::benchmark_data_smithi080_80848_object62786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-26T05:05:12.141 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:12.142332+0000", 2024-03-26T05:05:12.142 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.001721946, 2024-03-26T05:05:12.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001835742, 2024-03-26T05:05:12.142 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:12.142 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:05:12.142 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 182285 2024-03-26T05:05:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142332+0000", 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142332+0000", 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142333+0000", 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3999999999999999e-06 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142339+0000", 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7169999999999996e-06 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142353+0000", 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4352000000000001e-05 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142363+0000", 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5019999999999998e-06 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142426+0000", 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2989000000000003e-05 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142519+0000", 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2928000000000002e-05 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142622+0000", 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103484 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:182256 7.7 7:ec717679:::benchmark_data_smithi080_80848_object62757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:12.114837+0000", 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.029216901, 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.029376065999999999, 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 182256 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.114837+0000", 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.114837+0000", 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.114837+0000", 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9999999999999997e-07 2024-03-26T05:05:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.114841+0000", 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.162e-06 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.114851+0000", 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0705e-05 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.114858+0000", 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7819999999999996e-06 2024-03-26T05:05:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.116907+0000", 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020484779999999998 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.117059+0000", 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000152397 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.117184+0000", 2024-03-26T05:05:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000125139 2024-03-26T05:05:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.127625+0000", 2024-03-26T05:05:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010441125000000001 2024-03-26T05:05:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142757+0000", 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.015131309000000001 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:182287 7.7 7:e0f23292:::benchmark_data_smithi080_80848_object62788:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:12.142792+0000", 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.001262219, 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0014714070000000001, 2024-03-26T05:05:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 182287 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142792+0000", 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142792+0000", 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142792+0000", 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.23e-07 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142796+0000", 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2050000000000004e-06 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142807+0000", 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0559e-05 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142815+0000", 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6240000000000002e-06 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142860+0000", 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5417999999999998e-05 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143001+0000", 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014049700000000001 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143107+0000", 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010628000000000001 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(client.24701.0:182265 7.7 e154/103)", 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:12.142949+0000", 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0011045060000000001, 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0013521469999999999, 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "queued for pg", 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142949+0000", 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142949+0000", 2024-03-26T05:05:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142950+0000", 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5300000000000004e-07 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142954+0000", 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1320000000000004e-06 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142963+0000", 2024-03-26T05:05:12.159 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4089999999999992e-06 2024-03-26T05:05:12.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.159 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:12.159 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142970+0000", 2024-03-26T05:05:12.159 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.725e-06 2024-03-26T05:05:12.159 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.159 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:12.159 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(client.24701.0:182272 7.7 e154/103)", 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:12.143018+0000", 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0010361019999999999, 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001308652, 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "queued for pg", 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143018+0000", 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143018+0000", 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143018+0000", 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4700000000000002e-07 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143021+0000", 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.6709999999999999e-06 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.162 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:12.162 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143026+0000", 2024-03-26T05:05:12.162 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.237e-06 2024-03-26T05:05:12.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.163 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:12.163 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143030+0000", 2024-03-26T05:05:12.163 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6959999999999998e-06 2024-03-26T05:05:12.163 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.163 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:12.163 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.24701.0:182288 7.1 e154/103 7:8e25c444:::benchmark_data_smithi080_80848_object62789:head v 154'11401, mlcod=154'11399)", 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:12.143415+0000", 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.00063912699999999997, 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00093976599999999997, 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143415+0000", 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.164 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143415+0000", 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143415+0000", 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.1500000000000003e-07 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143421+0000", 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0109999999999997e-06 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143433+0000", 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1423000000000001e-05 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143440+0000", 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5719999999999997e-06 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143493+0000", 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2108000000000003e-05 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.166 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.167 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:12.167 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.143511+0000", 2024-03-26T05:05:12.167 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.8635e-05 2024-03-26T05:05:12.167 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.167 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:12.167 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.167 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.168 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.24701.0:182289 7.b e154/103 7:d1e841be:::benchmark_data_smithi080_80848_object62790:head v 154'11461, mlcod=154'11459)", 2024-03-26T05:05:12.168 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:12.144032+0000", 2024-03-26T05:05:12.168 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.1248000000000001e-05, 2024-03-26T05:05:12.168 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00036383800000000001, 2024-03-26T05:05:12.168 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:12.168 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:05:12.168 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:12.168 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.169 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:12.169 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.144032+0000", 2024-03-26T05:05:12.169 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.169 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.169 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:12.169 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.144032+0000", 2024-03-26T05:05:12.169 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.169 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.169 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:12.169 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.144033+0000", 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0699999999999996e-07 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.144042+0000", 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7809999999999994e-06 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.144052+0000", 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0506000000000001e-05 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.170 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:12.171 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.144059+0000", 2024-03-26T05:05:12.171 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5860000000000001e-06 2024-03-26T05:05:12.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.171 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.171 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:12.171 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.144113+0000", 2024-03-26T05:05:12.171 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4085000000000001e-05 2024-03-26T05:05:12.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.171 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.144132+0000", 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.8683000000000001e-05 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:182263 7.7 7:ea4de553:::benchmark_data_smithi080_80848_object62764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:12.118286+0000", 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.025767406, 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026144896000000001, 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:12.172 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 182263 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.118286+0000", 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.173 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.118286+0000", 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.118287+0000", 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5600000000000005e-07 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.118291+0000", 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.675e-06 2024-03-26T05:05:12.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.118299+0000", 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.118306+0000", 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3219999999999998e-06 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.120496+0000", 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021904469999999999 2024-03-26T05:05:12.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.120611+0000", 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115461 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.120739+0000", 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000127909 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:12.176 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.131096+0000", 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010356621 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.142800+0000", 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011704409000000001 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:182265 7.7 7:eeb0ed42:::benchmark_data_smithi080_80848_object62766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:12.118508+0000", 2024-03-26T05:05:12.177 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.025545399999999999, 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.025971432999999999, 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 182265 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.118508+0000", 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.118508+0000", 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.118509+0000", 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4000000000000001e-07 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.118512+0000", 2024-03-26T05:05:12.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.3560000000000001e-06 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.118520+0000", 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0439999999999996e-06 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.118527+0000", 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4169999999999997e-06 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.124135+0000", 2024-03-26T05:05:12.180 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0056078009999999999 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.124277+0000", 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014220500000000001 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.124446+0000", 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016894600000000001 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:12.181 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:12.131103+0000", 2024-03-26T05:05:12.182 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0066575760000000001 2024-03-26T05:05:12.182 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.182 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:12.182 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.182 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:12.182 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-03-26T05:05:12.182 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 10 2024-03-26T05:05:12.182 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:05:12.211 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:12.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:12.244 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:12.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:12.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:12 smithi132 ceph-mon[33500]: 5.1b scrub starts 2024-03-26T05:05:12.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:12 smithi132 ceph-mon[33500]: 5.1b scrub ok 2024-03-26T05:05:12.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:12 smithi132 ceph-mon[33500]: 5.3 deep-scrub starts 2024-03-26T05:05:12.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:12 smithi132 ceph-mon[33500]: 5.3 deep-scrub ok 2024-03-26T05:05:12.345 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:12.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:12 smithi080 ceph-mon[29691]: 5.1b scrub starts 2024-03-26T05:05:12.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:12 smithi080 ceph-mon[29691]: 5.1b scrub ok 2024-03-26T05:05:12.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:12 smithi080 ceph-mon[29691]: 5.3 deep-scrub starts 2024-03-26T05:05:12.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:12 smithi080 ceph-mon[29691]: 5.3 deep-scrub ok 2024-03-26T05:05:12.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:12 smithi080 ceph-mon[40404]: 5.1b scrub starts 2024-03-26T05:05:12.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:12 smithi080 ceph-mon[40404]: 5.1b scrub ok 2024-03-26T05:05:12.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:12 smithi080 ceph-mon[40404]: 5.3 deep-scrub starts 2024-03-26T05:05:12.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:12 smithi080 ceph-mon[40404]: 5.3 deep-scrub ok 2024-03-26T05:05:12.375 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:12.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:12.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:12.506 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:12.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:12.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:12.557 INFO:tasks.thrashosds.thrasher:j is {'epoch': 153, 'fsid': 'dd09e9cc-eb2c-11ee-95d0-87774f69a715', 'created': '2024-03-26T04:54:51.106954+0000', 'modified': '2024-03-26T05:05:05.893008+0000', 'last_up_change': '2024-03-26T05:00:08.801055+0000', 'last_in_change': '2024-03-26T04:59:50.693588+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T04:58:06.876819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T05:00:20.705417+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T05:00:21.150368+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T05:00:23.638065+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T05:00:25.181000+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T05:00:27.249998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-26T05:02:23.280375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '35a52544-0b0b-47d6-8d6a-571c48ceb366', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6802', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6803', 'nonce': 1840686622}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6804', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6805', 'nonce': 1840686622}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6808', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6809', 'nonce': 1840686622}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6806', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6807', 'nonce': 1840686622}]}, 'public_addr': '172.21.15.80:6803/1840686622', 'cluster_addr': '172.21.15.80:6805/1840686622', 'heartbeat_back_addr': '172.21.15.80:6809/1840686622', 'heartbeat_front_addr': '172.21.15.80:6807/1840686622', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efa5ea33-4f11-49b7-a3f3-f60d5329db23', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6810', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6811', 'nonce': 1726833996}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6812', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6813', 'nonce': 1726833996}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6816', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6817', 'nonce': 1726833996}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6814', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6815', 'nonce': 1726833996}]}, 'public_addr': '172.21.15.80:6811/1726833996', 'cluster_addr': '172.21.15.80:6813/1726833996', 'heartbeat_back_addr': '172.21.15.80:6817/1726833996', 'heartbeat_front_addr': '172.21.15.80:6815/1726833996', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '536bec9b-9636-4e67-b452-ed1a55c22ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6818', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6819', 'nonce': 128713757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6820', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6821', 'nonce': 128713757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6824', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6825', 'nonce': 128713757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6822', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6823', 'nonce': 128713757}]}, 'public_addr': '172.21.15.80:6819/128713757', 'cluster_addr': '172.21.15.80:6821/128713757', 'heartbeat_back_addr': '172.21.15.80:6825/128713757', 'heartbeat_front_addr': '172.21.15.80:6823/128713757', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '640cfdb6-b9c5-446d-8312-504ae12c7818', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6826', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6827', 'nonce': 418771441}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6828', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6829', 'nonce': 418771441}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6832', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6833', 'nonce': 418771441}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6830', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6831', 'nonce': 418771441}]}, 'public_addr': '172.21.15.80:6827/418771441', 'cluster_addr': '172.21.15.80:6829/418771441', 'heartbeat_back_addr': '172.21.15.80:6833/418771441', 'heartbeat_front_addr': '172.21.15.80:6831/418771441', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0cc7822a-ed73-4c79-a30d-1e151ddd6639', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6800', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6801', 'nonce': 1441071423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6802', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6803', 'nonce': 1441071423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6806', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6807', 'nonce': 1441071423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6804', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6805', 'nonce': 1441071423}]}, 'public_addr': '172.21.15.132:6801/1441071423', 'cluster_addr': '172.21.15.132:6803/1441071423', 'heartbeat_back_addr': '172.21.15.132:6807/1441071423', 'heartbeat_front_addr': '172.21.15.132:6805/1441071423', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6808', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6809', 'nonce': 794504625}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6810', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6811', 'nonce': 794504625}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6814', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6815', 'nonce': 794504625}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6812', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6813', 'nonce': 794504625}]}, 'public_addr': '172.21.15.132:6809/794504625', 'cluster_addr': '172.21.15.132:6811/794504625', 'heartbeat_back_addr': '172.21.15.132:6815/794504625', 'heartbeat_front_addr': '172.21.15.132:6813/794504625', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2970bbab-11b2-4d1b-b57e-037bdac089b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6816', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6817', 'nonce': 2209078972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6818', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6819', 'nonce': 2209078972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6822', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6823', 'nonce': 2209078972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6820', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6821', 'nonce': 2209078972}]}, 'public_addr': '172.21.15.132:6817/2209078972', 'cluster_addr': '172.21.15.132:6819/2209078972', 'heartbeat_back_addr': '172.21.15.132:6823/2209078972', 'heartbeat_front_addr': '172.21.15.132:6821/2209078972', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '12f4b466-28f9-469b-8fd0-65b91c6dc22d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6824', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6825', 'nonce': 3655388282}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6826', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6827', 'nonce': 3655388282}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6830', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6831', 'nonce': 3655388282}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6828', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6829', 'nonce': 3655388282}]}, 'public_addr': '172.21.15.132:6825/3655388282', 'cluster_addr': '172.21.15.132:6827/3655388282', 'heartbeat_back_addr': '172.21.15.132:6831/3655388282', 'heartbeat_front_addr': '172.21.15.132:6829/3655388282', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:04:25.384118+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:04:42.933605+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:04:23.556968+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:04:50.016444+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:03:58.633037+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:04:08.450522+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:04:46.632559+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:05:05.645006+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.16', 'mappings': [{'from': 5, 'to': 6}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.80:0/3992539385': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/2770889532': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:6801/2466284312': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/490056571': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1986135300': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/3796611170': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1993138634': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1832498181': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/2466284312': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1650870847': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6801/1418683843': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:0/3147301987': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6801/2770889532': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/571905941': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1451012560': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1235125152': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/1418683843': '2024-03-27T05:01:03.689739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T05:05:12.557 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:05:12.608 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:12.667 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:12.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:12.768 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:12.833 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:12.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:12.934 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:12.968 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:12.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:12.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:13.070 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:13.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[40404]: pgmap v193: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-26T05:05:13.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[40404]: 7.0 scrub starts 2024-03-26T05:05:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[40404]: 4.16 scrub starts 2024-03-26T05:05:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[40404]: 4.16 scrub ok 2024-03-26T05:05:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[40404]: 5.8 deep-scrub starts 2024-03-26T05:05:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[40404]: 5.8 deep-scrub ok 2024-03-26T05:05:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[40404]: 5.10 scrub starts 2024-03-26T05:05:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[40404]: 5.10 scrub ok 2024-03-26T05:05:13.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[40404]: 7.0 scrub ok 2024-03-26T05:05:13.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[29691]: pgmap v193: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-26T05:05:13.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[29691]: 7.0 scrub starts 2024-03-26T05:05:13.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[29691]: 4.16 scrub starts 2024-03-26T05:05:13.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[29691]: 4.16 scrub ok 2024-03-26T05:05:13.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[29691]: 5.8 deep-scrub starts 2024-03-26T05:05:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[29691]: 5.8 deep-scrub ok 2024-03-26T05:05:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[29691]: 5.10 scrub starts 2024-03-26T05:05:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[29691]: 5.10 scrub ok 2024-03-26T05:05:13.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:13 smithi080 ceph-mon[29691]: 7.0 scrub ok 2024-03-26T05:05:13.125 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:13.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:13.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:13.226 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:13.228 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:13.256 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:13.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:13.358 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:13.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:13.402 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:13.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:13.503 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:13.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:13 smithi132 ceph-mon[33500]: pgmap v193: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-26T05:05:13.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:13 smithi132 ceph-mon[33500]: 7.0 scrub starts 2024-03-26T05:05:13.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:13 smithi132 ceph-mon[33500]: 4.16 scrub starts 2024-03-26T05:05:13.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:13 smithi132 ceph-mon[33500]: 4.16 scrub ok 2024-03-26T05:05:13.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:13 smithi132 ceph-mon[33500]: 5.8 deep-scrub starts 2024-03-26T05:05:13.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:13 smithi132 ceph-mon[33500]: 5.8 deep-scrub ok 2024-03-26T05:05:13.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:13 smithi132 ceph-mon[33500]: 5.10 scrub starts 2024-03-26T05:05:13.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:13 smithi132 ceph-mon[33500]: 5.10 scrub ok 2024-03-26T05:05:13.544 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:13 smithi132 ceph-mon[33500]: 7.0 scrub ok 2024-03-26T05:05:13.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:13.550 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:13.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:13.640 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T05:05:13.652 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:13.695 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:13.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:13.797 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:13.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:13.839 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:13.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:13.940 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:13.980 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:13.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:14.013 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:05:14.018 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:05:14.083 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:14.128 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:05:14.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:14.165 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:14.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:14.177 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:14 smithi132 ceph-mon[33500]: 3.1e deep-scrub starts 2024-03-26T05:05:14.177 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:14 smithi132 ceph-mon[33500]: 3.1e deep-scrub ok 2024-03-26T05:05:14.177 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:14 smithi132 ceph-mon[33500]: 6.19 scrub starts 2024-03-26T05:05:14.177 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:14 smithi132 ceph-mon[33500]: 6.19 scrub ok 2024-03-26T05:05:14.177 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:14 smithi132 ceph-mon[33500]: 4.1e scrub starts 2024-03-26T05:05:14.177 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:14 smithi132 ceph-mon[33500]: 4.1e scrub ok 2024-03-26T05:05:14.177 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:14 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1729271672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:05:14.177 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:14 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:05:14.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:14 smithi132 ceph-mon[33500]: 5.1a scrub starts 2024-03-26T05:05:14.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:14 smithi132 ceph-mon[33500]: 5.1a scrub ok 2024-03-26T05:05:14.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:14 smithi132 ceph-mon[33500]: 7.d scrub starts 2024-03-26T05:05:14.266 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:14.295 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:14.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:14.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[40404]: 3.1e deep-scrub starts 2024-03-26T05:05:14.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[40404]: 3.1e deep-scrub ok 2024-03-26T05:05:14.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[40404]: 6.19 scrub starts 2024-03-26T05:05:14.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[40404]: 6.19 scrub ok 2024-03-26T05:05:14.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[40404]: 4.1e scrub starts 2024-03-26T05:05:14.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[40404]: 4.1e scrub ok 2024-03-26T05:05:14.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1729271672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:05:14.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:05:14.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[40404]: 5.1a scrub starts 2024-03-26T05:05:14.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[40404]: 5.1a scrub ok 2024-03-26T05:05:14.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[40404]: 7.d scrub starts 2024-03-26T05:05:14.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[29691]: 3.1e deep-scrub starts 2024-03-26T05:05:14.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[29691]: 3.1e deep-scrub ok 2024-03-26T05:05:14.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[29691]: 6.19 scrub starts 2024-03-26T05:05:14.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[29691]: 6.19 scrub ok 2024-03-26T05:05:14.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[29691]: 4.1e scrub starts 2024-03-26T05:05:14.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[29691]: 4.1e scrub ok 2024-03-26T05:05:14.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1729271672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:05:14.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:05:14.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[29691]: 5.1a scrub starts 2024-03-26T05:05:14.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[29691]: 5.1a scrub ok 2024-03-26T05:05:14.369 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-mon[29691]: 7.d scrub starts 2024-03-26T05:05:14.370 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:05:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:05:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:05:14.397 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:14.441 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:14.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:14.542 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:14.572 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:14.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:14.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:14.674 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:14.706 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:14.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:14.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:14.832 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:14.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:14.934 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:14.966 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:14.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:15.067 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:15.106 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:15.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:15.203 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:15 smithi132 ceph-mon[33500]: 4.0 scrub starts 2024-03-26T05:05:15.203 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:15 smithi132 ceph-mon[33500]: 4.0 scrub ok 2024-03-26T05:05:15.203 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:15 smithi132 ceph-mon[33500]: pgmap v194: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 626 op/s 2024-03-26T05:05:15.203 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:15 smithi132 ceph-mon[33500]: 4.12 scrub starts 2024-03-26T05:05:15.204 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:15 smithi132 ceph-mon[33500]: 4.12 scrub ok 2024-03-26T05:05:15.204 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:15 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:05:15.204 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:15 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:05:15.204 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:15 smithi132 ceph-mon[33500]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T05:05:15.204 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:05:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:05:15.208 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:15.248 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:15.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:15.350 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:15.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[40404]: 4.0 scrub starts 2024-03-26T05:05:15.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[40404]: 4.0 scrub ok 2024-03-26T05:05:15.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[40404]: pgmap v194: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 626 op/s 2024-03-26T05:05:15.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[40404]: 4.12 scrub starts 2024-03-26T05:05:15.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[40404]: 4.12 scrub ok 2024-03-26T05:05:15.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:05:15.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:05:15.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[40404]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T05:05:15.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[29691]: 4.0 scrub starts 2024-03-26T05:05:15.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[29691]: 4.0 scrub ok 2024-03-26T05:05:15.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[29691]: pgmap v194: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 626 op/s 2024-03-26T05:05:15.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[29691]: 4.12 scrub starts 2024-03-26T05:05:15.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[29691]: 4.12 scrub ok 2024-03-26T05:05:15.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:05:15.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:05:15.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:15 smithi080 ceph-mon[29691]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T05:05:15.381 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:15.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:15.483 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:15.511 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:15.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:15.613 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:15.655 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:15.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:15.757 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:15.793 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:15.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:15.897 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:15.917 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:05:15.917 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:05:15.917 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2024-03-26T05:05:15.917 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2024-03-26T05:05:15.918 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:05:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:15.941 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:15.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:16.042 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:16.080 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:16.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:16.182 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:16.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:16.222 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:16.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:16.323 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:16.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:16.347 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:16.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:16.370 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:16 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:16.370 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:16 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:16.449 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:16.466 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:16 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:16.483 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:16.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:16.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:16.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:16.618 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:16.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:16.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:16.710 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T05:05:16.721 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:16.781 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:16.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:16.883 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:16.920 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:16.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:17.020 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:17.041 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:05:17.060 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:17.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:17.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:17.162 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:17.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:17.202 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:17.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:17.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:17 smithi132 ceph-mon[33500]: pgmap v196: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-26T05:05:17.303 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:17.337 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:17.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:17.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:17 smithi080 ceph-mon[29691]: pgmap v196: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-26T05:05:17.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:17 smithi080 ceph-mon[40404]: pgmap v196: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-26T05:05:17.439 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:17.472 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:17.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:17.573 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:17.610 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:17.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:17.711 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:17.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:17.741 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:17.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:17.842 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:17.878 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:17.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:17.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:17.980 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:18.015 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:18.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:18.116 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:18.153 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:18.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:18.254 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:18.304 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:18.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:18.405 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:18.443 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:18.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:18.545 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:18.574 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:18.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:18.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:18.676 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:18.716 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:18.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:18.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:18.818 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:18.850 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:18.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:18.952 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:18.996 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:18.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:19.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:19.129 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:19.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:19.230 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:19.259 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:19.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:19.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:19.288 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:19 smithi132 ceph-mon[33500]: pgmap v197: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T05:05:19.361 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:19.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:19 smithi080 ceph-mon[29691]: pgmap v197: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T05:05:19.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:19 smithi080 ceph-mon[40404]: pgmap v197: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T05:05:19.399 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:19.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:19.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:19.457 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:05:19.457 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:05:19.457 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22071 7.7 7:efcbc97c:::benchmark_data_smithi080_80848_object22070:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.028083+0000", 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 152.43511709200001, 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11719394700000001, 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22071 2024-03-26T05:05:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.028083+0000", 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.028083+0000", 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.028084+0000", 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.215e-06 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.028163+0000", 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8566999999999995e-05 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.028177+0000", 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4679e-05 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.028187+0000", 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5810000000000003e-06 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.028232+0000", 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5328999999999999e-05 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.028357+0000", 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012492100000000001 2024-03-26T05:05:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.028502+0000", 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014456199999999999 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.029420+0000", 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000918304 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.031210+0000", 2024-03-26T05:05:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0017894129999999999 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.145201+0000", 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.113991627 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.145246+0000", 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5012000000000002e-05 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.145277+0000", 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0737000000000001e-05 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22080 7.d 7:bdabe7f2:::benchmark_data_smithi080_80848_object22079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.030530+0000", 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 152.432670011, 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.119740192, 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22080 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.030530+0000", 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.030530+0000", 2024-03-26T05:05:19.465 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.030531+0000", 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3099999999999996e-07 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.030601+0000", 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0129999999999994e-05 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.030610+0000", 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6629999999999999e-06 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.030615+0000", 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7359999999999998e-06 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.030670+0000", 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4203000000000003e-05 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.467 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.030785+0000", 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000114911 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.030931+0000", 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014686500000000001 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.031896+0000", 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096429600000000001 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.032350+0000", 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00045379099999999999 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.150159+0000", 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11780913799999999 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.150221+0000", 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2697e-05 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.150270+0000", 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8931000000000003e-05 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22566 7.c 7:39b9bfaa:::benchmark_data_smithi080_80848_object22565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.344062+0000", 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 152.119138217, 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14012134700000001, 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.470 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.471 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22566 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344062+0000", 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344062+0000", 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.473 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.473 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344063+0000", 2024-03-26T05:05:19.473 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5899999999999995e-07 2024-03-26T05:05:19.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.473 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.474 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344391+0000", 2024-03-26T05:05:19.474 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00032832899999999998 2024-03-26T05:05:19.474 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.474 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.474 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.474 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344402+0000", 2024-03-26T05:05:19.474 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0961999999999999e-05 2024-03-26T05:05:19.474 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.474 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.475 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.475 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344410+0000", 2024-03-26T05:05:19.475 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8080000000000005e-06 2024-03-26T05:05:19.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.475 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344468+0000", 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7772999999999997e-05 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344596+0000", 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012822500000000001 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344723+0000", 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012701499999999999 2024-03-26T05:05:19.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.346541+0000", 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001818442 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.346698+0000", 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000156188 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.484093+0000", 2024-03-26T05:05:19.477 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13739572 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.484138+0000", 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4437000000000002e-05 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.484183+0000", 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5589e-05 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22569 7.c 7:350aed00:::benchmark_data_smithi080_80848_object22568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.345174+0000", 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 152.11802567699999, 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13912730000000001, 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22569 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.479 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345174+0000", 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345174+0000", 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345175+0000", 2024-03-26T05:05:19.480 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5799999999999998e-07 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345207+0000", 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.1272000000000001e-05 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345217+0000", 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9480000000000003e-06 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.481 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345223+0000", 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9129999999999997e-06 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345625+0000", 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040126199999999999 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345710+0000", 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5051999999999998e-05 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345853+0000", 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000143602 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.346703+0000", 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084936400000000002 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.346871+0000", 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016788400000000001 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.484228+0000", 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13735725600000001 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.484264+0000", 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6599e-05 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.484302+0000", 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7289999999999997e-05 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.484 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22574 7.7 7:e024eb0a:::benchmark_data_smithi080_80848_object22573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.347763+0000", 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 152.115436843, 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14220207000000001, 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22574 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.485 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347763+0000", 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347763+0000", 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.486 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347764+0000", 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2799999999999995e-07 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347850+0000", 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5749999999999997e-05 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347864+0000", 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4154999999999999e-05 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347871+0000", 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4340000000000003e-06 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347935+0000", 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3559999999999995e-05 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.348029+0000", 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3696999999999994e-05 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.348119+0000", 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0000000000000006e-05 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.349313+0000", 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011938999999999999 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.349659+0000", 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034683400000000002 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.489846+0000", 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.140186273 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.489910+0000", 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4615000000000004e-05 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.489965+0000", 2024-03-26T05:05:19.490 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5024e-05 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31380 7.d 7:bbfbc35c:::benchmark_data_smithi080_80848_object31379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.249199+0000", 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 147.21400092299999, 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13902161499999999, 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31380 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249199+0000", 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249199+0000", 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.492 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249200+0000", 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8899999999999998e-07 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249321+0000", 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120458 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249333+0000", 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2082000000000001e-05 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249338+0000", 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5790000000000003e-06 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249394+0000", 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5948e-05 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249535+0000", 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000141225 2024-03-26T05:05:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249671+0000", 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000135594 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.385763+0000", 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13609241499999999 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388062+0000", 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002299054 2024-03-26T05:05:19.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388115+0000", 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2466999999999998e-05 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388175+0000", 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0009000000000001e-05 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388221+0000", 2024-03-26T05:05:19.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5895e-05 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31387 7.d 7:b9b9bc8e:::benchmark_data_smithi080_80848_object31386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.249860+0000", 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 147.21334035000001, 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14199305400000001, 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31387 2024-03-26T05:05:19.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249860+0000", 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249860+0000", 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.498 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249861+0000", 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4700000000000001e-07 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249943+0000", 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2377000000000001e-05 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249953+0000", 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8660000000000001e-06 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249961+0000", 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4740000000000005e-06 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250798+0000", 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00083652499999999996 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.382512+0000", 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.131714359 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.382690+0000", 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017777799999999999 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388005+0000", 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053151559999999997 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391615+0000", 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.003609929 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391752+0000", 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013692399999999999 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391811+0000", 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9286999999999997e-05 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391853+0000", 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1631999999999997e-05 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.502 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31388 7.d 7:b6deb006:::benchmark_data_smithi080_80848_object31387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.249986+0000", 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 147.213214561, 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14203564299999999, 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31388 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.503 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249986+0000", 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249986+0000", 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249986+0000", 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3799999999999997e-07 2024-03-26T05:05:19.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250070+0000", 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3956999999999996e-05 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250078+0000", 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0299999999999994e-06 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250083+0000", 2024-03-26T05:05:19.505 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.442e-06 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384264+0000", 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13418155200000001 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384385+0000", 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012047100000000001 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384584+0000", 2024-03-26T05:05:19.506 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00019931699999999999 2024-03-26T05:05:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388074+0000", 2024-03-26T05:05:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00349016 2024-03-26T05:05:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391670+0000", 2024-03-26T05:05:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0035954469999999999 2024-03-26T05:05:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.508 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.508 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391939+0000", 2024-03-26T05:05:19.508 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000269466 2024-03-26T05:05:19.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.508 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.508 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.508 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391985+0000", 2024-03-26T05:05:19.508 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6331000000000001e-05 2024-03-26T05:05:19.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.508 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.508 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.392021+0000", 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5831999999999998e-05 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31394 7.d 7:bacf0b5e:::benchmark_data_smithi080_80848_object31393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.250540+0000", 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 147.21266011399999, 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14170165000000001, 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.509 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31394 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250540+0000", 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.510 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250540+0000", 2024-03-26T05:05:19.511 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.511 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.511 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250541+0000", 2024-03-26T05:05:19.511 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0500000000000002e-07 2024-03-26T05:05:19.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.511 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.511 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250698+0000", 2024-03-26T05:05:19.511 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015699799999999999 2024-03-26T05:05:19.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250709+0000", 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0939e-05 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250721+0000", 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2531000000000001e-05 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.385794+0000", 2024-03-26T05:05:19.512 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13507309000000001 2024-03-26T05:05:19.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.513 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.513 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.385910+0000", 2024-03-26T05:05:19.513 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115531 2024-03-26T05:05:19.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.513 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:05:19.513 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.386052+0000", 2024-03-26T05:05:19.513 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014174899999999999 2024-03-26T05:05:19.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.514 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:19.515 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.515 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.515 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391713+0000", 2024-03-26T05:05:19.515 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.005661831 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.392060+0000", 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000346998 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.392154+0000", 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3131e-05 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.516 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.392202+0000", 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8689000000000001e-05 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.392242+0000", 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9357999999999998e-05 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.24701.0:31392 7.b e111/103 7:d4d557cd:::benchmark_data_smithi080_80848_object31391:head v 111'1944, mlcod=111'1941)", 2024-03-26T05:05:19.517 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.252059+0000", 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 147.211141037, 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13566467500000001, 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252059+0000", 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.518 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252059+0000", 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252060+0000", 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9199999999999999e-07 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252111+0000", 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1381000000000002e-05 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384561+0000", 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13244936700000001 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384572+0000", 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0983e-05 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384626+0000", 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4063000000000001e-05 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384648+0000", 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.2167999999999998e-05 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387664+0000", 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00301623 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387724+0000", 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9490999999999997e-05 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.521 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.24701.0:31393 7.1 e111/103 7:8ed06db3:::benchmark_data_smithi080_80848_object31392:head v 111'1956, mlcod=111'1954)", 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.252281+0000", 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 147.21091953800001, 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13488787599999999, 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252281+0000", 2024-03-26T05:05:19.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252281+0000", 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252281+0000", 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8400000000000004e-07 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252301+0000", 2024-03-26T05:05:19.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.9760999999999999e-05 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383132+0000", 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13083125600000001 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383147+0000", 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4747e-05 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383219+0000", 2024-03-26T05:05:19.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1748999999999994e-05 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383258+0000", 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8977999999999998e-05 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387124+0000", 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0038662129999999999 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387168+0000", 2024-03-26T05:05:19.525 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4487999999999998e-05 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(client.24701.0:31380 7.d e111/103 ondisk, result = 0)", 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.252755+0000", 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 147.210444975, 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13531982300000001, 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252755+0000", 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252755+0000", 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252756+0000", 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3000000000000001e-07 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252758+0000", 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.6759999999999999e-06 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384584+0000", 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.131825893 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384592+0000", 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8320000000000006e-06 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388049+0000", 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0034569150000000001 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388056+0000", 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2069999999999998e-06 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388075+0000", 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.8770000000000002e-05 2024-03-26T05:05:19.529 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:122222 7.7 7:e926a959:::benchmark_data_smithi080_80848_object2723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:11.511588+0000", 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 67.951611798000002, 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13703875800000001, 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.530 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 122222 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511588+0000", 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511588+0000", 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.531 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511590+0000", 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.392e-06 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511595+0000", 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.772e-06 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511610+0000", 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4817000000000001e-05 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511619+0000", 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3850000000000002e-06 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511679+0000", 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9858000000000002e-05 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511808+0000", 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012915100000000001 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511953+0000", 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000144878 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.514360+0000", 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0024071679999999999 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.566701+0000", 2024-03-26T05:05:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.052341721000000001 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.648533+0000", 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.081831423 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.648577+0000", 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3704e-05 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.648627+0000", 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0488999999999997e-05 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:157212 7.d 7:b26765b3:::benchmark_data_smithi080_80848_object37713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:46.726199+0000", 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 32.737001116000002, 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.12664176299999999, 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 157212 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.726199+0000", 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.726199+0000", 2024-03-26T05:05:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.726201+0000", 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5629999999999999e-06 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.726207+0000", 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8390000000000001e-06 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.726222+0000", 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4562000000000001e-05 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.726231+0000", 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6969999999999999e-06 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.727331+0000", 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011001489999999999 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.539 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.727455+0000", 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000123843 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.727569+0000", 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011413000000000001 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.730942+0000", 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0033731759999999999 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.733268+0000", 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0023263250000000002 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.852771+0000", 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.119502763 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.852808+0000", 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7085000000000002e-05 2024-03-26T05:05:19.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.852841+0000", 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2631e-05 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:157220 7.7 7:e3aaf5eb:::benchmark_data_smithi080_80848_object37721:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:46.732130+0000", 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 32.731069660999999, 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.128509334, 2024-03-26T05:05:19.542 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 157220 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732130+0000", 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.543 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732130+0000", 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732132+0000", 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.784e-06 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732137+0000", 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9830000000000004e-06 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.544 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732148+0000", 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0933e-05 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732155+0000", 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9469999999999997e-06 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732237+0000", 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.1747000000000002e-05 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.545 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732372+0000", 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000134697 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732509+0000", 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013783099999999999 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.739488+0000", 2024-03-26T05:05:19.546 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0069783199999999997 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.859410+0000", 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11992253899999999 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.860516+0000", 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011055870000000001 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.547 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.860589+0000", 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3193000000000002e-05 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.860640+0000", 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0773000000000003e-05 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:157222 7.c 7:34243d50:::benchmark_data_smithi080_80848_object37723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:46.732599+0000", 2024-03-26T05:05:19.548 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 32.730601544999999, 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.12243174699999999, 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 157222 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732599+0000", 2024-03-26T05:05:19.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732599+0000", 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732599+0000", 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5199999999999996e-07 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.550 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732603+0000", 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8659999999999999e-06 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732611+0000", 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5449999999999998e-06 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732617+0000", 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5200000000000003e-06 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732663+0000", 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5843000000000001e-05 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732759+0000", 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6314e-05 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732858+0000", 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8486999999999994e-05 2024-03-26T05:05:19.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.736572+0000", 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0037138869999999999 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.737677+0000", 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011053110000000001 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.854932+0000", 2024-03-26T05:05:19.553 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.117254436 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.854985+0000", 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3235000000000001e-05 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.855030+0000", 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5550999999999998e-05 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:157223 7.7 7:ea38fec5:::benchmark_data_smithi080_80848_object37724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:46.734236+0000", 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 32.728963667000002, 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126551473, 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 157223 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.555 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.734236+0000", 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.734236+0000", 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.734238+0000", 2024-03-26T05:05:19.556 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3549999999999999e-06 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.734243+0000", 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7029999999999997e-06 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.734253+0000", 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0710000000000001e-05 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.734260+0000", 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2339999999999997e-06 2024-03-26T05:05:19.557 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.736485+0000", 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0022243499999999999 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.736612+0000", 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012725099999999999 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.736733+0000", 2024-03-26T05:05:19.558 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000121206 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.739564+0000", 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0028307889999999998 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.859455+0000", 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.119890754 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.559 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.860687+0000", 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0012317739999999999 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.860743+0000", 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6700999999999998e-05 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.860788+0000", 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4645999999999999e-05 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.560 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:186914 7.d 7:b07c0e48:::benchmark_data_smithi080_80848_object67415:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:16.759339+0000", 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.7038612029999998, 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13061483300000001, 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 186914 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.561 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759339+0000", 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759339+0000", 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759340+0000", 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4869999999999999e-06 2024-03-26T05:05:19.562 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759346+0000", 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0820000000000004e-06 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759363+0000", 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6432000000000001e-05 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759371+0000", 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.439e-06 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.563 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759437+0000", 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5537000000000006e-05 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759562+0000", 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012494399999999999 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759685+0000", 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012313600000000001 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.564 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.766825+0000", 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0071396130000000004 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.769012+0000", 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002187446 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.889841+0000", 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.120828515 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.565 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.889895+0000", 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4341e-05 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.889954+0000", 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8860999999999998e-05 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:186918 7.d 7:b8fb59a9:::benchmark_data_smithi080_80848_object67419:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:05:19.566 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:16.763188+0000", 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.7000121400000001, 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.136308077, 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 186918 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763188+0000", 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763188+0000", 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763189+0000", 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2720000000000001e-06 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763194+0000", 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8280000000000002e-06 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.568 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763206+0000", 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1960000000000001e-05 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763214+0000", 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9589999999999995e-06 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763287+0000", 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3464e-05 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.569 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763436+0000", 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000148677 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763564+0000", 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012827199999999999 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.766835+0000", 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.003270102 2024-03-26T05:05:19.570 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.769152+0000", 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0023173080000000001 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.899215+0000", 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13006367099999999 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.571 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.899456+0000", 2024-03-26T05:05:19.572 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00024002800000000001 2024-03-26T05:05:19.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.572 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.572 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.572 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.899496+0000", 2024-03-26T05:05:19.572 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0535999999999999e-05 2024-03-26T05:05:19.572 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.572 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.572 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.572 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:188393 7.7 7:e4e756db:::benchmark_data_smithi080_80848_object68894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:18.441094+0000", 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 1.0221057579999999, 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.114167216, 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 188393 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.573 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.441094+0000", 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.441094+0000", 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.441096+0000", 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.314e-06 2024-03-26T05:05:19.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.441100+0000", 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4530000000000004e-06 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.441112+0000", 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.152e-05 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.441119+0000", 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3259999999999998e-06 2024-03-26T05:05:19.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.441171+0000", 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1842000000000003e-05 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.441285+0000", 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000113985 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.441399+0000", 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011395699999999999 2024-03-26T05:05:19.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.444422+0000", 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0030230439999999999 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.445159+0000", 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00073759499999999996 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.555168+0000", 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.110008626 2024-03-26T05:05:19.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.555214+0000", 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6396000000000001e-05 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.555262+0000", 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7157999999999998e-05 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:19.579 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:05:19.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:19.609 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:19.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:19.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:19.741 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:19.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:19.834 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:05:19.837 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:05:19.843 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:19.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:19.880 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:19.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:19.981 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:20.010 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:20.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:20.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:20.112 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:20.153 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:20.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:20.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:20.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:20.286 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:20.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:20.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:20.388 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:20.430 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:20.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:20.531 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:20.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:20.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:20.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:20 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:05:20.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:20 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:05:20.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:20.573 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:20.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:20.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:20.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:20.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:20 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:05:20.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:20 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:05:20.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:20.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:20.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:20 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:05:20.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:20 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:05:20.675 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:20.748 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T05:05:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:20.770 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:20.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:20.873 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:20.913 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:20.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:20.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:21.002 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-03-26T05:05:19.795949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3418823030,"num_objects":52389,"num_object_clones":0,"num_object_copies":157167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52389,"num_whiteouts":0,"num_read":1514,"num_read_kb":1328,"num_write":187642,"num_write_kb":7650206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90467,"ondisk_log_size":90467,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10992760,"kb_used_data":10130672,"kb_used_omap":12,"kb_used_meta":861555,"kb_avail":738968456,"statfs":{"total":767960285184,"available":756703698944,"internally_reserved":0,"allocated":10373808128,"data_stored":10371451830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":882232880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,2,9,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-662044672,"num_objects":-10102,"num_object_clones":0,"num_object_copies":-30306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10102,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10102,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002928"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":190,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108263+0000","last_change":"2024-03-26T05:04:18.210889+0000","last_active":"2024-03-26T05:05:14.108263+0000","last_peered":"2024-03-26T05:05:14.108263+0000","last_clean":"2024-03-26T05:05:14.108263+0000","last_became_active":"2024-03-26T05:00:28.972969+0000","last_became_peered":"2024-03-26T05:00:28.972969+0000","last_unstale":"2024-03-26T05:05:14.108263+0000","last_undegraded":"2024-03-26T05:05:14.108263+0000","last_fullsized":"2024-03-26T05:05:14.108263+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:18.210754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:18.210754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:32.582721+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.127438+0000","last_change":"2024-03-26T05:05:02.872166+0000","last_active":"2024-03-26T05:05:14.127438+0000","last_peered":"2024-03-26T05:05:14.127438+0000","last_clean":"2024-03-26T05:05:14.127438+0000","last_became_active":"2024-03-26T05:00:26.953154+0000","last_became_peered":"2024-03-26T05:00:26.953154+0000","last_unstale":"2024-03-26T05:05:14.127438+0000","last_undegraded":"2024-03-26T05:05:14.127438+0000","last_fullsized":"2024-03-26T05:05:14.127438+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:02.872031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:02.872031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:28.808012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":211,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.130211+0000","last_change":"2024-03-26T05:05:04.016954+0000","last_active":"2024-03-26T05:05:14.130211+0000","last_peered":"2024-03-26T05:05:14.130211+0000","last_clean":"2024-03-26T05:05:14.130211+0000","last_became_active":"2024-03-26T05:00:23.346607+0000","last_became_peered":"2024-03-26T05:00:23.346607+0000","last_unstale":"2024-03-26T05:05:14.130211+0000","last_undegraded":"2024-03-26T05:05:14.130211+0000","last_fullsized":"2024-03-26T05:05:14.130211+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.016730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.016730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:32.208697+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'15","reported_seq":227,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108418+0000","last_change":"2024-03-26T05:05:06.144447+0000","last_active":"2024-03-26T05:05:14.108418+0000","last_peered":"2024-03-26T05:05:14.108418+0000","last_clean":"2024-03-26T05:05:14.108418+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:05:14.108418+0000","last_undegraded":"2024-03-26T05:05:14.108418+0000","last_fullsized":"2024-03-26T05:05:14.108418+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:05:06.144208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:06.144208+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:11.936404+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":181,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.141504+0000","last_change":"2024-03-26T05:03:00.192980+0000","last_active":"2024-03-26T05:05:14.141504+0000","last_peered":"2024-03-26T05:05:14.141504+0000","last_clean":"2024-03-26T05:05:14.141504+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:05:14.141504+0000","last_undegraded":"2024-03-26T05:05:14.141504+0000","last_fullsized":"2024-03-26T05:05:14.141504+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:00.192732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:00.192732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":195,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108275+0000","last_change":"2024-03-26T05:05:09.143423+0000","last_active":"2024-03-26T05:05:14.108275+0000","last_peered":"2024-03-26T05:05:14.108275+0000","last_clean":"2024-03-26T05:05:14.108275+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:05:14.108275+0000","last_undegraded":"2024-03-26T05:05:14.108275+0000","last_fullsized":"2024-03-26T05:05:14.108275+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:09.143305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:09.143305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:34.674421+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.142130+0000","last_change":"2024-03-26T05:05:12.713135+0000","last_active":"2024-03-26T05:05:14.142130+0000","last_peered":"2024-03-26T05:05:14.142130+0000","last_clean":"2024-03-26T05:05:14.142130+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:05:14.142130+0000","last_undegraded":"2024-03-26T05:05:14.142130+0000","last_fullsized":"2024-03-26T05:05:14.142130+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.713011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:30.694974+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'39","reported_seq":270,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.148704+0000","last_change":"2024-03-26T05:04:20.637111+0000","last_active":"2024-03-26T05:05:14.148704+0000","last_peered":"2024-03-26T05:05:14.148704+0000","last_clean":"2024-03-26T05:05:14.148704+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:05:14.148704+0000","last_undegraded":"2024-03-26T05:05:14.148704+0000","last_fullsized":"2024-03-26T05:05:14.148704+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'39","last_scrub_stamp":"2024-03-26T05:04:20.636687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:20.636687+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:31.492803+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":198,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.123531+0000","last_change":"2024-03-26T05:04:58.964528+0000","last_active":"2024-03-26T05:05:14.123531+0000","last_peered":"2024-03-26T05:05:14.123531+0000","last_clean":"2024-03-26T05:05:14.123531+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:05:14.123531+0000","last_undegraded":"2024-03-26T05:05:14.123531+0000","last_fullsized":"2024-03-26T05:05:14.123531+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:58.963801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.963801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:21.663238+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.131155+0000","last_change":"2024-03-26T05:05:10.923094+0000","last_active":"2024-03-26T05:05:14.131155+0000","last_peered":"2024-03-26T05:05:14.131155+0000","last_clean":"2024-03-26T05:05:14.131155+0000","last_became_active":"2024-03-26T05:00:26.956013+0000","last_became_peered":"2024-03-26T05:00:26.956013+0000","last_unstale":"2024-03-26T05:05:14.131155+0000","last_undegraded":"2024-03-26T05:05:14.131155+0000","last_fullsized":"2024-03-26T05:05:14.131155+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:10.922975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:10.922975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:18.665098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.130421+0000","last_change":"2024-03-26T05:04:18.640724+0000","last_active":"2024-03-26T05:05:14.130421+0000","last_peered":"2024-03-26T05:05:14.130421+0000","last_clean":"2024-03-26T05:05:14.130421+0000","last_became_active":"2024-03-26T05:00:22.768098+0000","last_became_peered":"2024-03-26T05:00:22.768098+0000","last_unstale":"2024-03-26T05:05:14.130421+0000","last_undegraded":"2024-03-26T05:05:14.130421+0000","last_fullsized":"2024-03-26T05:05:14.130421+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:18.640616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:18.640616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:32.003176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'21","reported_seq":238,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108501+0000","last_change":"2024-03-26T05:03:18.506773+0000","last_active":"2024-03-26T05:05:14.108501+0000","last_peered":"2024-03-26T05:05:14.108501+0000","last_clean":"2024-03-26T05:05:14.108501+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:05:14.108501+0000","last_undegraded":"2024-03-26T05:05:14.108501+0000","last_fullsized":"2024-03-26T05:05:14.108501+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:03:18.506680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:18.506680+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":205,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.129503+0000","last_change":"2024-03-26T05:05:12.931165+0000","last_active":"2024-03-26T05:05:14.129503+0000","last_peered":"2024-03-26T05:05:14.129503+0000","last_clean":"2024-03-26T05:05:14.129503+0000","last_became_active":"2024-03-26T05:00:28.982276+0000","last_became_peered":"2024-03-26T05:00:28.982276+0000","last_unstale":"2024-03-26T05:05:14.129503+0000","last_undegraded":"2024-03-26T05:05:14.129503+0000","last_fullsized":"2024-03-26T05:05:14.129503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:12.931045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.931045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:35.489502+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":200,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.117658+0000","last_change":"2024-03-26T05:05:13.726750+0000","last_active":"2024-03-26T05:05:14.117658+0000","last_peered":"2024-03-26T05:05:14.117658+0000","last_clean":"2024-03-26T05:05:14.117658+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:05:14.117658+0000","last_undegraded":"2024-03-26T05:05:14.117658+0000","last_fullsized":"2024-03-26T05:05:14.117658+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:13.726612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:13.726612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:35.028669+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.130172+0000","last_change":"2024-03-26T05:04:10.613465+0000","last_active":"2024-03-26T05:05:14.130172+0000","last_peered":"2024-03-26T05:05:14.130172+0000","last_clean":"2024-03-26T05:05:14.130172+0000","last_became_active":"2024-03-26T05:00:22.766675+0000","last_became_peered":"2024-03-26T05:00:22.766675+0000","last_unstale":"2024-03-26T05:05:14.130172+0000","last_undegraded":"2024-03-26T05:05:14.130172+0000","last_fullsized":"2024-03-26T05:05:14.130172+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:10.613346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:10.613346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:19.750451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'11","reported_seq":212,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108434+0000","last_change":"2024-03-26T05:03:10.376790+0000","last_active":"2024-03-26T05:05:14.108434+0000","last_peered":"2024-03-26T05:05:14.108434+0000","last_clean":"2024-03-26T05:05:14.108434+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:05:14.108434+0000","last_undegraded":"2024-03-26T05:05:14.108434+0000","last_fullsized":"2024-03-26T05:05:14.108434+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-26T05:03:10.376657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:10.376657+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":199,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.109083+0000","last_change":"2024-03-26T05:04:56.073891+0000","last_active":"2024-03-26T05:05:14.109083+0000","last_peered":"2024-03-26T05:05:14.109083+0000","last_clean":"2024-03-26T05:05:14.109083+0000","last_became_active":"2024-03-26T05:00:29.461032+0000","last_became_peered":"2024-03-26T05:00:29.461032+0000","last_unstale":"2024-03-26T05:05:14.109083+0000","last_undegraded":"2024-03-26T05:05:14.109083+0000","last_fullsized":"2024-03-26T05:05:14.109083+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:56.073763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:04:56.073763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:25.502690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":193,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.128784+0000","last_change":"2024-03-26T05:03:15.886716+0000","last_active":"2024-03-26T05:05:14.128784+0000","last_peered":"2024-03-26T05:05:14.128784+0000","last_clean":"2024-03-26T05:05:14.128784+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:05:14.128784+0000","last_undegraded":"2024-03-26T05:05:14.128784+0000","last_fullsized":"2024-03-26T05:05:14.128784+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:15.885586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:15.885586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.121360+0000","last_change":"2024-03-26T05:05:08.044042+0000","last_active":"2024-03-26T05:05:14.121360+0000","last_peered":"2024-03-26T05:05:14.121360+0000","last_clean":"2024-03-26T05:05:14.121360+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:05:14.121360+0000","last_undegraded":"2024-03-26T05:05:14.121360+0000","last_fullsized":"2024-03-26T05:05:14.121360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:08.043732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:09.914470+0000","last_clean_scrub_stamp":"2024-03-26T05:05:08.043732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:25.734644+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":230,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.123255+0000","last_change":"2024-03-26T05:05:00.356865+0000","last_active":"2024-03-26T05:05:14.123255+0000","last_peered":"2024-03-26T05:05:14.123255+0000","last_clean":"2024-03-26T05:05:14.123255+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:05:14.123255+0000","last_undegraded":"2024-03-26T05:05:14.123255+0000","last_fullsized":"2024-03-26T05:05:14.123255+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:05:00.356704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:00.356704+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:22.884134+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":190,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.143283+0000","last_change":"2024-03-26T05:03:05.268194+0000","last_active":"2024-03-26T05:05:14.143283+0000","last_peered":"2024-03-26T05:05:14.143283+0000","last_clean":"2024-03-26T05:05:14.143283+0000","last_became_active":"2024-03-26T05:00:29.463238+0000","last_became_peered":"2024-03-26T05:00:29.463238+0000","last_unstale":"2024-03-26T05:05:14.143283+0000","last_undegraded":"2024-03-26T05:05:14.143283+0000","last_fullsized":"2024-03-26T05:05:14.143283+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:05.268043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:05.268043+0000","last_clean_scrub_stamp":"2024-03-26T05:03:05.268043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":194,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108507+0000","last_change":"2024-03-26T05:04:21.167015+0000","last_active":"2024-03-26T05:05:14.108507+0000","last_peered":"2024-03-26T05:05:14.108507+0000","last_clean":"2024-03-26T05:05:14.108507+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:05:14.108507+0000","last_undegraded":"2024-03-26T05:05:14.108507+0000","last_fullsized":"2024-03-26T05:05:14.108507+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:21.166893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.166893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:45.803222+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108602+0000","last_change":"2024-03-26T05:04:01.071006+0000","last_active":"2024-03-26T05:05:14.108602+0000","last_peered":"2024-03-26T05:05:14.108602+0000","last_clean":"2024-03-26T05:05:14.108602+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:05:14.108602+0000","last_undegraded":"2024-03-26T05:05:14.108602+0000","last_fullsized":"2024-03-26T05:05:14.108602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:01.070878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:04:01.070878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:14.920867+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'20","reported_seq":234,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.140249+0000","last_change":"2024-03-26T05:04:07.528781+0000","last_active":"2024-03-26T05:05:14.140249+0000","last_peered":"2024-03-26T05:05:14.140249+0000","last_clean":"2024-03-26T05:05:14.140249+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:05:14.140249+0000","last_undegraded":"2024-03-26T05:05:14.140249+0000","last_fullsized":"2024-03-26T05:05:14.140249+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:04:07.528619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:07.528619+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:36.984300+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":189,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.118466+0000","last_change":"2024-03-26T05:03:13.901836+0000","last_active":"2024-03-26T05:05:14.118466+0000","last_peered":"2024-03-26T05:05:14.118466+0000","last_clean":"2024-03-26T05:05:14.118466+0000","last_became_active":"2024-03-26T05:00:28.981505+0000","last_became_peered":"2024-03-26T05:00:28.981505+0000","last_unstale":"2024-03-26T05:05:14.118466+0000","last_undegraded":"2024-03-26T05:05:14.118466+0000","last_fullsized":"2024-03-26T05:05:14.118466+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:03:13.901714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:13.901714+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":272,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.129431+0000","last_change":"2024-03-26T05:04:18.963951+0000","last_active":"2024-03-26T05:05:14.129431+0000","last_peered":"2024-03-26T05:05:14.129431+0000","last_clean":"2024-03-26T05:05:14.129431+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:05:14.129431+0000","last_undegraded":"2024-03-26T05:05:14.129431+0000","last_fullsized":"2024-03-26T05:05:14.129431+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:18.963855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:18.963855+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:20.821553+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.127531+0000","last_change":"2024-03-26T05:04:07.578139+0000","last_active":"2024-03-26T05:05:14.127531+0000","last_peered":"2024-03-26T05:05:14.127531+0000","last_clean":"2024-03-26T05:05:14.127531+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:05:14.127531+0000","last_undegraded":"2024-03-26T05:05:14.127531+0000","last_fullsized":"2024-03-26T05:05:14.127531+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:04:07.578016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:07.578016+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:24.792226+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'16","reported_seq":234,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.132616+0000","last_change":"2024-03-26T05:05:13.083706+0000","last_active":"2024-03-26T05:05:14.132616+0000","last_peered":"2024-03-26T05:05:14.132616+0000","last_clean":"2024-03-26T05:05:14.132616+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:05:14.132616+0000","last_undegraded":"2024-03-26T05:05:14.132616+0000","last_fullsized":"2024-03-26T05:05:14.132616+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:05:13.083420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:13.083420+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:30.125899+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":197,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.129135+0000","last_change":"2024-03-26T05:04:55.814469+0000","last_active":"2024-03-26T05:05:14.129135+0000","last_peered":"2024-03-26T05:05:14.129135+0000","last_clean":"2024-03-26T05:05:14.129135+0000","last_became_active":"2024-03-26T05:00:28.981049+0000","last_became_peered":"2024-03-26T05:00:28.981049+0000","last_unstale":"2024-03-26T05:05:14.129135+0000","last_undegraded":"2024-03-26T05:05:14.129135+0000","last_fullsized":"2024-03-26T05:05:14.129135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:55.814330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:55.814330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:11.096973+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":195,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.133093+0000","last_change":"2024-03-26T05:03:59.829911+0000","last_active":"2024-03-26T05:05:14.133093+0000","last_peered":"2024-03-26T05:05:14.133093+0000","last_clean":"2024-03-26T05:05:14.133093+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:05:14.133093+0000","last_undegraded":"2024-03-26T05:05:14.133093+0000","last_fullsized":"2024-03-26T05:05:14.133093+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:59.829595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:59.829595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:14.748766+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.140622+0000","last_change":"2024-03-26T05:03:17.225355+0000","last_active":"2024-03-26T05:05:14.140622+0000","last_peered":"2024-03-26T05:05:14.140622+0000","last_clean":"2024-03-26T05:05:14.140622+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:05:14.140622+0000","last_undegraded":"2024-03-26T05:05:14.140622+0000","last_fullsized":"2024-03-26T05:05:14.140622+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:17.225243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:17.225243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'17","reported_seq":256,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.129287+0000","last_change":"2024-03-26T05:04:55.688482+0000","last_active":"2024-03-26T05:05:14.129287+0000","last_peered":"2024-03-26T05:05:14.129287+0000","last_clean":"2024-03-26T05:05:14.129287+0000","last_became_active":"2024-03-26T05:00:49.643308+0000","last_became_peered":"2024-03-26T05:00:49.643308+0000","last_unstale":"2024-03-26T05:05:14.129287+0000","last_undegraded":"2024-03-26T05:05:14.129287+0000","last_fullsized":"2024-03-26T05:05:14.129287+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:04:55.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:55.688341+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:04.276553+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":189,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.117552+0000","last_change":"2024-03-26T05:04:27.007041+0000","last_active":"2024-03-26T05:05:14.117552+0000","last_peered":"2024-03-26T05:05:14.117552+0000","last_clean":"2024-03-26T05:05:14.117552+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:05:14.117552+0000","last_undegraded":"2024-03-26T05:05:14.117552+0000","last_fullsized":"2024-03-26T05:05:14.117552+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:27.006904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:27.006904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:56.433967+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.128465+0000","last_change":"2024-03-26T05:05:01.841808+0000","last_active":"2024-03-26T05:05:14.128465+0000","last_peered":"2024-03-26T05:05:14.128465+0000","last_clean":"2024-03-26T05:05:14.128465+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:05:14.128465+0000","last_undegraded":"2024-03-26T05:05:14.128465+0000","last_fullsized":"2024-03-26T05:05:14.128465+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:01.841718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:01.841718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:03.537079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.130453+0000","last_change":"2024-03-26T05:04:14.715840+0000","last_active":"2024-03-26T05:05:14.130453+0000","last_peered":"2024-03-26T05:05:14.130453+0000","last_clean":"2024-03-26T05:05:14.130453+0000","last_became_active":"2024-03-26T05:00:22.774783+0000","last_became_peered":"2024-03-26T05:00:22.774783+0000","last_unstale":"2024-03-26T05:05:14.130453+0000","last_undegraded":"2024-03-26T05:05:14.130453+0000","last_fullsized":"2024-03-26T05:05:14.130453+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:14.715721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:14.715721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:39.237899+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"100'6","reported_seq":206,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.140711+0000","last_change":"2024-03-26T05:04:01.370962+0000","last_active":"2024-03-26T05:05:14.140711+0000","last_peered":"2024-03-26T05:05:14.140711+0000","last_clean":"2024-03-26T05:05:14.140711+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:05:14.140711+0000","last_undegraded":"2024-03-26T05:05:14.140711+0000","last_fullsized":"2024-03-26T05:05:14.140711+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'6","last_scrub_stamp":"2024-03-26T05:04:01.370804+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:04:01.370804+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":190,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.141785+0000","last_change":"2024-03-26T05:04:10.585914+0000","last_active":"2024-03-26T05:05:14.141785+0000","last_peered":"2024-03-26T05:05:14.141785+0000","last_clean":"2024-03-26T05:05:14.141785+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:05:14.141785+0000","last_undegraded":"2024-03-26T05:05:14.141785+0000","last_fullsized":"2024-03-26T05:05:14.141785+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:10.585750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:10.585750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:23.988184+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.118125+0000","last_change":"2024-03-26T05:05:12.747692+0000","last_active":"2024-03-26T05:05:14.118125+0000","last_peered":"2024-03-26T05:05:14.118125+0000","last_clean":"2024-03-26T05:05:14.118125+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:05:14.118125+0000","last_undegraded":"2024-03-26T05:05:14.118125+0000","last_fullsized":"2024-03-26T05:05:14.118125+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:12.747554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.747554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:36.623696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":118,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.129043+0000","last_change":"2024-03-26T05:04:20.919136+0000","last_active":"2024-03-26T05:05:14.129043+0000","last_peered":"2024-03-26T05:05:14.129043+0000","last_clean":"2024-03-26T05:05:14.129043+0000","last_became_active":"2024-03-26T05:03:05.521331+0000","last_became_peered":"2024-03-26T05:03:05.521331+0000","last_unstale":"2024-03-26T05:05:14.129043+0000","last_undegraded":"2024-03-26T05:05:14.129043+0000","last_fullsized":"2024-03-26T05:05:14.129043+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:20.919020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:20.919020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:50.626775+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"75'17","reported_seq":229,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.148615+0000","last_change":"2024-03-26T05:04:09.559560+0000","last_active":"2024-03-26T05:05:14.148615+0000","last_peered":"2024-03-26T05:05:14.148615+0000","last_clean":"2024-03-26T05:05:14.148615+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:05:14.148615+0000","last_undegraded":"2024-03-26T05:05:14.148615+0000","last_fullsized":"2024-03-26T05:05:14.148615+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:04:09.559412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:09.559412+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:38.053118+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":199,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.121955+0000","last_change":"2024-03-26T05:05:03.036378+0000","last_active":"2024-03-26T05:05:14.121955+0000","last_peered":"2024-03-26T05:05:14.121955+0000","last_clean":"2024-03-26T05:05:14.121955+0000","last_became_active":"2024-03-26T05:00:28.985158+0000","last_became_peered":"2024-03-26T05:00:28.985158+0000","last_unstale":"2024-03-26T05:05:14.121955+0000","last_undegraded":"2024-03-26T05:05:14.121955+0000","last_fullsized":"2024-03-26T05:05:14.121955+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:03.036264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:03.036264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:05.425521+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":194,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.141459+0000","last_change":"2024-03-26T05:05:04.767968+0000","last_active":"2024-03-26T05:05:14.141459+0000","last_peered":"2024-03-26T05:05:14.141459+0000","last_clean":"2024-03-26T05:05:14.141459+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:05:14.141459+0000","last_undegraded":"2024-03-26T05:05:14.141459+0000","last_fullsized":"2024-03-26T05:05:14.141459+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.767790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:11.978808+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.117810+0000","last_change":"2024-03-26T05:05:04.764827+0000","last_active":"2024-03-26T05:05:14.117810+0000","last_peered":"2024-03-26T05:05:14.117810+0000","last_clean":"2024-03-26T05:05:14.117810+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:05:14.117810+0000","last_undegraded":"2024-03-26T05:05:14.117810+0000","last_fullsized":"2024-03-26T05:05:14.117810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.764742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:21.274757+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":222,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.127631+0000","last_change":"2024-03-26T05:05:14.064119+0000","last_active":"2024-03-26T05:05:14.127631+0000","last_peered":"2024-03-26T05:05:14.127631+0000","last_clean":"2024-03-26T05:05:14.127631+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:05:14.127631+0000","last_undegraded":"2024-03-26T05:05:14.127631+0000","last_fullsized":"2024-03-26T05:05:14.127631+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-26T05:05:14.063125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:14.063125+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:16.289673+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":190,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.148289+0000","last_change":"2024-03-26T05:04:58.834254+0000","last_active":"2024-03-26T05:05:14.148289+0000","last_peered":"2024-03-26T05:05:14.148289+0000","last_clean":"2024-03-26T05:05:14.148289+0000","last_became_active":"2024-03-26T05:00:28.983417+0000","last_became_peered":"2024-03-26T05:00:28.983417+0000","last_unstale":"2024-03-26T05:05:14.148289+0000","last_undegraded":"2024-03-26T05:05:14.148289+0000","last_fullsized":"2024-03-26T05:05:14.148289+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:58.834141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.834141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:12.740874+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.127291+0000","last_change":"2024-03-26T05:04:21.579387+0000","last_active":"2024-03-26T05:05:14.127291+0000","last_peered":"2024-03-26T05:05:14.127291+0000","last_clean":"2024-03-26T05:05:14.127291+0000","last_became_active":"2024-03-26T05:00:26.954653+0000","last_became_peered":"2024-03-26T05:00:26.954653+0000","last_unstale":"2024-03-26T05:05:14.127291+0000","last_undegraded":"2024-03-26T05:05:14.127291+0000","last_fullsized":"2024-03-26T05:05:14.127291+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:21.579155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.579155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:39.504032+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":195,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108398+0000","last_change":"2024-03-26T05:03:03.408937+0000","last_active":"2024-03-26T05:05:14.108398+0000","last_peered":"2024-03-26T05:05:14.108398+0000","last_clean":"2024-03-26T05:05:14.108398+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:05:14.108398+0000","last_undegraded":"2024-03-26T05:05:14.108398+0000","last_fullsized":"2024-03-26T05:05:14.108398+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:03.408801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:03.408801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"95'16","reported_seq":223,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108923+0000","last_change":"2024-03-26T05:03:22.426319+0000","last_active":"2024-03-26T05:05:14.108923+0000","last_peered":"2024-03-26T05:05:14.108923+0000","last_clean":"2024-03-26T05:05:14.108923+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:05:14.108923+0000","last_undegraded":"2024-03-26T05:05:14.108923+0000","last_fullsized":"2024-03-26T05:05:14.108923+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:03:22.426170+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:03:22.426170+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":190,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.132201+0000","last_change":"2024-03-26T05:04:10.806723+0000","last_active":"2024-03-26T05:05:14.132201+0000","last_peered":"2024-03-26T05:05:14.132201+0000","last_clean":"2024-03-26T05:05:14.132201+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:05:14.132201+0000","last_undegraded":"2024-03-26T05:05:14.132201+0000","last_fullsized":"2024-03-26T05:05:14.132201+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:10.806588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:10.806588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:36.340744+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":262,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.749344+0000","last_change":"2024-03-26T05:04:05.662804+0000","last_active":"2024-03-26T05:05:14.749344+0000","last_peered":"2024-03-26T05:05:14.749344+0000","last_clean":"2024-03-26T05:05:14.749344+0000","last_became_active":"2024-03-26T05:00:26.955836+0000","last_became_peered":"2024-03-26T05:00:26.955836+0000","last_unstale":"2024-03-26T05:05:14.749344+0000","last_undegraded":"2024-03-26T05:05:14.749344+0000","last_fullsized":"2024-03-26T05:05:14.749344+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:05.662699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:05.662699+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:21.460708+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.117888+0000","last_change":"2024-03-26T05:05:08.856522+0000","last_active":"2024-03-26T05:05:14.117888+0000","last_peered":"2024-03-26T05:05:14.117888+0000","last_clean":"2024-03-26T05:05:14.117888+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:05:14.117888+0000","last_undegraded":"2024-03-26T05:05:14.117888+0000","last_fullsized":"2024-03-26T05:05:14.117888+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:08.856401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:08.856401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:36.518630+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'22","reported_seq":245,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.140796+0000","last_change":"2024-03-26T05:05:08.701087+0000","last_active":"2024-03-26T05:05:14.140796+0000","last_peered":"2024-03-26T05:05:14.140796+0000","last_clean":"2024-03-26T05:05:14.140796+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:05:14.140796+0000","last_undegraded":"2024-03-26T05:05:14.140796+0000","last_fullsized":"2024-03-26T05:05:14.140796+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:05:08.700945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:08.700945+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:13.232719+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":183,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.109307+0000","last_change":"2024-03-26T05:03:14.456132+0000","last_active":"2024-03-26T05:05:14.109307+0000","last_peered":"2024-03-26T05:05:14.109307+0000","last_clean":"2024-03-26T05:05:14.109307+0000","last_became_active":"2024-03-26T05:00:28.979477+0000","last_became_peered":"2024-03-26T05:00:28.979477+0000","last_unstale":"2024-03-26T05:05:14.109307+0000","last_undegraded":"2024-03-26T05:05:14.109307+0000","last_fullsized":"2024-03-26T05:05:14.109307+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:14.456016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:14.456016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:21.072142+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"120'9","reported_seq":265,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.141216+0000","last_change":"2024-03-26T05:03:20.187175+0000","last_active":"2024-03-26T05:05:14.141216+0000","last_peered":"2024-03-26T05:05:14.141216+0000","last_clean":"2024-03-26T05:05:14.141216+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:05:14.141216+0000","last_undegraded":"2024-03-26T05:05:14.141216+0000","last_fullsized":"2024-03-26T05:05:14.141216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'9","last_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:03:20.186818+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:19.648724+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":211,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.121796+0000","last_change":"2024-03-26T05:05:10.047756+0000","last_active":"2024-03-26T05:05:14.121796+0000","last_peered":"2024-03-26T05:05:14.121796+0000","last_clean":"2024-03-26T05:05:14.121796+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:05:14.121796+0000","last_undegraded":"2024-03-26T05:05:14.121796+0000","last_fullsized":"2024-03-26T05:05:14.121796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:10.047628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:10.047628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:19.492245+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"98'20","reported_seq":239,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.130747+0000","last_change":"2024-03-26T05:04:12.659452+0000","last_active":"2024-03-26T05:05:14.130747+0000","last_peered":"2024-03-26T05:05:14.130747+0000","last_clean":"2024-03-26T05:05:14.130747+0000","last_became_active":"2024-03-26T05:00:25.648284+0000","last_became_peered":"2024-03-26T05:00:25.648284+0000","last_unstale":"2024-03-26T05:05:14.130747+0000","last_undegraded":"2024-03-26T05:05:14.130747+0000","last_fullsized":"2024-03-26T05:05:14.130747+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'20","last_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:04:12.659322+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:26.101052+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":198,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.123575+0000","last_change":"2024-03-26T05:04:13.174305+0000","last_active":"2024-03-26T05:05:14.123575+0000","last_peered":"2024-03-26T05:05:14.123575+0000","last_clean":"2024-03-26T05:05:14.123575+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:05:14.123575+0000","last_undegraded":"2024-03-26T05:05:14.123575+0000","last_fullsized":"2024-03-26T05:05:14.123575+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:13.174161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:13.174161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:29.585590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":193,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.143158+0000","last_change":"2024-03-26T05:04:59.846243+0000","last_active":"2024-03-26T05:05:14.143158+0000","last_peered":"2024-03-26T05:05:14.143158+0000","last_clean":"2024-03-26T05:05:14.143158+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:05:14.143158+0000","last_undegraded":"2024-03-26T05:05:14.143158+0000","last_fullsized":"2024-03-26T05:05:14.143158+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:59.846123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:04:59.846123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:00.016038+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.117718+0000","last_change":"2024-03-26T05:04:15.786624+0000","last_active":"2024-03-26T05:05:14.117718+0000","last_peered":"2024-03-26T05:05:14.117718+0000","last_clean":"2024-03-26T05:05:14.117718+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:05:14.117718+0000","last_undegraded":"2024-03-26T05:05:14.117718+0000","last_fullsized":"2024-03-26T05:05:14.117718+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:15.786545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:15.786545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:45.427035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'21","reported_seq":247,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.132985+0000","last_change":"2024-03-26T05:05:12.090691+0000","last_active":"2024-03-26T05:05:14.132985+0000","last_peered":"2024-03-26T05:05:14.132985+0000","last_clean":"2024-03-26T05:05:14.132985+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:05:14.132985+0000","last_undegraded":"2024-03-26T05:05:14.132985+0000","last_fullsized":"2024-03-26T05:05:14.132985+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:05:12.089325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.089325+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:33.911096+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":189,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.117702+0000","last_change":"2024-03-26T05:04:05.932455+0000","last_active":"2024-03-26T05:05:14.117702+0000","last_peered":"2024-03-26T05:05:14.117702+0000","last_clean":"2024-03-26T05:05:14.117702+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:05:14.117702+0000","last_undegraded":"2024-03-26T05:05:14.117702+0000","last_fullsized":"2024-03-26T05:05:14.117702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:05.932270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:05.932270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:20.787403+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.131075+0000","last_change":"2024-03-26T05:05:09.946831+0000","last_active":"2024-03-26T05:05:14.131075+0000","last_peered":"2024-03-26T05:05:14.131075+0000","last_clean":"2024-03-26T05:05:14.131075+0000","last_became_active":"2024-03-26T05:00:26.953416+0000","last_became_peered":"2024-03-26T05:00:26.953416+0000","last_unstale":"2024-03-26T05:05:14.131075+0000","last_undegraded":"2024-03-26T05:05:14.131075+0000","last_fullsized":"2024-03-26T05:05:14.131075+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:09.946734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:09.946734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:38.455388+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":215,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.128019+0000","last_change":"2024-03-26T05:05:02.846889+0000","last_active":"2024-03-26T05:05:14.128019+0000","last_peered":"2024-03-26T05:05:14.128019+0000","last_clean":"2024-03-26T05:05:14.128019+0000","last_became_active":"2024-03-26T05:00:22.771080+0000","last_became_peered":"2024-03-26T05:00:22.771080+0000","last_unstale":"2024-03-26T05:05:14.128019+0000","last_undegraded":"2024-03-26T05:05:14.128019+0000","last_fullsized":"2024-03-26T05:05:14.128019+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:05:02.846756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:02.846756+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:11.935444+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'12","reported_seq":219,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.143396+0000","last_change":"2024-03-26T05:03:18.221430+0000","last_active":"2024-03-26T05:05:14.143396+0000","last_peered":"2024-03-26T05:05:14.143396+0000","last_clean":"2024-03-26T05:05:14.143396+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:05:14.143396+0000","last_undegraded":"2024-03-26T05:05:14.143396+0000","last_fullsized":"2024-03-26T05:05:14.143396+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:03:18.221280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:18.221280+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:19.648753+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"155'11635","reported_seq":11827,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.868508+0000","last_change":"2024-03-26T05:04:21.563737+0000","last_active":"2024-03-26T05:05:14.868508+0000","last_peered":"2024-03-26T05:05:14.868508+0000","last_clean":"2024-03-26T05:05:14.868508+0000","last_became_active":"2024-03-26T05:02:24.354658+0000","last_became_peered":"2024-03-26T05:02:24.354658+0000","last_unstale":"2024-03-26T05:05:14.868508+0000","last_undegraded":"2024-03-26T05:05:14.868508+0000","last_fullsized":"2024-03-26T05:05:14.868508+0000","mapping_epoch":103,"log_start":"133'6600","ondisk_log_start":"133'6600","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'8267","last_scrub_stamp":"2024-03-26T05:04:21.563226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.563226+0000","objects_scrubbed":6873,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:33.838683+0000","scrub_duration":2071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226033664,"num_objects":3449,"num_object_clones":0,"num_object_copies":10347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":482688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":200,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.131331+0000","last_change":"2024-03-26T05:04:59.901150+0000","last_active":"2024-03-26T05:05:14.131331+0000","last_peered":"2024-03-26T05:05:14.131331+0000","last_clean":"2024-03-26T05:05:14.131331+0000","last_became_active":"2024-03-26T05:00:29.461059+0000","last_became_peered":"2024-03-26T05:00:29.461059+0000","last_unstale":"2024-03-26T05:05:14.131331+0000","last_undegraded":"2024-03-26T05:05:14.131331+0000","last_fullsized":"2024-03-26T05:05:14.131331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:04:59.901027+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:04:59.901027+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:24.311519+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":230,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.119236+0000","last_change":"2024-03-26T05:05:06.832886+0000","last_active":"2024-03-26T05:05:14.119236+0000","last_peered":"2024-03-26T05:05:14.119236+0000","last_clean":"2024-03-26T05:05:14.119236+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:05:14.119236+0000","last_undegraded":"2024-03-26T05:05:14.119236+0000","last_fullsized":"2024-03-26T05:05:14.119236+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:05:06.832744+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:05:06.832744+0000","last_clean_scrub_stamp":"2024-03-26T05:05:06.832744+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:09.446229+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'27","reported_seq":255,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.129432+0000","last_change":"2024-03-26T05:04:13.682488+0000","last_active":"2024-03-26T05:05:14.129432+0000","last_peered":"2024-03-26T05:05:14.129432+0000","last_clean":"2024-03-26T05:05:14.129432+0000","last_became_active":"2024-03-26T05:00:25.651328+0000","last_became_peered":"2024-03-26T05:00:25.651328+0000","last_unstale":"2024-03-26T05:05:14.129432+0000","last_undegraded":"2024-03-26T05:05:14.129432+0000","last_fullsized":"2024-03-26T05:05:14.129432+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'27","last_scrub_stamp":"2024-03-26T05:04:13.682349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:13.682349+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:18.855246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":261,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.118549+0000","last_change":"2024-03-26T05:03:10.843641+0000","last_active":"2024-03-26T05:05:14.118549+0000","last_peered":"2024-03-26T05:05:14.118549+0000","last_clean":"2024-03-26T05:05:14.118549+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:05:14.118549+0000","last_undegraded":"2024-03-26T05:05:14.118549+0000","last_fullsized":"2024-03-26T05:05:14.118549+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:03:10.843497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:10.843497+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:19.761444+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"155'11707","reported_seq":11902,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:15.033613+0000","last_change":"2024-03-26T05:04:16.685339+0000","last_active":"2024-03-26T05:05:15.033613+0000","last_peered":"2024-03-26T05:05:15.033613+0000","last_clean":"2024-03-26T05:05:15.033613+0000","last_became_active":"2024-03-26T05:02:24.352844+0000","last_became_peered":"2024-03-26T05:02:24.352844+0000","last_unstale":"2024-03-26T05:05:15.033613+0000","last_undegraded":"2024-03-26T05:05:15.033613+0000","last_fullsized":"2024-03-26T05:05:15.033613+0000","mapping_epoch":103,"log_start":"136'7000","ondisk_log_start":"136'7000","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'8019","last_scrub_stamp":"2024-03-26T05:04:16.685114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:16.685114+0000","objects_scrubbed":7208,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:28.742298+0000","scrub_duration":3134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224460800,"num_objects":3425,"num_object_clones":0,"num_object_copies":10275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11707,"num_write_kb":484224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":190,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.141585+0000","last_change":"2024-03-26T05:03:16.177929+0000","last_active":"2024-03-26T05:05:14.141585+0000","last_peered":"2024-03-26T05:05:14.141585+0000","last_clean":"2024-03-26T05:05:14.141585+0000","last_became_active":"2024-03-26T05:00:28.985917+0000","last_became_peered":"2024-03-26T05:00:28.985917+0000","last_unstale":"2024-03-26T05:05:14.141585+0000","last_undegraded":"2024-03-26T05:05:14.141585+0000","last_fullsized":"2024-03-26T05:05:14.141585+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:16.177807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:16.177807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.118841+0000","last_change":"2024-03-26T05:03:17.962503+0000","last_active":"2024-03-26T05:05:14.118841+0000","last_peered":"2024-03-26T05:05:14.118841+0000","last_clean":"2024-03-26T05:05:14.118841+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:05:14.118841+0000","last_undegraded":"2024-03-26T05:05:14.118841+0000","last_fullsized":"2024-03-26T05:05:14.118841+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:17.962388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:17.962388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'24","reported_seq":246,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108974+0000","last_change":"2024-03-26T05:05:01.010769+0000","last_active":"2024-03-26T05:05:14.108974+0000","last_peered":"2024-03-26T05:05:14.108974+0000","last_clean":"2024-03-26T05:05:14.108974+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:05:14.108974+0000","last_undegraded":"2024-03-26T05:05:14.108974+0000","last_fullsized":"2024-03-26T05:05:14.108974+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'24","last_scrub_stamp":"2024-03-26T05:05:01.010617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:01.010617+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:02.462916+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.123848+0000","last_change":"2024-03-26T05:05:12.432880+0000","last_active":"2024-03-26T05:05:14.123848+0000","last_peered":"2024-03-26T05:05:14.123848+0000","last_clean":"2024-03-26T05:05:14.123848+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:05:14.123848+0000","last_undegraded":"2024-03-26T05:05:14.123848+0000","last_fullsized":"2024-03-26T05:05:14.123848+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:12.432732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.432732+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.432732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:30.731148+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"155'11427","reported_seq":11615,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:15.020496+0000","last_change":"2024-03-26T05:04:27.315840+0000","last_active":"2024-03-26T05:05:15.020496+0000","last_peered":"2024-03-26T05:05:15.020496+0000","last_clean":"2024-03-26T05:05:15.020496+0000","last_became_active":"2024-03-26T05:02:24.355930+0000","last_became_peered":"2024-03-26T05:02:24.355930+0000","last_unstale":"2024-03-26T05:05:15.020496+0000","last_undegraded":"2024-03-26T05:05:15.020496+0000","last_fullsized":"2024-03-26T05:05:15.020496+0000","mapping_epoch":103,"log_start":"129'5800","ondisk_log_start":"129'5800","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8447","last_scrub_stamp":"2024-03-26T05:04:27.315394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:27.315394+0000","objects_scrubbed":6429,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:56.899987+0000","scrub_duration":2530,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221708288,"num_objects":3383,"num_object_clones":0,"num_object_copies":10149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11427,"num_write_kb":473920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":189,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.117570+0000","last_change":"2024-03-26T05:03:23.071801+0000","last_active":"2024-03-26T05:05:14.117570+0000","last_peered":"2024-03-26T05:05:14.117570+0000","last_clean":"2024-03-26T05:05:14.117570+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:05:14.117570+0000","last_undegraded":"2024-03-26T05:05:14.117570+0000","last_fullsized":"2024-03-26T05:05:14.117570+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:23.071667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:23.071667+0000","last_clean_scrub_stamp":"2024-03-26T05:03:23.071667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.125449+0000","last_change":"2024-03-26T05:04:21.231029+0000","last_active":"2024-03-26T05:05:14.125449+0000","last_peered":"2024-03-26T05:05:14.125449+0000","last_clean":"2024-03-26T05:05:14.125449+0000","last_became_active":"2024-03-26T05:00:26.961730+0000","last_became_peered":"2024-03-26T05:00:26.961730+0000","last_unstale":"2024-03-26T05:05:14.125449+0000","last_undegraded":"2024-03-26T05:05:14.125449+0000","last_fullsized":"2024-03-26T05:05:14.125449+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:21.230901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.230901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:36.141812+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.118840+0000","last_change":"2024-03-26T05:04:23.884517+0000","last_active":"2024-03-26T05:05:14.118840+0000","last_peered":"2024-03-26T05:05:14.118840+0000","last_clean":"2024-03-26T05:05:14.118840+0000","last_became_active":"2024-03-26T05:00:22.772590+0000","last_became_peered":"2024-03-26T05:00:22.772590+0000","last_unstale":"2024-03-26T05:05:14.118840+0000","last_undegraded":"2024-03-26T05:05:14.118840+0000","last_fullsized":"2024-03-26T05:05:14.118840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:23.884406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:23.884406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:49.946181+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'25","reported_seq":248,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.128310+0000","last_change":"2024-03-26T05:05:04.815941+0000","last_active":"2024-03-26T05:05:14.128310+0000","last_peered":"2024-03-26T05:05:14.128310+0000","last_clean":"2024-03-26T05:05:14.128310+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:05:14.128310+0000","last_undegraded":"2024-03-26T05:05:14.128310+0000","last_fullsized":"2024-03-26T05:05:14.128310+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:05:04.815826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.815826+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:08.411978+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"155'11650","reported_seq":11848,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.903582+0000","last_change":"2024-03-26T05:04:11.124139+0000","last_active":"2024-03-26T05:05:14.903582+0000","last_peered":"2024-03-26T05:05:14.903582+0000","last_clean":"2024-03-26T05:05:14.903582+0000","last_became_active":"2024-03-26T05:02:24.353523+0000","last_became_peered":"2024-03-26T05:02:24.353523+0000","last_unstale":"2024-03-26T05:05:14.903582+0000","last_undegraded":"2024-03-26T05:05:14.903582+0000","last_fullsized":"2024-03-26T05:05:14.903582+0000","mapping_epoch":103,"log_start":"121'4800","ondisk_log_start":"121'4800","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7680","last_scrub_stamp":"2024-03-26T05:04:11.123962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:11.123962+0000","objects_scrubbed":7433,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:32.441783+0000","scrub_duration":2854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221904896,"num_objects":3386,"num_object_clones":0,"num_object_copies":10158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11650,"num_write_kb":481152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":191,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.117115+0000","last_change":"2024-03-26T05:04:00.814564+0000","last_active":"2024-03-26T05:05:14.117115+0000","last_peered":"2024-03-26T05:05:14.117115+0000","last_clean":"2024-03-26T05:05:14.117115+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:05:14.117115+0000","last_undegraded":"2024-03-26T05:05:14.117115+0000","last_fullsized":"2024-03-26T05:05:14.117115+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:00.814254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:00.814254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:21.361433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.133395+0000","last_change":"2024-03-26T05:05:09.447081+0000","last_active":"2024-03-26T05:05:14.133395+0000","last_peered":"2024-03-26T05:05:14.133395+0000","last_clean":"2024-03-26T05:05:14.133395+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:05:14.133395+0000","last_undegraded":"2024-03-26T05:05:14.133395+0000","last_fullsized":"2024-03-26T05:05:14.133395+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:09.446507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:09.446507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:29.377678+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.129982+0000","last_change":"2024-03-26T05:05:01.867783+0000","last_active":"2024-03-26T05:05:14.129982+0000","last_peered":"2024-03-26T05:05:14.129982+0000","last_clean":"2024-03-26T05:05:14.129982+0000","last_became_active":"2024-03-26T05:00:22.773770+0000","last_became_peered":"2024-03-26T05:00:22.773770+0000","last_unstale":"2024-03-26T05:05:14.129982+0000","last_undegraded":"2024-03-26T05:05:14.129982+0000","last_fullsized":"2024-03-26T05:05:14.129982+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:01.867674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:01.867674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:30.089609+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'15","reported_seq":227,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.121482+0000","last_change":"2024-03-26T05:03:15.852751+0000","last_active":"2024-03-26T05:05:14.121482+0000","last_peered":"2024-03-26T05:05:14.121482+0000","last_clean":"2024-03-26T05:05:14.121482+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:05:14.121482+0000","last_undegraded":"2024-03-26T05:05:14.121482+0000","last_fullsized":"2024-03-26T05:05:14.121482+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:03:15.852614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:15.852614+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"155'11848","reported_seq":12043,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:19.755447+0000","last_change":"2024-03-26T05:04:07.239552+0000","last_active":"2024-03-26T05:05:19.755447+0000","last_peered":"2024-03-26T05:05:19.755447+0000","last_clean":"2024-03-26T05:05:19.755447+0000","last_became_active":"2024-03-26T05:02:24.353248+0000","last_became_peered":"2024-03-26T05:02:24.353248+0000","last_unstale":"2024-03-26T05:05:19.755447+0000","last_undegraded":"2024-03-26T05:05:19.755447+0000","last_fullsized":"2024-03-26T05:05:19.755447+0000","mapping_epoch":103,"log_start":"128'5700","ondisk_log_start":"128'5700","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7371","last_scrub_stamp":"2024-03-26T05:04:07.239265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:07.239265+0000","objects_scrubbed":7298,"log_size":6148,"log_dups_size":0,"ondisk_log_size":6148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:31.785702+0000","scrub_duration":2188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":3008,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11848,"num_write_kb":475392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":190,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.142414+0000","last_change":"2024-03-26T05:04:11.571539+0000","last_active":"2024-03-26T05:05:14.142414+0000","last_peered":"2024-03-26T05:05:14.142414+0000","last_clean":"2024-03-26T05:05:14.142414+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:05:14.142414+0000","last_undegraded":"2024-03-26T05:05:14.142414+0000","last_fullsized":"2024-03-26T05:05:14.142414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:11.571425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:11.571425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:33.966929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":221,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.131181+0000","last_change":"2024-03-26T05:04:58.935696+0000","last_active":"2024-03-26T05:05:14.131181+0000","last_peered":"2024-03-26T05:05:14.131181+0000","last_clean":"2024-03-26T05:05:14.131181+0000","last_became_active":"2024-03-26T05:00:21.760521+0000","last_became_peered":"2024-03-26T05:00:21.760521+0000","last_unstale":"2024-03-26T05:05:14.131181+0000","last_undegraded":"2024-03-26T05:05:14.131181+0000","last_fullsized":"2024-03-26T05:05:14.131181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:04:58.935568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.935568+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:28.155396+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":195,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108691+0000","last_change":"2024-03-26T05:03:11.352008+0000","last_active":"2024-03-26T05:05:14.108691+0000","last_peered":"2024-03-26T05:05:14.108691+0000","last_clean":"2024-03-26T05:05:14.108691+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:05:14.108691+0000","last_undegraded":"2024-03-26T05:05:14.108691+0000","last_fullsized":"2024-03-26T05:05:14.108691+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:11.351820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:11.351820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'34","reported_seq":325,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.127396+0000","last_change":"2024-03-26T05:04:22.562805+0000","last_active":"2024-03-26T05:05:14.127396+0000","last_peered":"2024-03-26T05:05:14.127396+0000","last_clean":"2024-03-26T05:05:14.127396+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:05:14.127396+0000","last_undegraded":"2024-03-26T05:05:14.127396+0000","last_fullsized":"2024-03-26T05:05:14.127396+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'34","last_scrub_stamp":"2024-03-26T05:04:22.562552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:22.562552+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:46.860857+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":195,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.119089+0000","last_change":"2024-03-26T05:04:54.973421+0000","last_active":"2024-03-26T05:05:14.119089+0000","last_peered":"2024-03-26T05:05:14.119089+0000","last_clean":"2024-03-26T05:05:14.119089+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:05:14.119089+0000","last_undegraded":"2024-03-26T05:05:14.119089+0000","last_fullsized":"2024-03-26T05:05:14.119089+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:54.973306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:54.973306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:56.082970+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"155'11396","reported_seq":11585,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:15.062785+0000","last_change":"2024-03-26T05:04:24.005788+0000","last_active":"2024-03-26T05:05:15.062785+0000","last_peered":"2024-03-26T05:05:15.062785+0000","last_clean":"2024-03-26T05:05:15.062785+0000","last_became_active":"2024-03-26T05:02:24.356658+0000","last_became_peered":"2024-03-26T05:02:24.356658+0000","last_unstale":"2024-03-26T05:05:15.062785+0000","last_undegraded":"2024-03-26T05:05:15.062785+0000","last_fullsized":"2024-03-26T05:05:15.062785+0000","mapping_epoch":103,"log_start":"129'5800","ondisk_log_start":"129'5800","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'8251","last_scrub_stamp":"2024-03-26T05:04:24.005629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:24.005629+0000","objects_scrubbed":6565,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:27.436033+0000","scrub_duration":2213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219807744,"num_objects":3354,"num_object_clones":0,"num_object_copies":10062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11396,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":189,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.118986+0000","last_change":"2024-03-26T05:02:59.958661+0000","last_active":"2024-03-26T05:05:14.118986+0000","last_peered":"2024-03-26T05:05:14.118986+0000","last_clean":"2024-03-26T05:05:14.118986+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:05:14.118986+0000","last_undegraded":"2024-03-26T05:05:14.118986+0000","last_fullsized":"2024-03-26T05:05:14.118986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:02:59.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:02:59.958579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.126974+0000","last_change":"2024-03-26T05:04:17.462393+0000","last_active":"2024-03-26T05:05:14.126974+0000","last_peered":"2024-03-26T05:05:14.126974+0000","last_clean":"2024-03-26T05:05:14.126974+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:05:14.126974+0000","last_undegraded":"2024-03-26T05:05:14.126974+0000","last_fullsized":"2024-03-26T05:05:14.126974+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:17.462281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:17.462281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:36.813984+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":219,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.122875+0000","last_change":"2024-03-26T05:04:57.410678+0000","last_active":"2024-03-26T05:05:14.122875+0000","last_peered":"2024-03-26T05:05:14.122875+0000","last_clean":"2024-03-26T05:05:14.122875+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:05:14.122875+0000","last_undegraded":"2024-03-26T05:05:14.122875+0000","last_fullsized":"2024-03-26T05:05:14.122875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:57.410553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:04:57.410553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:06.632271+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'19","reported_seq":237,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.129804+0000","last_change":"2024-03-26T05:04:11.649338+0000","last_active":"2024-03-26T05:05:14.129804+0000","last_peered":"2024-03-26T05:05:14.129804+0000","last_clean":"2024-03-26T05:05:14.129804+0000","last_became_active":"2024-03-26T05:00:24.937145+0000","last_became_peered":"2024-03-26T05:00:24.937145+0000","last_unstale":"2024-03-26T05:05:14.129804+0000","last_undegraded":"2024-03-26T05:05:14.129804+0000","last_fullsized":"2024-03-26T05:05:14.129804+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-26T05:04:11.649163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:11.649163+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:15.906395+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.134327+0000","last_change":"2024-03-26T05:04:52.294165+0000","last_active":"2024-03-26T05:05:14.134327+0000","last_peered":"2024-03-26T05:05:14.134327+0000","last_clean":"2024-03-26T05:05:14.134327+0000","last_became_active":"2024-03-26T05:00:26.957936+0000","last_became_peered":"2024-03-26T05:00:26.957936+0000","last_unstale":"2024-03-26T05:05:14.134327+0000","last_undegraded":"2024-03-26T05:05:14.134327+0000","last_fullsized":"2024-03-26T05:05:14.134327+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:52.294056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:52.294056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:55.374354+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"155'11402","reported_seq":11599,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:15.024733+0000","last_change":"2024-03-26T05:04:12.245145+0000","last_active":"2024-03-26T05:05:15.024733+0000","last_peered":"2024-03-26T05:05:15.024733+0000","last_clean":"2024-03-26T05:05:15.024733+0000","last_became_active":"2024-03-26T05:02:24.356129+0000","last_became_peered":"2024-03-26T05:02:24.356129+0000","last_unstale":"2024-03-26T05:05:15.024733+0000","last_undegraded":"2024-03-26T05:05:15.024733+0000","last_fullsized":"2024-03-26T05:05:15.024733+0000","mapping_epoch":103,"log_start":"129'5800","ondisk_log_start":"129'5800","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7587","last_scrub_stamp":"2024-03-26T05:04:12.244918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:12.244918+0000","objects_scrubbed":7304,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:29.947538+0000","scrub_duration":3431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222035968,"num_objects":3388,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11402,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":198,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.131259+0000","last_change":"2024-03-26T05:05:03.848333+0000","last_active":"2024-03-26T05:05:14.131259+0000","last_peered":"2024-03-26T05:05:14.131259+0000","last_clean":"2024-03-26T05:05:14.131259+0000","last_became_active":"2024-03-26T05:00:28.979373+0000","last_became_peered":"2024-03-26T05:00:28.979373+0000","last_unstale":"2024-03-26T05:05:14.131259+0000","last_undegraded":"2024-03-26T05:05:14.131259+0000","last_fullsized":"2024-03-26T05:05:14.131259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_clean_scrub_stamp":"2024-03-26T05:05:03.848208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:06.228959+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":456,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.118005+0000","last_change":"2024-03-26T05:04:17.821154+0000","last_active":"2024-03-26T05:05:14.118005+0000","last_peered":"2024-03-26T05:05:14.118005+0000","last_clean":"2024-03-26T05:05:14.118005+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:05:14.118005+0000","last_undegraded":"2024-03-26T05:05:14.118005+0000","last_fullsized":"2024-03-26T05:05:14.118005+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:04:17.820870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:04:17.820870+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:31.192171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.140259+0000","last_change":"2024-03-26T05:05:07.744502+0000","last_active":"2024-03-26T05:05:14.140259+0000","last_peered":"2024-03-26T05:05:14.140259+0000","last_clean":"2024-03-26T05:05:14.140259+0000","last_became_active":"2024-03-26T05:00:22.774298+0000","last_became_peered":"2024-03-26T05:00:22.774298+0000","last_unstale":"2024-03-26T05:05:14.140259+0000","last_undegraded":"2024-03-26T05:05:14.140259+0000","last_fullsized":"2024-03-26T05:05:14.140259+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:07.743708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:07.743708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:16.422498+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'24","reported_seq":295,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.137881+0000","last_change":"2024-03-26T05:04:24.867946+0000","last_active":"2024-03-26T05:05:14.137881+0000","last_peered":"2024-03-26T05:05:14.137881+0000","last_clean":"2024-03-26T05:05:14.137881+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:05:14.137881+0000","last_undegraded":"2024-03-26T05:05:14.137881+0000","last_fullsized":"2024-03-26T05:05:14.137881+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'24","last_scrub_stamp":"2024-03-26T05:04:24.867838+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:04:24.867838+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:48.459336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":193,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.118563+0000","last_change":"2024-03-26T05:04:02.971643+0000","last_active":"2024-03-26T05:05:14.118563+0000","last_peered":"2024-03-26T05:05:14.118563+0000","last_clean":"2024-03-26T05:05:14.118563+0000","last_became_active":"2024-03-26T05:00:26.961724+0000","last_became_peered":"2024-03-26T05:00:26.961724+0000","last_unstale":"2024-03-26T05:05:14.118563+0000","last_undegraded":"2024-03-26T05:05:14.118563+0000","last_fullsized":"2024-03-26T05:05:14.118563+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:02.971526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:02.971526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:16.073532+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"155'11781","reported_seq":11948,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:19.720156+0000","last_change":"2024-03-26T05:04:59.106118+0000","last_active":"2024-03-26T05:05:19.720156+0000","last_peered":"2024-03-26T05:05:19.720156+0000","last_clean":"2024-03-26T05:05:19.720156+0000","last_became_active":"2024-03-26T05:02:24.354784+0000","last_became_peered":"2024-03-26T05:02:24.354784+0000","last_unstale":"2024-03-26T05:05:19.720156+0000","last_undegraded":"2024-03-26T05:05:19.720156+0000","last_fullsized":"2024-03-26T05:05:19.720156+0000","mapping_epoch":103,"log_start":"133'6600","ondisk_log_start":"133'6600","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'10450","last_scrub_stamp":"2024-03-26T05:04:59.105579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:59.105579+0000","objects_scrubbed":4395,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:22.819768+0000","scrub_duration":1358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197591040,"num_objects":3015,"num_object_clones":0,"num_object_copies":9045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11781,"num_write_kb":473472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":198,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.126964+0000","last_change":"2024-03-26T05:04:53.760276+0000","last_active":"2024-03-26T05:05:14.126964+0000","last_peered":"2024-03-26T05:05:14.126964+0000","last_clean":"2024-03-26T05:05:14.126964+0000","last_became_active":"2024-03-26T05:00:28.983202+0000","last_became_peered":"2024-03-26T05:00:28.983202+0000","last_unstale":"2024-03-26T05:05:14.126964+0000","last_undegraded":"2024-03-26T05:05:14.126964+0000","last_fullsized":"2024-03-26T05:05:14.126964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:53.760171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:53.760171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:07.693633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":211,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.117402+0000","last_change":"2024-03-26T05:04:59.995006+0000","last_active":"2024-03-26T05:05:14.117402+0000","last_peered":"2024-03-26T05:05:14.117402+0000","last_clean":"2024-03-26T05:05:14.117402+0000","last_became_active":"2024-03-26T05:00:22.773081+0000","last_became_peered":"2024-03-26T05:00:22.773081+0000","last_unstale":"2024-03-26T05:05:14.117402+0000","last_undegraded":"2024-03-26T05:05:14.117402+0000","last_fullsized":"2024-03-26T05:05:14.117402+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:59.994887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:59.994887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:06.597851+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":505,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:18.502774+0000","last_change":"2024-03-26T05:03:00.940387+0000","last_active":"2024-03-26T05:05:18.502774+0000","last_peered":"2024-03-26T05:05:18.502774+0000","last_clean":"2024-03-26T05:05:18.502774+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:05:18.502774+0000","last_undegraded":"2024-03-26T05:05:18.502774+0000","last_fullsized":"2024-03-26T05:05:18.502774+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:03:00.940299+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:03:00.940299+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:19.761281+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":291,"num_read_kb":286,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":237,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.132298+0000","last_change":"2024-03-26T05:04:56.020273+0000","last_active":"2024-03-26T05:05:14.132298+0000","last_peered":"2024-03-26T05:05:14.132298+0000","last_clean":"2024-03-26T05:05:14.132298+0000","last_became_active":"2024-03-26T05:00:49.642276+0000","last_became_peered":"2024-03-26T05:00:49.642276+0000","last_unstale":"2024-03-26T05:05:14.132298+0000","last_undegraded":"2024-03-26T05:05:14.132298+0000","last_fullsized":"2024-03-26T05:05:14.132298+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:04:56.020141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:56.020141+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:08.790686+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.131674+0000","last_change":"2024-03-26T05:04:22.633678+0000","last_active":"2024-03-26T05:05:14.131674+0000","last_peered":"2024-03-26T05:05:14.131674+0000","last_clean":"2024-03-26T05:05:14.131674+0000","last_became_active":"2024-03-26T05:00:26.956677+0000","last_became_peered":"2024-03-26T05:00:26.956677+0000","last_unstale":"2024-03-26T05:05:14.131674+0000","last_undegraded":"2024-03-26T05:05:14.131674+0000","last_fullsized":"2024-03-26T05:05:14.131674+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:22.633581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:22.633581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:30.314139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"155'11834","reported_seq":12072,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:19.724258+0000","last_change":"2024-03-26T05:05:12.881454+0000","last_active":"2024-03-26T05:05:19.724258+0000","last_peered":"2024-03-26T05:05:19.724258+0000","last_clean":"2024-03-26T05:05:19.724258+0000","last_became_active":"2024-03-26T05:02:24.353979+0000","last_became_peered":"2024-03-26T05:02:24.353979+0000","last_unstale":"2024-03-26T05:05:19.724258+0000","last_undegraded":"2024-03-26T05:05:19.724258+0000","last_fullsized":"2024-03-26T05:05:19.724258+0000","mapping_epoch":103,"log_start":"130'5900","ondisk_log_start":"130'5900","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'11430","last_scrub_stamp":"2024-03-26T05:05:12.881337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.881337+0000","objects_scrubbed":3464,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:21.234576+0000","scrub_duration":943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199360512,"num_objects":3042,"num_object_clones":0,"num_object_copies":9126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11834,"num_write_kb":476032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":190,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.127351+0000","last_change":"2024-03-26T05:03:20.739630+0000","last_active":"2024-03-26T05:05:14.127351+0000","last_peered":"2024-03-26T05:05:14.127351+0000","last_clean":"2024-03-26T05:05:14.127351+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:05:14.127351+0000","last_undegraded":"2024-03-26T05:05:14.127351+0000","last_fullsized":"2024-03-26T05:05:14.127351+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:03:20.739242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.127298+0000","last_change":"2024-03-26T05:05:05.892471+0000","last_active":"2024-03-26T05:05:14.127298+0000","last_peered":"2024-03-26T05:05:14.127298+0000","last_clean":"2024-03-26T05:05:14.127298+0000","last_became_active":"2024-03-26T05:00:23.346665+0000","last_became_peered":"2024-03-26T05:00:23.346665+0000","last_unstale":"2024-03-26T05:05:14.127298+0000","last_undegraded":"2024-03-26T05:05:14.127298+0000","last_fullsized":"2024-03-26T05:05:14.127298+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:05.892241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:05:05.892241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:11.884582+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'25","reported_seq":255,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.121260+0000","last_change":"2024-03-26T05:05:02.062360+0000","last_active":"2024-03-26T05:05:14.121260+0000","last_peered":"2024-03-26T05:05:14.121260+0000","last_clean":"2024-03-26T05:05:14.121260+0000","last_became_active":"2024-03-26T05:00:25.648253+0000","last_became_peered":"2024-03-26T05:00:25.648253+0000","last_unstale":"2024-03-26T05:05:14.121260+0000","last_undegraded":"2024-03-26T05:05:14.121260+0000","last_fullsized":"2024-03-26T05:05:14.121260+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:05:02.062204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:02.062204+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:24.541196+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":193,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.128405+0000","last_change":"2024-03-26T05:04:04.071060+0000","last_active":"2024-03-26T05:05:14.128405+0000","last_peered":"2024-03-26T05:05:14.128405+0000","last_clean":"2024-03-26T05:05:14.128405+0000","last_became_active":"2024-03-26T05:00:26.958689+0000","last_became_peered":"2024-03-26T05:00:26.958689+0000","last_unstale":"2024-03-26T05:05:14.128405+0000","last_undegraded":"2024-03-26T05:05:14.128405+0000","last_fullsized":"2024-03-26T05:05:14.128405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:04.070935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:04:04.070935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:18.040979+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"155'11891","reported_seq":12060,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:19.750296+0000","last_change":"2024-03-26T05:04:56.303089+0000","last_active":"2024-03-26T05:05:19.750296+0000","last_peered":"2024-03-26T05:05:19.750296+0000","last_clean":"2024-03-26T05:05:19.750296+0000","last_became_active":"2024-03-26T05:02:24.352730+0000","last_became_peered":"2024-03-26T05:02:24.352730+0000","last_unstale":"2024-03-26T05:05:19.750296+0000","last_undegraded":"2024-03-26T05:05:19.750296+0000","last_fullsized":"2024-03-26T05:05:19.750296+0000","mapping_epoch":103,"log_start":"137'7100","ondisk_log_start":"137'7100","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'10463","last_scrub_stamp":"2024-03-26T05:04:56.302865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:56.302865+0000","objects_scrubbed":4618,"log_size":4791,"log_dups_size":0,"ondisk_log_size":4791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:00.980986+0000","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206241792,"num_objects":3147,"num_object_clones":0,"num_object_copies":9441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11891,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":191,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.123243+0000","last_change":"2024-03-26T05:03:19.935218+0000","last_active":"2024-03-26T05:05:14.123243+0000","last_peered":"2024-03-26T05:05:14.123243+0000","last_clean":"2024-03-26T05:05:14.123243+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:05:14.123243+0000","last_undegraded":"2024-03-26T05:05:14.123243+0000","last_fullsized":"2024-03-26T05:05:14.123243+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:19.935094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:19.935094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.130586+0000","last_change":"2024-03-26T05:05:07.893979+0000","last_active":"2024-03-26T05:05:14.130586+0000","last_peered":"2024-03-26T05:05:14.130586+0000","last_clean":"2024-03-26T05:05:14.130586+0000","last_became_active":"2024-03-26T05:00:23.346876+0000","last_became_peered":"2024-03-26T05:00:23.346876+0000","last_unstale":"2024-03-26T05:05:14.130586+0000","last_undegraded":"2024-03-26T05:05:14.130586+0000","last_fullsized":"2024-03-26T05:05:14.130586+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:07.893841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:07.893841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:13.117675+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'16","reported_seq":196,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.133633+0000","last_change":"2024-03-26T05:05:03.406106+0000","last_active":"2024-03-26T05:05:14.133633+0000","last_peered":"2024-03-26T05:05:14.133633+0000","last_clean":"2024-03-26T05:05:14.133633+0000","last_became_active":"2024-03-26T05:00:49.641444+0000","last_became_peered":"2024-03-26T05:00:49.641444+0000","last_unstale":"2024-03-26T05:05:14.133633+0000","last_undegraded":"2024-03-26T05:05:14.133633+0000","last_fullsized":"2024-03-26T05:05:14.133633+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:05:03.405828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:03.405828+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:23.315672+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":271,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.121571+0000","last_change":"2024-03-26T05:05:11.014715+0000","last_active":"2024-03-26T05:05:14.121571+0000","last_peered":"2024-03-26T05:05:14.121571+0000","last_clean":"2024-03-26T05:05:14.121571+0000","last_became_active":"2024-03-26T05:00:26.961907+0000","last_became_peered":"2024-03-26T05:00:26.961907+0000","last_unstale":"2024-03-26T05:05:14.121571+0000","last_undegraded":"2024-03-26T05:05:14.121571+0000","last_fullsized":"2024-03-26T05:05:14.121571+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:05:11.014574+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:20.408109+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"155'11660","reported_seq":11852,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:15.045824+0000","last_change":"2024-03-26T05:04:19.068828+0000","last_active":"2024-03-26T05:05:15.045824+0000","last_peered":"2024-03-26T05:05:15.045824+0000","last_clean":"2024-03-26T05:05:15.045824+0000","last_became_active":"2024-03-26T05:02:24.354286+0000","last_became_peered":"2024-03-26T05:02:24.354286+0000","last_unstale":"2024-03-26T05:05:15.045824+0000","last_undegraded":"2024-03-26T05:05:15.045824+0000","last_fullsized":"2024-03-26T05:05:15.045824+0000","mapping_epoch":103,"log_start":"137'7000","ondisk_log_start":"137'7000","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'8121","last_scrub_stamp":"2024-03-26T05:04:19.068650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:19.068650+0000","objects_scrubbed":7009,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:25.300640+0000","scrub_duration":3488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221642752,"num_objects":3382,"num_object_clones":0,"num_object_copies":10146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11660,"num_write_kb":481344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":189,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.118215+0000","last_change":"2024-03-26T05:04:24.924461+0000","last_active":"2024-03-26T05:05:14.118215+0000","last_peered":"2024-03-26T05:05:14.118215+0000","last_clean":"2024-03-26T05:05:14.118215+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:05:14.118215+0000","last_undegraded":"2024-03-26T05:05:14.118215+0000","last_fullsized":"2024-03-26T05:05:14.118215+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:24.924377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:24.924377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:30.780658+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":212,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.132384+0000","last_change":"2024-03-26T05:05:01.049762+0000","last_active":"2024-03-26T05:05:14.132384+0000","last_peered":"2024-03-26T05:05:14.132384+0000","last_clean":"2024-03-26T05:05:14.132384+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:05:14.132384+0000","last_undegraded":"2024-03-26T05:05:14.132384+0000","last_fullsized":"2024-03-26T05:05:14.132384+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:05:01.049437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:01.049437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:08.380023+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'20","reported_seq":232,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108743+0000","last_change":"2024-03-26T05:04:25.126002+0000","last_active":"2024-03-26T05:05:14.108743+0000","last_peered":"2024-03-26T05:05:14.108743+0000","last_clean":"2024-03-26T05:05:14.108743+0000","last_became_active":"2024-03-26T05:00:24.936315+0000","last_became_peered":"2024-03-26T05:00:24.936315+0000","last_unstale":"2024-03-26T05:05:14.108743+0000","last_undegraded":"2024-03-26T05:05:14.108743+0000","last_fullsized":"2024-03-26T05:05:14.108743+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:04:25.125864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:25.125864+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:33.277048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":261,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.140934+0000","last_change":"2024-03-26T05:04:26.670277+0000","last_active":"2024-03-26T05:05:14.140934+0000","last_peered":"2024-03-26T05:05:14.140934+0000","last_clean":"2024-03-26T05:05:14.140934+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:05:14.140934+0000","last_undegraded":"2024-03-26T05:05:14.140934+0000","last_fullsized":"2024-03-26T05:05:14.140934+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:26.669895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:26.669895+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:47.751228+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"155'11575","reported_seq":11744,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:15.032185+0000","last_change":"2024-03-26T05:04:58.534809+0000","last_active":"2024-03-26T05:05:15.032185+0000","last_peered":"2024-03-26T05:05:15.032185+0000","last_clean":"2024-03-26T05:05:15.032185+0000","last_became_active":"2024-03-26T05:02:24.352725+0000","last_became_peered":"2024-03-26T05:02:24.352725+0000","last_unstale":"2024-03-26T05:05:15.032185+0000","last_undegraded":"2024-03-26T05:05:15.032185+0000","last_fullsized":"2024-03-26T05:05:15.032185+0000","mapping_epoch":103,"log_start":"135'6900","ondisk_log_start":"135'6900","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'10511","last_scrub_stamp":"2024-03-26T05:04:58.534644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.534644+0000","objects_scrubbed":4512,"log_size":4675,"log_dups_size":0,"ondisk_log_size":4675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:22.836586+0000","scrub_duration":1678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223936512,"num_objects":3417,"num_object_clones":0,"num_object_copies":10251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11575,"num_write_kb":479744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":191,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108403+0000","last_change":"2024-03-26T05:04:10.112906+0000","last_active":"2024-03-26T05:05:14.108403+0000","last_peered":"2024-03-26T05:05:14.108403+0000","last_clean":"2024-03-26T05:05:14.108403+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:05:14.108403+0000","last_undegraded":"2024-03-26T05:05:14.108403+0000","last_fullsized":"2024-03-26T05:05:14.108403+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:10.112830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:10.112830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.142974+0000","last_change":"2024-03-26T05:04:54.805782+0000","last_active":"2024-03-26T05:05:14.142974+0000","last_peered":"2024-03-26T05:05:14.142974+0000","last_clean":"2024-03-26T05:05:14.142974+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:05:14.142974+0000","last_undegraded":"2024-03-26T05:05:14.142974+0000","last_fullsized":"2024-03-26T05:05:14.142974+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:54.805655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:54.805655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:16.316884+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"96'40","reported_seq":272,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.140842+0000","last_change":"2024-03-26T05:05:13.704632+0000","last_active":"2024-03-26T05:05:14.140842+0000","last_peered":"2024-03-26T05:05:14.140842+0000","last_clean":"2024-03-26T05:05:14.140842+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:05:14.140842+0000","last_undegraded":"2024-03-26T05:05:14.140842+0000","last_fullsized":"2024-03-26T05:05:14.140842+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:05:13.704484+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:05:13.704484+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:15.521470+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":187,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108227+0000","last_change":"2024-03-26T05:03:09.332660+0000","last_active":"2024-03-26T05:05:14.108227+0000","last_peered":"2024-03-26T05:05:14.108227+0000","last_clean":"2024-03-26T05:05:14.108227+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:05:14.108227+0000","last_undegraded":"2024-03-26T05:05:14.108227+0000","last_fullsized":"2024-03-26T05:05:14.108227+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:09.332533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.332533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"155'11811","reported_seq":12006,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:19.762888+0000","last_change":"2024-03-26T05:04:14.777321+0000","last_active":"2024-03-26T05:05:19.762888+0000","last_peered":"2024-03-26T05:05:19.762888+0000","last_clean":"2024-03-26T05:05:19.762888+0000","last_became_active":"2024-03-26T05:02:24.351780+0000","last_became_peered":"2024-03-26T05:02:24.351780+0000","last_unstale":"2024-03-26T05:05:19.762888+0000","last_undegraded":"2024-03-26T05:05:19.762888+0000","last_fullsized":"2024-03-26T05:05:19.762888+0000","mapping_epoch":103,"log_start":"127'5600","ondisk_log_start":"127'5600","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7828","last_scrub_stamp":"2024-03-26T05:04:14.777151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:14.777151+0000","objects_scrubbed":7129,"log_size":6211,"log_dups_size":0,"ondisk_log_size":6211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:37.260876+0000","scrub_duration":1692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203227136,"num_objects":3101,"num_object_clones":0,"num_object_copies":9303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11811,"num_write_kb":477184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":198,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.130327+0000","last_change":"2024-03-26T05:05:04.886043+0000","last_active":"2024-03-26T05:05:14.130327+0000","last_peered":"2024-03-26T05:05:14.130327+0000","last_clean":"2024-03-26T05:05:14.130327+0000","last_became_active":"2024-03-26T05:00:28.982825+0000","last_became_peered":"2024-03-26T05:00:28.982825+0000","last_unstale":"2024-03-26T05:05:14.130327+0000","last_undegraded":"2024-03-26T05:05:14.130327+0000","last_fullsized":"2024-03-26T05:05:14.130327+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.885959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.885959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:32.680637+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":263,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.109164+0000","last_change":"2024-03-26T05:04:09.071212+0000","last_active":"2024-03-26T05:05:14.109164+0000","last_peered":"2024-03-26T05:05:14.109164+0000","last_clean":"2024-03-26T05:05:14.109164+0000","last_became_active":"2024-03-26T05:00:26.956596+0000","last_became_peered":"2024-03-26T05:00:26.956596+0000","last_unstale":"2024-03-26T05:05:14.109164+0000","last_undegraded":"2024-03-26T05:05:14.109164+0000","last_fullsized":"2024-03-26T05:05:14.109164+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:09.071057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:09.071057+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.140796+0000","last_change":"2024-03-26T05:04:06.525151+0000","last_active":"2024-03-26T05:05:14.140796+0000","last_peered":"2024-03-26T05:05:14.140796+0000","last_clean":"2024-03-26T05:05:14.140796+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:05:14.140796+0000","last_undegraded":"2024-03-26T05:05:14.140796+0000","last_fullsized":"2024-03-26T05:05:14.140796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:06.525047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:06.525047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:21.294383+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":231,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.129934+0000","last_change":"2024-03-26T05:04:24.625537+0000","last_active":"2024-03-26T05:05:14.129934+0000","last_peered":"2024-03-26T05:05:14.129934+0000","last_clean":"2024-03-26T05:05:14.129934+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:05:14.129934+0000","last_undegraded":"2024-03-26T05:05:14.129934+0000","last_fullsized":"2024-03-26T05:05:14.129934+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:04:24.625406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:24.625406+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:47.792081+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"155'12081","reported_seq":12294,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:19.761730+0000","last_change":"2024-03-26T05:05:14.154844+0000","last_active":"2024-03-26T05:05:19.761730+0000","last_peered":"2024-03-26T05:05:19.761730+0000","last_clean":"2024-03-26T05:05:19.761730+0000","last_became_active":"2024-03-26T05:02:24.352008+0000","last_became_peered":"2024-03-26T05:02:24.352008+0000","last_unstale":"2024-03-26T05:05:19.761730+0000","last_undegraded":"2024-03-26T05:05:19.761730+0000","last_fullsized":"2024-03-26T05:05:19.761730+0000","mapping_epoch":103,"log_start":"128'5900","ondisk_log_start":"128'5900","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7544","last_scrub_stamp":"2024-03-26T05:04:08.237042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:08.237042+0000","objects_scrubbed":1149,"log_size":6181,"log_dups_size":0,"ondisk_log_size":6181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197001240,"num_objects":3007,"num_object_clones":0,"num_object_copies":9021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12081,"num_write_kb":482753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":190,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.140679+0000","last_change":"2024-03-26T05:04:25.636127+0000","last_active":"2024-03-26T05:05:14.140679+0000","last_peered":"2024-03-26T05:05:14.140679+0000","last_clean":"2024-03-26T05:05:14.140679+0000","last_became_active":"2024-03-26T05:00:29.461236+0000","last_became_peered":"2024-03-26T05:00:29.461236+0000","last_unstale":"2024-03-26T05:05:14.140679+0000","last_undegraded":"2024-03-26T05:05:14.140679+0000","last_fullsized":"2024-03-26T05:05:14.140679+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:25.636012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:25.636012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:37.693140+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.131594+0000","last_change":"2024-03-26T05:05:05.919957+0000","last_active":"2024-03-26T05:05:14.131594+0000","last_peered":"2024-03-26T05:05:14.131594+0000","last_clean":"2024-03-26T05:05:14.131594+0000","last_became_active":"2024-03-26T05:00:26.955805+0000","last_became_peered":"2024-03-26T05:00:26.955805+0000","last_unstale":"2024-03-26T05:05:14.131594+0000","last_undegraded":"2024-03-26T05:05:14.131594+0000","last_fullsized":"2024-03-26T05:05:14.131594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:05.919860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:05.919860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:26.915238+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108599+0000","last_change":"2024-03-26T05:03:19.466684+0000","last_active":"2024-03-26T05:05:14.108599+0000","last_peered":"2024-03-26T05:05:14.108599+0000","last_clean":"2024-03-26T05:05:14.108599+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:05:14.108599+0000","last_undegraded":"2024-03-26T05:05:14.108599+0000","last_fullsized":"2024-03-26T05:05:14.108599+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:19.466546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:19.466546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'17","reported_seq":230,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.109054+0000","last_change":"2024-03-26T05:03:23.417863+0000","last_active":"2024-03-26T05:05:14.109054+0000","last_peered":"2024-03-26T05:05:14.109054+0000","last_clean":"2024-03-26T05:05:14.109054+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:05:14.109054+0000","last_undegraded":"2024-03-26T05:05:14.109054+0000","last_fullsized":"2024-03-26T05:05:14.109054+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:03:23.417719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:03:23.417719+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"155'11655","reported_seq":11853,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.935635+0000","last_change":"2024-03-26T05:04:09.590882+0000","last_active":"2024-03-26T05:05:14.935635+0000","last_peered":"2024-03-26T05:05:14.935635+0000","last_clean":"2024-03-26T05:05:14.935635+0000","last_became_active":"2024-03-26T05:02:24.357742+0000","last_became_peered":"2024-03-26T05:02:24.357742+0000","last_unstale":"2024-03-26T05:05:14.935635+0000","last_undegraded":"2024-03-26T05:05:14.935635+0000","last_fullsized":"2024-03-26T05:05:14.935635+0000","mapping_epoch":103,"log_start":"122'4800","ondisk_log_start":"122'4800","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7576","last_scrub_stamp":"2024-03-26T05:04:09.590405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:04:09.590405+0000","objects_scrubbed":7467,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:26.578806+0000","scrub_duration":3319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220135424,"num_objects":3359,"num_object_clones":0,"num_object_copies":10077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11655,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":190,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.123917+0000","last_change":"2024-03-26T05:03:21.940017+0000","last_active":"2024-03-26T05:05:14.123917+0000","last_peered":"2024-03-26T05:05:14.123917+0000","last_clean":"2024-03-26T05:05:14.123917+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:05:14.123917+0000","last_undegraded":"2024-03-26T05:05:14.123917+0000","last_fullsized":"2024-03-26T05:05:14.123917+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:21.939782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:03:21.939782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":194,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.127569+0000","last_change":"2024-03-26T05:03:18.783710+0000","last_active":"2024-03-26T05:05:14.127569+0000","last_peered":"2024-03-26T05:05:14.127569+0000","last_clean":"2024-03-26T05:05:14.127569+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:05:14.127569+0000","last_undegraded":"2024-03-26T05:05:14.127569+0000","last_fullsized":"2024-03-26T05:05:14.127569+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:18.783471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:03:18.783471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.128426+0000","last_change":"2024-03-26T05:04:21.884967+0000","last_active":"2024-03-26T05:05:14.128426+0000","last_peered":"2024-03-26T05:05:14.128426+0000","last_clean":"2024-03-26T05:05:14.128426+0000","last_became_active":"2024-03-26T05:00:22.768610+0000","last_became_peered":"2024-03-26T05:00:22.768610+0000","last_unstale":"2024-03-26T05:05:14.128426+0000","last_undegraded":"2024-03-26T05:05:14.128426+0000","last_fullsized":"2024-03-26T05:05:14.128426+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:21.884876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_clean_scrub_stamp":"2024-03-26T05:04:21.884876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:37.934924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":227,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108863+0000","last_change":"2024-03-26T05:05:04.091067+0000","last_active":"2024-03-26T05:05:14.108863+0000","last_peered":"2024-03-26T05:05:14.108863+0000","last_clean":"2024-03-26T05:05:14.108863+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:05:14.108863+0000","last_undegraded":"2024-03-26T05:05:14.108863+0000","last_fullsized":"2024-03-26T05:05:14.108863+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:05:04.090960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.090960+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:27.771650+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"155'11482","reported_seq":11648,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:15.038022+0000","last_change":"2024-03-26T05:05:00.025500+0000","last_active":"2024-03-26T05:05:15.038022+0000","last_peered":"2024-03-26T05:05:15.038022+0000","last_clean":"2024-03-26T05:05:15.038022+0000","last_became_active":"2024-03-26T05:02:24.352196+0000","last_became_peered":"2024-03-26T05:02:24.352196+0000","last_unstale":"2024-03-26T05:05:15.038022+0000","last_undegraded":"2024-03-26T05:05:15.038022+0000","last_fullsized":"2024-03-26T05:05:15.038022+0000","mapping_epoch":103,"log_start":"130'5900","ondisk_log_start":"130'5900","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'10519","last_scrub_stamp":"2024-03-26T05:05:00.025304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:23.329282+0000","last_clean_scrub_stamp":"2024-03-26T05:05:00.025304+0000","objects_scrubbed":4328,"log_size":5582,"log_dups_size":0,"ondisk_log_size":5582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:06:29.127404+0000","scrub_duration":2074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216006656,"num_objects":3296,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11482,"num_write_kb":472896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":191,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.109178+0000","last_change":"2024-03-26T05:04:22.211543+0000","last_active":"2024-03-26T05:05:14.109178+0000","last_peered":"2024-03-26T05:05:14.109178+0000","last_clean":"2024-03-26T05:05:14.109178+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:05:14.109178+0000","last_undegraded":"2024-03-26T05:05:14.109178+0000","last_fullsized":"2024-03-26T05:05:14.109178+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:22.211431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:22.211431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:23.405861+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"117'9","reported_seq":262,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.123269+0000","last_change":"2024-03-26T05:03:09.035963+0000","last_active":"2024-03-26T05:05:14.123269+0000","last_peered":"2024-03-26T05:05:14.123269+0000","last_clean":"2024-03-26T05:05:14.123269+0000","last_became_active":"2024-03-26T05:00:26.957142+0000","last_became_peered":"2024-03-26T05:00:26.957142+0000","last_unstale":"2024-03-26T05:05:14.123269+0000","last_undegraded":"2024-03-26T05:05:14.123269+0000","last_fullsized":"2024-03-26T05:05:14.123269+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'9","last_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.035843+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.141338+0000","last_change":"2024-03-26T05:03:09.192328+0000","last_active":"2024-03-26T05:05:14.141338+0000","last_peered":"2024-03-26T05:05:14.141338+0000","last_clean":"2024-03-26T05:05:14.141338+0000","last_became_active":"2024-03-26T05:00:22.778719+0000","last_became_peered":"2024-03-26T05:00:22.778719+0000","last_unstale":"2024-03-26T05:05:14.141338+0000","last_undegraded":"2024-03-26T05:05:14.141338+0000","last_fullsized":"2024-03-26T05:05:14.141338+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:03:09.192105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:03:09.192105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":214,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-26T05:05:14.108489+0000","last_change":"2024-03-26T05:04:16.153198+0000","last_active":"2024-03-26T05:05:14.108489+0000","last_peered":"2024-03-26T05:05:14.108489+0000","last_clean":"2024-03-26T05:05:14.108489+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:05:14.108489+0000","last_undegraded":"2024-03-26T05:05:14.108489+0000","last_fullsized":"2024-03-26T05:05:14.108489+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:04:16.153054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:16.153054+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:05:39.530782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3418226712,"num_objects":52159,"num_object_clones":0,"num_object_copies":156477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186835,"num_write_kb":7647745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10235686912,"data_stored":10235674696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89635,"ondisk_log_size":89635,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":291,"num_read_kb":286,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":617,"ondisk_log_size":617,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005120,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1106320,"kb_used_data":1002952,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92638832,"statfs":{"total":95995035648,"available":94862163968,"internally_reserved":0,"allocated":1027022848,"data_stored":1026729368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332165,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1097348,"kb_used_data":989820,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92647804,"statfs":{"total":95995035648,"available":94871351296,"internally_reserved":0,"allocated":1013575680,"data_stored":1013292996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1417320,"kb_used_data":1313824,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92327832,"statfs":{"total":95995035648,"available":94543699968,"internally_reserved":0,"allocated":1345355776,"data_stored":1345028846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1861140,"kb_used_data":1735308,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":91884012,"statfs":{"total":95995035648,"available":94089228288,"internally_reserved":0,"allocated":1776955392,"data_stored":1776649891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1414264,"kb_used_data":1310828,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92330888,"statfs":{"total":95995035648,"available":94546829312,"internally_reserved":0,"allocated":1342287872,"data_stored":1342020818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117075,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2094236,"kb_used_data":1958928,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91650916,"statfs":{"total":95995035648,"available":93850537984,"internally_reserved":0,"allocated":2005942272,"data_stored":2005639020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138541511},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1118612,"kb_used_data":1011084,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92626540,"statfs":{"total":95995035648,"available":94849576960,"internally_reserved":0,"allocated":1035350016,"data_stored":1035051449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659210,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":883520,"kb_used_data":807928,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92861632,"statfs":{"total":95995035648,"available":95090311168,"internally_reserved":0,"allocated":827318272,"data_stored":827039442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1743130624,"data_stored":1743126552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1324945408,"data_stored":1324941336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1329397760,"data_stored":1329397760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1987837952,"data_stored":1987837952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1019936768,"data_stored":1019936768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":812318720,"data_stored":812318720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1010962432,"data_stored":1010958360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1007157248,"data_stored":1007157248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:05:21.008 INFO:tasks.thrashosds.thrasher:Setting 3.1 to [4, 0, 5] 2024-03-26T05:05:21.008 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.1', '4', '0', '5'] 2024-03-26T05:05:21.008 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pg-upmap 3.1 4 0 5 2024-03-26T05:05:21.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:21.034 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:21.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:21.062 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:21.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:21.096 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:05:21.163 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:21.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:21 smithi080 ceph-mon[29691]: pgmap v198: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-26T05:05:21.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:21 smithi080 ceph-mon[29691]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:05:21.216 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:21 smithi080 ceph-mon[29691]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T05:05:21.216 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:21 smithi080 ceph-mon[40404]: pgmap v198: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-26T05:05:21.216 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:21 smithi080 ceph-mon[40404]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:05:21.216 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:21 smithi080 ceph-mon[40404]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T05:05:21.219 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:21.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:21.321 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:21.380 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:21.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:21.481 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:21.511 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:21 smithi132 ceph-mon[33500]: pgmap v198: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-26T05:05:21.511 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:21 smithi132 ceph-mon[33500]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:05:21.512 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:21 smithi132 ceph-mon[33500]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T05:05:21.528 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:21.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:21.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:21.630 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:21.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:21.673 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:21.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:21.774 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:21.802 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:21.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:21.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:21.904 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:21.922 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:21.931 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:21.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:22.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:22.063 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:22.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:22.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:22.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:22.194 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:22.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:22.295 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:22.311 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:05:22.324 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:22.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:22.426 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:22.471 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:22.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:22.574 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:22.607 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:22.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:22.708 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:22.736 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:22.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:22.838 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:22.865 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:22.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:22.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:22.968 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:23.003 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:23.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:23.104 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:23.146 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:23.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:23.247 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:23.278 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:23.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:23.380 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:23.432 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:23.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:23.533 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:23.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:23.573 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:23.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:23.607 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:05:23.637 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:05:23.638 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:23 smithi080 ceph-mon[40404]: pgmap v200: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-26T05:05:23.638 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:23 smithi080 ceph-mon[29691]: pgmap v200: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-26T05:05:23.675 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:23.682 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:05:23.712 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:23.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:23.723 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:05:23.763 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:05:23.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:23 smithi132 ceph-mon[33500]: pgmap v200: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-26T05:05:23.800 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:05:23.815 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:23.850 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:23.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:23.863 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:05:23.902 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:05:23.912 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:05:23.912 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:05:23.912 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:05:23.912 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:05:23.912 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:05:23.912 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:05:23.913 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:05:23.913 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:05:23.951 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:23.978 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:23.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:24.080 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:24.110 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:24.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:24.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:24.212 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:24.241 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:05:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:05:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:05:24.247 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:24.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:24.350 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:24.391 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:24.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:24.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:24.493 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:24.522 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:24.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:24.531 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:24 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/951816001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:05:24.532 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:24 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/951816001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:05:24.624 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:24.671 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:24.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:24.772 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:24.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/951816001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:05:24.810 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:24.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:24.912 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:24.942 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:24.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:24.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:25.044 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:25.088 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:25.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:25.128 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:25.189 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:25.215 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:25.216 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:05:25.216 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:05:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:25.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:25.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:05:24 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:05:25.318 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:25.346 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:25.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:25.448 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:25.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:25.493 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:25.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:25.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:25 smithi132 ceph-mon[33500]: pgmap v201: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-26T05:05:25.595 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:25.624 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:25.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:25.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:25.749 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:25.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:25.850 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:25.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:25 smithi080 ceph-mon[29691]: pgmap v201: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-26T05:05:25.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:25 smithi080 ceph-mon[40404]: pgmap v201: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-26T05:05:25.893 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:25.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:25.995 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:26.023 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:26.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:26.091 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T05:05:26.125 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:26.159 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:26.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:26.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:26.261 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:26.305 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:26.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:26.406 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:26.407 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:05:26.435 INFO:teuthology.orchestra.run.smithi080.stderr:set 3.1 pg_upmap mapping to [4,0,5] 2024-03-26T05:05:26.447 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:26.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:26.528 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:26 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:26.528 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:26 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3320777162' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.1", "id": [4, 0, 5]}]: dispatch 2024-03-26T05:05:26.528 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:26 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.1", "id": [4, 0, 5]}]: dispatch 2024-03-26T05:05:26.528 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:26 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:26.528 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:26 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3320777162' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.1", "id": [4, 0, 5]}]: dispatch 2024-03-26T05:05:26.528 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:26 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.1", "id": [4, 0, 5]}]: dispatch 2024-03-26T05:05:26.549 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:26.578 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:26.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:26.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:26.679 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:26.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:26.713 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:26.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:26.715 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:26 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:26.715 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:26 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3320777162' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.1", "id": [4, 0, 5]}]: dispatch 2024-03-26T05:05:26.715 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:26 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.1", "id": [4, 0, 5]}]: dispatch 2024-03-26T05:05:26.818 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:26.848 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:05:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:26.850 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:26.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:26.951 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:27.000 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:27.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:27.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:27.102 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:27.153 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:27.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:27.254 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:27.297 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:27.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:27.398 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:27.431 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:27.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:27.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:27.533 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:27.579 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:27.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:27.680 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:27.707 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:27.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:27.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:27.719 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:27 smithi132 ceph-mon[33500]: pgmap v202: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-26T05:05:27.719 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:27 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.1", "id": [4, 0, 5]}]': finished 2024-03-26T05:05:27.719 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:27 smithi132 ceph-mon[33500]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T05:05:27.740 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:27.810 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:27.845 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:27 smithi080 ceph-mon[29691]: pgmap v202: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-26T05:05:27.845 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:27 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.1", "id": [4, 0, 5]}]': finished 2024-03-26T05:05:27.845 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:27 smithi080 ceph-mon[29691]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T05:05:27.845 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:27 smithi080 ceph-mon[40404]: pgmap v202: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-26T05:05:27.846 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:27 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.1", "id": [4, 0, 5]}]': finished 2024-03-26T05:05:27.846 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:27 smithi080 ceph-mon[40404]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T05:05:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:27.870 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:27.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:27.971 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:28.032 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:28.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:28.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:28.134 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:28.176 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:28.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:28.278 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:28.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:28.326 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:28.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:28.411 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:05:28.411 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:05:28.427 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:28.462 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:28.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:28.563 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:28.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:28.604 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:28.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:28.705 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:28.733 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:28.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:28.778 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:28 smithi080 ceph-mon[40404]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T05:05:28.778 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:28 smithi080 ceph-mon[29691]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T05:05:28.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:28 smithi132 ceph-mon[33500]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T05:05:28.836 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:28.865 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:28.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:28.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:28.993 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:28.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:29.095 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:29.122 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:29.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:29.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:29.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:29.252 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:29.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:29.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:29.354 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:29.394 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:29.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:29.419 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T05:05:29.451 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:05:29.495 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:29.537 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:29.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:29.639 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:29.674 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:29.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:29.736 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:05:29.779 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:29.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:29 smithi132 ceph-mon[33500]: pgmap v205: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 671 B/s rd, 0 B/s wr, 694 op/s 2024-03-26T05:05:29.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:29 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4044841159' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:05:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:29.820 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:29.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:29.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:29 smithi080 ceph-mon[29691]: pgmap v205: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 671 B/s rd, 0 B/s wr, 694 op/s 2024-03-26T05:05:29.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:29 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4044841159' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:05:29.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:29 smithi080 ceph-mon[40404]: pgmap v205: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 671 B/s rd, 0 B/s wr, 694 op/s 2024-03-26T05:05:29.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:29 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4044841159' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:05:29.922 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:29.960 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:29.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:30.061 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:30.102 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:30.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:30.204 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:30.235 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:30.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:30.336 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:30.363 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:30.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:30.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:30.464 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:30.505 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:30 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:05:30.506 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:30 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4044841159' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:05:30.506 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:30 smithi080 ceph-mon[29691]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T05:05:30.506 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:30 smithi080 ceph-mon[29691]: pgmap v207: 148 pgs: 148 active+clean; 2.6 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-03-26T05:05:30.506 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:30 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:05:30.506 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:30 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4044841159' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:05:30.506 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:30 smithi080 ceph-mon[40404]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T05:05:30.506 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:30 smithi080 ceph-mon[40404]: pgmap v207: 148 pgs: 148 active+clean; 2.6 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-03-26T05:05:30.520 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:30.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:30.621 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:30.649 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:30.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:30.751 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:30.781 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:30.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:30 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:05:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:30 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4044841159' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:05:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:30 smithi132 ceph-mon[33500]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T05:05:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:30 smithi132 ceph-mon[33500]: pgmap v207: 148 pgs: 148 active+clean; 2.6 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-03-26T05:05:30.884 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:30.892 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:05:30.918 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:30.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:30.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:30.939 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:05:30.980 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:05:31.020 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:31.031 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:05:31.062 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:31.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:31.072 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:05:31.110 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:05:31.152 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:05:31.164 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:31.190 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:05:31.192 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:31.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:31.193 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:05:31.193 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:05:31.193 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:05:31.193 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:05:31.194 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:05:31.194 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:05:31.194 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:05:31.194 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:05:31.293 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:31.321 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:31.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:31.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:31.424 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:31.460 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:31.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:31.561 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:31.575 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:31 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1205521915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:05:31.594 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:31.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:31.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:31.696 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:31.728 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:31.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:31.830 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:31.864 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:31 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1205521915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:05:31.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:31 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1205521915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:05:31.870 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:31.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:31.972 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:32.010 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:05:32.010 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:05:32.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:05:32.010 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:05:32.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.010 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22065 7.5 7:a0e81304:::benchmark_data_smithi080_80848_object22064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.025527+0000", 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 164.98903412799999, 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11927265099999999, 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22065 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.025527+0000", 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.025527+0000", 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.025528+0000", 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3200000000000004e-07 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.025602+0000", 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3906999999999996e-05 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.025611+0000", 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2469999999999997e-06 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.025618+0000", 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2129999999999996e-06 2024-03-26T05:05:32.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.025649+0000", 2024-03-26T05:05:32.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.1047999999999998e-05 2024-03-26T05:05:32.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.025732+0000", 2024-03-26T05:05:32.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3398000000000003e-05 2024-03-26T05:05:32.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.025815+0000", 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3158999999999994e-05 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.026285+0000", 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00046959399999999999 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.027715+0000", 2024-03-26T05:05:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001429909 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.144715+0000", 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.117000701 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.144755+0000", 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9107999999999999e-05 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.144800+0000", 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5534999999999997e-05 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22091 7.5 7:aa4a2012:::benchmark_data_smithi080_80848_object22090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.064325+0000", 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 164.95023640799999, 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.082468866000000002, 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.017 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.018 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.018 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22091 2024-03-26T05:05:32.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.018 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.018 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.018 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.064325+0000", 2024-03-26T05:05:32.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.064325+0000", 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.064326+0000", 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.232e-06 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.064410+0000", 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3195999999999994e-05 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.064426+0000", 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6371e-05 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.064435+0000", 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.9900000000000003e-06 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.064496+0000", 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1036000000000003e-05 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.064632+0000", 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013615799999999999 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.064787+0000", 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015512499999999999 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.066269+0000", 2024-03-26T05:05:32.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0014814450000000001 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.066298+0000", 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.9335999999999998e-05 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.146714+0000", 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.080415887000000005 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.146755+0000", 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0970999999999998e-05 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.146794+0000", 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9119000000000003e-05 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22092 7.5 7:abb4457b:::benchmark_data_smithi080_80848_object22091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.065382+0000", 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 164.94917959399999, 2024-03-26T05:05:32.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.083644660999999995, 2024-03-26T05:05:32.025 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.025 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.025 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.025 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.025 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22092 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.065382+0000", 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.065382+0000", 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.065383+0000", 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2399999999999997e-07 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.065476+0000", 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2961999999999995e-05 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.065486+0000", 2024-03-26T05:05:32.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0631999999999999e-05 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.065493+0000", 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.657e-06 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.065573+0000", 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0211999999999997e-05 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.028 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.065689+0000", 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011538499999999999 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.065776+0000", 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7001000000000003e-05 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.066373+0000", 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00059783099999999995 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.066577+0000", 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00020372899999999999 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.148911+0000", 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.082334236000000005 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.148982+0000", 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0792999999999998e-05 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.030 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.149027+0000", 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4499000000000002e-05 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:121788 7.5 7:a9476dbb:::benchmark_data_smithi080_80848_object2289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:11.001969+0000", 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 81.012592307999995, 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.092351599000000006, 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.031 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 121788 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.001969+0000", 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.001969+0000", 2024-03-26T05:05:32.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.001971+0000", 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.499e-06 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.001976+0000", 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5589999999999996e-06 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.001990+0000", 2024-03-26T05:05:32.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3416e-05 2024-03-26T05:05:32.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.002000+0000", 2024-03-26T05:05:32.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0135e-05 2024-03-26T05:05:32.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.002054+0000", 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4205e-05 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.002187+0000", 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013291999999999999 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.002310+0000", 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012310500000000001 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.004817+0000", 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0025065059999999999 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.047390+0000", 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.042573277999999999 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.094203+0000", 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046812694000000002 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.094268+0000", 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5790000000000005e-05 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.094321+0000", 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2491999999999998e-05 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:124904 7.5 7:a48172ee:::benchmark_data_smithi080_80848_object5405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-26T05:05:32.037 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:14.463088+0000", 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 77.551473740999995, 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.083546386, 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 124904 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.463088+0000", 2024-03-26T05:05:32.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.463088+0000", 2024-03-26T05:05:32.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.463089+0000", 2024-03-26T05:05:32.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3480000000000001e-06 2024-03-26T05:05:32.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:32.041 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:32.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:32.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.463095+0000", 2024-03-26T05:05:32.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.643e-06 2024-03-26T05:05:32.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.463108+0000", 2024-03-26T05:05:32.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3349e-05 2024-03-26T05:05:32.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.463117+0000", 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3329999999999997e-06 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.463162+0000", 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5076e-05 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.463266+0000", 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010425900000000001 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.463379+0000", 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000113186 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.465669+0000", 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0022898860000000001 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.466599+0000", 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00092982099999999995 2024-03-26T05:05:32.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.546530+0000", 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.079930876999999997 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.546590+0000", 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0591000000000003e-05 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.546634+0000", 2024-03-26T05:05:32.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4017000000000002e-05 2024-03-26T05:05:32.046 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.046 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.046 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.046 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:124910 7.5 7:a037fdd6:::benchmark_data_smithi080_80848_object5411:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-26T05:05:32.046 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:14.470466+0000", 2024-03-26T05:05:32.046 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 77.544095425999998, 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.090918735000000001, 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 124910 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.470466+0000", 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.470466+0000", 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.470468+0000", 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.621e-06 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.470476+0000", 2024-03-26T05:05:32.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3000000000000002e-06 2024-03-26T05:05:32.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.470491+0000", 2024-03-26T05:05:32.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4939e-05 2024-03-26T05:05:32.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.470500+0000", 2024-03-26T05:05:32.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.9490000000000002e-06 2024-03-26T05:05:32.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.470559+0000", 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9123000000000003e-05 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.470687+0000", 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000128322 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.470799+0000", 2024-03-26T05:05:32.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000111946 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.475221+0000", 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.004421726 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.476625+0000", 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0014036999999999999 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.561170+0000", 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.084544776000000002 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.561234+0000", 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4299000000000002e-05 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.561385+0000", 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000151034 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.052 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:124912 7.5 7:a31a0a81:::benchmark_data_smithi080_80848_object5413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:14.471734+0000", 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 77.542828005000004, 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.089798830999999996, 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 124912 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.471734+0000", 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.471734+0000", 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.471734+0000", 2024-03-26T05:05:32.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5199999999999995e-07 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.471738+0000", 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9369999999999997e-06 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.471751+0000", 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2449999999999999e-05 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.471758+0000", 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2230000000000001e-06 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.473042+0000", 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0012842730000000001 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.473167+0000", 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000124762 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.473287+0000", 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119781 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.475377+0000", 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020897720000000002 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.476760+0000", 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001383312 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.561433+0000", 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.084673100000000001 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.561489+0000", 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5779999999999998e-05 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.561532+0000", 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3588999999999999e-05 2024-03-26T05:05:32.058 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:124918 7.5 7:aa63349a:::benchmark_data_smithi080_80848_object5419:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:14.477697+0000", 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 77.536864746000006, 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.084006745999999993, 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 124918 2024-03-26T05:05:32.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.477697+0000", 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.477697+0000", 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.477698+0000", 2024-03-26T05:05:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3120000000000001e-06 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.477704+0000", 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9689999999999999e-06 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.477714+0000", 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0030000000000001e-05 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.477720+0000", 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1399999999999997e-06 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.477773+0000", 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2225999999999998e-05 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.477903+0000", 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013052699999999999 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.478033+0000", 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013028599999999999 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.480036+0000", 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020023250000000001 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.480842+0000", 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00080613500000000003 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.561598+0000", 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.080755808999999998 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.561658+0000", 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0494000000000001e-05 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:14.561704+0000", 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5493e-05 2024-03-26T05:05:32.064 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:128550 7.5 7:a078bc82:::benchmark_data_smithi080_80848_object9051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:18.289857+0000", 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 73.724705032000003, 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.082931901000000002, 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 128550 2024-03-26T05:05:32.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.289857+0000", 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.289857+0000", 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.289858+0000", 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.344e-06 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.289862+0000", 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4680000000000003e-06 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.289876+0000", 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3878e-05 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.289884+0000", 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6510000000000001e-06 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.289938+0000", 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3749999999999999e-05 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.290053+0000", 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115605 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.290168+0000", 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011434600000000001 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.294907+0000", 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0047390219999999999 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.295498+0000", 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00059099399999999996 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.372717+0000", 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.077219615000000005 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.372756+0000", 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8717999999999998e-05 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:18.372788+0000", 2024-03-26T05:05:32.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2509999999999999e-05 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:129384 7.5 7:a4a8da78:::benchmark_data_smithi080_80848_object9885:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:19.344424+0000", 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 72.670137741999994, 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.085540466999999995, 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 129384 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.344424+0000", 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.344424+0000", 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.344426+0000", 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.674e-06 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.344431+0000", 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5339999999999999e-06 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.344455+0000", 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.351e-05 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.344464+0000", 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.533e-06 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.344524+0000", 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9741000000000003e-05 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.344653+0000", 2024-03-26T05:05:32.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000129424 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.344789+0000", 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000135446 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.349592+0000", 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0048031999999999997 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.351350+0000", 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0017581840000000001 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.429864+0000", 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.078513437000000005 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.429911+0000", 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7777e-05 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.429964+0000", 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3007000000000003e-05 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:129386 7.5 7:a8349627:::benchmark_data_smithi080_80848_object9887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:19.345511+0000", 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 72.669050476999999, 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.084605009999999994, 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.077 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 129386 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.345511+0000", 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.345511+0000", 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.345512+0000", 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5000000000000001e-07 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.345515+0000", 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.1319999999999998e-06 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.345526+0000", 2024-03-26T05:05:32.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0849000000000001e-05 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.345534+0000", 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8480000000000001e-06 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.347263+0000", 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0017294249999999999 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.347387+0000", 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012382499999999999 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.347511+0000", 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012411099999999999 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.349707+0000", 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021954140000000001 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.357684+0000", 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0079776840000000005 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.430018+0000", 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.072333691000000006 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.430074+0000", 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6042999999999999e-05 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:19.430116+0000", 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2138000000000002e-05 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:157214 7.5 7:a5448770:::benchmark_data_smithi080_80848_object37715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:46.725077+0000", 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 45.289484721999997, 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.12596942899999999, 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.083 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 157214 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.725077+0000", 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.725077+0000", 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.725078+0000", 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4169999999999999e-06 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.725083+0000", 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.408e-06 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.725093+0000", 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0662000000000001e-05 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.725101+0000", 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8830000000000006e-06 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.725185+0000", 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3677999999999994e-05 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.725318+0000", 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013278199999999999 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.725503+0000", 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000185236 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.729666+0000", 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0041629329999999997 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.729778+0000", 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000112204 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.850957+0000", 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.12117905299999999 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.851003+0000", 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5998999999999998e-05 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.851046+0000", 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3173999999999997e-05 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:157221 7.5 7:a0e04ea7:::benchmark_data_smithi080_80848_object37722:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:46.732395+0000", 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 45.282166861999997, 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.124014191, 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 157221 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732395+0000", 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732395+0000", 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732396+0000", 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1400000000000001e-06 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732400+0000", 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7050000000000001e-06 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732410+0000", 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0468e-05 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732418+0000", 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6790000000000005e-06 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732474+0000", 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6725999999999999e-05 2024-03-26T05:05:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732553+0000", 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8999999999999996e-05 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732623+0000", 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9377999999999999e-05 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.738114+0000", 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0054907619999999997 2024-03-26T05:05:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.738128+0000", 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4759999999999999e-05 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.856311+0000", 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.118182675 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.856355+0000", 2024-03-26T05:05:32.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3649000000000002e-05 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.856409+0000", 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4249000000000002e-05 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:157226 7.5 7:a77f1c76:::benchmark_data_smithi080_80848_object37727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:46.735442+0000", 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 45.279119553000001, 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.121633981, 2024-03-26T05:05:32.095 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.096 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.096 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.096 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.096 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.096 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 157226 2024-03-26T05:05:32.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.096 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.735442+0000", 2024-03-26T05:05:32.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.735442+0000", 2024-03-26T05:05:32.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.735443+0000", 2024-03-26T05:05:32.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4449999999999999e-06 2024-03-26T05:05:32.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.735448+0000", 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7779999999999998e-06 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.735460+0000", 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1364e-05 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.735530+0000", 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9898000000000001e-05 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.735595+0000", 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5184000000000005e-05 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.735734+0000", 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013931100000000001 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.735868+0000", 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013392699999999999 2024-03-26T05:05:32.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.738181+0000", 2024-03-26T05:05:32.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0023132040000000001 2024-03-26T05:05:32.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.738623+0000", 2024-03-26T05:05:32.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00044138899999999998 2024-03-26T05:05:32.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.857010+0000", 2024-03-26T05:05:32.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.118387771 2024-03-26T05:05:32.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.102 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.857044+0000", 2024-03-26T05:05:32.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.3319999999999999e-05 2024-03-26T05:05:32.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.102 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.857076+0000", 2024-03-26T05:05:32.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.239e-05 2024-03-26T05:05:32.103 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.103 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.103 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.103 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:157481 7.5 7:a9c13086:::benchmark_data_smithi080_80848_object37982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:05:32.103 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:47.070434+0000", 2024-03-26T05:05:32.103 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 44.944127745000003, 2024-03-26T05:05:32.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.131676552, 2024-03-26T05:05:32.103 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.103 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 157481 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.070434+0000", 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.070434+0000", 2024-03-26T05:05:32.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.070435+0000", 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3510000000000001e-06 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.070440+0000", 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5800000000000002e-06 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.070452+0000", 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1924e-05 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.070462+0000", 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0319999999999999e-05 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.070512+0000", 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0263999999999999e-05 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.070616+0000", 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103966 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.070721+0000", 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010484099999999999 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.073486+0000", 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0027648540000000002 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.126749+0000", 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.053262608000000003 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.202006+0000", 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.075257764000000005 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.202067+0000", 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1017000000000002e-05 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:47.202110+0000", 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3062999999999997e-05 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:186910 7.5 7:abca9db6:::benchmark_data_smithi080_80848_object67411:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:16.758728+0000", 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 15.255833952, 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.121751925, 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.109 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 186910 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.758728+0000", 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.758728+0000", 2024-03-26T05:05:32.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.758730+0000", 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.939e-06 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.758735+0000", 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6640000000000003e-06 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.758747+0000", 2024-03-26T05:05:32.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2024e-05 2024-03-26T05:05:32.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.758754+0000", 2024-03-26T05:05:32.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4400000000000002e-06 2024-03-26T05:05:32.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759423+0000", 2024-03-26T05:05:32.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00066898000000000001 2024-03-26T05:05:32.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.113 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759538+0000", 2024-03-26T05:05:32.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011565600000000001 2024-03-26T05:05:32.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.113 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759685+0000", 2024-03-26T05:05:32.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014625099999999999 2024-03-26T05:05:32.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.114 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.762496+0000", 2024-03-26T05:05:32.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0028110040000000002 2024-03-26T05:05:32.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.114 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.765715+0000", 2024-03-26T05:05:32.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0032194599999999999 2024-03-26T05:05:32.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.880359+0000", 2024-03-26T05:05:32.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.114644077 2024-03-26T05:05:32.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.880424+0000", 2024-03-26T05:05:32.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5052e-05 2024-03-26T05:05:32.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.880480+0000", 2024-03-26T05:05:32.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5377999999999997e-05 2024-03-26T05:05:32.116 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.116 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.116 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.117 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:186920 7.5 7:a6bf0922:::benchmark_data_smithi080_80848_object67421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:05:32.117 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:16.767874+0000", 2024-03-26T05:05:32.117 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 15.246688017, 2024-03-26T05:05:32.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.12864367299999999, 2024-03-26T05:05:32.117 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.117 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.117 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.117 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.117 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.118 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 186920 2024-03-26T05:05:32.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.118 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.767874+0000", 2024-03-26T05:05:32.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.767874+0000", 2024-03-26T05:05:32.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.767875+0000", 2024-03-26T05:05:32.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5090000000000001e-06 2024-03-26T05:05:32.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.767881+0000", 2024-03-26T05:05:32.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7259999999999999e-06 2024-03-26T05:05:32.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.767897+0000", 2024-03-26T05:05:32.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6215e-05 2024-03-26T05:05:32.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.767907+0000", 2024-03-26T05:05:32.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6339999999999996e-06 2024-03-26T05:05:32.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.767978+0000", 2024-03-26T05:05:32.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1211000000000006e-05 2024-03-26T05:05:32.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.122 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.768127+0000", 2024-03-26T05:05:32.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014901500000000001 2024-03-26T05:05:32.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.122 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.122 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.768248+0000", 2024-03-26T05:05:32.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000121082 2024-03-26T05:05:32.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.123 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.772442+0000", 2024-03-26T05:05:32.123 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.004193563 2024-03-26T05:05:32.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.123 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.772923+0000", 2024-03-26T05:05:32.123 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00048103000000000002 2024-03-26T05:05:32.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.896424+0000", 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.123501167 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.896474+0000", 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0334999999999998e-05 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.896517+0000", 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3186000000000003e-05 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.124 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:186925 7.5 7:a7cc1812:::benchmark_data_smithi080_80848_object67426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:16.772835+0000", 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 15.241726259, 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.131348508, 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 186925 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.125 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.772835+0000", 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.772835+0000", 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.772837+0000", 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4649999999999999e-06 2024-03-26T05:05:32.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.772846+0000", 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6779999999999995e-06 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.772868+0000", 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.1891000000000001e-05 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.772880+0000", 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2082000000000001e-05 2024-03-26T05:05:32.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.772962+0000", 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.1154000000000003e-05 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.773116+0000", 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000154587 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.773249+0000", 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013252500000000001 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.776245+0000", 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0029961810000000001 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.776356+0000", 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011121899999999999 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.904034+0000", 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.12767806100000001 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.904128+0000", 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3894999999999995e-05 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.904184+0000", 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5770000000000003e-05 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.130 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:186927 7.5 7:a1b1ccbd:::benchmark_data_smithi080_80848_object67428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:16.777266+0000", 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 15.237295103999999, 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.127059592, 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 186927 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.131 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.777266+0000", 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.777266+0000", 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.777268+0000", 2024-03-26T05:05:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.389e-06 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.777273+0000", 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9790000000000001e-06 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.777285+0000", 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.205e-05 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.777292+0000", 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3640000000000001e-06 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.777353+0000", 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0640000000000002e-05 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.777535+0000", 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018251899999999999 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.777631+0000", 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6002999999999995e-05 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.779956+0000", 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0023245000000000002 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.780106+0000", 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000150447 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.904223+0000", 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.12411706 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.904284+0000", 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1068999999999994e-05 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.904326+0000", 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1572000000000001e-05 2024-03-26T05:05:32.136 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:188352 7.5 7:a351d751:::benchmark_data_smithi080_80848_object68853:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:18.319355+0000", 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 13.695206474000001, 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.099329799999999996, 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:05:32.137 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 188352 2024-03-26T05:05:32.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.138 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:05:32.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:05:32.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.319355+0000", 2024-03-26T05:05:32.138 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:05:32.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.319355+0000", 2024-03-26T05:05:32.138 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:05:32.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:05:32.139 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.319356+0000", 2024-03-26T05:05:32.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0699999999999999e-06 2024-03-26T05:05:32.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:05:32.139 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.319360+0000", 2024-03-26T05:05:32.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.579e-06 2024-03-26T05:05:32.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.140 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:05:32.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.319370+0000", 2024-03-26T05:05:32.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.007e-05 2024-03-26T05:05:32.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.140 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:05:32.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.319378+0000", 2024-03-26T05:05:32.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2870000000000004e-06 2024-03-26T05:05:32.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.140 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:05:32.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.319426+0000", 2024-03-26T05:05:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7873000000000001e-05 2024-03-26T05:05:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:05:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.319508+0000", 2024-03-26T05:05:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.1539000000000007e-05 2024-03-26T05:05:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:05:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.319583+0000", 2024-03-26T05:05:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5974000000000001e-05 2024-03-26T05:05:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.321767+0000", 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021834379999999998 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.323001+0000", 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001233636 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:05:32.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.418559+0000", 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095558460999999997 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.418626+0000", 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6975000000000002e-05 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.418685+0000", 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8897999999999998e-05 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:05:32.144 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:05:32.144 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:05:32.144 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:32.177 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:32.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:32.206 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:05:32.206 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:05:32.206 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:05:32.207 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd out 5 2024-03-26T05:05:32.278 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:32.291 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:05:32.310 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:32.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:32.413 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:32.473 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:32.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:32.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:32.574 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:32.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:32 smithi080 ceph-mon[29691]: pgmap v208: 148 pgs: 148 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 811 op/s 2024-03-26T05:05:32.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:32 smithi080 ceph-mon[40404]: pgmap v208: 148 pgs: 148 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 811 op/s 2024-03-26T05:05:32.643 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:32.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:32.699 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:32.746 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:32.763 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:32.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:32 smithi132 ceph-mon[33500]: pgmap v208: 148 pgs: 148 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 811 op/s 2024-03-26T05:05:32.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:32.810 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:32.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:32.912 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:32.957 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:32.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:33.059 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:33.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:33.099 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:33.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:33.200 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:33.237 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:33.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:33.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T05:05:33.339 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:33.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:33.369 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:33.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:33.470 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:33.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:33.601 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:33.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:33.646 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:33.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:33.747 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:33.778 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:33.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:33.880 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:33.912 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:05:33.918 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:33.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:33.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:34.020 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:34.048 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:34.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:34.150 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:34.179 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:34.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:34.233 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:05:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:05:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:05:34.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:34.310 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:34.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:34.412 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:34.453 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:34.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:34.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:34.555 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:34.609 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:34.619 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:34.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:34.720 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:34.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:34.750 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:34.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:34.851 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:34.879 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:34.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:34.981 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:34.995 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:34 smithi080 ceph-mon[29691]: pgmap v209: 148 pgs: 148 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 693 B/s rd, 0 B/s wr, 659 op/s 2024-03-26T05:05:34.995 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:34 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:34.995 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:34 smithi080 ceph-mon[40404]: pgmap v209: 148 pgs: 148 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 693 B/s rd, 0 B/s wr, 659 op/s 2024-03-26T05:05:34.995 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:34 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:35.013 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:35.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:35.115 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:35.136 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:35.140 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:35.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:35.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:35.242 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:35.270 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:35.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:34 smithi132 ceph-mon[33500]: pgmap v209: 148 pgs: 148 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 693 B/s rd, 0 B/s wr, 659 op/s 2024-03-26T05:05:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:34 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:35.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:05:34 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:05:35.372 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:35.416 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:35.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:35.517 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:35.559 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:35.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:35.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:35.662 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:35.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:35.720 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:35.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:35.821 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:35.851 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:35.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:35.859 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:35 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a[29687]: 2024-03-26T05:05:35.632+0000 7f70d2504700 -1 mon.a@0(leader).osd e159 definitely_dead 0 2024-03-26T05:05:35.877 INFO:teuthology.orchestra.run.smithi080.stderr:marked out osd.5. 2024-03-26T05:05:35.953 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:35.976 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:35.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:35.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:36.078 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:36.102 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:36.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:36.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:36.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:35 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:36.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:35 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3613574884' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T05:05:36.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:35 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T05:05:36.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:35 smithi080 ceph-mon[29691]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-26T05:05:36.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:35 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:36.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:35 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3613574884' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T05:05:36.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:35 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T05:05:36.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:35 smithi080 ceph-mon[40404]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-26T05:05:36.204 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:36.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:36.229 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:36.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:36.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:35 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:36.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:35 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3613574884' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T05:05:36.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:35 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T05:05:36.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:35 smithi132 ceph-mon[33500]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-26T05:05:36.331 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:36.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:36.356 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:36.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:36.457 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:36.482 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:36.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:36.583 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:36.607 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:36.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:36.708 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:36.733 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:36.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:36.834 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:36.858 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:36.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:36.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:36.960 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:36.985 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:36.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:37.087 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:37.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:37.112 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:37.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:37.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:36 smithi080 ceph-mon[29691]: pgmap v210: 148 pgs: 148 active+clean; 2.1 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T05:05:37.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:36 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-26T05:05:37.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:36 smithi080 ceph-mon[29691]: osdmap e160: 8 total, 8 up, 7 in 2024-03-26T05:05:37.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:36 smithi080 ceph-mon[29691]: osdmap e161: 8 total, 8 up, 7 in 2024-03-26T05:05:37.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:36 smithi080 ceph-mon[40404]: pgmap v210: 148 pgs: 148 active+clean; 2.1 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T05:05:37.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:36 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-26T05:05:37.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:36 smithi080 ceph-mon[40404]: osdmap e160: 8 total, 8 up, 7 in 2024-03-26T05:05:37.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:36 smithi080 ceph-mon[40404]: osdmap e161: 8 total, 8 up, 7 in 2024-03-26T05:05:37.212 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:37.238 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:37.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:36 smithi132 ceph-mon[33500]: pgmap v210: 148 pgs: 148 active+clean; 2.1 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T05:05:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:36 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-26T05:05:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:36 smithi132 ceph-mon[33500]: osdmap e160: 8 total, 8 up, 7 in 2024-03-26T05:05:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:36 smithi132 ceph-mon[33500]: osdmap e161: 8 total, 8 up, 7 in 2024-03-26T05:05:37.340 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:37.363 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:37.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:37.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:37.466 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:37.496 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:37.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:37.597 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:37.625 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:37.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:37.727 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:37.770 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:37.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:37.872 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:37.899 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:37.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:37.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:38.000 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:38.027 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:38.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:38.129 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:38.176 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:38.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:38.277 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:38.306 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:38.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:38.407 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:38.412 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:05:38.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:38.451 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:38.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:38.552 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:38.581 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:38.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:38.683 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:38.722 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:38.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:38.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:38 smithi132 ceph-mon[33500]: 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 159'12832 2024-03-26T05:05:38.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:38 smithi132 ceph-mon[33500]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 159'12973 2024-03-26T05:05:38.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:38 smithi132 ceph-mon[33500]: osdmap e162: 8 total, 8 up, 7 in 2024-03-26T05:05:38.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:38 smithi132 ceph-mon[33500]: 7.a starting backfill to osd.1 from (0'0,0'0] MAX to 159'13073 2024-03-26T05:05:38.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:38 smithi132 ceph-mon[33500]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 159'12960 2024-03-26T05:05:38.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:38 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1047451746' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:05:38.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:38 smithi080 ceph-mon[40404]: 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 159'12832 2024-03-26T05:05:38.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:38 smithi080 ceph-mon[40404]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 159'12973 2024-03-26T05:05:38.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:38 smithi080 ceph-mon[40404]: osdmap e162: 8 total, 8 up, 7 in 2024-03-26T05:05:38.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:38 smithi080 ceph-mon[40404]: 7.a starting backfill to osd.1 from (0'0,0'0] MAX to 159'13073 2024-03-26T05:05:38.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:38 smithi080 ceph-mon[40404]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 159'12960 2024-03-26T05:05:38.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:38 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1047451746' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:05:38.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:38 smithi080 ceph-mon[29691]: 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 159'12832 2024-03-26T05:05:38.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:38 smithi080 ceph-mon[29691]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 159'12973 2024-03-26T05:05:38.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:38 smithi080 ceph-mon[29691]: osdmap e162: 8 total, 8 up, 7 in 2024-03-26T05:05:38.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:38 smithi080 ceph-mon[29691]: 7.a starting backfill to osd.1 from (0'0,0'0] MAX to 159'13073 2024-03-26T05:05:38.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:38 smithi080 ceph-mon[29691]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 159'12960 2024-03-26T05:05:38.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:38 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1047451746' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:05:38.824 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:38.851 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:38.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:38.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:38.953 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:38.991 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:38.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:39.093 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:39.135 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:39.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:39.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:39.237 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:39.264 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:39.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:39.366 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:39.408 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:39.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:39.510 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:39.552 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:39.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:39.654 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:39.695 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:39.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:39.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:39.773 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:39 smithi080 ceph-mon[29691]: pgmap v214: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 858 op/s 2024-03-26T05:05:39.773 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:39 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:05:39.773 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:39 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1047451746' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:05:39.773 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:39 smithi080 ceph-mon[29691]: osdmap e163: 8 total, 8 up, 7 in 2024-03-26T05:05:39.773 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:39 smithi080 ceph-mon[29691]: 2.0 scrub starts 2024-03-26T05:05:39.773 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:39 smithi080 ceph-mon[29691]: 2.0 scrub ok 2024-03-26T05:05:39.773 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:39 smithi080 ceph-mon[40404]: pgmap v214: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 858 op/s 2024-03-26T05:05:39.773 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:39 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:05:39.773 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:39 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1047451746' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:05:39.774 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:39 smithi080 ceph-mon[40404]: osdmap e163: 8 total, 8 up, 7 in 2024-03-26T05:05:39.774 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:39 smithi080 ceph-mon[40404]: 2.0 scrub starts 2024-03-26T05:05:39.774 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:39 smithi080 ceph-mon[40404]: 2.0 scrub ok 2024-03-26T05:05:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:39 smithi132 ceph-mon[33500]: pgmap v214: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 858 op/s 2024-03-26T05:05:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:39 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:05:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:39 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1047451746' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:05:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:39 smithi132 ceph-mon[33500]: osdmap e163: 8 total, 8 up, 7 in 2024-03-26T05:05:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:39 smithi132 ceph-mon[33500]: 2.0 scrub starts 2024-03-26T05:05:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:39 smithi132 ceph-mon[33500]: 2.0 scrub ok 2024-03-26T05:05:39.797 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:39.833 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:39.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:39.937 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:39.966 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:39.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:40.068 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:40.096 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:40.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:40.198 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:40.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:40.226 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:40.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:40.327 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:40.355 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:40.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:40.457 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:40.485 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:40.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:40.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:40.587 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:40.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:40.616 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:40.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:40.717 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:40.744 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:40.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:40.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:40.846 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:40.851 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:05:40.851 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:05:40.851 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:05:40.851 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:05:40.851 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:05:40.869 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:05:40.912 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:40.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:40.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:40.915 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:05:40.958 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:05:40.966 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:05:40.966 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:05:40.966 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:05:40.966 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:05:40.966 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:05:40.966 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:05:40.966 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:05:40.966 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:05:41.015 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:41.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:41.042 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:41.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:41.143 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:41.183 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:41.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:41.341 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:41.383 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:41.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:41.484 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:41.522 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:41.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:41.568 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:41 smithi080 ceph-mon[29691]: pgmap v216: 148 pgs: 148 active+clean; 2.0 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 242 op/s; 277 B/s, 7 objects/s recovering 2024-03-26T05:05:41.568 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:41 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1007723905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:05:41.569 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:41 smithi080 ceph-mon[40404]: pgmap v216: 148 pgs: 148 active+clean; 2.0 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 242 op/s; 277 B/s, 7 objects/s recovering 2024-03-26T05:05:41.569 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:41 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1007723905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:05:41.624 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:41.655 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:41.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:41.757 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:41.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:41 smithi132 ceph-mon[33500]: pgmap v216: 148 pgs: 148 active+clean; 2.0 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 242 op/s; 277 B/s, 7 objects/s recovering 2024-03-26T05:05:41.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:41 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1007723905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:05:41.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:41.795 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:41.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:41.896 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:41.909 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:05:41.909 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:05:41.934 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:41.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:42.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:42.064 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:42.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:42.166 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:42.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:42.208 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:42.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:42.282 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:05:42.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 0 1 2024-03-26T05:05:42.309 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:42.367 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:42.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:42.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:42.469 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:42.517 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:42.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:42.618 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:42 smithi080 ceph-mon[29691]: 4.13 scrub starts 2024-03-26T05:05:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:42 smithi080 ceph-mon[29691]: 4.13 scrub ok 2024-03-26T05:05:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:42 smithi080 ceph-mon[29691]: 6.1c scrub starts 2024-03-26T05:05:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:42 smithi080 ceph-mon[29691]: 6.1c scrub ok 2024-03-26T05:05:42.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:42 smithi080 ceph-mon[40404]: 4.13 scrub starts 2024-03-26T05:05:42.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:42 smithi080 ceph-mon[40404]: 4.13 scrub ok 2024-03-26T05:05:42.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:42 smithi080 ceph-mon[40404]: 6.1c scrub starts 2024-03-26T05:05:42.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:42 smithi080 ceph-mon[40404]: 6.1c scrub ok 2024-03-26T05:05:42.644 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:42.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:42.746 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:42.792 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:42.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:42 smithi132 ceph-mon[33500]: 4.13 scrub starts 2024-03-26T05:05:42.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:42 smithi132 ceph-mon[33500]: 4.13 scrub ok 2024-03-26T05:05:42.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:42 smithi132 ceph-mon[33500]: 6.1c scrub starts 2024-03-26T05:05:42.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:42 smithi132 ceph-mon[33500]: 6.1c scrub ok 2024-03-26T05:05:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:42.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:42.895 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:42.937 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:42.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:42.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:43.039 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:43.081 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:43.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:43.183 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:43.225 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:43.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:43.327 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:43.346 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:05:43.357 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:43.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:43.459 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:43.484 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:43.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:43.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:43.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:43.638 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:43.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:43.740 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:43.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:43 smithi132 ceph-mon[33500]: pgmap v217: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.9 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 690 B/s rd, 0 B/s wr, 552 op/s; 7258/92502 objects misplaced (7.846%); 4.0 MiB/s, 74 objects/s recovering 2024-03-26T05:05:43.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:43 smithi132 ceph-mon[33500]: 3.14 scrub starts 2024-03-26T05:05:43.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:43 smithi132 ceph-mon[33500]: 3.14 scrub ok 2024-03-26T05:05:43.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:43 smithi132 ceph-mon[33500]: 4.b scrub starts 2024-03-26T05:05:43.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:43 smithi132 ceph-mon[33500]: 4.b scrub ok 2024-03-26T05:05:43.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:43 smithi132 ceph-mon[33500]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:05:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:43.770 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:43.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:43.848 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:43 smithi080 ceph-mon[29691]: pgmap v217: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.9 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 690 B/s rd, 0 B/s wr, 552 op/s; 7258/92502 objects misplaced (7.846%); 4.0 MiB/s, 74 objects/s recovering 2024-03-26T05:05:43.848 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:43 smithi080 ceph-mon[29691]: 3.14 scrub starts 2024-03-26T05:05:43.848 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:43 smithi080 ceph-mon[29691]: 3.14 scrub ok 2024-03-26T05:05:43.848 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:43 smithi080 ceph-mon[29691]: 4.b scrub starts 2024-03-26T05:05:43.848 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:43 smithi080 ceph-mon[29691]: 4.b scrub ok 2024-03-26T05:05:43.848 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:43 smithi080 ceph-mon[29691]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:05:43.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:43 smithi080 ceph-mon[40404]: pgmap v217: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.9 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 690 B/s rd, 0 B/s wr, 552 op/s; 7258/92502 objects misplaced (7.846%); 4.0 MiB/s, 74 objects/s recovering 2024-03-26T05:05:43.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:43 smithi080 ceph-mon[40404]: 3.14 scrub starts 2024-03-26T05:05:43.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:43 smithi080 ceph-mon[40404]: 3.14 scrub ok 2024-03-26T05:05:43.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:43 smithi080 ceph-mon[40404]: 4.b scrub starts 2024-03-26T05:05:43.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:43 smithi080 ceph-mon[40404]: 4.b scrub ok 2024-03-26T05:05:43.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:43 smithi080 ceph-mon[40404]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:05:43.871 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:43.900 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:43.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:44.002 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:44.030 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:44.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:44.117 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:05:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:05:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:05:44.132 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:44.169 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:44.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:44.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:44.243 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T05:05:44.271 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:44.274 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:44.324 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:44.346 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:44.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:44.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:44.448 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:44.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:44.520 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:44.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:44.621 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:44.647 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:44.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:44.748 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:44.754 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:44 smithi080 ceph-mon[29691]: 6.5 scrub starts 2024-03-26T05:05:44.754 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:44 smithi080 ceph-mon[29691]: 6.5 scrub ok 2024-03-26T05:05:44.754 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:44 smithi080 ceph-mon[40404]: 6.5 scrub starts 2024-03-26T05:05:44.754 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:44 smithi080 ceph-mon[40404]: 6.5 scrub ok 2024-03-26T05:05:44.775 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:44.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:44 smithi132 ceph-mon[33500]: 6.5 scrub starts 2024-03-26T05:05:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:44 smithi132 ceph-mon[33500]: 6.5 scrub ok 2024-03-26T05:05:44.878 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:44.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:44.920 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:44.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:45.021 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:45.057 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:45.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:45.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:45.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:45.193 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:45.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:45.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:05:44 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:05:45.295 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:45.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:45.337 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:45.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:45.438 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:45.467 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:45.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:45.503 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:05:45.569 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:45.598 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:45.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:45.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:45.700 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:45.727 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:45.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:45.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:45 smithi132 ceph-mon[33500]: pgmap v218: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.9 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 552 B/s rd, 0 B/s wr, 442 op/s; 7258/92502 objects misplaced (7.846%); 3.2 MiB/s, 59 objects/s recovering 2024-03-26T05:05:45.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:45 smithi132 ceph-mon[33500]: 4.1b scrub starts 2024-03-26T05:05:45.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:45 smithi132 ceph-mon[33500]: 4.1b scrub ok 2024-03-26T05:05:45.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:45 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:05:45.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:45 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:05:45.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:45 smithi132 ceph-mon[33500]: osdmap e164: 8 total, 8 up, 7 in 2024-03-26T05:05:45.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:45 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:45.829 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:45.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[29691]: pgmap v218: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.9 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 552 B/s rd, 0 B/s wr, 442 op/s; 7258/92502 objects misplaced (7.846%); 3.2 MiB/s, 59 objects/s recovering 2024-03-26T05:05:45.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[29691]: 4.1b scrub starts 2024-03-26T05:05:45.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[29691]: 4.1b scrub ok 2024-03-26T05:05:45.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:05:45.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:05:45.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[29691]: osdmap e164: 8 total, 8 up, 7 in 2024-03-26T05:05:45.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:45.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[40404]: pgmap v218: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.9 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 552 B/s rd, 0 B/s wr, 442 op/s; 7258/92502 objects misplaced (7.846%); 3.2 MiB/s, 59 objects/s recovering 2024-03-26T05:05:45.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[40404]: 4.1b scrub starts 2024-03-26T05:05:45.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[40404]: 4.1b scrub ok 2024-03-26T05:05:45.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:05:45.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:05:45.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[40404]: osdmap e164: 8 total, 8 up, 7 in 2024-03-26T05:05:45.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:45 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:45.877 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:45.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:45.979 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:46.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:46.026 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:46.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:46.127 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:46.152 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:46.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:46.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:46.282 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:46.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:46.383 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:46.408 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:46.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:46.510 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:46.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:46.540 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:46.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:46.641 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:46.678 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:46.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:46.780 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:46.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:46 smithi132 ceph-mon[33500]: 7.d scrub starts 2024-03-26T05:05:46.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:46 smithi132 ceph-mon[33500]: pgmap v220: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.6 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.00k op/s; 5389/78282 objects misplaced (6.884%); 9.3 MiB/s, 157 objects/s recovering 2024-03-26T05:05:46.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:46 smithi132 ceph-mon[33500]: 7.d scrub ok 2024-03-26T05:05:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:46.809 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:46.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:46.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:46 smithi080 ceph-mon[40404]: 7.d scrub starts 2024-03-26T05:05:46.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:46 smithi080 ceph-mon[40404]: pgmap v220: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.6 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.00k op/s; 5389/78282 objects misplaced (6.884%); 9.3 MiB/s, 157 objects/s recovering 2024-03-26T05:05:46.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:46 smithi080 ceph-mon[40404]: 7.d scrub ok 2024-03-26T05:05:46.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:46 smithi080 ceph-mon[29691]: 7.d scrub starts 2024-03-26T05:05:46.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:46 smithi080 ceph-mon[29691]: pgmap v220: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.6 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.00k op/s; 5389/78282 objects misplaced (6.884%); 9.3 MiB/s, 157 objects/s recovering 2024-03-26T05:05:46.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:46 smithi080 ceph-mon[29691]: 7.d scrub ok 2024-03-26T05:05:46.909 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:46.946 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:46.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:47.048 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:47.072 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:47.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:47.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:47.174 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:47.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:47.203 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:47.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:47.304 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:47.341 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:47.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:47.443 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:47.478 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:47.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:47.579 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:47.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:47 smithi080 ceph-mon[40404]: 3.3 scrub starts 2024-03-26T05:05:47.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:47 smithi080 ceph-mon[40404]: 3.3 scrub ok 2024-03-26T05:05:47.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:47 smithi080 ceph-mon[40404]: 6.8 scrub starts 2024-03-26T05:05:47.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:47 smithi080 ceph-mon[40404]: 6.8 scrub ok 2024-03-26T05:05:47.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:47 smithi080 ceph-mon[29691]: 3.3 scrub starts 2024-03-26T05:05:47.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:47 smithi080 ceph-mon[29691]: 3.3 scrub ok 2024-03-26T05:05:47.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:47 smithi080 ceph-mon[29691]: 6.8 scrub starts 2024-03-26T05:05:47.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:47 smithi080 ceph-mon[29691]: 6.8 scrub ok 2024-03-26T05:05:47.622 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:47.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:47.724 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:47.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:47.753 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:47.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:47 smithi132 ceph-mon[33500]: 3.3 scrub starts 2024-03-26T05:05:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:47 smithi132 ceph-mon[33500]: 3.3 scrub ok 2024-03-26T05:05:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:47 smithi132 ceph-mon[33500]: 6.8 scrub starts 2024-03-26T05:05:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:47 smithi132 ceph-mon[33500]: 6.8 scrub ok 2024-03-26T05:05:47.854 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:47.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:47.879 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:47.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:47.980 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:48.018 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:48.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:48.119 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:48.147 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:48.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:48.249 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:48.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:48.275 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:48.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:48.376 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:48.417 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:48.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:48.518 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:48.547 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:48.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:48.648 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:48.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:48.672 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:48.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:48.773 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:48.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:48 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:05:48.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:48 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:05:48.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:48 smithi132 ceph-mon[33500]: 5.1 scrub starts 2024-03-26T05:05:48.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:48 smithi132 ceph-mon[33500]: 5.1 scrub ok 2024-03-26T05:05:48.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:48 smithi132 ceph-mon[33500]: osdmap e165: 8 total, 8 up, 7 in 2024-03-26T05:05:48.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:48 smithi132 ceph-mon[33500]: pgmap v222: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.6 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 881 op/s; 5389/78282 objects misplaced (6.884%); 9.3 MiB/s, 153 objects/s recovering 2024-03-26T05:05:48.801 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:48.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:48.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:48 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:05:48.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:48 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:05:48.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:48 smithi080 ceph-mon[40404]: 5.1 scrub starts 2024-03-26T05:05:48.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:48 smithi080 ceph-mon[40404]: 5.1 scrub ok 2024-03-26T05:05:48.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:48 smithi080 ceph-mon[40404]: osdmap e165: 8 total, 8 up, 7 in 2024-03-26T05:05:48.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:48 smithi080 ceph-mon[40404]: pgmap v222: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.6 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 881 op/s; 5389/78282 objects misplaced (6.884%); 9.3 MiB/s, 153 objects/s recovering 2024-03-26T05:05:48.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:48 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:05:48.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:48 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:05:48.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:48 smithi080 ceph-mon[29691]: 5.1 scrub starts 2024-03-26T05:05:48.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:48 smithi080 ceph-mon[29691]: 5.1 scrub ok 2024-03-26T05:05:48.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:48 smithi080 ceph-mon[29691]: osdmap e165: 8 total, 8 up, 7 in 2024-03-26T05:05:48.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:48 smithi080 ceph-mon[29691]: pgmap v222: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.6 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 881 op/s; 5389/78282 objects misplaced (6.884%); 9.3 MiB/s, 153 objects/s recovering 2024-03-26T05:05:48.903 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:48.945 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:48.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:49.046 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:49.082 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:49.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:49.183 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:49.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:49.211 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:49.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:49.312 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:49.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:49.340 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:49.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:49.441 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:49.466 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:49.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:49.567 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:49.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:49.605 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:49.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:49.706 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:49.730 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:49.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:49.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:49.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:49 smithi132 ceph-mon[33500]: 5.1f scrub starts 2024-03-26T05:05:49.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:49 smithi132 ceph-mon[33500]: 5.1f scrub ok 2024-03-26T05:05:49.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:49.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:49.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:49 smithi132 ceph-mon[33500]: 3.9 scrub starts 2024-03-26T05:05:49.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:49 smithi132 ceph-mon[33500]: 3.9 scrub ok 2024-03-26T05:05:49.832 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:49.861 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:49.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:49.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:49.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:49 smithi080 ceph-mon[40404]: 5.1f scrub starts 2024-03-26T05:05:49.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:49 smithi080 ceph-mon[40404]: 5.1f scrub ok 2024-03-26T05:05:49.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:49.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:49.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:49 smithi080 ceph-mon[40404]: 3.9 scrub starts 2024-03-26T05:05:49.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:49 smithi080 ceph-mon[40404]: 3.9 scrub ok 2024-03-26T05:05:49.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:49 smithi080 ceph-mon[29691]: 5.1f scrub starts 2024-03-26T05:05:49.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:49 smithi080 ceph-mon[29691]: 5.1f scrub ok 2024-03-26T05:05:49.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:05:49.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:49.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:49 smithi080 ceph-mon[29691]: 3.9 scrub starts 2024-03-26T05:05:49.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:49 smithi080 ceph-mon[29691]: 3.9 scrub ok 2024-03-26T05:05:49.963 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:49.992 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:49.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:49.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:50.094 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:50.122 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:50.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:50.223 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:50.252 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:50.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:50.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:50.353 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:50.390 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:50.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:50.492 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:50.539 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:50.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:50.639 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:50.679 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:50.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:50.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:50.769 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:50 smithi080 ceph-mon[29691]: pgmap v223: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.5 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 755 op/s; 5389/74403 objects misplaced (7.243%); 6.4 MiB/s, 102 objects/s recovering 2024-03-26T05:05:50.769 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:50 smithi080 ceph-mon[40404]: pgmap v223: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.5 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 755 op/s; 5389/74403 objects misplaced (7.243%); 6.4 MiB/s, 102 objects/s recovering 2024-03-26T05:05:50.782 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:50.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:50 smithi132 ceph-mon[33500]: pgmap v223: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.5 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 755 op/s; 5389/74403 objects misplaced (7.243%); 6.4 MiB/s, 102 objects/s recovering 2024-03-26T05:05:50.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:50.840 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:50.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:50.941 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:50.969 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:50.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:51.071 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:51.099 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:51.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:51.140 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:51.201 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:51.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:51.259 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:51.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:51.361 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:51.387 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:51.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:51.490 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:51.532 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:51.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:51.633 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:51.662 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:51.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:51.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:51.764 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:51.808 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:51.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:51.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:51.936 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:51.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:52.038 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:52.085 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:52.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:52.186 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:52.230 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:52.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:52.331 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:52.360 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:52.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:52.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:52.399 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:05:52.462 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:52.492 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:52.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:52.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:52.625 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:52.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:52.723 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:52 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2497962340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:05:52.723 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:52 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:05:52.723 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:52 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2497962340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:05:52.724 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:52 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:05:52.726 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:52.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:52 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2497962340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:05:52.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:52 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:05:52.798 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:52.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:52.899 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:52.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:52.950 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:52.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:53.051 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:53.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:53.079 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:53.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:53.180 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:53.209 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:53.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:53.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:53.311 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:53.339 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:53.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:53.441 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:53.468 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:53.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:53.569 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:53.595 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:53.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:53.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:53.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:53 smithi080 ceph-mon[29691]: pgmap v224: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.3 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 3807/65880 objects misplaced (5.779%); 12 MiB/s, 198 objects/s recovering 2024-03-26T05:05:53.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:53 smithi080 ceph-mon[29691]: 6.0 scrub starts 2024-03-26T05:05:53.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:53 smithi080 ceph-mon[29691]: 6.0 scrub ok 2024-03-26T05:05:53.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:53 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:05:53.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:53 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:05:53.620 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:53 smithi080 ceph-mon[29691]: osdmap e166: 8 total, 8 up, 7 in 2024-03-26T05:05:53.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:53 smithi080 ceph-mon[40404]: pgmap v224: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.3 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 3807/65880 objects misplaced (5.779%); 12 MiB/s, 198 objects/s recovering 2024-03-26T05:05:53.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:53 smithi080 ceph-mon[40404]: 6.0 scrub starts 2024-03-26T05:05:53.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:53 smithi080 ceph-mon[40404]: 6.0 scrub ok 2024-03-26T05:05:53.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:53 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:05:53.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:53 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:05:53.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:53 smithi080 ceph-mon[40404]: osdmap e166: 8 total, 8 up, 7 in 2024-03-26T05:05:53.697 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:53.724 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:53.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:53.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:53 smithi132 ceph-mon[33500]: pgmap v224: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.3 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 3807/65880 objects misplaced (5.779%); 12 MiB/s, 198 objects/s recovering 2024-03-26T05:05:53.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:53 smithi132 ceph-mon[33500]: 6.0 scrub starts 2024-03-26T05:05:53.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:53 smithi132 ceph-mon[33500]: 6.0 scrub ok 2024-03-26T05:05:53.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:53 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:05:53.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:53 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:05:53.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:53 smithi132 ceph-mon[33500]: osdmap e166: 8 total, 8 up, 7 in 2024-03-26T05:05:53.825 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:53.853 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:53.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:53.911 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:53.955 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:53.982 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:53.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:54.011 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:05:54.011 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:05:54.084 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:54.111 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:54.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:54.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:54.213 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:54.230 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:05:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:05:54.252 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:54.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:54.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:54.354 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:54.382 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:54.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:54.484 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:54.510 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:54.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:54.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:54.612 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:54.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-mon[29691]: 5.1e scrub starts 2024-03-26T05:05:54.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-mon[29691]: 5.1e scrub ok 2024-03-26T05:05:54.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-mon[29691]: 4.1a scrub starts 2024-03-26T05:05:54.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-mon[29691]: 4.1a scrub ok 2024-03-26T05:05:54.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:05:54.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:05:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-mon[40404]: 5.1e scrub starts 2024-03-26T05:05:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-mon[40404]: 5.1e scrub ok 2024-03-26T05:05:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-mon[40404]: 4.1a scrub starts 2024-03-26T05:05:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-mon[40404]: 4.1a scrub ok 2024-03-26T05:05:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:05:54.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:54 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:05:54.639 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:54.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:54.741 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:54.768 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:54.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:54.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:54 smithi132 ceph-mon[33500]: 5.1e scrub starts 2024-03-26T05:05:54.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:54 smithi132 ceph-mon[33500]: 5.1e scrub ok 2024-03-26T05:05:54.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:54 smithi132 ceph-mon[33500]: 4.1a scrub starts 2024-03-26T05:05:54.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:54 smithi132 ceph-mon[33500]: 4.1a scrub ok 2024-03-26T05:05:54.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:54 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:05:54.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:54 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:05:54.870 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:54.895 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:54.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:54.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:54.997 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:55.022 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:55.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:55.123 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:55.148 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:55.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:55.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:55.275 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:55.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:55.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:05:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:05:55.375 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:55.400 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:55.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:55.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:55.502 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:55.543 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:55.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:55.645 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:55.669 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:55.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:55.771 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:55.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:55 smithi132 ceph-mon[33500]: pgmap v226: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.3 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 517 op/s; 3807/65880 objects misplaced (5.779%); 6.0 MiB/s, 96 objects/s recovering 2024-03-26T05:05:55.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:55 smithi132 ceph-mon[33500]: 3.e scrub starts 2024-03-26T05:05:55.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:55 smithi132 ceph-mon[33500]: 3.e scrub ok 2024-03-26T05:05:55.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:55 smithi132 ceph-mon[33500]: 6.17 scrub starts 2024-03-26T05:05:55.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:55 smithi132 ceph-mon[33500]: 6.17 scrub ok 2024-03-26T05:05:55.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:55 smithi132 ceph-mon[33500]: 7.7 scrub starts 2024-03-26T05:05:55.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:55 smithi132 ceph-mon[33500]: osdmap e167: 8 total, 8 up, 7 in 2024-03-26T05:05:55.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:55 smithi132 ceph-mon[33500]: 7.7 scrub ok 2024-03-26T05:05:55.794 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:55.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:55.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[29691]: pgmap v226: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.3 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 517 op/s; 3807/65880 objects misplaced (5.779%); 6.0 MiB/s, 96 objects/s recovering 2024-03-26T05:05:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[29691]: 3.e scrub starts 2024-03-26T05:05:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[29691]: 3.e scrub ok 2024-03-26T05:05:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[29691]: 6.17 scrub starts 2024-03-26T05:05:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[29691]: 6.17 scrub ok 2024-03-26T05:05:55.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[29691]: 7.7 scrub starts 2024-03-26T05:05:55.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[29691]: osdmap e167: 8 total, 8 up, 7 in 2024-03-26T05:05:55.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[29691]: 7.7 scrub ok 2024-03-26T05:05:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[40404]: pgmap v226: 148 pgs: 2 active+remapped+backfilling, 144 active+clean, 2 active+remapped+backfill_toofull; 1.3 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 517 op/s; 3807/65880 objects misplaced (5.779%); 6.0 MiB/s, 96 objects/s recovering 2024-03-26T05:05:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[40404]: 3.e scrub starts 2024-03-26T05:05:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[40404]: 3.e scrub ok 2024-03-26T05:05:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[40404]: 6.17 scrub starts 2024-03-26T05:05:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[40404]: 6.17 scrub ok 2024-03-26T05:05:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[40404]: 7.7 scrub starts 2024-03-26T05:05:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[40404]: osdmap e167: 8 total, 8 up, 7 in 2024-03-26T05:05:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:55 smithi080 ceph-mon[40404]: 7.7 scrub ok 2024-03-26T05:05:55.896 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:55.920 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:55.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:56.022 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:56.047 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:56.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:56.148 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:56.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:56.174 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:56.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:56.275 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:56.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:56.300 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:56.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:56.401 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:56.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:56.433 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:56.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:56.534 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:56.581 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:56.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:56.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:56.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:56 smithi080 ceph-mon[29691]: 5.d scrub starts 2024-03-26T05:05:56.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:56 smithi080 ceph-mon[29691]: 5.d scrub ok 2024-03-26T05:05:56.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:56 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:56.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:56 smithi080 ceph-mon[29691]: 6.9 scrub starts 2024-03-26T05:05:56.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:56 smithi080 ceph-mon[29691]: 6.9 scrub ok 2024-03-26T05:05:56.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:56 smithi080 ceph-mon[29691]: osdmap e168: 8 total, 8 up, 7 in 2024-03-26T05:05:56.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:56 smithi080 ceph-mon[40404]: 5.d scrub starts 2024-03-26T05:05:56.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:56 smithi080 ceph-mon[40404]: 5.d scrub ok 2024-03-26T05:05:56.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:56 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:56.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:56 smithi080 ceph-mon[40404]: 6.9 scrub starts 2024-03-26T05:05:56.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:56 smithi080 ceph-mon[40404]: 6.9 scrub ok 2024-03-26T05:05:56.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:56 smithi080 ceph-mon[40404]: osdmap e168: 8 total, 8 up, 7 in 2024-03-26T05:05:56.683 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:56.710 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:56.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:56.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:56 smithi132 ceph-mon[33500]: 5.d scrub starts 2024-03-26T05:05:56.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:56 smithi132 ceph-mon[33500]: 5.d scrub ok 2024-03-26T05:05:56.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:56 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:56.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:56 smithi132 ceph-mon[33500]: 6.9 scrub starts 2024-03-26T05:05:56.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:56 smithi132 ceph-mon[33500]: 6.9 scrub ok 2024-03-26T05:05:56.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:56 smithi132 ceph-mon[33500]: osdmap e168: 8 total, 8 up, 7 in 2024-03-26T05:05:56.813 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:56.841 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:56.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:56.943 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:56.981 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:56.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:57.009 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T05:05:57.084 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:57.165 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:57.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:57.266 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:57.334 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:57.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:57.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:57.436 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:57.486 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:57.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:57.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:57.617 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:57.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:57.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:57 smithi080 ceph-mon[40404]: pgmap v229: 148 pgs: 1 peering, 1 active+clean+remapped, 144 active+clean, 2 active+remapped+backfill_toofull; 1.0 GiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.24k op/s; 2152/52023 objects misplaced (4.137%); 18 MiB/s, 284 objects/s recovering 2024-03-26T05:05:57.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:57 smithi080 ceph-mon[40404]: 6.f scrub starts 2024-03-26T05:05:57.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:57 smithi080 ceph-mon[40404]: 6.f scrub ok 2024-03-26T05:05:57.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:57 smithi080 ceph-mon[40404]: 4.e scrub starts 2024-03-26T05:05:57.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:57 smithi080 ceph-mon[40404]: 4.e scrub ok 2024-03-26T05:05:57.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:57 smithi080 ceph-mon[40404]: osdmap e169: 8 total, 8 up, 7 in 2024-03-26T05:05:57.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:57 smithi080 ceph-mon[29691]: pgmap v229: 148 pgs: 1 peering, 1 active+clean+remapped, 144 active+clean, 2 active+remapped+backfill_toofull; 1.0 GiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.24k op/s; 2152/52023 objects misplaced (4.137%); 18 MiB/s, 284 objects/s recovering 2024-03-26T05:05:57.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:57 smithi080 ceph-mon[29691]: 6.f scrub starts 2024-03-26T05:05:57.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:57 smithi080 ceph-mon[29691]: 6.f scrub ok 2024-03-26T05:05:57.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:57 smithi080 ceph-mon[29691]: 4.e scrub starts 2024-03-26T05:05:57.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:57 smithi080 ceph-mon[29691]: 4.e scrub ok 2024-03-26T05:05:57.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:57 smithi080 ceph-mon[29691]: osdmap e169: 8 total, 8 up, 7 in 2024-03-26T05:05:57.718 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:57.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:57.749 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:57.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:57.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:57 smithi132 ceph-mon[33500]: pgmap v229: 148 pgs: 1 peering, 1 active+clean+remapped, 144 active+clean, 2 active+remapped+backfill_toofull; 1.0 GiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.24k op/s; 2152/52023 objects misplaced (4.137%); 18 MiB/s, 284 objects/s recovering 2024-03-26T05:05:57.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:57 smithi132 ceph-mon[33500]: 6.f scrub starts 2024-03-26T05:05:57.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:57 smithi132 ceph-mon[33500]: 6.f scrub ok 2024-03-26T05:05:57.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:57 smithi132 ceph-mon[33500]: 4.e scrub starts 2024-03-26T05:05:57.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:57 smithi132 ceph-mon[33500]: 4.e scrub ok 2024-03-26T05:05:57.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:57 smithi132 ceph-mon[33500]: osdmap e169: 8 total, 8 up, 7 in 2024-03-26T05:05:57.850 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:05:57.878 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:57.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:05:57.950 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:05:57.980 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:58.007 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:58.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:58.108 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:05:58.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:58.152 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:58.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:05:58.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:05:58.281 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:58.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:58.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:05:58.383 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:58.429 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:58.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:58.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:58.446 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.0 to 1 (10000) 2024-03-26T05:05:58.532 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:58.571 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:58.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:05:58.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[40404]: 6.2 scrub starts 2024-03-26T05:05:58.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[40404]: 6.2 scrub ok 2024-03-26T05:05:58.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[40404]: 7.c scrub starts 2024-03-26T05:05:58.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[40404]: 7.c scrub ok 2024-03-26T05:05:58.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/116610182' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:05:58.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:05:58.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[40404]: 5.9 scrub starts 2024-03-26T05:05:58.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[40404]: 5.9 scrub ok 2024-03-26T05:05:58.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[29691]: 6.2 scrub starts 2024-03-26T05:05:58.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[29691]: 6.2 scrub ok 2024-03-26T05:05:58.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[29691]: 7.c scrub starts 2024-03-26T05:05:58.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[29691]: 7.c scrub ok 2024-03-26T05:05:58.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/116610182' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:05:58.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:05:58.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[29691]: 5.9 scrub starts 2024-03-26T05:05:58.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:58 smithi080 ceph-mon[29691]: 5.9 scrub ok 2024-03-26T05:05:58.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:58.699 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:58.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:58.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:58 smithi132 ceph-mon[33500]: 6.2 scrub starts 2024-03-26T05:05:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:58 smithi132 ceph-mon[33500]: 6.2 scrub ok 2024-03-26T05:05:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:58 smithi132 ceph-mon[33500]: 7.c scrub starts 2024-03-26T05:05:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:58 smithi132 ceph-mon[33500]: 7.c scrub ok 2024-03-26T05:05:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/116610182' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:05:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:58 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:05:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:58 smithi132 ceph-mon[33500]: 5.9 scrub starts 2024-03-26T05:05:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:58 smithi132 ceph-mon[33500]: 5.9 scrub ok 2024-03-26T05:05:58.802 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:58.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:58.834 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:58.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:58.935 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:58.962 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:58.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:59.064 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:05:59.089 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:05:59.091 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:59.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:05:59.193 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:59.237 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:59.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:59.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:59.338 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:05:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:59.382 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:59.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:05:59.483 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:05:59.524 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:59.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:59.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:05:59.626 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:59.631 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T05:05:59.654 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:59.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:59.734 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[40404]: pgmap v231: 148 pgs: 1 peering, 1 active+clean+remapped, 144 active+clean, 2 active+remapped+backfill_toofull; 1.0 GiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 945 B/s rd, 0 B/s wr, 853 op/s; 2152/52023 objects misplaced (4.137%); 11 MiB/s, 172 objects/s recovering 2024-03-26T05:05:59.735 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[40404]: 5.b scrub starts 2024-03-26T05:05:59.735 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[40404]: 5.b scrub ok 2024-03-26T05:05:59.735 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:05:59.735 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[40404]: osdmap e170: 8 total, 8 up, 7 in 2024-03-26T05:05:59.735 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[40404]: 6.15 scrub starts 2024-03-26T05:05:59.735 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[40404]: 6.15 scrub ok 2024-03-26T05:05:59.735 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[29691]: pgmap v231: 148 pgs: 1 peering, 1 active+clean+remapped, 144 active+clean, 2 active+remapped+backfill_toofull; 1.0 GiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 945 B/s rd, 0 B/s wr, 853 op/s; 2152/52023 objects misplaced (4.137%); 11 MiB/s, 172 objects/s recovering 2024-03-26T05:05:59.735 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[29691]: 5.b scrub starts 2024-03-26T05:05:59.736 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[29691]: 5.b scrub ok 2024-03-26T05:05:59.736 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:05:59.736 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[29691]: osdmap e170: 8 total, 8 up, 7 in 2024-03-26T05:05:59.736 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[29691]: 6.15 scrub starts 2024-03-26T05:05:59.736 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:05:59 smithi080 ceph-mon[29691]: 6.15 scrub ok 2024-03-26T05:05:59.755 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:05:59.781 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:05:59.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:59.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:05:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:59 smithi132 ceph-mon[33500]: pgmap v231: 148 pgs: 1 peering, 1 active+clean+remapped, 144 active+clean, 2 active+remapped+backfill_toofull; 1.0 GiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 945 B/s rd, 0 B/s wr, 853 op/s; 2152/52023 objects misplaced (4.137%); 11 MiB/s, 172 objects/s recovering 2024-03-26T05:05:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:59 smithi132 ceph-mon[33500]: 5.b scrub starts 2024-03-26T05:05:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:59 smithi132 ceph-mon[33500]: 5.b scrub ok 2024-03-26T05:05:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:59 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:05:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:59 smithi132 ceph-mon[33500]: osdmap e170: 8 total, 8 up, 7 in 2024-03-26T05:05:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:59 smithi132 ceph-mon[33500]: 6.15 scrub starts 2024-03-26T05:05:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:05:59 smithi132 ceph-mon[33500]: 6.15 scrub ok 2024-03-26T05:05:59.883 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:05:59.933 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:05:59.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:05:59.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:00.035 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:00.079 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:00.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:00.182 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:00.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:00.245 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:00.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:00.346 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:00.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:00.400 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:00.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:00.501 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:00.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:00.539 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:00.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:00.548 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:00 smithi080 ceph-mon[40404]: 6.14 scrub starts 2024-03-26T05:06:00.548 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:00 smithi080 ceph-mon[40404]: 6.14 scrub ok 2024-03-26T05:06:00.548 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:00 smithi080 ceph-mon[40404]: 3.17 scrub starts 2024-03-26T05:06:00.548 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:00 smithi080 ceph-mon[40404]: 3.17 scrub ok 2024-03-26T05:06:00.548 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:00 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3469935089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:00.549 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:00 smithi080 ceph-mon[29691]: 6.14 scrub starts 2024-03-26T05:06:00.549 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:00 smithi080 ceph-mon[29691]: 6.14 scrub ok 2024-03-26T05:06:00.549 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:00 smithi080 ceph-mon[29691]: 3.17 scrub starts 2024-03-26T05:06:00.549 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:00 smithi080 ceph-mon[29691]: 3.17 scrub ok 2024-03-26T05:06:00.549 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:00 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3469935089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:00.640 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:00.677 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:00.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:00.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:00.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:00 smithi132 ceph-mon[33500]: 6.14 scrub starts 2024-03-26T05:06:00.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:00 smithi132 ceph-mon[33500]: 6.14 scrub ok 2024-03-26T05:06:00.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:00 smithi132 ceph-mon[33500]: 3.17 scrub starts 2024-03-26T05:06:00.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:00 smithi132 ceph-mon[33500]: 3.17 scrub ok 2024-03-26T05:06:00.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:00 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3469935089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:00.794 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:06:00.794 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:06:00.794 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:06:00.794 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:06:00.794 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:06:00.794 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:06:00.794 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:06:00.803 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:06:00.806 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:00.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:00.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:00.809 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:06:00.809 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:06:00.809 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:06:00.809 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:06:00.810 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:06:00.810 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:06:00.810 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:06:00.810 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:06:00.909 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:00.938 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:00.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:01.039 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:01.066 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:01.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:01.167 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:01.194 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:01.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:01.296 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:01.338 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:01.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:01.439 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:01.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:01.468 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:01.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:01.569 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:01.615 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:01.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:01.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:01.717 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:01.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:01.753 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:01.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:01.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:01 smithi132 ceph-mon[33500]: pgmap v233: 148 pgs: 1 peering, 1 active+clean+remapped, 144 active+clean, 2 active+remapped+backfill_toofull; 1001 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 1.06k op/s; 2152/48711 objects misplaced (4.418%); 11 MiB/s, 173 objects/s recovering 2024-03-26T05:06:01.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:01 smithi132 ceph-mon[33500]: 7.8 scrub starts 2024-03-26T05:06:01.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:01 smithi132 ceph-mon[33500]: 3.1a scrub starts 2024-03-26T05:06:01.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:01 smithi132 ceph-mon[33500]: 3.1a scrub ok 2024-03-26T05:06:01.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:01 smithi132 ceph-mon[33500]: 5.1d scrub starts 2024-03-26T05:06:01.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:01 smithi132 ceph-mon[33500]: 5.1d scrub ok 2024-03-26T05:06:01.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:01 smithi132 ceph-mon[33500]: 7.8 scrub ok 2024-03-26T05:06:01.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:01 smithi132 ceph-mon[33500]: 5.4 scrub starts 2024-03-26T05:06:01.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:01 smithi132 ceph-mon[33500]: 5.4 scrub ok 2024-03-26T05:06:01.854 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:01.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[29691]: pgmap v233: 148 pgs: 1 peering, 1 active+clean+remapped, 144 active+clean, 2 active+remapped+backfill_toofull; 1001 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 1.06k op/s; 2152/48711 objects misplaced (4.418%); 11 MiB/s, 173 objects/s recovering 2024-03-26T05:06:01.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[29691]: 7.8 scrub starts 2024-03-26T05:06:01.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[29691]: 3.1a scrub starts 2024-03-26T05:06:01.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[29691]: 3.1a scrub ok 2024-03-26T05:06:01.870 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[29691]: 5.1d scrub starts 2024-03-26T05:06:01.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[29691]: 5.1d scrub ok 2024-03-26T05:06:01.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[29691]: 7.8 scrub ok 2024-03-26T05:06:01.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[29691]: 5.4 scrub starts 2024-03-26T05:06:01.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[29691]: 5.4 scrub ok 2024-03-26T05:06:01.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[40404]: pgmap v233: 148 pgs: 1 peering, 1 active+clean+remapped, 144 active+clean, 2 active+remapped+backfill_toofull; 1001 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 1.06k op/s; 2152/48711 objects misplaced (4.418%); 11 MiB/s, 173 objects/s recovering 2024-03-26T05:06:01.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[40404]: 7.8 scrub starts 2024-03-26T05:06:01.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[40404]: 3.1a scrub starts 2024-03-26T05:06:01.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[40404]: 3.1a scrub ok 2024-03-26T05:06:01.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[40404]: 5.1d scrub starts 2024-03-26T05:06:01.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[40404]: 5.1d scrub ok 2024-03-26T05:06:01.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[40404]: 7.8 scrub ok 2024-03-26T05:06:01.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[40404]: 5.4 scrub starts 2024-03-26T05:06:01.872 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:01 smithi080 ceph-mon[40404]: 5.4 scrub ok 2024-03-26T05:06:01.895 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:01.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:01.998 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:02.046 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:02.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:02.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:02.148 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:02.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:02.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:02.279 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:02.313 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:02.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:02.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:02.415 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:02.446 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:02.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:02.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:02.548 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:02.577 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:02.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:02.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:02.659 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:02.662 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:02.680 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:02.745 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:02.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:02.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:02 smithi132 ceph-mon[33500]: 3.18 scrub starts 2024-03-26T05:06:02.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:02 smithi132 ceph-mon[33500]: 3.18 scrub ok 2024-03-26T05:06:02.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:02 smithi132 ceph-mon[33500]: 6.1 scrub starts 2024-03-26T05:06:02.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:02 smithi132 ceph-mon[33500]: 6.1 scrub ok 2024-03-26T05:06:02.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:02 smithi132 ceph-mon[33500]: pgmap v234: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+remapped+backfill_toofull; 830 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 800 B/s rd, 0 B/s wr, 601 op/s; 1673/40497 objects misplaced (4.131%) 2024-03-26T05:06:02.800 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:02 smithi080 ceph-mon[29691]: 3.18 scrub starts 2024-03-26T05:06:02.800 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:02 smithi080 ceph-mon[29691]: 3.18 scrub ok 2024-03-26T05:06:02.800 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:02 smithi080 ceph-mon[29691]: 6.1 scrub starts 2024-03-26T05:06:02.800 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:02 smithi080 ceph-mon[29691]: 6.1 scrub ok 2024-03-26T05:06:02.800 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:02 smithi080 ceph-mon[29691]: pgmap v234: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+remapped+backfill_toofull; 830 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 800 B/s rd, 0 B/s wr, 601 op/s; 1673/40497 objects misplaced (4.131%) 2024-03-26T05:06:02.800 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:02 smithi080 ceph-mon[40404]: 3.18 scrub starts 2024-03-26T05:06:02.801 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:02 smithi080 ceph-mon[40404]: 3.18 scrub ok 2024-03-26T05:06:02.801 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:02 smithi080 ceph-mon[40404]: 6.1 scrub starts 2024-03-26T05:06:02.801 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:02 smithi080 ceph-mon[40404]: 6.1 scrub ok 2024-03-26T05:06:02.801 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:02 smithi080 ceph-mon[40404]: pgmap v234: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+remapped+backfill_toofull; 830 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 800 B/s rd, 0 B/s wr, 601 op/s; 1673/40497 objects misplaced (4.131%) 2024-03-26T05:06:02.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:02.877 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:02.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:02.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:02.979 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:03.031 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:03.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:03.132 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:03.161 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:03.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:03.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:03.262 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:03.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:03.291 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:03.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:03.392 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:03.421 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:03.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:03.523 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:03.567 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:03.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:03.669 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:03.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:03 smithi080 ceph-mon[40404]: 7.4 scrub starts 2024-03-26T05:06:03.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:03 smithi080 ceph-mon[40404]: 7.4 scrub ok 2024-03-26T05:06:03.674 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:03 smithi080 ceph-mon[29691]: 7.4 scrub starts 2024-03-26T05:06:03.674 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:03 smithi080 ceph-mon[29691]: 7.4 scrub ok 2024-03-26T05:06:03.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:03.699 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:03.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:03.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:03 smithi132 ceph-mon[33500]: 7.4 scrub starts 2024-03-26T05:06:03.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:03 smithi132 ceph-mon[33500]: 7.4 scrub ok 2024-03-26T05:06:03.800 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:03.829 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:03.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:03.931 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:03.958 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:03.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:03.990 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:06:04.059 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:04.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:04.086 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:04.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:04.187 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:04.217 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:04.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:04.319 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:04.348 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:06:04.349 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:06:04.349 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:06:04.349 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:06:04.349 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.349 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:22570 7.6 7:67f0f67e:::benchmark_data_smithi080_80848_object22569:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:06:04.349 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:47.345694+0000", 2024-03-26T05:06:04.349 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 196.296444753, 2024-03-26T05:06:04.349 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14342458899999999, 2024-03-26T05:06:04.349 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.349 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.349 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:04.349 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 22570 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.345694+0000", 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.345694+0000", 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.345695+0000", 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.342e-06 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.345715+0000", 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0047999999999998e-05 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.345727+0000", 2024-03-26T05:06:04.351 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2048999999999999e-05 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.345735+0000", 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0779999999999996e-06 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.346952+0000", 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0012164929999999999 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.352 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.347073+0000", 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012146699999999999 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.347201+0000", 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012805299999999999 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.348668+0000", 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014665030000000001 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.353 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.348705+0000", 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6881000000000001e-05 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.489019+0000", 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14031427799999999 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.489074+0000", 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4576000000000001e-05 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.354 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.489119+0000", 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4821000000000003e-05 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:22573 7.f 7:f965c4fd:::benchmark_data_smithi080_80848_object22572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:02:47.347454+0000", 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 196.29468483799999, 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13941943000000001, 2024-03-26T05:06:04.355 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 22573 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.347454+0000", 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.356 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.347454+0000", 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.347455+0000", 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1599999999999999e-06 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.347475+0000", 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0313e-05 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.357 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.347486+0000", 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1056999999999999e-05 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.347494+0000", 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7670000000000003e-06 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.347570+0000", 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5628000000000002e-05 2024-03-26T05:06:04.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.347690+0000", 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000120728 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.347844+0000", 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015330899999999999 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.349053+0000", 2024-03-26T05:06:04.359 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0012093049999999999 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.349176+0000", 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000122948 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.486756+0000", 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13758018899999999 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.486821+0000", 2024-03-26T05:06:04.360 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4539e-05 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:02:47.486873+0000", 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2487000000000001e-05 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.24701.0:122210 7.e e140/103 7:791c132a:::benchmark_data_smithi080_80848_object2711:head v 140'7676, mlcod=140'7674)", 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:11.503528+0000", 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 112.13861108499999, 2024-03-26T05:06:04.361 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.15073949, 2024-03-26T05:06:04.362 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.362 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.362 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.362 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.362 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.362 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.503528+0000", 2024-03-26T05:06:04.364 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.364 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.364 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.364 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.364 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.503528+0000", 2024-03-26T05:06:04.364 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.364 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.364 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.364 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.503528+0000", 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.34e-07 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.503532+0000", 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3579999999999996e-06 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.503538+0000", 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2110000000000003e-06 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.365 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.366 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.366 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.503543+0000", 2024-03-26T05:06:04.366 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7570000000000004e-06 2024-03-26T05:06:04.366 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.366 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.366 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.366 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504492+0000", 2024-03-26T05:06:04.366 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00094837100000000004 2024-03-26T05:06:04.366 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.366 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504511+0000", 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9065000000000001e-05 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.654228+0000", 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14971720599999999 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.654267+0000", 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9187999999999998e-05 2024-03-26T05:06:04.367 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:122213 7.9 7:93734bcd:::benchmark_data_smithi080_80848_object2714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:11.504227+0000", 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 112.13791120400001, 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14399373800000001, 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:04.368 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 122213 2024-03-26T05:06:04.369 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.369 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.369 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.369 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.369 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504227+0000", 2024-03-26T05:06:04.369 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.369 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.369 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.369 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.370 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504227+0000", 2024-03-26T05:06:04.370 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.370 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.370 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.370 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.370 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504229+0000", 2024-03-26T05:06:04.370 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5510000000000001e-06 2024-03-26T05:06:04.370 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.370 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.370 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504234+0000", 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8860000000000003e-06 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504246+0000", 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1729000000000001e-05 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504254+0000", 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.943e-06 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.371 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504300+0000", 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6483999999999997e-05 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504409+0000", 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000109029 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,4", 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504539+0000", 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012967 2024-03-26T05:06:04.372 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648089+0000", 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.143549753 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648099+0000", 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0910999999999999e-05 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648134+0000", 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4140000000000002e-05 2024-03-26T05:06:04.373 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648179+0000", 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4997000000000003e-05 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648221+0000", 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2645000000000003e-05 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.374 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:122214 7.9 7:90784465:::benchmark_data_smithi080_80848_object2715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:11.504480+0000", 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 112.137658605, 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14529752000000001, 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 122214 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.375 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504480+0000", 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504480+0000", 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504481+0000", 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6300000000000005e-07 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.376 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504485+0000", 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9700000000000001e-06 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504495+0000", 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.027e-05 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.504502+0000", 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3209999999999999e-06 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.377 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507624+0000", 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0031215330000000001 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507744+0000", 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011985 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,4", 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507859+0000", 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000114963 2024-03-26T05:06:04.378 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648104+0000", 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14024552900000001 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648256+0000", 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000151955 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.649220+0000", 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000964075 2024-03-26T05:06:04.379 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.649730+0000", 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000510287 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.649778+0000", 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7104000000000002e-05 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.380 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:122215 7.6 7:6407bc53:::benchmark_data_smithi080_80848_object2716:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:11.506466+0000", 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 112.13567233400001, 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.15195119600000001, 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 122215 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.381 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.506466+0000", 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.506466+0000", 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.506467+0000", 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.091e-06 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.382 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.383 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.383 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.506472+0000", 2024-03-26T05:06:04.383 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.403e-06 2024-03-26T05:06:04.383 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.383 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.383 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.383 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.506483+0000", 2024-03-26T05:06:04.383 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0967e-05 2024-03-26T05:06:04.383 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.383 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.383 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.383 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.506491+0000", 2024-03-26T05:06:04.384 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8930000000000005e-06 2024-03-26T05:06:04.384 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.384 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.384 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.384 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507224+0000", 2024-03-26T05:06:04.384 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00073336899999999999 2024-03-26T05:06:04.384 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.384 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.384 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.384 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507362+0000", 2024-03-26T05:06:04.384 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000137823 2024-03-26T05:06:04.384 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507843+0000", 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00048081799999999999 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.646808+0000", 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.138964904 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.385 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.647128+0000", 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00032003199999999999 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.658309+0000", 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.011181244999999999 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.658357+0000", 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8195000000000002e-05 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.658418+0000", 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0455999999999999e-05 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "rep_scrubmap(7.6 e103 from shard 7)", 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:11.506802+0000", 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 112.135336896, 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13996625100000001, 2024-03-26T05:06:04.387 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.506802+0000", 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.506802+0000", 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.388 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.506803+0000", 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9599999999999998e-07 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507192+0000", 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000389904 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507203+0000", 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0088999999999999e-05 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.389 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507209+0000", 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3969999999999999e-06 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.646567+0000", 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13935820700000001 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.646579+0000", 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2251e-05 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.390 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.646768+0000", 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018860700000000001 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop_reply(client.24701.0:122213 7.9 e140/103 ondisk, result = 0)", 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:11.507058+0000", 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 112.135080184, 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14104656700000001, 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:06:04.391 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507058+0000", 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507058+0000", 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507059+0000", 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6899999999999997e-07 2024-03-26T05:06:04.392 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.393 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.393 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507063+0000", 2024-03-26T05:06:04.393 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.912e-06 2024-03-26T05:06:04.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.393 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.393 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507073+0000", 2024-03-26T05:06:04.393 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6199999999999994e-06 2024-03-26T05:06:04.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.393 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.507079+0000", 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6449999999999999e-06 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648077+0000", 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.140997385 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648083+0000", 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4479999999999999e-06 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.394 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648105+0000", 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1988e-05 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop_reply(client.24701.0:122213 7.9 e140/103 ondisk, result = 0)", 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:11.508058+0000", 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 112.134080789, 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14016747199999999, 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:06:04.395 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.508058+0000", 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.508058+0000", 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.508059+0000", 2024-03-26T05:06:04.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8899999999999999e-07 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.508063+0000", 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.42e-06 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.508072+0000", 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0699999999999996e-06 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.397 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.508079+0000", 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6309999999999997e-06 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648124+0000", 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14004541400000001 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648129+0000", 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2809999999999999e-06 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.648225+0000", 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5966999999999998e-05 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.24701.0:122211 7.e e140/103 7:76b0ff28:::benchmark_data_smithi080_80848_object2712:head v 140'7677, mlcod=140'7674)", 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:11.508783+0000", 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 112.133356058, 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.15831055499999999, 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.399 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.508783+0000", 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.508783+0000", 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.508783+0000", 2024-03-26T05:06:04.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6599999999999996e-07 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.508789+0000", 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3970000000000002e-06 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.642726+0000", 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13393691099999999 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.642751+0000", 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5242999999999999e-05 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.646842+0000", 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0040915910000000003 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.646861+0000", 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8468999999999999e-05 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.402 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.667049+0000", 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.020187878999999999 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.667093+0000", 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4399e-05 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop_reply(client.24701.0:122215 7.6 e140/103 ondisk, result = 0)", 2024-03-26T05:06:04.403 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:11.511503+0000", 2024-03-26T05:06:04.404 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 112.13063608100001, 2024-03-26T05:06:04.404 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13531253800000001, 2024-03-26T05:06:04.404 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.404 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:06:04.404 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.404 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.404 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.511503+0000", 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.511503+0000", 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.511503+0000", 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1600000000000001e-07 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.511507+0000", 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.98e-06 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.642707+0000", 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13119977799999999 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.642737+0000", 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.008e-05 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.646794+0000", 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0040568879999999998 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.646802+0000", 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6199999999999999e-06 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:11.646815+0000", 2024-03-26T05:06:04.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3475999999999999e-05 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:157203 7.6 7:6072c2c6:::benchmark_data_smithi080_80848_object37704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:46.717954+0000", 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 76.924184296999996, 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.140032082, 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:04.408 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 157203 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.717954+0000", 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.717954+0000", 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.717955+0000", 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.122e-06 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.717959+0000", 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0040000000000001e-06 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.717968+0000", 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4509999999999992e-06 2024-03-26T05:06:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.717975+0000", 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5729999999999996e-06 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.720763+0000", 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027884540000000001 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.720890+0000", 2024-03-26T05:06:04.411 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000126895 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.720999+0000", 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010925999999999999 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.731363+0000", 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010364141 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:04.412 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.731441+0000", 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8220000000000007e-05 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.857893+0000", 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.126451754 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.857941+0000", 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7329e-05 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.413 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.857986+0000", 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5878999999999999e-05 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:157205 7.f 7:fb4a3cb8:::benchmark_data_smithi080_80848_object37706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:46.720271+0000", 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 76.921867642999999, 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.12793711399999999, 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.414 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 157205 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.720271+0000", 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.415 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.720271+0000", 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.720272+0000", 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2899999999999999e-06 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.720277+0000", 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0340000000000004e-06 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.720289+0000", 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1284000000000001e-05 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.720297+0000", 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9480000000000008e-06 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.721421+0000", 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0011246369999999999 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.721539+0000", 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011731 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.721650+0000", 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000111817 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.726638+0000", 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.004987806 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.730379+0000", 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.003740373 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.848113+0000", 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.117734489 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.848164+0000", 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1137999999999997e-05 2024-03-26T05:06:04.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.848208+0000", 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3988e-05 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:157208 7.6 7:60841499:::benchmark_data_smithi080_80848_object37709:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:46.724479+0000", 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 76.917660081999998, 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13362568499999999, 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.420 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 157208 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.724479+0000", 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.421 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.724479+0000", 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.724480+0000", 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6470000000000001e-06 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.724484+0000", 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2780000000000002e-06 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.724492+0000", 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4430000000000003e-06 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.724498+0000", 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2689999999999996e-06 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.724870+0000", 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00037216300000000001 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.423 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.724983+0000", 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011304 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.725107+0000", 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012365499999999999 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.731427+0000", 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.006319641 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.857805+0000", 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.126378607 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.858029+0000", 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000223537 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.858071+0000", 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1879999999999999e-05 2024-03-26T05:06:04.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.858104+0000", 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3525000000000002e-05 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:157217 7.6 7:6d195129:::benchmark_data_smithi080_80848_object37718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:04:46.730831+0000", 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 76.911307358000002, 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.12894355800000001, 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.426 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 157217 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.730831+0000", 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.427 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.730831+0000", 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.730837+0000", 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4829999999999999e-06 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.730841+0000", 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9389999999999999e-06 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.428 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.730850+0000", 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2969999999999992e-06 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.730857+0000", 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7299999999999999e-06 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.731493+0000", 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00063587099999999998 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.429 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.731596+0000", 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103353 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.731698+0000", 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000102401 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.857850+0000", 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.126151175 2024-03-26T05:06:04.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.857861+0000", 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1605e-05 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.859664+0000", 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0018029020000000001 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.859734+0000", 2024-03-26T05:06:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9839999999999995e-05 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:04:46.859775+0000", 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0961999999999998e-05 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.24701.0:186914 7.d e155/103 7:b07c0e48:::benchmark_data_smithi080_80848_object67415:head v 155'11904, mlcod=155'11902)", 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:16.763021+0000", 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 46.879117876000002, 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.12594856700000001, 2024-03-26T05:06:04.432 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.763021+0000", 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.763021+0000", 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.763022+0000", 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3300000000000001e-07 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.763028+0000", 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2990000000000002e-06 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.763040+0000", 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1608e-05 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.763047+0000", 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8180000000000004e-06 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.867272+0000", 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.104224366 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.867291+0000", 2024-03-26T05:06:04.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9746999999999999e-05 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.888890+0000", 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.021598938000000002 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.888969+0000", 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8957999999999999e-05 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.24701.0:186918 7.d e155/103 7:b8fb59a9:::benchmark_data_smithi080_80848_object67419:head v 155'11905, mlcod=155'11902)", 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:16.763682+0000", 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 46.878456219999997, 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.135617021, 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.763682+0000", 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.763682+0000", 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.763683+0000", 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8999999999999996e-07 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.763688+0000", 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7129999999999996e-06 2024-03-26T05:06:04.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.763696+0000", 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7680000000000008e-06 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.763701+0000", 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9569999999999999e-06 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.873715+0000", 2024-03-26T05:06:04.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.110013428 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.873734+0000", 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9565e-05 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.898945+0000", 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.025210165 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:16.899299+0000", 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00035483500000000001 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:188377 7.6 7:6c0eb783:::benchmark_data_smithi080_80848_object68878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:18.428279+0000", 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 45.213859237999998, 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.128660731, 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:04.441 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 188377 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.428279+0000", 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.428279+0000", 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.428280+0000", 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.03e-07 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.428282+0000", 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2970000000000002e-06 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.428288+0000", 2024-03-26T05:06:04.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1179999999999997e-06 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.428293+0000", 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2499999999999997e-06 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.435038+0000", 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0067440089999999996 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.435157+0000", 2024-03-26T05:06:04.444 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000119116 2024-03-26T05:06:04.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:06:04.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.435270+0000", 2024-03-26T05:06:04.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000113066 2024-03-26T05:06:04.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:04.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.441622+0000", 2024-03-26T05:06:04.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0063518960000000001 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.442485+0000", 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00086352200000000003 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.556540+0000", 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.114054818 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.446 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.556745+0000", 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020457799999999999 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.556940+0000", 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000195658 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:188378 7.6 7:600c27c8:::benchmark_data_smithi080_80848_object68879:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:18.428409+0000", 2024-03-26T05:06:04.447 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 45.213729592999996, 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.12882237599999999, 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 188378 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.428409+0000", 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.428409+0000", 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.428410+0000", 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6699999999999999e-07 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.428412+0000", 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5629999999999999e-06 2024-03-26T05:06:04.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.428419+0000", 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0180000000000004e-06 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.428425+0000", 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5180000000000004e-06 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.438556+0000", 2024-03-26T05:06:04.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010131569 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.438690+0000", 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000134225 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.438811+0000", 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012076199999999999 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.441764+0000", 2024-03-26T05:06:04.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0029526769999999999 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.442736+0000", 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00097243299999999998 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.557008+0000", 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.114272053 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:04.452 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.557188+0000", 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000179612 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:18.557231+0000", 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3479000000000002e-05 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24356.0:1902 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94789128791168 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.871899+0000", 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 27.770239615000001, 2024-03-26T05:06:04.453 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.55856863400000001, 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24356", 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 1902 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.871899+0000", 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.454 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.871899+0000", 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.871901+0000", 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4529999999999999e-06 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.871904+0000", 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2329999999999998e-06 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.455 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.871914+0000", 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0441e-05 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.871921+0000", 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5139999999999998e-06 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872073+0000", 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015249 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.456 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op must wait for map", 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872079+0000", 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9750000000000004e-06 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.882805+0000", 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010726051 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.882814+0000", 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6090000000000001e-06 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.423086+0000", 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.54027256800000001 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for degraded object", 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.423362+0000", 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00027567500000000002 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.428581+0000", 2024-03-26T05:06:04.458 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0052186919999999996 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.428689+0000", 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010869 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.430468+0000", 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0017782430000000001 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.459 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:04.460 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:04.460 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:06:04.460 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:06:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:06:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:04.461 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:04.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:04.562 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:04.591 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:04.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:04.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: 5.6 scrub starts 2024-03-26T05:06:04.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: 5.6 scrub ok 2024-03-26T05:06:04.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: 6.1f scrub starts 2024-03-26T05:06:04.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: 6.1f scrub ok 2024-03-26T05:06:04.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: 5.c deep-scrub starts 2024-03-26T05:06:04.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: 5.c deep-scrub ok 2024-03-26T05:06:04.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: 7.6 scrub starts 2024-03-26T05:06:04.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: 7.6 scrub ok 2024-03-26T05:06:04.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: pgmap v235: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+remapped+backfill_toofull; 830 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 480 op/s; 1673/40497 objects misplaced (4.131%) 2024-03-26T05:06:04.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: 4.8 scrub starts 2024-03-26T05:06:04.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: 4.8 scrub ok 2024-03-26T05:06:04.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:04.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:06:04.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: 5.6 scrub starts 2024-03-26T05:06:04.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: 5.6 scrub ok 2024-03-26T05:06:04.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: 6.1f scrub starts 2024-03-26T05:06:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: 6.1f scrub ok 2024-03-26T05:06:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: 5.c deep-scrub starts 2024-03-26T05:06:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: 5.c deep-scrub ok 2024-03-26T05:06:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: 7.6 scrub starts 2024-03-26T05:06:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: 7.6 scrub ok 2024-03-26T05:06:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: pgmap v235: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+remapped+backfill_toofull; 830 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 480 op/s; 1673/40497 objects misplaced (4.131%) 2024-03-26T05:06:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: 4.8 scrub starts 2024-03-26T05:06:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: 4.8 scrub ok 2024-03-26T05:06:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:04.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:04 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:06:04.692 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:04.718 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:04.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:04.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:04.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: 5.6 scrub starts 2024-03-26T05:06:04.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: 5.6 scrub ok 2024-03-26T05:06:04.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: 6.1f scrub starts 2024-03-26T05:06:04.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: 6.1f scrub ok 2024-03-26T05:06:04.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: 5.c deep-scrub starts 2024-03-26T05:06:04.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: 5.c deep-scrub ok 2024-03-26T05:06:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: 7.6 scrub starts 2024-03-26T05:06:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: 7.6 scrub ok 2024-03-26T05:06:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: pgmap v235: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+remapped+backfill_toofull; 830 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 480 op/s; 1673/40497 objects misplaced (4.131%) 2024-03-26T05:06:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: 4.8 scrub starts 2024-03-26T05:06:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: 4.8 scrub ok 2024-03-26T05:06:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:04.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:06:04.819 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:04.847 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:04.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:04.949 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:04.977 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:04.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:05.079 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:05.119 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:05.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:05.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:05.221 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:05.257 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:05.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:05.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:06:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:06:05.358 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:05.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:05.396 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:05.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:05.497 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:05.537 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:05.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:05.639 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:05.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:05.666 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:05.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:05.767 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:05 smithi132 ceph-mon[33500]: 3.19 scrub starts 2024-03-26T05:06:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:05 smithi132 ceph-mon[33500]: 3.19 scrub ok 2024-03-26T05:06:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:05 smithi132 ceph-mon[33500]: 6.16 deep-scrub starts 2024-03-26T05:06:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:05 smithi132 ceph-mon[33500]: 6.16 deep-scrub ok 2024-03-26T05:06:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 0]}]: dispatch 2024-03-26T05:06:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 1]}]: dispatch 2024-03-26T05:06:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]: dispatch 2024-03-26T05:06:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-26T05:06:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:05 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:05 smithi132 ceph-mon[33500]: 6.e scrub starts 2024-03-26T05:06:05.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:05 smithi132 ceph-mon[33500]: 6.e scrub ok 2024-03-26T05:06:05.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:05.809 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:05.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:05.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[29691]: 3.19 scrub starts 2024-03-26T05:06:05.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[29691]: 3.19 scrub ok 2024-03-26T05:06:05.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[29691]: 6.16 deep-scrub starts 2024-03-26T05:06:05.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[29691]: 6.16 deep-scrub ok 2024-03-26T05:06:05.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 0]}]: dispatch 2024-03-26T05:06:05.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 1]}]: dispatch 2024-03-26T05:06:05.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]: dispatch 2024-03-26T05:06:05.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-26T05:06:05.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:05.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[29691]: 6.e scrub starts 2024-03-26T05:06:05.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[29691]: 6.e scrub ok 2024-03-26T05:06:05.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[40404]: 3.19 scrub starts 2024-03-26T05:06:05.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[40404]: 3.19 scrub ok 2024-03-26T05:06:05.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[40404]: 6.16 deep-scrub starts 2024-03-26T05:06:05.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[40404]: 6.16 deep-scrub ok 2024-03-26T05:06:05.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 0]}]: dispatch 2024-03-26T05:06:05.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 1]}]: dispatch 2024-03-26T05:06:05.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]: dispatch 2024-03-26T05:06:05.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-26T05:06:05.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:05.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[40404]: 6.e scrub starts 2024-03-26T05:06:05.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:05 smithi080 ceph-mon[40404]: 6.e scrub ok 2024-03-26T05:06:05.910 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:05.947 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:05.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:06.048 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:06.075 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:06.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:06.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:06.177 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:06.203 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:06.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:06.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:06.305 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:06.339 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:06.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:06.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:06.441 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:06.467 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:06.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:06.569 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:06.597 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:06.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:06.699 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:06.725 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:06.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:06.732 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T05:06:06.771 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[29691]: 3.b scrub starts 2024-03-26T05:06:06.771 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[29691]: 3.b scrub ok 2024-03-26T05:06:06.771 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[29691]: 5.5 scrub starts 2024-03-26T05:06:06.771 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[29691]: 5.5 scrub ok 2024-03-26T05:06:06.771 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 0]}]': finished 2024-03-26T05:06:06.771 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 1]}]': finished 2024-03-26T05:06:06.771 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]': finished 2024-03-26T05:06:06.771 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-26T05:06:06.772 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[29691]: osdmap e171: 8 total, 8 up, 7 in 2024-03-26T05:06:06.772 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[29691]: pgmap v237: 148 pgs: 146 active+clean, 2 active+remapped+backfill_toofull; 531 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 1069/26151 objects misplaced (4.088%) 2024-03-26T05:06:06.772 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[40404]: 3.b scrub starts 2024-03-26T05:06:06.772 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[40404]: 3.b scrub ok 2024-03-26T05:06:06.772 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[40404]: 5.5 scrub starts 2024-03-26T05:06:06.772 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[40404]: 5.5 scrub ok 2024-03-26T05:06:06.772 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 0]}]': finished 2024-03-26T05:06:06.772 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 1]}]': finished 2024-03-26T05:06:06.772 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]': finished 2024-03-26T05:06:06.772 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-26T05:06:06.772 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[40404]: osdmap e171: 8 total, 8 up, 7 in 2024-03-26T05:06:06.772 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:06 smithi080 ceph-mon[40404]: pgmap v237: 148 pgs: 146 active+clean, 2 active+remapped+backfill_toofull; 531 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 1069/26151 objects misplaced (4.088%) 2024-03-26T05:06:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:06 smithi132 ceph-mon[33500]: 3.b scrub starts 2024-03-26T05:06:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:06 smithi132 ceph-mon[33500]: 3.b scrub ok 2024-03-26T05:06:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:06 smithi132 ceph-mon[33500]: 5.5 scrub starts 2024-03-26T05:06:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:06 smithi132 ceph-mon[33500]: 5.5 scrub ok 2024-03-26T05:06:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 0]}]': finished 2024-03-26T05:06:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 1]}]': finished 2024-03-26T05:06:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]': finished 2024-03-26T05:06:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-26T05:06:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:06 smithi132 ceph-mon[33500]: osdmap e171: 8 total, 8 up, 7 in 2024-03-26T05:06:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:06 smithi132 ceph-mon[33500]: pgmap v237: 148 pgs: 146 active+clean, 2 active+remapped+backfill_toofull; 531 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 1069/26151 objects misplaced (4.088%) 2024-03-26T05:06:06.827 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:06.852 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:06.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:06.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:06.954 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:07.020 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:07.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:07.121 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:07.147 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:07.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:07.249 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:07.298 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:07.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:07.399 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:07.427 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:07.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:07.529 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:07.558 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:07.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:07.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:07.660 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:07.717 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:07.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:07 smithi132 ceph-mon[33500]: 7.9 scrub starts 2024-03-26T05:06:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:07 smithi132 ceph-mon[33500]: 7.b scrub starts 2024-03-26T05:06:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:07 smithi132 ceph-mon[33500]: 7.9 scrub ok 2024-03-26T05:06:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:07 smithi132 ceph-mon[33500]: 7.b scrub ok 2024-03-26T05:06:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:07 smithi132 ceph-mon[33500]: osdmap e172: 8 total, 8 up, 7 in 2024-03-26T05:06:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:07 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:06:07.818 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:07 smithi080 ceph-mon[40404]: 7.9 scrub starts 2024-03-26T05:06:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:07 smithi080 ceph-mon[40404]: 7.b scrub starts 2024-03-26T05:06:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:07 smithi080 ceph-mon[40404]: 7.9 scrub ok 2024-03-26T05:06:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:07 smithi080 ceph-mon[40404]: 7.b scrub ok 2024-03-26T05:06:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:07 smithi080 ceph-mon[40404]: osdmap e172: 8 total, 8 up, 7 in 2024-03-26T05:06:07.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:07 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:06:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:07 smithi080 ceph-mon[29691]: 7.9 scrub starts 2024-03-26T05:06:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:07 smithi080 ceph-mon[29691]: 7.b scrub starts 2024-03-26T05:06:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:07 smithi080 ceph-mon[29691]: 7.9 scrub ok 2024-03-26T05:06:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:07 smithi080 ceph-mon[29691]: 7.b scrub ok 2024-03-26T05:06:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:07 smithi080 ceph-mon[29691]: osdmap e172: 8 total, 8 up, 7 in 2024-03-26T05:06:07.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:07 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:06:07.886 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:07.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:07.987 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:08.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:08.040 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:08.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:08.141 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:08.168 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:08.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:08.270 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:08.298 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:08.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:08.400 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:08.467 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:08.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:08.568 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:08.595 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:08.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:08.687 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:08.696 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:08.761 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:08.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:08.788 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[40404]: 3.1c scrub starts 2024-03-26T05:06:08.789 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[40404]: 3.1c scrub ok 2024-03-26T05:06:08.789 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:06:08.789 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:06:08.789 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[40404]: pgmap v239: 148 pgs: 146 active+clean, 2 active+remapped+backfill_toofull; 531 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 940 op/s; 1069/26151 objects misplaced (4.088%) 2024-03-26T05:06:08.789 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[40404]: 6.1a scrub starts 2024-03-26T05:06:08.789 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[40404]: 6.1a scrub ok 2024-03-26T05:06:08.789 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[29691]: 3.1c scrub starts 2024-03-26T05:06:08.789 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[29691]: 3.1c scrub ok 2024-03-26T05:06:08.790 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:06:08.790 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:06:08.790 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[29691]: pgmap v239: 148 pgs: 146 active+clean, 2 active+remapped+backfill_toofull; 531 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 940 op/s; 1069/26151 objects misplaced (4.088%) 2024-03-26T05:06:08.790 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[29691]: 6.1a scrub starts 2024-03-26T05:06:08.790 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:08 smithi080 ceph-mon[29691]: 6.1a scrub ok 2024-03-26T05:06:08.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:08 smithi132 ceph-mon[33500]: 3.1c scrub starts 2024-03-26T05:06:08.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:08 smithi132 ceph-mon[33500]: 3.1c scrub ok 2024-03-26T05:06:08.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:08 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:06:08.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:08 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:06:08.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:08 smithi132 ceph-mon[33500]: pgmap v239: 148 pgs: 146 active+clean, 2 active+remapped+backfill_toofull; 531 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 940 op/s; 1069/26151 objects misplaced (4.088%) 2024-03-26T05:06:08.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:08 smithi132 ceph-mon[33500]: 6.1a scrub starts 2024-03-26T05:06:08.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:08 smithi132 ceph-mon[33500]: 6.1a scrub ok 2024-03-26T05:06:08.863 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:08.919 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:08.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:09.020 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:09.045 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:09.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:09.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:09.147 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:09.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:09.212 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:09.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:09.313 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:09.338 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:09.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:09.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:09.440 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:09.469 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:09.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:09.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:09.571 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:09.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:09.598 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:09.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:09.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:09 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:09.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:09 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:09.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:09 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:06:09.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:09 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:06:09.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:09 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:09.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:09 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:09.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:09 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:06:09.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:09 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:06:09.698 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:09.751 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:09.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:09.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:09.853 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:09.905 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:09.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:10.006 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:10.034 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:10.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:10.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:09 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:10.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:09 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:10.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:09 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:06:10.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:09 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:06:10.135 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:10.163 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:10.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:10.265 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:10.294 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:10.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:10.395 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:10.421 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:10.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:10.518 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:10.523 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:10.565 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:10.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:10.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:06:10.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:06:10.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[29691]: 4.5 scrub starts 2024-03-26T05:06:10.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[29691]: 4.5 scrub ok 2024-03-26T05:06:10.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[29691]: osdmap e173: 8 total, 8 up, 7 in 2024-03-26T05:06:10.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:06:10.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:06:10.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:10.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[29691]: pgmap v241: 148 pgs: 146 active+clean, 2 active+remapped+backfill_toofull; 439 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s; 1069/21759 objects misplaced (4.913%) 2024-03-26T05:06:10.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1110731664' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:06:10.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[29691]: 5.e scrub starts 2024-03-26T05:06:10.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[29691]: 5.e scrub ok 2024-03-26T05:06:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:06:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:06:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[40404]: 4.5 scrub starts 2024-03-26T05:06:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[40404]: 4.5 scrub ok 2024-03-26T05:06:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[40404]: osdmap e173: 8 total, 8 up, 7 in 2024-03-26T05:06:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:06:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:06:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[40404]: pgmap v241: 148 pgs: 146 active+clean, 2 active+remapped+backfill_toofull; 439 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s; 1069/21759 objects misplaced (4.913%) 2024-03-26T05:06:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1110731664' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:06:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[40404]: 5.e scrub starts 2024-03-26T05:06:10.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:10 smithi080 ceph-mon[40404]: 5.e scrub ok 2024-03-26T05:06:10.667 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:10.716 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:06:10.717 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:10.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:10.819 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:10.846 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:10.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:10.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:10.948 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:10.977 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:10.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:10.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:11.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:10 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:06:11.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:10 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:06:11.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:10 smithi132 ceph-mon[33500]: 4.5 scrub starts 2024-03-26T05:06:11.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:10 smithi132 ceph-mon[33500]: 4.5 scrub ok 2024-03-26T05:06:11.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:10 smithi132 ceph-mon[33500]: osdmap e173: 8 total, 8 up, 7 in 2024-03-26T05:06:11.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:10 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:06:11.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:10 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:06:11.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:10 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:11.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:10 smithi132 ceph-mon[33500]: pgmap v241: 148 pgs: 146 active+clean, 2 active+remapped+backfill_toofull; 439 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s; 1069/21759 objects misplaced (4.913%) 2024-03-26T05:06:11.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:10 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1110731664' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:06:11.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:10 smithi132 ceph-mon[33500]: 5.e scrub starts 2024-03-26T05:06:11.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:10 smithi132 ceph-mon[33500]: 5.e scrub ok 2024-03-26T05:06:11.079 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:11.109 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:11.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:11.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:11.211 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:11.251 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:11.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:11.352 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:11.380 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:11.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:11.402 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T05:06:11.482 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:11.524 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:11.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:11.625 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:11.655 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:11.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:11.757 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:11.781 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:11.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:11.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:11.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:11 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:06:11.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:11 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1110731664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:06:11.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:11 smithi080 ceph-mon[29691]: osdmap e174: 8 total, 8 up, 7 in 2024-03-26T05:06:11.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:11 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3435197612' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:06:11.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:11 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3435197612' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:06:11.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:11 smithi080 ceph-mon[29691]: osdmap e175: 8 total, 8 up, 7 in 2024-03-26T05:06:11.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:11 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:06:11.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:11 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1110731664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:06:11.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:11 smithi080 ceph-mon[40404]: osdmap e174: 8 total, 8 up, 7 in 2024-03-26T05:06:11.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:11 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3435197612' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:06:11.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:11 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3435197612' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:06:11.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:11 smithi080 ceph-mon[40404]: osdmap e175: 8 total, 8 up, 7 in 2024-03-26T05:06:11.883 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:11.975 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:11.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:12.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:11 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:06:12.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:11 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1110731664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:06:12.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:11 smithi132 ceph-mon[33500]: osdmap e174: 8 total, 8 up, 7 in 2024-03-26T05:06:12.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:11 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3435197612' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:06:12.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:11 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3435197612' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:06:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:11 smithi132 ceph-mon[33500]: osdmap e175: 8 total, 8 up, 7 in 2024-03-26T05:06:12.076 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:12.101 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:12.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:12.201 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:12.226 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:12.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:12.327 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:12.355 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:12.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:12.456 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:12.490 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:12.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:12.592 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:12.618 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:12.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:12.719 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:12.743 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:12.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:12.844 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:12.868 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:12.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:12.970 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:12.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:12.997 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:12.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:13.098 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:13.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:13.123 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:13.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:13.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:13.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:13.251 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:13.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:13.353 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:13.380 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:13.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:13.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:13.482 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:13.524 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:13.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:13.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:13 smithi080 ceph-mon[40404]: pgmap v244: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 237 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 770 B/s rd, 0 B/s wr, 885 op/s; 219/12048 objects misplaced (1.818%); 3.6 MiB/s, 57 objects/s recovering 2024-03-26T05:06:13.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:13 smithi080 ceph-mon[40404]: osdmap e176: 8 total, 8 up, 7 in 2024-03-26T05:06:13.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:13 smithi080 ceph-mon[40404]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T05:06:13.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:13 smithi080 ceph-mon[29691]: pgmap v244: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 237 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 770 B/s rd, 0 B/s wr, 885 op/s; 219/12048 objects misplaced (1.818%); 3.6 MiB/s, 57 objects/s recovering 2024-03-26T05:06:13.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:13 smithi080 ceph-mon[29691]: osdmap e176: 8 total, 8 up, 7 in 2024-03-26T05:06:13.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:13 smithi080 ceph-mon[29691]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T05:06:13.626 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:13.653 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:13.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:13.755 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:13.767 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T05:06:13.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:13 smithi132 ceph-mon[33500]: pgmap v244: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 237 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 770 B/s rd, 0 B/s wr, 885 op/s; 219/12048 objects misplaced (1.818%); 3.6 MiB/s, 57 objects/s recovering 2024-03-26T05:06:13.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:13 smithi132 ceph-mon[33500]: osdmap e176: 8 total, 8 up, 7 in 2024-03-26T05:06:13.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:13 smithi132 ceph-mon[33500]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T05:06:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:13.786 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:13.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:13.888 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:13.915 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:13.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:14.016 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:14.029 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:06:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:06:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:06:14.042 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:14.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:14.066 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:06:14.116 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:06:14.143 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:14.153 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:06:14.195 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:06:14.200 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:14.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:14.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:14.237 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:06:14.279 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:06:14.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:14.317 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:06:14.329 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:14.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:14.358 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:06:14.363 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:06:14.363 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:06:14.363 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:06:14.363 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:06:14.363 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:06:14.364 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:06:14.364 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:06:14.364 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:06:14.431 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:14.459 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:14.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:14.560 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:14.586 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:14.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:14.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:14 smithi080 ceph-mon[40404]: osdmap e177: 8 total, 8 up, 7 in 2024-03-26T05:06:14.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:14 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1836356958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:14.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:14 smithi080 ceph-mon[29691]: osdmap e177: 8 total, 8 up, 7 in 2024-03-26T05:06:14.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:14 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1836356958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:14.688 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:14.725 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:14.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:14.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:14 smithi132 ceph-mon[33500]: osdmap e177: 8 total, 8 up, 7 in 2024-03-26T05:06:14.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:14 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1836356958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:14.826 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:14.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:14.871 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:14.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:14.972 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:14.998 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:14.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:14.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:15.100 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:15.125 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:15.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:15.227 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:15.268 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:15.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:15.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:06:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:06:15.331 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Removed 119496 objects 2024-03-26T05:06:15.331 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Clean up completed and total clean up time :127.102 2024-03-26T05:06:15.372 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:15.400 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:15.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:15.501 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:15.525 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:15.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:15.627 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:15.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:15.652 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:15.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:15.753 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:15.777 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:15.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:15.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:15 smithi132 ceph-mon[33500]: pgmap v247: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 237 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 219/12048 objects misplaced (1.818%); 4.8 MiB/s, 76 objects/s recovering 2024-03-26T05:06:15.825 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:15 smithi080 ceph-mon[40404]: pgmap v247: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 237 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 219/12048 objects misplaced (1.818%); 4.8 MiB/s, 76 objects/s recovering 2024-03-26T05:06:15.826 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:15 smithi080 ceph-mon[29691]: pgmap v247: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 237 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 219/12048 objects misplaced (1.818%); 4.8 MiB/s, 76 objects/s recovering 2024-03-26T05:06:15.878 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:15.901 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:15.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:16.003 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:16.030 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:16.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:16.131 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:16.158 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:16.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:16.259 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:16.284 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:16.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:16.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:16.386 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:16.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:16.416 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:16.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:16.517 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:16.542 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:16.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:16.643 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:16.667 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:16.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:16.768 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:16.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:16 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:16.796 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:16.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:16 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:16.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:16 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:16.898 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:16.925 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:16.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:17.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:17.055 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:17.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:17.144 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:06:17.157 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:17.217 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:17.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:17.319 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:17.346 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:17.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:17.448 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:17.476 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:17.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:17.578 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:17.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:17.603 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:17.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:17.704 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:17.728 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:17.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:17.770 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-26T05:06:17.770 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-26T05:06:17.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:17 smithi132 ceph-mon[33500]: pgmap v248: 148 pgs: 148 active+clean; 22 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1003 B/s rd, 0 B/s wr, 674 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-03-26T05:06:17.794 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:17 smithi080 ceph-mon[29691]: pgmap v248: 148 pgs: 148 active+clean; 22 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1003 B/s rd, 0 B/s wr, 674 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-03-26T05:06:17.795 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:17 smithi080 ceph-mon[40404]: pgmap v248: 148 pgs: 148 active+clean; 22 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1003 B/s rd, 0 B/s wr, 674 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-03-26T05:06:17.830 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:17.850 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:06:17.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:17.856 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:17.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:17.874 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:17.958 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:17.984 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:17.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:18.085 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:18.110 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:18.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:18.211 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:18.239 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:18.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:18.341 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:18.368 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:18.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:18.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:18.470 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:18.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:18.496 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:18.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:18.561 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:18.597 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:18.621 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:18.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:18.722 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:18.747 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:18.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:18.848 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:18.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:18.877 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:18.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:18.978 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:19.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:19.003 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:19.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:19.104 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:19.129 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:19.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:19.230 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:19.254 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:19.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:19.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:19.356 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:19.384 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:19.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:19.487 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:19.514 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:19.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:19.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:19.554 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:19 smithi080 ceph-mon[29691]: pgmap v249: 148 pgs: 148 active+clean; 22 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 798 B/s rd, 0 B/s wr, 536 op/s; 1.7 MiB/s, 27 objects/s recovering 2024-03-26T05:06:19.554 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:19.554 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:06:19.555 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:19 smithi080 ceph-mon[40404]: pgmap v249: 148 pgs: 148 active+clean; 22 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 798 B/s rd, 0 B/s wr, 536 op/s; 1.7 MiB/s, 27 objects/s recovering 2024-03-26T05:06:19.555 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:19.555 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:06:19.615 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:19.643 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:19.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:19.744 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:19.771 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:19.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:19.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:19.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:19 smithi132 ceph-mon[33500]: pgmap v249: 148 pgs: 148 active+clean; 22 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 798 B/s rd, 0 B/s wr, 536 op/s; 1.7 MiB/s, 27 objects/s recovering 2024-03-26T05:06:19.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:19.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:06:19.873 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:19.900 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:19.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:20.002 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:20.027 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:20.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:20.128 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:20.157 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:20.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:20.211 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:06:20.211 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:06:20.259 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:20.284 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:20.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:20.385 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:20.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:20.411 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:20.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:20.512 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:20.545 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:20.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:20.647 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:20.675 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:20.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:20.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:20.704 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:20 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:06:20.705 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:20 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:06:20.705 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:20 smithi080 ceph-mon[40404]: osdmap e178: 8 total, 8 up, 7 in 2024-03-26T05:06:20.705 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:20 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:06:20.705 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:20 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:06:20.705 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:20 smithi080 ceph-mon[29691]: osdmap e178: 8 total, 8 up, 7 in 2024-03-26T05:06:20.777 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:20.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:20 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:06:20.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:20 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:06:20.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:20 smithi132 ceph-mon[33500]: osdmap e178: 8 total, 8 up, 7 in 2024-03-26T05:06:20.804 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:20.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:20.905 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:20.933 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:20.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:20.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:21.035 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:21.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:21.060 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:21.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:21.161 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:21.186 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:21.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:21.287 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:21.315 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:21.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:21.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:21.417 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:21.444 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:21.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:21.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:21.445 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T05:06:21.546 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:21.571 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:21.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:21.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:21 smithi080 ceph-mon[29691]: pgmap v251: 148 pgs: 148 active+clean; 9.5 MiB data, 969 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 492 op/s; 1.5 MiB/s, 23 objects/s recovering 2024-03-26T05:06:21.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:21 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/848753936' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:06:21.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:21 smithi080 ceph-mon[40404]: pgmap v251: 148 pgs: 148 active+clean; 9.5 MiB data, 969 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 492 op/s; 1.5 MiB/s, 23 objects/s recovering 2024-03-26T05:06:21.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:21 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/848753936' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:06:21.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:21.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:21.701 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:21.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:21.788 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T05:06:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:21 smithi132 ceph-mon[33500]: pgmap v251: 148 pgs: 148 active+clean; 9.5 MiB data, 969 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 492 op/s; 1.5 MiB/s, 23 objects/s recovering 2024-03-26T05:06:21.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:21 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/848753936' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:06:21.803 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:21.855 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:21.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:21.956 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:21.982 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:21.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:22.021 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:22.083 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:22.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:22.112 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:22.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:22.213 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:22.242 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:22.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:22.343 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:22.372 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:22.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:22.473 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:22.483 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:22.499 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 3 1 2024-03-26T05:06:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:22.500 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:22.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:22.601 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:22.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:22.628 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:22.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:22.729 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:22.756 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:22.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:22.773 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:22 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/848753936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:06:22.773 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:22 smithi080 ceph-mon[29691]: osdmap e179: 8 total, 8 up, 7 in 2024-03-26T05:06:22.773 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:22 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/848753936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:06:22.773 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:22 smithi080 ceph-mon[40404]: osdmap e179: 8 total, 8 up, 7 in 2024-03-26T05:06:22.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:22 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/848753936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:06:22.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:22 smithi132 ceph-mon[33500]: osdmap e179: 8 total, 8 up, 7 in 2024-03-26T05:06:22.857 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:22.883 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:22.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:22.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:22.986 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:23.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:23.012 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:23.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:23.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:23.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:23.137 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:23.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:23.238 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:23.266 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:23.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:23.368 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:23.397 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:23.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:23.499 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:23.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:23.524 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:23.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:23.625 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:23.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:23.650 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:23.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:23.727 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:23 smithi080 ceph-mon[29691]: pgmap v253: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 474 op/s; 1.4 MiB/s, 22 objects/s recovering 2024-03-26T05:06:23.727 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:23 smithi080 ceph-mon[40404]: pgmap v253: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 474 op/s; 1.4 MiB/s, 22 objects/s recovering 2024-03-26T05:06:23.750 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:23.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:23.775 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:23.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:23.795 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:23 smithi132 ceph-mon[33500]: pgmap v253: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 474 op/s; 1.4 MiB/s, 22 objects/s recovering 2024-03-26T05:06:23.876 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:23.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:23.901 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:23.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:24.002 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:24.026 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:24.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:24.127 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:24.156 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:24.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:24.258 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:24.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:24.287 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:24.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:24.368 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:06:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:06:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:06:24.387 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:24.415 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:24.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:24.517 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:24.544 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:24.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:24.553 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:24.646 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:24.671 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:24.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:24.772 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:24.798 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:24.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:24.899 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:24.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:24.928 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:24.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:25.029 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:25.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:25.058 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:25.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:25.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:25.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:25.183 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:25.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:25.284 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:25.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:06:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:06:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:25.313 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:25.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:25.415 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:25.458 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:25.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:25.560 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:25.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:25.584 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:25.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:25.685 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:25.707 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:25 smithi080 ceph-mon[40404]: pgmap v254: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 44 op/s 2024-03-26T05:06:25.707 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:25 smithi080 ceph-mon[29691]: pgmap v254: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 44 op/s 2024-03-26T05:06:25.713 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:25.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:25.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:25.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:25 smithi132 ceph-mon[33500]: pgmap v254: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 44 op/s 2024-03-26T05:06:25.815 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:25.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:25.843 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:25.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:25.944 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:25.968 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:25.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:25.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:26.009 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:06:26.029 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:06:26.049 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:06:26.067 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:06:26.070 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:26.087 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:06:26.098 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:26.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:26.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:26.107 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:06:26.126 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:06:26.146 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:06:26.147 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:06:26.147 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:06:26.147 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:06:26.147 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:06:26.148 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:06:26.148 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:06:26.148 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:06:26.148 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:06:26.200 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:26.228 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:26.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:26.329 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:26.354 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:26.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:26.455 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:26.465 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:06:26.476 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:06:26.476 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:06:26.476 INFO:teuthology.orchestra.run.smithi080.stdout: "complaint_time": 30, 2024-03-26T05:06:26.477 INFO:teuthology.orchestra.run.smithi080.stdout: "num_blocked_ops": 0 2024-03-26T05:06:26.477 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:06:26.481 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:26.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:26.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:26.612 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:26.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:26.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:26 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:26.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:26 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1570990537' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:06:26.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:26 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1925398265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:26.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:26 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:26.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:26 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1570990537' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:06:26.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:26 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1925398265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:26.713 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:26.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:26.737 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:26.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:26.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:26 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:26.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:26 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1570990537' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:06:26.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:26 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1925398265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:26.838 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:26.862 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:26.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:26.963 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:26.991 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:26.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:27.093 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:27.118 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:27.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:27.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:27.220 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:27.244 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:27.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:27.312 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:27.345 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:27.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:27.370 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:27.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:27.471 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:27.495 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:27.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:27.596 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:27.620 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:27.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:27.721 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:27.747 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:27.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:27.760 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:27 smithi080 ceph-mon[29691]: pgmap v255: 148 pgs: 148 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 44 op/s 2024-03-26T05:06:27.760 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:27 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:06:27.760 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:27 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1570990537' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:06:27.760 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:27 smithi080 ceph-mon[29691]: osdmap e180: 8 total, 8 up, 7 in 2024-03-26T05:06:27.761 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:27 smithi080 ceph-mon[40404]: pgmap v255: 148 pgs: 148 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 44 op/s 2024-03-26T05:06:27.761 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:27 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:06:27.761 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:27 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1570990537' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:06:27.761 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:27 smithi080 ceph-mon[40404]: osdmap e180: 8 total, 8 up, 7 in 2024-03-26T05:06:27.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:27 smithi132 ceph-mon[33500]: pgmap v255: 148 pgs: 148 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 44 op/s 2024-03-26T05:06:27.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:27 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:06:27.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:27 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1570990537' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:06:27.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:27 smithi132 ceph-mon[33500]: osdmap e180: 8 total, 8 up, 7 in 2024-03-26T05:06:27.849 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:27.874 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:27.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:27.976 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:28.005 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:28.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:28.107 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:28.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:28.132 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:28.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:28.233 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:28.261 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:28.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:28.362 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:28.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:28.387 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:28.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:28.488 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:28.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:28.517 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:28.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:28.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:28 smithi080 ceph-mon[29691]: pgmap v257: 148 pgs: 148 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 18 op/s 2024-03-26T05:06:28.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:28 smithi080 ceph-mon[40404]: pgmap v257: 148 pgs: 148 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 18 op/s 2024-03-26T05:06:28.618 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:28.643 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:28.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:28.744 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:28.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:28.769 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:28.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:28.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:28 smithi132 ceph-mon[33500]: pgmap v257: 148 pgs: 148 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 18 op/s 2024-03-26T05:06:28.869 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:28.894 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:28.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:28.995 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:29.019 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:29.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:29.121 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:29.127 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:06:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:29.151 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:29.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:29.242 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T05:06:29.251 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:29.309 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:29.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:29.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:29.434 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:29.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:29.537 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:29.565 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:29.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:29.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:29.667 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:29.696 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:29.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:29.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:29.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:29.826 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:29.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:29.927 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:29.953 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:29.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:30.055 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:30.084 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:30.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:30.186 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:30.215 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:30.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:30.316 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:30.344 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:30.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:30.446 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:30.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:30.472 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:30.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:30.573 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:30.601 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:30.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:30.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:30.703 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:30.731 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:30.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:30.833 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:30.859 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:30.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:30.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:30.884 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'unique_pool_0' removed 2024-03-26T05:06:30.961 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:30.989 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:30.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:31.091 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:31.117 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:31.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:30 smithi080 ceph-mon[40404]: pgmap v258: 148 pgs: 148 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:06:31.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:30 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1942088484' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:06:31.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:30 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:06:31.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:30 smithi080 ceph-mon[29691]: pgmap v258: 148 pgs: 148 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:06:31.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:30 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1942088484' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:06:31.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:30 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:06:31.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:31.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:31.219 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:31.243 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:31.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:31.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:30 smithi132 ceph-mon[33500]: pgmap v258: 148 pgs: 148 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:06:31.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:30 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1942088484' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:06:31.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:30 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:06:31.344 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:31.367 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:31.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:31.469 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:31.497 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:31.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:31.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:31.545 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:31.545 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:31.600 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:31.628 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:31.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:31.729 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:31.753 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:31.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:31.854 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:31.880 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:31.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:31.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:31.982 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:31.983 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:31 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:06:31.983 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:31 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:06:31.983 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:31 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:06:31.983 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:31 smithi080 ceph-mon[40404]: osdmap e181: 8 total, 8 up, 7 in 2024-03-26T05:06:31.984 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:31 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:06:31.984 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:31 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:06:31.984 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:31 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:06:31.984 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:31 smithi080 ceph-mon[29691]: osdmap e181: 8 total, 8 up, 7 in 2024-03-26T05:06:32.008 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:32.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:32.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:32.011 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T05:06:32.012 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T05:06:32.012 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-26T05:06:32.012 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-26T05:06:32.110 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:32.133 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:32.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:32.235 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:32.259 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:32.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:32.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:31 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:06:32.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:31 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:06:32.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:31 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:06:32.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:31 smithi132 ceph-mon[33500]: osdmap e181: 8 total, 8 up, 7 in 2024-03-26T05:06:32.361 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:32.390 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:32.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:32.492 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:32.517 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:32.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:32.619 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:32.647 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:32.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:32.750 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:32.778 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:32.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:32.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:32.784 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:06:32.880 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:32.904 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:32.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:33.005 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:33.029 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:33.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:33.130 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:33.151 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:32 smithi080 ceph-mon[40404]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:06:33.151 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:32 smithi080 ceph-mon[29691]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:06:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:33.156 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:33.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:33.257 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:33.281 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:33.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:33.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:32 smithi132 ceph-mon[33500]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:06:33.381 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:33.407 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:33.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:33.508 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:33.532 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:33.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:33.633 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:33.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:33.657 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:33.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:33.758 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:33.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:33.783 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:33.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:33.884 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:33.912 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:33.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:34.014 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:34.043 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:34.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:34.117 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:06:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:06:34.144 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:34.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:34.169 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:34.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:34.270 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:34.295 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:34.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:34.395 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:34.419 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:34.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:34.520 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:34.545 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:34.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:34.646 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:34.677 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:34.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:34.778 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:34.806 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:34.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:34.908 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:34.933 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:34.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:35.034 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:35.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:35.062 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:35.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:35.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:06:35.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:06:35.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[29691]: pgmap v261: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T05:06:35.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:06:35.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:06:35.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[29691]: osdmap e182: 8 total, 8 up, 7 in 2024-03-26T05:06:35.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:06:35.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:06:35.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:06:35.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[40404]: pgmap v261: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T05:06:35.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:06:35.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:06:35.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[40404]: osdmap e182: 8 total, 8 up, 7 in 2024-03-26T05:06:35.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:34 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:06:35.164 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:35.191 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:35.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:35.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:35.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:06:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:06:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:34 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:06:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:34 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:06:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:34 smithi132 ceph-mon[33500]: pgmap v261: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T05:06:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:34 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:06:35.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:34 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:06:35.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:34 smithi132 ceph-mon[33500]: osdmap e182: 8 total, 8 up, 7 in 2024-03-26T05:06:35.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:34 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:06:35.294 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:35.318 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:35.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:35.419 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:35.447 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:35.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:35.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:35.549 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:35.577 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:35.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:35.672 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:35.682 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:35.699 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:06:35.699 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:06:35.699 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:06:35.699 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:06:35.699 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.699 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206018 7.b 7:d4295363:::benchmark_data_smithi080_80848_object86519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-03-26T05:06:35.699 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.872555+0000", 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 59.832801121999999, 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.670752888, 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206018 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872555+0000", 2024-03-26T05:06:35.700 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872555+0000", 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872556+0000", 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3629999999999999e-06 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.701 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872562+0000", 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5949999999999998e-06 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872577+0000", 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5183e-05 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872587+0000", 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4620000000000002e-06 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.702 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872681+0000", 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4724999999999999e-05 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.873453+0000", 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00077158499999999998 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.873565+0000", 2024-03-26T05:06:35.703 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011171900000000001 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.900679+0000", 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.027114513 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.900691+0000", 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2386e-05 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392310+0000", 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.491618318 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392323+0000", 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2852e-05 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.543230+0000", 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1509076469999999 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.705 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.543308+0000", 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7540000000000006e-05 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206009 7.0 7:0aae643b:::benchmark_data_smithi080_80848_object86510:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.886333+0000", 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 59.819022926999999, 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6852875250000001, 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:06:35.706 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206009 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886333+0000", 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886333+0000", 2024-03-26T05:06:35.707 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886335+0000", 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2790000000000001e-06 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886341+0000", 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6509999999999998e-06 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886354+0000", 2024-03-26T05:06:35.708 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2736e-05 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886364+0000", 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0159999999999999e-05 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.887501+0000", 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0011366639999999999 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.709 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.887510+0000", 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8609999999999993e-06 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.394783+0000", 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.50727263899999997 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.394988+0000", 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020464800000000001 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.710 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.569805+0000", 2024-03-26T05:06:35.711 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1748175009999999 2024-03-26T05:06:35.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.711 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.711 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.711 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.571621+0000", 2024-03-26T05:06:35.711 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001815386 2024-03-26T05:06:35.711 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.711 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.711 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206029 7.0 7:0428147a:::benchmark_data_smithi080_80848_object86530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.897565+0000", 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 59.807790902000001, 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7163614869999999, 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206029 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.712 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.713 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.713 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.713 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897565+0000", 2024-03-26T05:06:35.713 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.713 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.713 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.713 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.713 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897565+0000", 2024-03-26T05:06:35.714 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.714 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.715 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.715 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.715 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897566+0000", 2024-03-26T05:06:35.715 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.319e-06 2024-03-26T05:06:35.715 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.715 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.715 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.715 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897569+0000", 2024-03-26T05:06:35.715 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.7920000000000001e-06 2024-03-26T05:06:35.715 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.715 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.716 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.716 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897578+0000", 2024-03-26T05:06:35.716 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4840000000000004e-06 2024-03-26T05:06:35.716 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.716 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.716 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.716 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897585+0000", 2024-03-26T05:06:35.716 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5750000000000003e-06 2024-03-26T05:06:35.716 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.716 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.898208+0000", 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00062269599999999995 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.898218+0000", 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0395e-05 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395017+0000", 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.496798144 2024-03-26T05:06:35.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395026+0000", 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6509999999999996e-06 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.571919+0000", 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1768926829999999 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.718 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.572211+0000", 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00029205600000000001 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.572422+0000", 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021075299999999999 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591634+0000", 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.019212002999999998 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.719 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596999+0000", 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0053655129999999997 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597046+0000", 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6810000000000001e-05 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613723+0000", 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016677112000000001 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.720 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613791+0000", 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8207999999999995e-05 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613927+0000", 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013529300000000001 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.721 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206032 7.b 7:db699848:::benchmark_data_smithi080_80848_object86533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.907812+0000", 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 59.797543595999997, 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.654457345, 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206032 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.722 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907812+0000", 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907812+0000", 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907814+0000", 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7710000000000001e-06 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.723 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907821+0000", 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0049999999999998e-06 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907845+0000", 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4151e-05 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907854+0000", 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8279999999999992e-06 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.724 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.909278+0000", 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014236120000000001 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.909289+0000", 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1054e-05 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392352+0000", 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.48306306100000002 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.725 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392361+0000", 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9190000000000005e-06 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.543561+0000", 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.151200486 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.543686+0000", 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012433600000000001 2024-03-26T05:06:35.726 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,5,6", 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.544119+0000", 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00043299000000000001 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558280+0000", 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014161366999999999 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558330+0000", 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0148000000000002e-05 2024-03-26T05:06:35.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558377+0000", 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6708000000000001e-05 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.561932+0000", 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0035554779999999999 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562152+0000", 2024-03-26T05:06:35.728 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021959599999999999 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562270+0000", 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117835 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206042 7.0 7:0cf7d89a:::benchmark_data_smithi080_80848_object86543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.917369+0000", 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 59.787987098000002, 2024-03-26T05:06:35.729 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.696765531, 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206042 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917369+0000", 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.730 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917369+0000", 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917371+0000", 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.232e-06 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917379+0000", 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2479999999999996e-06 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.731 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917402+0000", 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2521999999999999e-05 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917444+0000", 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1974999999999997e-05 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917648+0000", 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020391800000000001 2024-03-26T05:06:35.732 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917662+0000", 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4052e-05 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395054+0000", 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.47739227899999997 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395062+0000", 2024-03-26T05:06:35.733 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2689999999999999e-06 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.586328+0000", 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.191265663 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.587220+0000", 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00089233699999999999 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:06:35.734 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.587549+0000", 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00032828700000000002 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597064+0000", 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0095153470000000004 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597117+0000", 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2428e-05 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.735 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613547+0000", 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016430177000000001 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613969+0000", 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00042261700000000002 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614022+0000", 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2505e-05 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614134+0000", 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000112675 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206043 7.0 7:0c4ab3a4:::benchmark_data_smithi080_80848_object86544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.917482+0000", 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 59.787874403000004, 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6968406790000001, 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.737 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206043 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917482+0000", 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.738 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917482+0000", 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917482+0000", 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7299999999999996e-07 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917487+0000", 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5020000000000003e-06 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.739 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917495+0000", 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3049999999999993e-06 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917501+0000", 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1340000000000001e-06 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917687+0000", 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018563299999999999 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.740 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917694+0000", 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.708e-06 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395094+0000", 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.47740021100000002 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395102+0000", 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2160000000000006e-06 2024-03-26T05:06:35.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.742 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.742 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.742 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591668+0000", 2024-03-26T05:06:35.742 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1965658779999999 2024-03-26T05:06:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:35.742 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:35.743 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:35.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:35.744 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.744 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.744 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591783+0000", 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011553 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591918+0000", 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000134303 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597070+0000", 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051523330000000003 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.745 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613593+0000", 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016523013 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613637+0000", 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4057999999999997e-05 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614180+0000", 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00054313899999999999 2024-03-26T05:06:35.746 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614232+0000", 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2014e-05 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614322+0000", 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0129000000000004e-05 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.747 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206045 7.0 7:0d078525:::benchmark_data_smithi080_80848_object86546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.919727+0000", 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 59.785628668999998, 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.707372347, 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206045 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.748 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919727+0000", 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919727+0000", 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919729+0000", 2024-03-26T05:06:35.749 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9149999999999999e-06 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919737+0000", 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.926e-06 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919757+0000", 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0190000000000002e-05 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.750 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919767+0000", 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9899999999999992e-06 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919857+0000", 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9819000000000002e-05 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919869+0000", 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2241e-05 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.751 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395136+0000", 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.47526629399999998 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395144+0000", 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1950000000000003e-06 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.594264+0000", 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.199119764 2024-03-26T05:06:35.752 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.594373+0000", 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010950699999999999 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.594495+0000", 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000122044 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597075+0000", 2024-03-26T05:06:35.753 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002579951 2024-03-26T05:06:35.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.754 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.754 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.754 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613680+0000", 2024-03-26T05:06:35.754 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016604739 2024-03-26T05:06:35.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.754 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.754 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.754 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614358+0000", 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00067832500000000002 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.626740+0000", 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.012381614000000001 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.626798+0000", 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8226000000000001e-05 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.755 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.627100+0000", 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00030160700000000001 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206049 7.b 7:d42fcd17:::benchmark_data_smithi080_80848_object86550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.923737+0000", 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 59.781618678000001, 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.638929509, 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.756 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206049 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923737+0000", 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923737+0000", 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.757 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923739+0000", 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9750000000000001e-06 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923747+0000", 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0439999999999996e-06 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.758 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923781+0000", 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4029000000000002e-05 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923793+0000", 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1834999999999999e-05 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923882+0000", 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8941000000000001e-05 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.759 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923895+0000", 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2989e-05 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392396+0000", 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.46850114399999998 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392404+0000", 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9589999999999995e-06 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.760 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.554545+0000", 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.162140891 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.554683+0000", 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013746099999999999 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,5,6", 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.555704+0000", 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001020927 2024-03-26T05:06:35.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558652+0000", 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0029487879999999999 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558667+0000", 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4798e-05 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558710+0000", 2024-03-26T05:06:35.762 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2973999999999999e-05 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562312+0000", 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036020919999999999 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562556+0000", 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000244057 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.763 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562667+0000", 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000110605 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206053 7.0 7:08b24a04:::benchmark_data_smithi080_80848_object86554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.926223+0000", 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 59.779132664999999, 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7011446530000001, 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.764 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206053 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926223+0000", 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926223+0000", 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.765 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926225+0000", 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5409999999999999e-06 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926230+0000", 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5219999999999998e-06 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926246+0000", 2024-03-26T05:06:35.766 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5339999999999999e-05 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926255+0000", 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7110000000000001e-06 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926346+0000", 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1246e-05 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.767 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926354+0000", 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0260000000000007e-06 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395168+0000", 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.46881425399999999 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395176+0000", 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0439999999999996e-06 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.768 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597154+0000", 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2019782939999999 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597287+0000", 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000132848 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597420+0000", 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000132917 2024-03-26T05:06:35.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.770 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.770 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.770 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614394+0000", 2024-03-26T05:06:35.770 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016974182000000001 2024-03-26T05:06:35.770 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.770 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.770 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.770 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614560+0000", 2024-03-26T05:06:35.770 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016532100000000001 2024-03-26T05:06:35.770 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.770 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.770 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.771 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.619983+0000", 2024-03-26T05:06:35.771 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0054232200000000003 2024-03-26T05:06:35.771 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.771 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.771 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.771 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.627148+0000", 2024-03-26T05:06:35.771 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0071655720000000003 2024-03-26T05:06:35.771 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.771 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.771 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.771 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.627250+0000", 2024-03-26T05:06:35.771 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010138599999999999 2024-03-26T05:06:35.772 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.772 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.772 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.772 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.627368+0000", 2024-03-26T05:06:35.772 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118229 2024-03-26T05:06:35.772 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.772 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.772 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.772 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.772 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.772 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236543 7.0 7:02068409:::benchmark_data_smithi080_80848_object117044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:06:35.772 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.452286+0000", 2024-03-26T05:06:35.773 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 24.253069568000001, 2024-03-26T05:06:35.773 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.001309335, 2024-03-26T05:06:35.773 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.773 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.773 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.773 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.773 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.773 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236543 2024-03-26T05:06:35.773 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.773 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.773 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.773 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452286+0000", 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452286+0000", 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452288+0000", 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4249999999999999e-06 2024-03-26T05:06:35.774 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.775 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.775 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.775 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452293+0000", 2024-03-26T05:06:35.775 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8969999999999999e-06 2024-03-26T05:06:35.775 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.775 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.775 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.775 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452305+0000", 2024-03-26T05:06:35.775 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2170999999999999e-05 2024-03-26T05:06:35.775 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.775 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.775 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452313+0000", 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6329999999999996e-06 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452604+0000", 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000290943 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452613+0000", 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0860000000000008e-06 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.776 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.447196+0000", 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99458339399999995 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.447355+0000", 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015935600000000001 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.447444+0000", 2024-03-26T05:06:35.777 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8975999999999996e-05 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.449713+0000", 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022690480000000001 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.449794+0000", 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1021999999999998e-05 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.778 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453503+0000", 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0037082650000000001 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453557+0000", 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4512000000000003e-05 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453596+0000", 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8606999999999998e-05 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.779 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236550 7.0 7:0e84af01:::benchmark_data_smithi080_80848_object117051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.460859+0000", 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 24.244497422999999, 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99296332399999998, 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236550 2024-03-26T05:06:35.780 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460859+0000", 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460859+0000", 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.781 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460860+0000", 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1090000000000001e-06 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460865+0000", 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5389999999999998e-06 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460881+0000", 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6152999999999999e-05 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.782 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460889+0000", 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7100000000000007e-06 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460950+0000", 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1356e-05 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460959+0000", 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6209999999999993e-06 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.783 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.449848+0000", 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98888869099999999 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.449965+0000", 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117137 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.450042+0000", 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7497999999999996e-05 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.784 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453686+0000", 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036440470000000001 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453744+0000", 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7787000000000001e-05 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453769+0000", 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4307e-05 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453799+0000", 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.0151999999999999e-05 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453822+0000", 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3217000000000001e-05 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.786 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236558 7.0 7:0d5d8d1f:::benchmark_data_smithi080_80848_object117059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.474483+0000", 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 24.230872708, 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97948798000000004, 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236558 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.787 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474483+0000", 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474483+0000", 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474485+0000", 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.412e-06 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.788 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474488+0000", 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6160000000000002e-06 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474497+0000", 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9619999999999999e-06 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474504+0000", 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4899999999999997e-06 2024-03-26T05:06:35.789 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474602+0000", 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8004999999999994e-05 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474611+0000", 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6840000000000002e-06 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.790 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.451610+0000", 2024-03-26T05:06:35.791 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97699924800000004 2024-03-26T05:06:35.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.791 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.791 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.451703+0000", 2024-03-26T05:06:35.791 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2864999999999999e-05 2024-03-26T05:06:35.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.791 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:06:35.791 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.451781+0000", 2024-03-26T05:06:35.791 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8761e-05 2024-03-26T05:06:35.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453735+0000", 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0019537589999999998 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453749+0000", 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4021e-05 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.792 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453879+0000", 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000129605 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453931+0000", 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1987999999999998e-05 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453971+0000", 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0564e-05 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.793 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236560 7.0 7:0afff4a7:::benchmark_data_smithi080_80848_object117061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.474521+0000", 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 24.23083497, 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98418719899999996, 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236560 2024-03-26T05:06:35.794 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474521+0000", 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474521+0000", 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.795 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474521+0000", 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1300000000000001e-07 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474524+0000", 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.289e-06 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474530+0000", 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2160000000000003e-06 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.796 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474535+0000", 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.011e-06 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474632+0000", 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6506000000000001e-05 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474637+0000", 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3539999999999999e-06 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.797 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.454019+0000", 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97938261800000004 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.454151+0000", 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013146800000000001 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.454275+0000", 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000123635 2024-03-26T05:06:35.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458530+0000", 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0042555229999999998 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458574+0000", 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4110999999999998e-05 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458589+0000", 2024-03-26T05:06:35.799 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.505e-05 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458635+0000", 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5636000000000001e-05 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458708+0000", 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3368999999999995e-05 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.800 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236566 7.0 7:0070f26d:::benchmark_data_smithi080_80848_object117067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.480345+0000", 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 24.225011229, 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98387816699999997, 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236566 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480345+0000", 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480345+0000", 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480347+0000", 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6750000000000001e-06 2024-03-26T05:06:35.802 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480351+0000", 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9309999999999999e-06 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480367+0000", 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5599000000000001e-05 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480376+0000", 2024-03-26T05:06:35.803 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7949999999999996e-06 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480438+0000", 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2456e-05 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480448+0000", 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4320000000000005e-06 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.804 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.456042+0000", 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97559416799999998 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.456124+0000", 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2108999999999996e-05 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.456202+0000", 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8367000000000004e-05 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.805 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458755+0000", 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0025522359999999998 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458765+0000", 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0862e-05 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464122+0000", 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0053569200000000003 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.806 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464175+0000", 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2556999999999998e-05 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464223+0000", 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8059999999999997e-05 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.807 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236568 7.0 7:07c96239:::benchmark_data_smithi080_80848_object117069:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.480947+0000", 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 24.224408660999998, 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98352315300000004, 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236568 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480947+0000", 2024-03-26T05:06:35.808 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480947+0000", 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480948+0000", 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0470000000000001e-06 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.809 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480953+0000", 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7589999999999997e-06 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480969+0000", 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5812e-05 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480977+0000", 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7659999999999998e-06 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.810 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.481028+0000", 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0794999999999997e-05 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.481035+0000", 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3629999999999996e-06 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458791+0000", 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97775615100000002 2024-03-26T05:06:35.811 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458955+0000", 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000164411 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.459033+0000", 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7306999999999997e-05 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464267+0000", 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0052343420000000003 2024-03-26T05:06:35.812 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464329+0000", 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1607999999999997e-05 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464386+0000", 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7380000000000003e-05 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464431+0000", 2024-03-26T05:06:35.813 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5321999999999997e-05 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464471+0000", 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.909e-05 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236570 7.0 7:03c0468e:::benchmark_data_smithi080_80848_object117071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.482326+0000", 2024-03-26T05:06:35.814 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 24.223030336000001, 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98740077800000003, 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236570 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482326+0000", 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.815 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482326+0000", 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482327+0000", 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.522e-06 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482332+0000", 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8600000000000001e-06 2024-03-26T05:06:35.816 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482347+0000", 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4577e-05 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482354+0000", 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7160000000000003e-06 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482406+0000", 2024-03-26T05:06:35.817 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1476e-05 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482415+0000", 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2310000000000002e-06 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.461304+0000", 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97888937600000003 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.818 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.461434+0000", 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012907000000000001 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.461601+0000", 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000167589 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464375+0000", 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027743030000000001 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.819 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464479+0000", 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103106 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469623+0000", 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051441489999999998 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469676+0000", 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3794e-05 2024-03-26T05:06:35.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469726+0000", 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0009000000000002e-05 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236639 7.0 7:0b36f35f:::benchmark_data_smithi080_80848_object117140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.545081+0000", 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 24.160274880999999, 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93260101100000004, 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.821 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236639 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.545081+0000", 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.822 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.545081+0000", 2024-03-26T05:06:35.823 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.823 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.823 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.823 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.823 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.545083+0000", 2024-03-26T05:06:35.823 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.491e-06 2024-03-26T05:06:35.823 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.823 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.823 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.823 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.545089+0000", 2024-03-26T05:06:35.823 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5830000000000003e-06 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.545104+0000", 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4684e-05 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.545113+0000", 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6149999999999997e-06 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.824 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.545183+0000", 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0933999999999995e-05 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.545193+0000", 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4590000000000004e-06 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464505+0000", 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.919312197 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.825 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464607+0000", 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010204600000000001 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464714+0000", 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000106399 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469739+0000", 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0050251200000000001 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.826 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469779+0000", 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0339000000000001e-05 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.477579+0000", 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0078001930000000004 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.477632+0000", 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2648999999999997e-05 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.827 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.828 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.828 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.477682+0000", 2024-03-26T05:06:35.828 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0302000000000001e-05 2024-03-26T05:06:35.828 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.828 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236649 7.0 7:05354b19:::benchmark_data_smithi080_80848_object117150:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.552318+0000", 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 24.153037638000001, 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93121235999999996, 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.829 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236649 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552318+0000", 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552318+0000", 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.830 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552320+0000", 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6249999999999999e-06 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552326+0000", 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.195e-06 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552339+0000", 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2292e-05 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.831 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552346+0000", 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7999999999999999e-06 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552428+0000", 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2100000000000003e-05 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552438+0000", 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9960000000000005e-06 2024-03-26T05:06:35.832 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.466946+0000", 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.91450766100000003 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.467070+0000", 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000124213 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.467201+0000", 2024-03-26T05:06:35.833 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013037199999999999 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469791+0000", 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0025900710000000002 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.477730+0000", 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0079393780000000004 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.834 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.483433+0000", 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0057026200000000003 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.483485+0000", 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2580999999999997e-05 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.483531+0000", 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5456e-05 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236652 7.0 7:05f9fd31:::benchmark_data_smithi080_80848_object117153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.552915+0000", 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 24.152441347, 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93075311699999996, 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236652 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.836 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552915+0000", 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552915+0000", 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.837 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552916+0000", 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2099999999999995e-07 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552920+0000", 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.087e-06 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552933+0000", 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3433e-05 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.838 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.552940+0000", 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7340000000000002e-06 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.553327+0000", 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000387153 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.553335+0000", 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5120000000000008e-06 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.839 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469848+0000", 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.91651284099999997 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469971+0000", 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000122311 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.470093+0000", 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012256499999999999 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.477537+0000", 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0074434380000000001 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.477774+0000", 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00023732500000000001 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.483586+0000", 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0058121539999999999 2024-03-26T05:06:35.841 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.483633+0000", 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7138999999999997e-05 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.483668+0000", 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4604000000000003e-05 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.842 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236654 7.0 7:028e4924:::benchmark_data_smithi080_80848_object117155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.555854+0000", 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 24.149502140999999, 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93320793999999996, 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236654 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.843 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555854+0000", 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555854+0000", 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555856+0000", 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6899999999999999e-06 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.844 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555861+0000", 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3449999999999996e-06 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555877+0000", 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5977e-05 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555885+0000", 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9959999999999993e-06 2024-03-26T05:06:35.845 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555943+0000", 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8131999999999998e-05 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555951+0000", 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9440000000000005e-06 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.472491+0000", 2024-03-26T05:06:35.846 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.91653987999999997 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.472604+0000", 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000113354 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.472720+0000", 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115812 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.483540+0000", 2024-03-26T05:06:35.847 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010820412 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.483703+0000", 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000162645 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.488951+0000", 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0052477599999999998 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489011+0000", 2024-03-26T05:06:35.848 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9954000000000002e-05 2024-03-26T05:06:35.849 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:06:35.849 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:06:35.849 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:06:35.849 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489062+0000", 2024-03-26T05:06:35.849 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1038999999999997e-05 2024-03-26T05:06:35.849 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.849 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.849 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.849 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:06:35.849 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:06:35.849 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:06:35.850 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:35.877 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:35.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:35.907 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.3 to 1 (10000) 2024-03-26T05:06:35.907 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:06:35.979 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:36.007 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:36.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:36.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:35 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:36.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:35 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2770625928' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:06:36.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:35 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:06:36.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:35 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3240038408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:06:36.035 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:35 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:06:36.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:35 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:36.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:35 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2770625928' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:06:36.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:35 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:06:36.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:35 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3240038408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:06:36.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:35 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:06:36.109 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:36.137 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:36.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:36.238 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:36.263 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:36.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:36.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:35 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:36.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:35 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2770625928' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:06:36.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:35 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:06:36.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:35 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3240038408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:06:36.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:35 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:06:36.365 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:36.390 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:36.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:36.491 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:36.516 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:36.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:36.617 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:36.642 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:36.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:36.743 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:36.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:36.767 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:36.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:36.868 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:36.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:36.899 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:36.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:37.000 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:37.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:37.028 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:37.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:37.052 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:36 smithi080 ceph-mon[40404]: pgmap v263: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:06:37.053 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:36 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:06:37.053 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:36 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:06:37.053 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:36 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:06:37.053 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:36 smithi080 ceph-mon[40404]: osdmap e183: 8 total, 8 up, 7 in 2024-03-26T05:06:37.053 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:36 smithi080 ceph-mon[29691]: pgmap v263: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:06:37.053 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:36 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:06:37.053 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:36 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:06:37.053 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:36 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:06:37.053 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:36 smithi080 ceph-mon[29691]: osdmap e183: 8 total, 8 up, 7 in 2024-03-26T05:06:37.129 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:37.156 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:37.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:37.258 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:37.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:37.282 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:37.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:36 smithi132 ceph-mon[33500]: pgmap v263: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:06:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:36 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:06:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:36 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:06:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:36 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:06:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:36 smithi132 ceph-mon[33500]: osdmap e183: 8 total, 8 up, 7 in 2024-03-26T05:06:37.383 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:37.407 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:37.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:37.454 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T05:06:37.508 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:37.569 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:37.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:37.670 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:37.849 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:37.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:37.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:37.951 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:37.985 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:37.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:38.077 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:37 smithi080 ceph-mon[40404]: 6.1b scrub starts 2024-03-26T05:06:38.077 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:37 smithi080 ceph-mon[40404]: 6.1b scrub ok 2024-03-26T05:06:38.077 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:37 smithi080 ceph-mon[29691]: 6.1b scrub starts 2024-03-26T05:06:38.077 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:37 smithi080 ceph-mon[29691]: 6.1b scrub ok 2024-03-26T05:06:38.089 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:38.117 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:38.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:38.219 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:38.243 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:38.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:38.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:37 smithi132 ceph-mon[33500]: 6.1b scrub starts 2024-03-26T05:06:38.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:37 smithi132 ceph-mon[33500]: 6.1b scrub ok 2024-03-26T05:06:38.345 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:38.375 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:38.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:38.476 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:38.501 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:38.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:38.602 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:38.630 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:38.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:38.732 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:38.761 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:38.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:38.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:38.863 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:38.890 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:38.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:38.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:38.992 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:39.021 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:39.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:39.122 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:39.147 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:39.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:39.248 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:39.271 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:39.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:39.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:38 smithi132 ceph-mon[33500]: pgmap v265: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 737 B/s rd, 0 op/s 2024-03-26T05:06:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:38 smithi132 ceph-mon[33500]: 6.b scrub starts 2024-03-26T05:06:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:38 smithi132 ceph-mon[33500]: 6.b scrub ok 2024-03-26T05:06:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:38 smithi132 ceph-mon[33500]: 4.7 scrub starts 2024-03-26T05:06:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:38 smithi132 ceph-mon[33500]: 4.7 scrub ok 2024-03-26T05:06:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:38 smithi132 ceph-mon[33500]: 3.1d scrub starts 2024-03-26T05:06:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:38 smithi132 ceph-mon[33500]: 3.1d scrub ok 2024-03-26T05:06:39.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[40404]: pgmap v265: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 737 B/s rd, 0 op/s 2024-03-26T05:06:39.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[40404]: 6.b scrub starts 2024-03-26T05:06:39.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[40404]: 6.b scrub ok 2024-03-26T05:06:39.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[40404]: 4.7 scrub starts 2024-03-26T05:06:39.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[40404]: 4.7 scrub ok 2024-03-26T05:06:39.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[40404]: 3.1d scrub starts 2024-03-26T05:06:39.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[40404]: 3.1d scrub ok 2024-03-26T05:06:39.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[29691]: pgmap v265: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 737 B/s rd, 0 op/s 2024-03-26T05:06:39.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[29691]: 6.b scrub starts 2024-03-26T05:06:39.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[29691]: 6.b scrub ok 2024-03-26T05:06:39.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[29691]: 4.7 scrub starts 2024-03-26T05:06:39.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[29691]: 4.7 scrub ok 2024-03-26T05:06:39.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[29691]: 3.1d scrub starts 2024-03-26T05:06:39.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:38 smithi080 ceph-mon[29691]: 3.1d scrub ok 2024-03-26T05:06:39.373 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:39.396 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:39.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:39.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:39.498 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:39.523 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:39.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:39.564 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T05:06:39.624 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:39.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:39.654 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:39.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:39.755 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:39.781 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:39.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:39.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:39.883 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:39.910 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:39.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:40.012 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:40.037 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:40.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:40.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:40.139 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:40.162 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:40.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:40.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:40.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:39 smithi080 ceph-mon[40404]: 3.1 scrub starts 2024-03-26T05:06:40.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:39 smithi080 ceph-mon[40404]: 3.1 scrub ok 2024-03-26T05:06:40.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:39 smithi080 ceph-mon[40404]: 5.14 scrub starts 2024-03-26T05:06:40.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:39 smithi080 ceph-mon[40404]: 5.14 scrub ok 2024-03-26T05:06:40.168 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:39 smithi080 ceph-mon[29691]: 3.1 scrub starts 2024-03-26T05:06:40.168 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:39 smithi080 ceph-mon[29691]: 3.1 scrub ok 2024-03-26T05:06:40.168 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:39 smithi080 ceph-mon[29691]: 5.14 scrub starts 2024-03-26T05:06:40.168 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:39 smithi080 ceph-mon[29691]: 5.14 scrub ok 2024-03-26T05:06:40.264 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:40.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:40.288 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:40.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:39 smithi132 ceph-mon[33500]: 3.1 scrub starts 2024-03-26T05:06:40.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:39 smithi132 ceph-mon[33500]: 3.1 scrub ok 2024-03-26T05:06:40.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:39 smithi132 ceph-mon[33500]: 5.14 scrub starts 2024-03-26T05:06:40.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:39 smithi132 ceph-mon[33500]: 5.14 scrub ok 2024-03-26T05:06:40.389 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:40.418 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:40.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:40.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:40.520 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:40.545 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:40.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:40.646 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:40.669 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:40.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:40.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:40.768 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:06:40.771 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:40.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:40.838 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:40.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:40.938 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'unique_pool_1' created 2024-03-26T05:06:40.939 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:40.972 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:40.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:40.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:41.074 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:41.102 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:41.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:41.204 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:41.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:41.233 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:41.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:41.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:40 smithi132 ceph-mon[33500]: pgmap v266: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:06:41.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:40 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1289984785' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T05:06:41.334 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:41.363 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:41.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:41.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:41.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:40 smithi080 ceph-mon[29691]: pgmap v266: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:06:41.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:40 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1289984785' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T05:06:41.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:40 smithi080 ceph-mon[40404]: pgmap v266: 132 pgs: 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:06:41.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:40 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1289984785' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T05:06:41.465 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:41.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:41.490 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:41.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:41.591 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:41.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:41.616 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:41.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:41.717 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:41.724 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-26T05:06:41.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:41.742 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:41.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:41.843 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:41.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:41.870 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:41.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:41.947 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:41 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1289984785' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-26T05:06:41.947 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:41 smithi080 ceph-mon[29691]: osdmap e184: 8 total, 8 up, 7 in 2024-03-26T05:06:41.947 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:41 smithi080 ceph-mon[29691]: 2.1 scrub starts 2024-03-26T05:06:41.947 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:41 smithi080 ceph-mon[29691]: 2.1 scrub ok 2024-03-26T05:06:41.968 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:41.972 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:41.989 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:06:41.998 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:41.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:42.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:42.010 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:42.011 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:06:42.035 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:06:42.055 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:06:42.075 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:06:42.095 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:06:42.100 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:42.115 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:06:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:42.125 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:42.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:42.135 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:06:42.136 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:06:42.136 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:06:42.137 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:06:42.137 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:06:42.137 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:06:42.137 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:06:42.137 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:06:42.137 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:06:42.226 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:42.250 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:41 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1289984785' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-26T05:06:42.254 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:41 smithi080 ceph-mon[40404]: osdmap e184: 8 total, 8 up, 7 in 2024-03-26T05:06:42.254 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:41 smithi080 ceph-mon[40404]: 2.1 scrub starts 2024-03-26T05:06:42.254 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:41 smithi080 ceph-mon[40404]: 2.1 scrub ok 2024-03-26T05:06:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:42.255 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:42.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:41 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1289984785' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-26T05:06:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:41 smithi132 ceph-mon[33500]: osdmap e184: 8 total, 8 up, 7 in 2024-03-26T05:06:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:41 smithi132 ceph-mon[33500]: 2.1 scrub starts 2024-03-26T05:06:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:41 smithi132 ceph-mon[33500]: 2.1 scrub ok 2024-03-26T05:06:42.357 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:42.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:42.382 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:42.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:42.483 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:42.511 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:42.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:42.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:42.614 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:42.639 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:42.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:42.741 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:42.766 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:42.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:42.867 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:42.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:42.894 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:42.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:42.996 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:43.030 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:43.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:43.131 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:43.155 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:43.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:43.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[29691]: pgmap v268: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:06:43.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[29691]: osdmap e185: 8 total, 8 up, 7 in 2024-03-26T05:06:43.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2859603959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:43.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[29691]: 4.17 scrub starts 2024-03-26T05:06:43.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[29691]: 4.17 scrub ok 2024-03-26T05:06:43.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[29691]: 5.12 scrub starts 2024-03-26T05:06:43.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[29691]: 5.12 scrub ok 2024-03-26T05:06:43.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[29691]: 4.1 scrub starts 2024-03-26T05:06:43.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[29691]: 4.1 scrub ok 2024-03-26T05:06:43.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[29691]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:06:43.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[29691]: 5.11 scrub starts 2024-03-26T05:06:43.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[29691]: 5.11 scrub ok 2024-03-26T05:06:43.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[40404]: pgmap v268: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:06:43.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[40404]: osdmap e185: 8 total, 8 up, 7 in 2024-03-26T05:06:43.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2859603959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:43.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[40404]: 4.17 scrub starts 2024-03-26T05:06:43.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[40404]: 4.17 scrub ok 2024-03-26T05:06:43.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[40404]: 5.12 scrub starts 2024-03-26T05:06:43.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[40404]: 5.12 scrub ok 2024-03-26T05:06:43.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[40404]: 4.1 scrub starts 2024-03-26T05:06:43.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[40404]: 4.1 scrub ok 2024-03-26T05:06:43.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[40404]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:06:43.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[40404]: 5.11 scrub starts 2024-03-26T05:06:43.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:42 smithi080 ceph-mon[40404]: 5.11 scrub ok 2024-03-26T05:06:43.256 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:43.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:43.282 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:43.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:43.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:42 smithi132 ceph-mon[33500]: pgmap v268: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:06:43.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:42 smithi132 ceph-mon[33500]: osdmap e185: 8 total, 8 up, 7 in 2024-03-26T05:06:43.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:42 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2859603959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:43.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:42 smithi132 ceph-mon[33500]: 4.17 scrub starts 2024-03-26T05:06:43.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:42 smithi132 ceph-mon[33500]: 4.17 scrub ok 2024-03-26T05:06:43.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:42 smithi132 ceph-mon[33500]: 5.12 scrub starts 2024-03-26T05:06:43.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:42 smithi132 ceph-mon[33500]: 5.12 scrub ok 2024-03-26T05:06:43.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:42 smithi132 ceph-mon[33500]: 4.1 scrub starts 2024-03-26T05:06:43.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:42 smithi132 ceph-mon[33500]: 4.1 scrub ok 2024-03-26T05:06:43.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:42 smithi132 ceph-mon[33500]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:06:43.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:42 smithi132 ceph-mon[33500]: 5.11 scrub starts 2024-03-26T05:06:43.295 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:42 smithi132 ceph-mon[33500]: 5.11 scrub ok 2024-03-26T05:06:43.383 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:43.411 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:43.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:43.513 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:43.540 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:43.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:43.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:43.642 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:43.668 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:43.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:43.770 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:43.796 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:43.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:43.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:43.898 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:43.922 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:43.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:43.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:44.024 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:44.051 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:44.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:44.117 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:06:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:06:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:06:44.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:43 smithi080 ceph-mon[40404]: osdmap e186: 8 total, 8 up, 7 in 2024-03-26T05:06:44.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:43 smithi080 ceph-mon[40404]: 6.3 scrub starts 2024-03-26T05:06:44.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:43 smithi080 ceph-mon[40404]: 6.3 scrub ok 2024-03-26T05:06:44.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:43 smithi080 ceph-mon[29691]: osdmap e186: 8 total, 8 up, 7 in 2024-03-26T05:06:44.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:43 smithi080 ceph-mon[29691]: 6.3 scrub starts 2024-03-26T05:06:44.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:43 smithi080 ceph-mon[29691]: 6.3 scrub ok 2024-03-26T05:06:44.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:44.180 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:44.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:44.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:44.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:44.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:43 smithi132 ceph-mon[33500]: osdmap e186: 8 total, 8 up, 7 in 2024-03-26T05:06:44.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:43 smithi132 ceph-mon[33500]: 6.3 scrub starts 2024-03-26T05:06:44.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:43 smithi132 ceph-mon[33500]: 6.3 scrub ok 2024-03-26T05:06:44.307 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:44.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:44.409 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:44.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:44.437 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:44.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:44.538 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:44.566 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:44.566 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:44.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:44.567 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:44.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:44.668 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:44.696 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:44.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:44.798 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:44.823 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:44.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:44.864 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:06:44.926 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:44.950 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:44.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:45.051 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:45.077 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:45.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:45.179 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:45.203 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:45.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:45.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:45.259 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:44 smithi080 ceph-mon[29691]: pgmap v271: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:45.260 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:44 smithi080 ceph-mon[29691]: 5.7 scrub starts 2024-03-26T05:06:45.260 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:44 smithi080 ceph-mon[29691]: 5.7 scrub ok 2024-03-26T05:06:45.260 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:44 smithi080 ceph-mon[40404]: pgmap v271: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:45.260 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:44 smithi080 ceph-mon[40404]: 5.7 scrub starts 2024-03-26T05:06:45.260 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:44 smithi080 ceph-mon[40404]: 5.7 scrub ok 2024-03-26T05:06:45.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:44 smithi132 ceph-mon[33500]: pgmap v271: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 872 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:45.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:44 smithi132 ceph-mon[33500]: 5.7 scrub starts 2024-03-26T05:06:45.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:44 smithi132 ceph-mon[33500]: 5.7 scrub ok 2024-03-26T05:06:45.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:06:45.306 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:45.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:45.336 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:45.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:45.437 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:45.465 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:45.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:45.568 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:45.595 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:45.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:45.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:45.697 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:45.725 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:45.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:45.827 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:45.852 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:45.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:45.954 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:45.969 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:06:45.979 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:45.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:45.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:46.082 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:46.107 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:46.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:46.208 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:46.237 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:46.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:46.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:46.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:46.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: 3.6 scrub starts 2024-03-26T05:06:46.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: 3.6 scrub ok 2024-03-26T05:06:46.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: 4.10 scrub starts 2024-03-26T05:06:46.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: 4.1c scrub starts 2024-03-26T05:06:46.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: 4.10 scrub ok 2024-03-26T05:06:46.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: 4.1c scrub ok 2024-03-26T05:06:46.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: 4.c scrub starts 2024-03-26T05:06:46.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: 4.c scrub ok 2024-03-26T05:06:46.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: 4.a scrub starts 2024-03-26T05:06:46.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: 4.a scrub ok 2024-03-26T05:06:46.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1303220940' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:06:46.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: 1.0 scrub starts 2024-03-26T05:06:46.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:45 smithi132 ceph-mon[33500]: 1.0 scrub ok 2024-03-26T05:06:46.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:46.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: 3.6 scrub starts 2024-03-26T05:06:46.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: 3.6 scrub ok 2024-03-26T05:06:46.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: 4.10 scrub starts 2024-03-26T05:06:46.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: 4.1c scrub starts 2024-03-26T05:06:46.302 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: 4.10 scrub ok 2024-03-26T05:06:46.302 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: 4.1c scrub ok 2024-03-26T05:06:46.302 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: 4.c scrub starts 2024-03-26T05:06:46.302 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: 4.c scrub ok 2024-03-26T05:06:46.302 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: 4.a scrub starts 2024-03-26T05:06:46.302 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: 4.a scrub ok 2024-03-26T05:06:46.302 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1303220940' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:06:46.303 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: 1.0 scrub starts 2024-03-26T05:06:46.303 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[40404]: 1.0 scrub ok 2024-03-26T05:06:46.303 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:46.303 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: 3.6 scrub starts 2024-03-26T05:06:46.303 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: 3.6 scrub ok 2024-03-26T05:06:46.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: 4.10 scrub starts 2024-03-26T05:06:46.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: 4.1c scrub starts 2024-03-26T05:06:46.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: 4.10 scrub ok 2024-03-26T05:06:46.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: 4.1c scrub ok 2024-03-26T05:06:46.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: 4.c scrub starts 2024-03-26T05:06:46.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: 4.c scrub ok 2024-03-26T05:06:46.305 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: 4.a scrub starts 2024-03-26T05:06:46.305 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: 4.a scrub ok 2024-03-26T05:06:46.305 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1303220940' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:06:46.305 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: 1.0 scrub starts 2024-03-26T05:06:46.305 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:45 smithi080 ceph-mon[29691]: 1.0 scrub ok 2024-03-26T05:06:46.339 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:46.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:46.366 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:46.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:46.468 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:46.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:46.494 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:46.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:46.595 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:46.620 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:46.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:46.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:46.722 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:46.748 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:46.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:46.849 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:46.873 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:46.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:46.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:46.975 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:47.001 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:47.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:47.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:47.094 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:46 smithi080 ceph-mon[40404]: pgmap v272: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:47.095 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:46 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:06:47.095 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:46 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1303220940' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:06:47.095 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:46 smithi080 ceph-mon[40404]: osdmap e187: 8 total, 8 up, 7 in 2024-03-26T05:06:47.095 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:46 smithi080 ceph-mon[29691]: pgmap v272: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:47.095 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:46 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:06:47.095 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:46 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1303220940' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:06:47.095 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:46 smithi080 ceph-mon[29691]: osdmap e187: 8 total, 8 up, 7 in 2024-03-26T05:06:47.103 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:47.131 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:47.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:47.232 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:47.259 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:47.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:47.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:46 smithi132 ceph-mon[33500]: pgmap v272: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:47.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:46 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:06:47.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:46 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1303220940' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:06:47.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:46 smithi132 ceph-mon[33500]: osdmap e187: 8 total, 8 up, 7 in 2024-03-26T05:06:47.361 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:47.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:47.388 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:47.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:47.489 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:47.514 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:47.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:47.615 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:47.642 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:47.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:47.743 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:47.768 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:47.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:47.869 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:47.892 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:47.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:47.994 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:48.026 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:48.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:48.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:48.128 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:48.156 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:48.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:48.257 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:48.285 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:48.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:48.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:47 smithi132 ceph-mon[33500]: 5.15 scrub starts 2024-03-26T05:06:48.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:47 smithi132 ceph-mon[33500]: 5.15 scrub ok 2024-03-26T05:06:48.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:47 smithi132 ceph-mon[33500]: 4.2 scrub starts 2024-03-26T05:06:48.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:47 smithi132 ceph-mon[33500]: 4.2 scrub ok 2024-03-26T05:06:48.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:47 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:06:48.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:47 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:06:48.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:47 smithi080 ceph-mon[29691]: 5.15 scrub starts 2024-03-26T05:06:48.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:47 smithi080 ceph-mon[29691]: 5.15 scrub ok 2024-03-26T05:06:48.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:47 smithi080 ceph-mon[29691]: 4.2 scrub starts 2024-03-26T05:06:48.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:47 smithi080 ceph-mon[29691]: 4.2 scrub ok 2024-03-26T05:06:48.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:47 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:06:48.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:47 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:06:48.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:47 smithi080 ceph-mon[40404]: 5.15 scrub starts 2024-03-26T05:06:48.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:47 smithi080 ceph-mon[40404]: 5.15 scrub ok 2024-03-26T05:06:48.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:47 smithi080 ceph-mon[40404]: 4.2 scrub starts 2024-03-26T05:06:48.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:47 smithi080 ceph-mon[40404]: 4.2 scrub ok 2024-03-26T05:06:48.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:47 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:06:48.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:47 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:06:48.387 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:48.412 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:48.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:48.513 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:48.537 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:48.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:48.638 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:48.662 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:48.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:48.764 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:48.792 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:48.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:48.894 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:48.918 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:48.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:48.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:49.019 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:49.044 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:49.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:49.121 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:06:49.145 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:49.174 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:49.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:49.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:49.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[29691]: pgmap v274: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:49.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[29691]: 3.8 scrub starts 2024-03-26T05:06:49.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[29691]: 3.8 scrub ok 2024-03-26T05:06:49.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[29691]: 5.1c scrub starts 2024-03-26T05:06:49.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[29691]: 5.1c scrub ok 2024-03-26T05:06:49.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[29691]: 6.a scrub starts 2024-03-26T05:06:49.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[29691]: 6.a scrub ok 2024-03-26T05:06:49.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[40404]: pgmap v274: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:49.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[40404]: 3.8 scrub starts 2024-03-26T05:06:49.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[40404]: 3.8 scrub ok 2024-03-26T05:06:49.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[40404]: 5.1c scrub starts 2024-03-26T05:06:49.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[40404]: 5.1c scrub ok 2024-03-26T05:06:49.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[40404]: 6.a scrub starts 2024-03-26T05:06:49.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:48 smithi080 ceph-mon[40404]: 6.a scrub ok 2024-03-26T05:06:49.277 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:49.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:48 smithi132 ceph-mon[33500]: pgmap v274: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:49.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:48 smithi132 ceph-mon[33500]: 3.8 scrub starts 2024-03-26T05:06:49.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:48 smithi132 ceph-mon[33500]: 3.8 scrub ok 2024-03-26T05:06:49.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:48 smithi132 ceph-mon[33500]: 5.1c scrub starts 2024-03-26T05:06:49.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:48 smithi132 ceph-mon[33500]: 5.1c scrub ok 2024-03-26T05:06:49.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:48 smithi132 ceph-mon[33500]: 6.a scrub starts 2024-03-26T05:06:49.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:48 smithi132 ceph-mon[33500]: 6.a scrub ok 2024-03-26T05:06:49.301 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:49.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:49.403 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:49.431 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:49.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:49.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:49.533 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:49.557 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:49.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:49.658 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:49.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:49.683 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:49.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:49.784 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:49.808 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:49.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:49.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:49.938 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:49.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:49.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:49.984 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T05:06:50.040 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:50.065 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:50.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:50.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: 4.6 deep-scrub starts 2024-03-26T05:06:50.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: 4.6 deep-scrub ok 2024-03-26T05:06:50.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2747677339' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:06:50.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:06:50.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: 3.0 scrub starts 2024-03-26T05:06:50.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: 3.0 scrub ok 2024-03-26T05:06:50.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: 6.13 deep-scrub starts 2024-03-26T05:06:50.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: 6.13 deep-scrub ok 2024-03-26T05:06:50.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:50.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:06:50.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:06:50.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:06:50.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: 4.15 scrub starts 2024-03-26T05:06:50.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[40404]: 4.15 scrub ok 2024-03-26T05:06:50.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: 4.6 deep-scrub starts 2024-03-26T05:06:50.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: 4.6 deep-scrub ok 2024-03-26T05:06:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2747677339' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:06:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:06:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: 3.0 scrub starts 2024-03-26T05:06:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: 3.0 scrub ok 2024-03-26T05:06:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: 6.13 deep-scrub starts 2024-03-26T05:06:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: 6.13 deep-scrub ok 2024-03-26T05:06:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:06:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:06:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:06:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: 4.15 scrub starts 2024-03-26T05:06:50.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:49 smithi080 ceph-mon[29691]: 4.15 scrub ok 2024-03-26T05:06:50.166 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:50.190 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:50.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:50.292 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:50.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: 4.6 deep-scrub starts 2024-03-26T05:06:50.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: 4.6 deep-scrub ok 2024-03-26T05:06:50.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2747677339' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:06:50.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:06:50.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: 3.0 scrub starts 2024-03-26T05:06:50.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: 3.0 scrub ok 2024-03-26T05:06:50.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: 6.13 deep-scrub starts 2024-03-26T05:06:50.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: 6.13 deep-scrub ok 2024-03-26T05:06:50.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:06:50.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:06:50.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:06:50.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:06:50.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: 4.15 scrub starts 2024-03-26T05:06:50.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:49 smithi132 ceph-mon[33500]: 4.15 scrub ok 2024-03-26T05:06:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:50.316 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:50.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:50.417 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:50.442 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:50.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:50.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:50.529 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:06:50.529 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:06:50.547 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:50.572 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:50.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:50.673 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:50.701 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:50.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:50.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:50.803 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:50.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:50.829 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:50.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:50.930 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:50.955 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:50.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:50.991 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-26T05:06:51.055 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:51.083 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:51.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:51.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[40404]: pgmap v275: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:06:51.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:06:51.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[40404]: osdmap e188: 8 total, 8 up, 7 in 2024-03-26T05:06:51.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1561008568' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:06:51.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:06:51.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[40404]: 4.9 deep-scrub starts 2024-03-26T05:06:51.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[40404]: 4.9 deep-scrub ok 2024-03-26T05:06:51.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[40404]: 3.10 deep-scrub starts 2024-03-26T05:06:51.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[40404]: 3.10 deep-scrub ok 2024-03-26T05:06:51.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[29691]: pgmap v275: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:06:51.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:06:51.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[29691]: osdmap e188: 8 total, 8 up, 7 in 2024-03-26T05:06:51.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1561008568' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:06:51.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:06:51.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[29691]: 4.9 deep-scrub starts 2024-03-26T05:06:51.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[29691]: 4.9 deep-scrub ok 2024-03-26T05:06:51.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[29691]: 3.10 deep-scrub starts 2024-03-26T05:06:51.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:50 smithi080 ceph-mon[29691]: 3.10 deep-scrub ok 2024-03-26T05:06:51.150 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 6 1 2024-03-26T05:06:51.186 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:51.215 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:51.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:50 smithi132 ceph-mon[33500]: pgmap v275: 148 pgs: 148 active+clean; 582 KiB data, 873 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:06:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:50 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:06:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:50 smithi132 ceph-mon[33500]: osdmap e188: 8 total, 8 up, 7 in 2024-03-26T05:06:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:50 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1561008568' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:06:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:50 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:06:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:50 smithi132 ceph-mon[33500]: 4.9 deep-scrub starts 2024-03-26T05:06:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:50 smithi132 ceph-mon[33500]: 4.9 deep-scrub ok 2024-03-26T05:06:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:50 smithi132 ceph-mon[33500]: 3.10 deep-scrub starts 2024-03-26T05:06:51.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:50 smithi132 ceph-mon[33500]: 3.10 deep-scrub ok 2024-03-26T05:06:51.316 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:51.344 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:51.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:51.446 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:51.471 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:51.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:51.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:51.573 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:51.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:51.597 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:51.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:51.698 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:51.728 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:51.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:51.829 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:51.854 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:51.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:51.957 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:51.981 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:51.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:52.082 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:52.110 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:52.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:52.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:51 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:06:52.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:51 smithi080 ceph-mon[29691]: osdmap e189: 8 total, 8 up, 7 in 2024-03-26T05:06:52.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:51 smithi080 ceph-mon[29691]: 5.a scrub starts 2024-03-26T05:06:52.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:51 smithi080 ceph-mon[29691]: 5.a scrub ok 2024-03-26T05:06:52.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:51 smithi080 ceph-mon[29691]: 6.1e scrub starts 2024-03-26T05:06:52.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:51 smithi080 ceph-mon[29691]: 6.1e scrub ok 2024-03-26T05:06:52.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:51 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:06:52.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:51 smithi080 ceph-mon[40404]: osdmap e189: 8 total, 8 up, 7 in 2024-03-26T05:06:52.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:51 smithi080 ceph-mon[40404]: 5.a scrub starts 2024-03-26T05:06:52.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:51 smithi080 ceph-mon[40404]: 5.a scrub ok 2024-03-26T05:06:52.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:51 smithi080 ceph-mon[40404]: 6.1e scrub starts 2024-03-26T05:06:52.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:51 smithi080 ceph-mon[40404]: 6.1e scrub ok 2024-03-26T05:06:52.212 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:52.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:52.238 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:52.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:52.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:51 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:06:52.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:51 smithi132 ceph-mon[33500]: osdmap e189: 8 total, 8 up, 7 in 2024-03-26T05:06:52.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:51 smithi132 ceph-mon[33500]: 5.a scrub starts 2024-03-26T05:06:52.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:51 smithi132 ceph-mon[33500]: 5.a scrub ok 2024-03-26T05:06:52.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:51 smithi132 ceph-mon[33500]: 6.1e scrub starts 2024-03-26T05:06:52.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:51 smithi132 ceph-mon[33500]: 6.1e scrub ok 2024-03-26T05:06:52.340 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:52.364 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:52.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:52.466 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:52.490 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:52.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:52.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:52.595 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:52.623 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:52.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:52.642 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:52.726 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:52.754 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:52.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:52.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:52.856 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:52.857 DEBUG:teuthology.orchestra.run.smithi080:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-26T05:06:52.860 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T05:06:52.860 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T05:06:52.883 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:52.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:52.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:52.907 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:hints = 1 2024-03-26T05:06:52.908 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T05:06:52.908 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Object prefix: benchmark_data_smithi080_109213 2024-03-26T05:06:52.908 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:06:52.908 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 0 0 0 0 0 0 - 0 2024-03-26T05:06:52.985 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:53.017 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:53.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:53.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:53.023 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:53.048 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T05:06:53.118 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:53.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:06:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:06:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[40404]: pgmap v278: 148 pgs: 148 active+clean; 582 KiB data, 877 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[40404]: 4.3 scrub starts 2024-03-26T05:06:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[40404]: 4.3 scrub ok 2024-03-26T05:06:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[40404]: 4.18 scrub starts 2024-03-26T05:06:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[40404]: 4.18 scrub ok 2024-03-26T05:06:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[40404]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:06:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[40404]: Cluster is now healthy 2024-03-26T05:06:53.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[40404]: osdmap e190: 8 total, 8 up, 7 in 2024-03-26T05:06:53.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:06:53.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:06:53.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[29691]: pgmap v278: 148 pgs: 148 active+clean; 582 KiB data, 877 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:53.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[29691]: 4.3 scrub starts 2024-03-26T05:06:53.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[29691]: 4.3 scrub ok 2024-03-26T05:06:53.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[29691]: 4.18 scrub starts 2024-03-26T05:06:53.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[29691]: 4.18 scrub ok 2024-03-26T05:06:53.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[29691]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:06:53.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[29691]: Cluster is now healthy 2024-03-26T05:06:53.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:52 smithi080 ceph-mon[29691]: osdmap e190: 8 total, 8 up, 7 in 2024-03-26T05:06:53.148 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:53.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:53.249 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:53.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:52 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:06:53.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:52 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:06:53.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:52 smithi132 ceph-mon[33500]: pgmap v278: 148 pgs: 148 active+clean; 582 KiB data, 877 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:53.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:52 smithi132 ceph-mon[33500]: 4.3 scrub starts 2024-03-26T05:06:53.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:52 smithi132 ceph-mon[33500]: 4.3 scrub ok 2024-03-26T05:06:53.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:52 smithi132 ceph-mon[33500]: 4.18 scrub starts 2024-03-26T05:06:53.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:52 smithi132 ceph-mon[33500]: 4.18 scrub ok 2024-03-26T05:06:53.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:52 smithi132 ceph-mon[33500]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:06:53.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:52 smithi132 ceph-mon[33500]: Cluster is now healthy 2024-03-26T05:06:53.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:52 smithi132 ceph-mon[33500]: osdmap e190: 8 total, 8 up, 7 in 2024-03-26T05:06:53.295 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:53.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:53.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:53.397 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:53.442 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:53.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:53.544 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:53.577 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:53.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:53.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:53.679 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:53.718 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:53.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:53.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:53.823 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:53.864 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:53.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:53.908 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 1 15 3115 3100 193.739 193.75 0.00436653 0.00514799 2024-03-26T05:06:53.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:54.000 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:54.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:54.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:54.102 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:54.154 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:54.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:54.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:54.256 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:54.263 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:06:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:06:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:06:54.263 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:54 smithi080 ceph-mon[29691]: 5.8 scrub starts 2024-03-26T05:06:54.263 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:54 smithi080 ceph-mon[29691]: 5.8 scrub ok 2024-03-26T05:06:54.263 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:54 smithi080 ceph-mon[29691]: 5.1f scrub starts 2024-03-26T05:06:54.264 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:54 smithi080 ceph-mon[29691]: 5.1f scrub ok 2024-03-26T05:06:54.264 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:54 smithi080 ceph-mon[40404]: 5.8 scrub starts 2024-03-26T05:06:54.264 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:54 smithi080 ceph-mon[40404]: 5.8 scrub ok 2024-03-26T05:06:54.264 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:54 smithi080 ceph-mon[40404]: 5.1f scrub starts 2024-03-26T05:06:54.264 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:54 smithi080 ceph-mon[40404]: 5.1f scrub ok 2024-03-26T05:06:54.286 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:54.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:54.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:54.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:54 smithi132 ceph-mon[33500]: 5.8 scrub starts 2024-03-26T05:06:54.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:54 smithi132 ceph-mon[33500]: 5.8 scrub ok 2024-03-26T05:06:54.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:54 smithi132 ceph-mon[33500]: 5.1f scrub starts 2024-03-26T05:06:54.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:54 smithi132 ceph-mon[33500]: 5.1f scrub ok 2024-03-26T05:06:54.387 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:54.419 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:54.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:54.521 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:54.554 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:54.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:54.655 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:54.686 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:54.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:54.788 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:54.825 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:54.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:54.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:54.908 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 2 16 6046 6030 188.424 183.125 0.00419146 0.00529648 2024-03-26T05:06:54.927 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:54.968 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:54.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:55.069 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:55.114 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:55.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:55.215 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:55.246 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:55.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:55.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:55 smithi132 ceph-mon[33500]: pgmap v280: 148 pgs: 148 active+clean; 582 KiB data, 877 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:55.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:55 smithi132 ceph-mon[33500]: 6.10 scrub starts 2024-03-26T05:06:55.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:55 smithi132 ceph-mon[33500]: 4.4 scrub starts 2024-03-26T05:06:55.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:55 smithi132 ceph-mon[33500]: 6.10 scrub ok 2024-03-26T05:06:55.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:55 smithi132 ceph-mon[33500]: 4.4 scrub ok 2024-03-26T05:06:55.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:06:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:06:55.308 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:55 smithi080 ceph-mon[29691]: pgmap v280: 148 pgs: 148 active+clean; 582 KiB data, 877 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:55.308 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:55 smithi080 ceph-mon[29691]: 6.10 scrub starts 2024-03-26T05:06:55.308 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:55 smithi080 ceph-mon[29691]: 4.4 scrub starts 2024-03-26T05:06:55.308 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:55 smithi080 ceph-mon[29691]: 6.10 scrub ok 2024-03-26T05:06:55.308 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:55 smithi080 ceph-mon[29691]: 4.4 scrub ok 2024-03-26T05:06:55.308 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:55 smithi080 ceph-mon[40404]: pgmap v280: 148 pgs: 148 active+clean; 582 KiB data, 877 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:06:55.308 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:55 smithi080 ceph-mon[40404]: 6.10 scrub starts 2024-03-26T05:06:55.308 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:55 smithi080 ceph-mon[40404]: 4.4 scrub starts 2024-03-26T05:06:55.308 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:55 smithi080 ceph-mon[40404]: 6.10 scrub ok 2024-03-26T05:06:55.309 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:55 smithi080 ceph-mon[40404]: 4.4 scrub ok 2024-03-26T05:06:55.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:55.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:55.396 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:55.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:55.497 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:55.524 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:55.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:55.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:55.576 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:06:55.620 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:06:55.626 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:55.663 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:06:55.670 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:55.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:55.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:55.714 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:06:55.747 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:06:55.773 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:55.786 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:06:55.800 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:55.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:55.829 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:06:55.871 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:06:55.881 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:06:55.881 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:06:55.881 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:06:55.881 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:06:55.881 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:06:55.881 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:06:55.881 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:06:55.881 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:06:55.902 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:55.908 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 3 16 8591 8575 178.632 159.062 0.00808428 0.00558436 2024-03-26T05:06:55.930 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:55.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:56.032 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:56.062 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:56.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:56.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:56.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:56 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:56.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:56 smithi080 ceph-mon[29691]: 3.12 scrub starts 2024-03-26T05:06:56.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:56 smithi080 ceph-mon[29691]: 3.12 scrub ok 2024-03-26T05:06:56.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:56 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1019590369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:56.083 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:56 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:56.084 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:56 smithi080 ceph-mon[40404]: 3.12 scrub starts 2024-03-26T05:06:56.084 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:56 smithi080 ceph-mon[40404]: 3.12 scrub ok 2024-03-26T05:06:56.084 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:56 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1019590369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:56.164 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:56.205 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:56.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:56.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:56 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:06:56.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:56 smithi132 ceph-mon[33500]: 3.12 scrub starts 2024-03-26T05:06:56.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:56 smithi132 ceph-mon[33500]: 3.12 scrub ok 2024-03-26T05:06:56.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:56 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1019590369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:06:56.307 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:56.360 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:56.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:56.462 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:56.505 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:56.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:56.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:56.607 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:56.638 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:56.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:56.740 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:56.772 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:56.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:56.874 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:56.908 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 4 16 11318 11302 176.58 170.438 0.0041215 0.00565304 2024-03-26T05:06:56.914 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:56.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:56.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:57.016 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:57.045 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:57.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:57.147 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:57.167 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:06:57.175 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:57.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:57.277 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:57.319 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:57.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:57.421 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:57.450 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:57.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:57.552 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:57.583 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:57.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:57.684 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:57.724 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:57.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:57.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:57.744 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[29691]: pgmap v281: 148 pgs: 148 active+clean; 396 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-26T05:06:57.744 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[29691]: 5.d scrub starts 2024-03-26T05:06:57.745 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[29691]: 5.d scrub ok 2024-03-26T05:06:57.745 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[29691]: 5.f deep-scrub starts 2024-03-26T05:06:57.745 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[29691]: 5.f deep-scrub ok 2024-03-26T05:06:57.745 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[29691]: 3.16 deep-scrub starts 2024-03-26T05:06:57.745 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[29691]: 3.16 deep-scrub ok 2024-03-26T05:06:57.745 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[40404]: pgmap v281: 148 pgs: 148 active+clean; 396 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-26T05:06:57.746 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[40404]: 5.d scrub starts 2024-03-26T05:06:57.746 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[40404]: 5.d scrub ok 2024-03-26T05:06:57.746 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[40404]: 5.f deep-scrub starts 2024-03-26T05:06:57.746 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[40404]: 5.f deep-scrub ok 2024-03-26T05:06:57.746 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[40404]: 3.16 deep-scrub starts 2024-03-26T05:06:57.746 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:57 smithi080 ceph-mon[40404]: 3.16 deep-scrub ok 2024-03-26T05:06:57.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:57 smithi132 ceph-mon[33500]: pgmap v281: 148 pgs: 148 active+clean; 396 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-26T05:06:57.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:57 smithi132 ceph-mon[33500]: 5.d scrub starts 2024-03-26T05:06:57.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:57 smithi132 ceph-mon[33500]: 5.d scrub ok 2024-03-26T05:06:57.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:57 smithi132 ceph-mon[33500]: 5.f deep-scrub starts 2024-03-26T05:06:57.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:57 smithi132 ceph-mon[33500]: 5.f deep-scrub ok 2024-03-26T05:06:57.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:57 smithi132 ceph-mon[33500]: 3.16 deep-scrub starts 2024-03-26T05:06:57.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:57 smithi132 ceph-mon[33500]: 3.16 deep-scrub ok 2024-03-26T05:06:57.825 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:06:57.855 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:57.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:06:57.909 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 5 15 13736 13721 171.499 151.188 0.00560744 0.00582365 2024-03-26T05:06:57.957 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:57.987 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:57.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:58.089 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:06:58.122 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:58.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:06:58.223 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:58.263 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:58.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:58.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:58.365 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:58.403 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:58.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:58.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:58.437 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:06:58.505 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:58.539 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:58.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:58.641 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:58.670 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:58.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:58.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:58.772 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:06:58.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:58 smithi132 ceph-mon[33500]: 4.f scrub starts 2024-03-26T05:06:58.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:58 smithi132 ceph-mon[33500]: 4.f scrub ok 2024-03-26T05:06:58.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2263158042' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:06:58.796 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:58.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:06:58.842 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:06:58.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:58 smithi080 ceph-mon[40404]: 4.f scrub starts 2024-03-26T05:06:58.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:58 smithi080 ceph-mon[40404]: 4.f scrub ok 2024-03-26T05:06:58.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2263158042' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:06:58.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:58 smithi080 ceph-mon[29691]: 4.f scrub starts 2024-03-26T05:06:58.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:58 smithi080 ceph-mon[29691]: 4.f scrub ok 2024-03-26T05:06:58.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2263158042' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:06:58.898 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:58.909 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 6 16 16022 16006 166.698 142.812 0.0114303 0.00599182 2024-03-26T05:06:58.927 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:58.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:59.029 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:06:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:59.070 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:59.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:06:59.171 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:59.218 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:59.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:59.321 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:06:59.346 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:59.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:06:59.449 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:59.478 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:59.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:59.580 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:06:59.618 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:59.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:06:59.720 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:06:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:59.760 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:06:59.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:06:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:59 smithi132 ceph-mon[33500]: pgmap v282: 148 pgs: 148 active+clean; 396 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 653 B/s rd, 50 MiB/s wr, 807 op/s 2024-03-26T05:06:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:59 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:06:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:59 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2263158042' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:06:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:06:59 smithi132 ceph-mon[33500]: osdmap e191: 8 total, 8 up, 7 in 2024-03-26T05:06:59.862 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:06:59.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:59 smithi080 ceph-mon[29691]: pgmap v282: 148 pgs: 148 active+clean; 396 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 653 B/s rd, 50 MiB/s wr, 807 op/s 2024-03-26T05:06:59.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:59 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:06:59.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:59 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2263158042' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:06:59.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:06:59 smithi080 ceph-mon[29691]: osdmap e191: 8 total, 8 up, 7 in 2024-03-26T05:06:59.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:59 smithi080 ceph-mon[40404]: pgmap v282: 148 pgs: 148 active+clean; 396 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 653 B/s rd, 50 MiB/s wr, 807 op/s 2024-03-26T05:06:59.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:59 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:06:59.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:59 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2263158042' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:06:59.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:06:59 smithi080 ceph-mon[40404]: osdmap e191: 8 total, 8 up, 7 in 2024-03-26T05:06:59.891 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:06:59.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:06:59.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:06:59.909 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 7 16 18118 18102 161.597 131 0.0137128 0.00618168 2024-03-26T05:06:59.994 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:00.025 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:00.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:00.126 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:00.156 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:00.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:00.262 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:00.292 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:00.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:00.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:00.394 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:00.417 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:00.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:00.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:00.519 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:00.552 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:00.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:00.653 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:00.676 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:00.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:00.778 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:00.808 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:00.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:00.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:00.910 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 8 15 20243 20228 158.005 132.875 0.00769698 0.006322 2024-03-26T05:07:00.910 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:00.948 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:00.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:00.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:01.050 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:01.074 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:01.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:01.175 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:01.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:01.215 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:01.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:01.317 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:01.341 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:01.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:01.442 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:01.466 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:01.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:01.568 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:01.605 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:01.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:01.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:01.709 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:01.751 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:01.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:01.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:01 smithi132 ceph-mon[33500]: pgmap v284: 148 pgs: 148 active+clean; 634 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-26T05:07:01.843 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:01 smithi080 ceph-mon[29691]: pgmap v284: 148 pgs: 148 active+clean; 634 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-26T05:07:01.843 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:01 smithi080 ceph-mon[40404]: pgmap v284: 148 pgs: 148 active+clean; 634 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-26T05:07:01.852 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:01.900 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:01.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:01.909 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 9 16 21995 21979 152.608 109.438 0.00462414 0.00654503 2024-03-26T05:07:02.002 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:02.052 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:02.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:02.154 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:02.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:02.285 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:02.328 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:02.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:02.429 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:02.457 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:02.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:02.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:02.559 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:02.605 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:02.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:02.707 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:02.764 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:02.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:02.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:02.866 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:02.897 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:02.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:02.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:02.909 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 10 16 23986 23970 149.79 124.438 0.00886131 0.00666904 2024-03-26T05:07:02.975 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:02.999 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:03.032 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:03.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:03.133 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:03.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:03.177 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:03.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:03.278 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:03.308 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:03.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:03.316 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:07:03.316 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:07:03.316 INFO:teuthology.orchestra.run.smithi080.stdout: "complaint_time": 30, 2024-03-26T05:07:03.316 INFO:teuthology.orchestra.run.smithi080.stdout: "num_blocked_ops": 0 2024-03-26T05:07:03.316 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:07:03.410 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:03.430 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.6 to 1 (10000) 2024-03-26T05:07:03.441 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:03.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:03.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:03.450 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:03 smithi080 ceph-mon[29691]: pgmap v285: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-03-26T05:07:03.450 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:03 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3943087303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:07:03.450 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:03 smithi080 ceph-mon[40404]: pgmap v285: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-03-26T05:07:03.450 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:03 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3943087303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:07:03.543 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:03.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:03.584 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:03.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:03.686 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:03.725 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:03.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:03.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:03 smithi132 ceph-mon[33500]: pgmap v285: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-03-26T05:07:03.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:03 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3943087303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:07:03.827 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:03.867 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:03.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:03.910 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 11 16 25869 25853 146.871 117.688 0.0048975 0.00680219 2024-03-26T05:07:03.968 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:03.999 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:04.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:04.101 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:04.117 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:07:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:07:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:07:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:04.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:04.231 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:04.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:04.272 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:04.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:04.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:04.406 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:04.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:04.507 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:04.546 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:04.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:04.649 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:04.670 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:07:04.676 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:04.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:04.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:04.712 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:04 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3943087303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:07:04.712 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:04 smithi080 ceph-mon[40404]: osdmap e192: 8 total, 8 up, 7 in 2024-03-26T05:07:04.713 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:04 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3943087303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:07:04.713 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:04 smithi080 ceph-mon[29691]: osdmap e192: 8 total, 8 up, 7 in 2024-03-26T05:07:04.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:04.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:04 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3943087303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:07:04.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:04 smithi132 ceph-mon[33500]: osdmap e192: 8 total, 8 up, 7 in 2024-03-26T05:07:04.805 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:04.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:04.907 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:04.910 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 12 16 27847 27831 144.933 123.625 0.00176498 0.00688341 2024-03-26T05:07:04.934 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:04.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:05.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:05.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:05.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:05.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:05.170 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:05.207 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:05.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:05.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:05.309 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:05.348 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:05.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:05.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:05.447 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:07:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:07:05.450 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:05.491 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:05.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:05.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:05.626 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:05.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:05.728 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:05.759 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:05.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:05.763 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:05 smithi080 ceph-mon[40404]: pgmap v287: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-26T05:07:05.763 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:05.763 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:07:05.763 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:05 smithi080 ceph-mon[29691]: pgmap v287: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-26T05:07:05.763 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:05.763 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:07:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:05 smithi132 ceph-mon[33500]: pgmap v287: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-26T05:07:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:07:05.819 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T05:07:05.861 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:05.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:05.886 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:05.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:05.910 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 13 16 29595 29579 142.187 109.25 0.00771175 0.00700262 2024-03-26T05:07:05.987 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:06.012 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:06.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:06.113 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:06.143 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:06.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:06.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:06.245 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:06.268 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:06.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:06.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:06.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:06.400 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:06.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:06.502 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:06.532 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:06.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:06.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:06.634 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:06.657 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:06.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:06.759 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:06 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:06.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:06 smithi132 ceph-mon[33500]: osdmap e193: 8 total, 8 up, 7 in 2024-03-26T05:07:06.799 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:06.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:06.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:06 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:06 smithi080 ceph-mon[40404]: osdmap e193: 8 total, 8 up, 7 in 2024-03-26T05:07:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:06 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:06 smithi080 ceph-mon[29691]: osdmap e193: 8 total, 8 up, 7 in 2024-03-26T05:07:06.901 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:06.911 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 14 16 30932 30916 137.991 83.5625 0.0076556 0.00724174 2024-03-26T05:07:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:06.931 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:06.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:07.032 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:07.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:07.164 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:07.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:07.192 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:07.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:07.293 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:07.341 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:07.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:07.443 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:07.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:07.475 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:07.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:07.576 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:07.600 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:07.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:07.702 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:07.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:07.731 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:07.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:07.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:07 smithi132 ceph-mon[33500]: pgmap v288: 148 pgs: 148 active+clean; 1.7 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.71k op/s 2024-03-26T05:07:07.833 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:07.862 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:07.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:07.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:07 smithi080 ceph-mon[29691]: pgmap v288: 148 pgs: 148 active+clean; 1.7 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.71k op/s 2024-03-26T05:07:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:07 smithi080 ceph-mon[40404]: pgmap v288: 148 pgs: 148 active+clean; 1.7 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.71k op/s 2024-03-26T05:07:07.911 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 15 16 32191 32175 134.037 78.6875 0.00709459 0.0074548 2024-03-26T05:07:07.964 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:07.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:07.996 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:07.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:08.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:08.137 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:08.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:08.239 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:08.277 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:08.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:08.379 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:08.403 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:08.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:08.504 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:08.530 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:08.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:08.632 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:08.656 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:08.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:08.757 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:08.786 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:08.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:08.888 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:08.911 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 16 16 33452 33436 130.586 78.8125 0.007455 0.00765251 2024-03-26T05:07:08.918 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:08.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:08.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:09.020 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:09.061 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:09.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:09.106 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 7 1 2024-03-26T05:07:09.163 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:09.191 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:09.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:09.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:09.293 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:09.335 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:09.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:09.437 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:09.467 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:09.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:09.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:09.569 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:09.594 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:09.595 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:09.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:09.674 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:09.697 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:09.743 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:09 smithi080 ceph-mon[29691]: pgmap v290: 148 pgs: 148 active+clean; 1.7 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-26T05:07:09.744 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:09 smithi080 ceph-mon[40404]: pgmap v290: 148 pgs: 148 active+clean; 1.7 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-26T05:07:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:09.749 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:09.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:09.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:09 smithi132 ceph-mon[33500]: pgmap v290: 148 pgs: 148 active+clean; 1.7 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-26T05:07:09.850 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:09.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:09.896 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:09.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:09.911 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 17 16 35125 35109 129.054 104.562 0.00549952 0.00774126 2024-03-26T05:07:09.998 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:10.046 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:10.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:10.147 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:10.176 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:10.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:10.278 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:10.306 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:10.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:10.408 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:10.455 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:10.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:10.556 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:10.614 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:10.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:10.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:10 smithi080 ceph-mon[29691]: pgmap v291: 148 pgs: 148 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-03-26T05:07:10.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:10 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:07:10.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:10 smithi080 ceph-mon[40404]: pgmap v291: 148 pgs: 148 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-03-26T05:07:10.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:10 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:07:10.716 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:10.749 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:10.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:10.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:10 smithi132 ceph-mon[33500]: pgmap v291: 148 pgs: 148 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-03-26T05:07:10.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:10 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:07:10.850 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:10.897 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:10.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:10.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:10.912 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 18 16 36655 36639 127.191 95.625 0.0103308 0.00785545 2024-03-26T05:07:11.000 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:11.041 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:11.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:11.143 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:11.176 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:11.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:11.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:11.278 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:11.309 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:11.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:11.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:11.412 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:11.451 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:11.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:11.553 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:11.594 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:11.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:11.696 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:11.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:11.726 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:11.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:11.827 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:11.855 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:11.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:11.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:11.857 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:11 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:11.857 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:11 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:11.857 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:11 smithi080 ceph-mon[40404]: osdmap e194: 8 total, 8 up, 7 in 2024-03-26T05:07:11.857 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:11 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:11.857 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:11 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:11.857 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:11 smithi080 ceph-mon[29691]: osdmap e194: 8 total, 8 up, 7 in 2024-03-26T05:07:11.912 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 19 16 38314 38298 125.954 103.688 0.00498319 0.00793351 2024-03-26T05:07:11.956 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:11.983 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:11.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:11.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:11.990 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:07:12.027 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:07:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:11 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:12.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:11 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:12.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:11 smithi132 ceph-mon[33500]: osdmap e194: 8 total, 8 up, 7 in 2024-03-26T05:07:12.069 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:07:12.085 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:12.109 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:12.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:12.136 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:07:12.158 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:07:12.188 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:07:12.211 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:12.226 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:07:12.240 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:12.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:12.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:12.266 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:07:12.270 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:07:12.270 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:07:12.270 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:07:12.270 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:07:12.270 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:07:12.270 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:07:12.270 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:07:12.270 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:07:12.342 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:12.372 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:12.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:12.474 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:12.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:12.506 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:12.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:12.608 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:12.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:12.639 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:12.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:12.686 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:12 smithi080 ceph-mon[40404]: pgmap v293: 148 pgs: 148 active+clean; 2.2 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-03-26T05:07:12.686 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:12 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/116211751' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:07:12.686 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:12 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:07:12.686 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:12 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:07:12.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:12 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:12.687 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:12 smithi080 ceph-mon[29691]: pgmap v293: 148 pgs: 148 active+clean; 2.2 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-03-26T05:07:12.687 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:12 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/116211751' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:07:12.687 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:12 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:07:12.687 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:12 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:07:12.687 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:12 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:12.740 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:12.785 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:12.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:12.886 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:12.912 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:07:12.921663+0000 min lat: 0.00124249 max lat: 0.162251 avg lat: 0.00804227 2024-03-26T05:07:12.912 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:07:12.912 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 20 16 39783 39767 124.247 91.8125 0.00752334 0.00804227 2024-03-26T05:07:12.916 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:12.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:12.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:13.018 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:12 smithi132 ceph-mon[33500]: pgmap v293: 148 pgs: 148 active+clean; 2.2 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-03-26T05:07:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:12 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/116211751' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:07:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:12 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:07:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:12 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:07:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:12 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:13.058 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:13.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:13.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:13.161 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:13.191 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:13.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:13.292 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:13.332 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:13.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:13.432 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:13.473 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:13.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:13.574 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:13.612 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:13.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:13.714 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:13.752 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:13.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:13.854 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:13.886 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:13.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:13.912 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 21 16 41495 41479 123.425 107 0.00555915 0.00809668 2024-03-26T05:07:13.986 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:14.012 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:14.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:14.114 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:14.159 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:14.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:14.261 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:14.304 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:14.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:14.367 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:07:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:07:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:07:14.406 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:14.435 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:14.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:14.537 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:14.577 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:14.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:14.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:14.679 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:14.725 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:14.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:14.827 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:14.856 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:14.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:14.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:14.912 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 22 16 43007 42991 122.11 94.5 0.0108879 0.00818376 2024-03-26T05:07:14.958 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:14.980 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:15.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:15.017 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:15.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:15.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:14 smithi080 ceph-mon[40404]: pgmap v294: 148 pgs: 148 active+clean; 2.2 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 906 op/s 2024-03-26T05:07:15.118 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:15.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:14 smithi080 ceph-mon[29691]: pgmap v294: 148 pgs: 148 active+clean; 2.2 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 906 op/s 2024-03-26T05:07:15.147 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:15.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:15.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:15.154 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:07:15.249 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:15.278 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:15.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:15.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:15.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:14 smithi132 ceph-mon[33500]: pgmap v294: 148 pgs: 148 active+clean; 2.2 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 906 op/s 2024-03-26T05:07:15.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:07:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:07:15.358 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:07:15.358 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:07:15.381 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:15.431 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:15.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:15.532 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:15.560 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:15.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:15.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:15.662 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:15.689 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:15.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:15.791 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:15.832 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:15.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:15.903 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:07:15.912 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 23 16 44352 44336 120.456 84.0625 0.0162028 0.00829386 2024-03-26T05:07:15.935 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:15.964 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:15.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:15.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:16.066 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:16.110 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:16.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:16.193 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:15 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:16.233 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:15 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2284876001' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:07:16.233 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:15 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:16.234 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:15 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2284876001' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:07:16.234 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:16.274 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:16.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:16.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:16.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:15 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:16.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:15 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2284876001' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:07:16.376 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:16.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:16.418 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:16.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:16.519 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:16.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:16.560 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:16.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:16.661 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:16.706 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:16.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:16.808 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:16.837 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:16.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:16.912 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 24 16 45937 45921 119.564 99.0625 0.00658822 0.00835793 2024-03-26T05:07:16.939 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:16.969 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:16.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:17.071 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:17.109 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:17.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:17.211 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:17.242 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:17.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:17.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:17.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:16 smithi080 ceph-mon[29691]: pgmap v295: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-26T05:07:17.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:16 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:07:17.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:16 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2284876001' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:07:17.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:16 smithi080 ceph-mon[29691]: osdmap e195: 8 total, 8 up, 7 in 2024-03-26T05:07:17.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:16 smithi080 ceph-mon[29691]: osdmap e196: 8 total, 8 up, 7 in 2024-03-26T05:07:17.263 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:16 smithi080 ceph-mon[40404]: pgmap v295: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-26T05:07:17.263 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:16 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:07:17.263 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:16 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2284876001' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:07:17.263 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:16 smithi080 ceph-mon[40404]: osdmap e195: 8 total, 8 up, 7 in 2024-03-26T05:07:17.263 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:16 smithi080 ceph-mon[40404]: osdmap e196: 8 total, 8 up, 7 in 2024-03-26T05:07:17.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:16 smithi132 ceph-mon[33500]: pgmap v295: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-26T05:07:17.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:16 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:07:17.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:16 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2284876001' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:07:17.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:16 smithi132 ceph-mon[33500]: osdmap e195: 8 total, 8 up, 7 in 2024-03-26T05:07:17.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:16 smithi132 ceph-mon[33500]: osdmap e196: 8 total, 8 up, 7 in 2024-03-26T05:07:17.344 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:17.381 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:17.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:17.483 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:17.513 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:17.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:17.615 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:17.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:17.645 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:17.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:17.746 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:17.776 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:17.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:17.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:17.877 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:17.905 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:17.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:17.912 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 25 16 47540 47524 118.789 100.188 0.00254418 0.00841235 2024-03-26T05:07:18.008 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:18.035 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:18.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:18.138 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:18.167 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:18.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:18.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:18.269 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:18.311 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:18.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:18.413 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:18.444 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:18.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:18.547 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:18.579 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:18.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:18.681 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:18.707 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:18.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:18.737 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T05:07:18.809 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:18.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:18.856 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:18.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:18.912 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 26 16 48947 48931 117.602 87.9375 0.007138 0.0084974 2024-03-26T05:07:18.958 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:18.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:18.982 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:18.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:19.083 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:19.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:18 smithi080 ceph-mon[40404]: pgmap v298: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 797 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-26T05:07:19.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:18 smithi080 ceph-mon[29691]: pgmap v298: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 797 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-26T05:07:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:19.134 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:19.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:19.235 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:19.265 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:19.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:19.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:18 smithi132 ceph-mon[33500]: pgmap v298: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 797 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-26T05:07:19.367 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:19.400 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:19.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:19.501 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:19.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:19.540 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:19.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:19.616 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:07:19.641 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:19.669 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:19.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:19.771 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:19.799 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:19.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:19.901 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:19.913 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 27 16 50142 50126 116.012 74.6875 0.0109543 0.00861415 2024-03-26T05:07:19.957 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:19.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:20.059 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:20.087 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:20.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:20.191 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:20.229 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:20.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:20.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:20.251 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:07:20.252 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:07:20.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:07:20.331 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:20.338 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:20.357 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:20.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:20.458 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:20.501 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:20.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:20.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:20.603 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:20.648 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:20.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:20.676 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:20.750 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:20.794 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:20.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:20.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:20.896 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:20.913 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 28 16 51540 51524 114.989 87.375 0.0112398 0.00869025 2024-03-26T05:07:20.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:20.941 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:20.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:21.042 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:21.071 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:21.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:21.173 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:21.218 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:21.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:21.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:21.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:20 smithi132 ceph-mon[33500]: pgmap v299: 148 pgs: 148 active+clean; 2.8 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-26T05:07:21.320 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:21.345 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:21.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:21.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:20 smithi080 ceph-mon[29691]: pgmap v299: 148 pgs: 148 active+clean; 2.8 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-26T05:07:21.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:20 smithi080 ceph-mon[40404]: pgmap v299: 148 pgs: 148 active+clean; 2.8 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-26T05:07:21.447 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:21.488 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:21.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:21.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:21.590 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:21.624 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:21.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:21.726 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:21.768 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:21.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:21.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:21.870 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:21.911 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:21.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:21.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:21.913 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 29 15 52967 52952 114.101 89.25 0.0063045 0.00875889 2024-03-26T05:07:22.012 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:22.043 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:22.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:22.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:22.145 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:22.177 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:22.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:22.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:22.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:22.309 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:22.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:22.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:22.449 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:22.456 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:22.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:22.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:22.558 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:22.589 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:22.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:22.591 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:22 smithi080 ceph-mon[40404]: osdmap e197: 8 total, 8 up, 7 in 2024-03-26T05:07:22.591 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:22 smithi080 ceph-mon[29691]: osdmap e197: 8 total, 8 up, 7 in 2024-03-26T05:07:22.691 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:22.734 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:22.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:22.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:22.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:22 smithi132 ceph-mon[33500]: osdmap e197: 8 total, 8 up, 7 in 2024-03-26T05:07:22.836 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:22.865 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:22.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:22.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:22.913 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 30 15 54313 54298 113.102 84.125 0.0110116 0.00883612 2024-03-26T05:07:22.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:22.994 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:22.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:23.096 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:23.137 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:23.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:23.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:23.239 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:23.266 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:23.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:23.369 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:23.424 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:23.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:23.453 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.7 to 1 (10000) 2024-03-26T05:07:23.525 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:23.575 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:23.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:23.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:23 smithi080 ceph-mon[40404]: pgmap v301: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T05:07:23.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:23 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2148917588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:07:23.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:23 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:07:23.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:23 smithi080 ceph-mon[29691]: pgmap v301: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T05:07:23.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:23 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2148917588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:07:23.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:23 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:07:23.676 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:23.713 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:23.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:23.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:23 smithi132 ceph-mon[33500]: pgmap v301: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T05:07:23.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:23 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2148917588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:07:23.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:23 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:07:23.814 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:23.840 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:23.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:23.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:23.913 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 31 16 55585 55569 112.015 79.4375 0.0173932 0.00892184 2024-03-26T05:07:23.942 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:23.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:23.988 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:23.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:24.086 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:07:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:07:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:07:24.089 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:24.118 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:24.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:24.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:24.220 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:24.262 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:24.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:24.364 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:24.410 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:24.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:24.511 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:24.554 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:24.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:24.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:24.656 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:24.684 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:24.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:24.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:24.786 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:24.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:24 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:07:24.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:24 smithi132 ceph-mon[33500]: osdmap e198: 8 total, 8 up, 7 in 2024-03-26T05:07:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:24.826 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:24.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:24.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:24 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:07:24.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:24 smithi080 ceph-mon[29691]: osdmap e198: 8 total, 8 up, 7 in 2024-03-26T05:07:24.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:24 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:07:24.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:24 smithi080 ceph-mon[40404]: osdmap e198: 8 total, 8 up, 7 in 2024-03-26T05:07:24.913 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 32 15 56798 56783 110.885 75.875 0.0126525 0.00901373 2024-03-26T05:07:24.927 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:24.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:24.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:25.059 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:25.096 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:25.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:25.198 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:25.223 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:25.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:25.325 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:25.352 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:25.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:25.453 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:07:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:07:25.453 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:25.485 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:25.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:25.587 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:25.627 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:25.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:25.677 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:25 smithi080 ceph-mon[29691]: pgmap v303: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 690 B/s rd, 61 MiB/s wr, 973 op/s 2024-03-26T05:07:25.677 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:25 smithi080 ceph-mon[40404]: pgmap v303: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 690 B/s rd, 61 MiB/s wr, 973 op/s 2024-03-26T05:07:25.728 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:25.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:25.775 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:25.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:25.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:25 smithi132 ceph-mon[33500]: pgmap v303: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 690 B/s rd, 61 MiB/s wr, 973 op/s 2024-03-26T05:07:25.877 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:25.914 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 33 16 58078 58062 109.945 79.9375 0.00385807 0.00909022 2024-03-26T05:07:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:25.915 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:25.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:26.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:26.055 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:26.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:26.156 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:26.196 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:26.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:26.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:26.335 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:26.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:26.437 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:26.474 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:26.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:26.576 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:26.600 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T05:07:26.600 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:07:26.603 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:26.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:26.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:26.640 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:26 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:26.640 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:26 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:07:26.640 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:26 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:07:26.640 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:26 smithi080 ceph-mon[40404]: osdmap e199: 8 total, 8 up, 7 in 2024-03-26T05:07:26.640 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:26 smithi080 ceph-mon[40404]: osdmap e200: 8 total, 8 up, 7 in 2024-03-26T05:07:26.642 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:26 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:26.642 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:26 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:07:26.642 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:26 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:07:26.642 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:26 smithi080 ceph-mon[29691]: osdmap e199: 8 total, 8 up, 7 in 2024-03-26T05:07:26.642 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:26 smithi080 ceph-mon[29691]: osdmap e200: 8 total, 8 up, 7 in 2024-03-26T05:07:26.705 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:26.732 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:26.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:26.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:26.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:26 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:26.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:26 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:07:26.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:26 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:07:26.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:26 smithi132 ceph-mon[33500]: osdmap e199: 8 total, 8 up, 7 in 2024-03-26T05:07:26.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:26 smithi132 ceph-mon[33500]: osdmap e200: 8 total, 8 up, 7 in 2024-03-26T05:07:26.834 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:26.902 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:26.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:26.914 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 34 16 59264 59248 108.891 74.125 0.0108416 0.00917687 2024-03-26T05:07:27.004 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:27.034 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:27.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:27.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:27.136 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:27.163 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:27.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:27.265 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:27.279 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:07:27.279 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:07:27.279 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:07:27.279 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:07:27.279 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:07:27.279 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:07:27.279 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:07:27.280 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:07:27.280 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:07:27.280 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:07:27.280 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:07:27.280 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:07:27.280 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:07:27.280 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:07:27.280 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:07:27.280 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:07:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:27.312 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:27.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:27.413 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:27.443 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:27.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:27.544 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:27.575 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:27.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:27.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:27.677 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:27.716 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:27.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:27.783 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:27 smithi080 ceph-mon[40404]: pgmap v305: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-26T05:07:27.783 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:27 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4178158610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:07:27.783 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:27 smithi080 ceph-mon[29691]: pgmap v305: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-26T05:07:27.783 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:27 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4178158610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:07:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:27 smithi132 ceph-mon[33500]: pgmap v305: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-26T05:07:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:27 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4178158610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:07:27.818 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:27.845 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:27.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:27.914 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 35 16 60560 60544 108.094 81 0.011665 0.00924686 2024-03-26T05:07:27.947 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:27.989 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:27.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:28.090 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:28.116 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:28.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:28.218 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:28.245 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:28.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:28.347 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:28.372 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:28.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:28.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:28.433 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:07:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:07:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:61118 8.f 8:f334d1d4:::benchmark_data_smithi080_109213_object61117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-26T05:07:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:28.428216+0000", 2024-03-26T05:07:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.012257964, 2024-03-26T05:07:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.012294181, 2024-03-26T05:07:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:07:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:07:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:07:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:07:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:07:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 61118 2024-03-26T05:07:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:07:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.428216+0000", 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.428216+0000", 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.428218+0000", 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5160000000000001e-06 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.428246+0000", 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.7504999999999998e-05 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.428260+0000", 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4222999999999999e-05 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.428269+0000", 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8230000000000001e-06 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.431214+0000", 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0029453389999999999 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.431346+0000", 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013171400000000001 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,2", 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.431482+0000", 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013588000000000001 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.437099+0000", 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0056174409999999999 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439252+0000", 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0021531060000000001 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:61119 8.f 8:f5f185fb:::benchmark_data_smithi080_109213_object61118:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:28.428288+0000", 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.012185967000000001, 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.012327223999999999, 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 61119 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:07:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.428288+0000", 2024-03-26T05:07:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:07:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.428288+0000", 2024-03-26T05:07:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:07:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.428289+0000", 2024-03-26T05:07:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.68e-07 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.428313+0000", 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.389e-05 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.428320+0000", 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9130000000000003e-06 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.428327+0000", 2024-03-26T05:07:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6159999999999998e-06 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.433508+0000", 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051809129999999997 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.433653+0000", 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014458799999999999 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,2", 2024-03-26T05:07:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.433796+0000", 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000143891 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.437104+0000", 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0033079350000000001 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439586+0000", 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0024819019999999998 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:07:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:61123 8.8 8:172e0508:::benchmark_data_smithi080_109213_object61122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:28.430558+0000", 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0099162239999999995, 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010160687, 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 61123 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.430558+0000", 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.430558+0000", 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.430560+0000", 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8160000000000001e-06 2024-03-26T05:07:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.430586+0000", 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6514999999999999e-05 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.430601+0000", 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4795e-05 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:07:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.430610+0000", 2024-03-26T05:07:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.3440000000000007e-06 2024-03-26T05:07:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:07:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.430673+0000", 2024-03-26T05:07:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2885000000000005e-05 2024-03-26T05:07:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:07:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.430838+0000", 2024-03-26T05:07:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016451200000000001 2024-03-26T05:07:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:07:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.431037+0000", 2024-03-26T05:07:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019933900000000001 2024-03-26T05:07:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:07:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:61126 8.b 8:d99fc516:::benchmark_data_smithi080_109213_object61125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:28.433521+0000", 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0069529730000000003, 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0072847909999999997, 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 61126 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:07:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.433521+0000", 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.433521+0000", 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.433522+0000", 2024-03-26T05:07:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.389e-06 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.433567+0000", 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4897000000000001e-05 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.433577+0000", 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7629999999999995e-06 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.433584+0000", 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0870000000000001e-06 2024-03-26T05:07:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.433633+0000", 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8599e-05 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.433752+0000", 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011954400000000001 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,4", 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.433881+0000", 2024-03-26T05:07:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012850000000000001 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop_reply(client.15093.0:61123 8.8 e200/184)", 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:28.436350+0000", 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0041238960000000002, 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0045381099999999997, 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "queued for pg", 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.436350+0000", 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.436350+0000", 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.436351+0000", 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5900000000000005e-07 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.436356+0000", 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4319999999999998e-06 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.436370+0000", 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4029000000000001e-05 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.436378+0000", 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.542e-06 2024-03-26T05:07:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop_reply(client.15093.0:61126 8.b e200/184)", 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:28.439082+0000", 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.001391963, 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0018575639999999999, 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "queued for pg", 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439082+0000", 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439082+0000", 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439083+0000", 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1689999999999999e-06 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439088+0000", 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8740000000000003e-06 2024-03-26T05:07:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439104+0000", 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6308999999999999e-05 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439113+0000", 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9400000000000008e-06 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:61129 8.f 8:f5e6474a:::benchmark_data_smithi080_109213_object61128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-26T05:07:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:28.439606+0000", 2024-03-26T05:07:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.00086806199999999996, 2024-03-26T05:07:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001390596, 2024-03-26T05:07:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:07:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:07:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:07:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:07:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:07:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 61129 2024-03-26T05:07:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439606+0000", 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439606+0000", 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439607+0000", 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3510000000000001e-06 2024-03-26T05:07:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439630+0000", 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3025000000000001e-05 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439640+0000", 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0225e-05 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439648+0000", 2024-03-26T05:07:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3660000000000003e-06 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439770+0000", 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000121962 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.439939+0000", 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000168892 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,2", 2024-03-26T05:07:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.440098+0000", 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015915100000000001 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop_reply(client.15093.0:61123 8.8 e200/184)", 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:28.440183+0000", 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.00029074599999999998, 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00089092000000000001, 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "queued for pg", 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.440183+0000", 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.440183+0000", 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.440183+0000", 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.28e-07 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.440186+0000", 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5339999999999998e-06 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.440193+0000", 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8260000000000004e-06 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:28.440200+0000", 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.866e-06 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:07:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:07:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:07:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 8 2024-03-26T05:07:28.468 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:07:28.474 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:28.503 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:28.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:28.604 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:28.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:28 smithi080 ceph-mon[29691]: pgmap v307: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-26T05:07:28.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:28 smithi080 ceph-mon[40404]: pgmap v307: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-26T05:07:28.651 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:28.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:28.753 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:28.780 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:28.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:28.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:28 smithi132 ceph-mon[33500]: pgmap v307: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-26T05:07:28.882 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:28.916 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 36 16 61700 61684 107.066 71.25 0.0102291 0.0093341 2024-03-26T05:07:28.929 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:28.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:28.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:29.030 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:29.061 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:29.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:29.163 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:29.192 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:29.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:29.293 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:29.324 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:29.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:29.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:29.364 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T05:07:29.426 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:29.497 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:29.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:29.515 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:07:29.597 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:29.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:29 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/121449570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:07:29.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:29 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/121449570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:07:29.627 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:29.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:29.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:29.729 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:29.759 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:29.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:29.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:29 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/121449570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:07:29.861 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:29.916 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 37 16 62915 62899 106.225 75.9375 0.0103437 0.00940877 2024-03-26T05:07:29.920 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:29.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:30.022 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:30.070 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:30.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:30.171 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:30.216 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:30.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:30.318 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:30.338 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:30.375 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:30.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:30.476 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:30.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:30.532 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:30.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:30.633 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:30.679 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:30.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:30.762 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:07:30.762 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/121449570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:07:30.762 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[29691]: osdmap e201: 8 total, 8 up, 7 in 2024-03-26T05:07:30.762 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[29691]: 5.4 scrub starts 2024-03-26T05:07:30.762 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[29691]: 5.4 scrub ok 2024-03-26T05:07:30.762 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[29691]: 6.17 scrub starts 2024-03-26T05:07:30.762 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[29691]: 6.17 scrub ok 2024-03-26T05:07:30.763 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[29691]: pgmap v309: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-26T05:07:30.763 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:07:30.763 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/121449570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:07:30.763 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[40404]: osdmap e201: 8 total, 8 up, 7 in 2024-03-26T05:07:30.763 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[40404]: 5.4 scrub starts 2024-03-26T05:07:30.763 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[40404]: 5.4 scrub ok 2024-03-26T05:07:30.763 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[40404]: 6.17 scrub starts 2024-03-26T05:07:30.763 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[40404]: 6.17 scrub ok 2024-03-26T05:07:30.763 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:30 smithi080 ceph-mon[40404]: pgmap v309: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-26T05:07:30.781 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:30 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:07:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:30 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/121449570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:07:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:30 smithi132 ceph-mon[33500]: osdmap e201: 8 total, 8 up, 7 in 2024-03-26T05:07:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:30 smithi132 ceph-mon[33500]: 5.4 scrub starts 2024-03-26T05:07:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:30 smithi132 ceph-mon[33500]: 5.4 scrub ok 2024-03-26T05:07:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:30 smithi132 ceph-mon[33500]: 6.17 scrub starts 2024-03-26T05:07:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:30 smithi132 ceph-mon[33500]: 6.17 scrub ok 2024-03-26T05:07:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:30 smithi132 ceph-mon[33500]: pgmap v309: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-26T05:07:30.808 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:30.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:30.909 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:30.916 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 38 16 64235 64219 105.6 82.5 0.00707661 0.00946408 2024-03-26T05:07:30.936 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:30.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:31.038 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:31.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:31.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:31.170 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:31.198 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:31.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:31.300 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:31.327 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:31.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:31.429 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:31.459 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:31.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:31.561 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:31.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:31.603 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:31.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:31.704 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:31.742 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:31.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:31.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:31.844 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:31.874 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:31.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:31.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:31.880 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:07:31.916 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 39 16 65484 65468 104.894 78.0625 0.0154914 0.00952824 2024-03-26T05:07:31.962 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:31.975 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:32.005 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:32.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:32.107 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:32.134 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:32.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:32.236 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:32.266 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:32.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:32.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:32.368 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:32.409 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:32.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:32.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:32.511 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:32.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:32.551 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:32.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:32.652 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:32.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:32 smithi080 ceph-mon[40404]: osdmap e202: 8 total, 8 up, 7 in 2024-03-26T05:07:32.682 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:32 smithi080 ceph-mon[29691]: osdmap e202: 8 total, 8 up, 7 in 2024-03-26T05:07:32.684 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:32.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:32.786 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:32.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:32 smithi132 ceph-mon[33500]: osdmap e202: 8 total, 8 up, 7 in 2024-03-26T05:07:32.819 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:32.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:32.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:32.918 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:07:32.926061+0000 min lat: 0.00124249 max lat: 0.162251 avg lat: 0.00959381 2024-03-26T05:07:32.919 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:07:32.919 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 40 15 66696 66681 104.167 75.8125 0.00475335 0.00959381 2024-03-26T05:07:32.921 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:32.953 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:32.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:33.055 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:33.093 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:33.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:33.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:33.195 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:33.223 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:33.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:33.228 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:07:33.324 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:33.354 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:33.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:33.456 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:33.485 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:33.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:33.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:33.587 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:33.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:33 smithi080 ceph-mon[40404]: pgmap v311: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-26T05:07:33.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:33 smithi080 ceph-mon[40404]: 4.5 scrub starts 2024-03-26T05:07:33.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:33 smithi080 ceph-mon[40404]: 4.5 scrub ok 2024-03-26T05:07:33.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:33 smithi080 ceph-mon[29691]: pgmap v311: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-26T05:07:33.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:33 smithi080 ceph-mon[29691]: 4.5 scrub starts 2024-03-26T05:07:33.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:33 smithi080 ceph-mon[29691]: 4.5 scrub ok 2024-03-26T05:07:33.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:33.635 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:33.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:33.736 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:33.764 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:33.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:33.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:33 smithi132 ceph-mon[33500]: pgmap v311: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-26T05:07:33.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:33 smithi132 ceph-mon[33500]: 4.5 scrub starts 2024-03-26T05:07:33.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:33 smithi132 ceph-mon[33500]: 4.5 scrub ok 2024-03-26T05:07:33.865 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:33.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:33.900 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:07:33.904 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:07:33.916 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 41 16 67782 67766 103.28 67.8125 0.0129837 0.00967702 2024-03-26T05:07:33.995 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:34.032 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:34.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:34.134 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:34.154 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:07:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:07:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:07:34.188 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:34.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:34.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:34.290 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:34.320 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:34.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:34.422 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:34.465 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:34.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:34.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:34.568 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:34.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:34.609 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:34.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:34.710 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:34.753 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:34.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:34.855 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:34.887 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:34.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:34.917 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 42 16 68987 68971 102.614 75.3125 0.0145751 0.00973938 2024-03-26T05:07:34.936 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-03-26T05:07:33.838831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4135393630,"num_objects":63322,"num_object_clones":0,"num_object_copies":189966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63322,"num_whiteouts":0,"num_read":1641,"num_read_kb":1455,"num_write":63899,"num_write_kb":4040349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":1773004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63942,"ondisk_log_size":63942,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13306096,"kb_used_data":11991144,"kb_used_omap":10,"kb_used_meta":1314869,"kb_avail":642909968,"statfs":{"total":671965249536,"available":658339807232,"internally_reserved":0,"allocated":12278931456,"data_stored":12276183612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1346425963},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,6,4,3,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":403046400,"num_objects":6150,"num_object_clones":0,"num_object_copies":18450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6150,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6150,"num_write_kb":393600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.409552"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":297,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.517886+0000","last_change":"2024-03-26T05:06:08.113386+0000","last_active":"2024-03-26T05:07:29.517886+0000","last_peered":"2024-03-26T05:07:29.517886+0000","last_clean":"2024-03-26T05:07:29.517886+0000","last_became_active":"2024-03-26T05:05:36.407230+0000","last_became_peered":"2024-03-26T05:05:36.407230+0000","last_unstale":"2024-03-26T05:07:29.517886+0000","last_undegraded":"2024-03-26T05:07:29.517886+0000","last_fullsized":"2024-03-26T05:07:29.517886+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:08.113253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:08.113253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604611+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":300,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531254+0000","last_change":"2024-03-26T05:05:36.404129+0000","last_active":"2024-03-26T05:07:29.531254+0000","last_peered":"2024-03-26T05:07:29.531254+0000","last_clean":"2024-03-26T05:07:29.531254+0000","last_became_active":"2024-03-26T05:05:36.403620+0000","last_became_peered":"2024-03-26T05:05:36.403620+0000","last_unstale":"2024-03-26T05:07:29.531254+0000","last_undegraded":"2024-03-26T05:07:29.531254+0000","last_fullsized":"2024-03-26T05:07:29.531254+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:02.872031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:02.872031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:33.316795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":310,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.539072+0000","last_change":"2024-03-26T05:05:36.411757+0000","last_active":"2024-03-26T05:07:29.539072+0000","last_peered":"2024-03-26T05:07:29.539072+0000","last_clean":"2024-03-26T05:07:29.539072+0000","last_became_active":"2024-03-26T05:05:36.411231+0000","last_became_peered":"2024-03-26T05:05:36.411231+0000","last_unstale":"2024-03-26T05:07:29.539072+0000","last_undegraded":"2024-03-26T05:07:29.539072+0000","last_fullsized":"2024-03-26T05:07:29.539072+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.016730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.016730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'15","reported_seq":327,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.517642+0000","last_change":"2024-03-26T05:06:52.298546+0000","last_active":"2024-03-26T05:07:29.517642+0000","last_peered":"2024-03-26T05:07:29.517642+0000","last_clean":"2024-03-26T05:07:29.517642+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:07:29.517642+0000","last_undegraded":"2024-03-26T05:07:29.517642+0000","last_fullsized":"2024-03-26T05:07:29.517642+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:06:52.298418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:52.298418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:17.770238+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":282,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.546322+0000","last_change":"2024-03-26T05:06:37.226644+0000","last_active":"2024-03-26T05:07:29.546322+0000","last_peered":"2024-03-26T05:07:29.546322+0000","last_clean":"2024-03-26T05:07:29.546322+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:07:29.546322+0000","last_undegraded":"2024-03-26T05:07:29.546322+0000","last_fullsized":"2024-03-26T05:07:29.546322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:37.226521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:37.226521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:50.929750+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":287,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.517513+0000","last_change":"2024-03-26T05:05:09.143423+0000","last_active":"2024-03-26T05:07:29.517513+0000","last_peered":"2024-03-26T05:07:29.517513+0000","last_clean":"2024-03-26T05:07:29.517513+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:07:29.517513+0000","last_undegraded":"2024-03-26T05:07:29.517513+0000","last_fullsized":"2024-03-26T05:07:29.517513+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:09.143305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:09.143305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.536714+0000","last_change":"2024-03-26T05:05:12.713135+0000","last_active":"2024-03-26T05:07:29.536714+0000","last_peered":"2024-03-26T05:07:29.536714+0000","last_clean":"2024-03-26T05:07:29.536714+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:07:29.536714+0000","last_undegraded":"2024-03-26T05:07:29.536714+0000","last_fullsized":"2024-03-26T05:07:29.536714+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.713011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551736+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'39","reported_seq":363,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.545380+0000","last_change":"2024-03-26T05:04:20.637111+0000","last_active":"2024-03-26T05:07:29.545380+0000","last_peered":"2024-03-26T05:07:29.545380+0000","last_clean":"2024-03-26T05:07:29.545380+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:07:29.545380+0000","last_undegraded":"2024-03-26T05:07:29.545380+0000","last_fullsized":"2024-03-26T05:07:29.545380+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'39","last_scrub_stamp":"2024-03-26T05:04:20.636687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:20.636687+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551331+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":291,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.539375+0000","last_change":"2024-03-26T05:04:58.964528+0000","last_active":"2024-03-26T05:07:29.539375+0000","last_peered":"2024-03-26T05:07:29.539375+0000","last_clean":"2024-03-26T05:07:29.539375+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:07:29.539375+0000","last_undegraded":"2024-03-26T05:07:29.539375+0000","last_fullsized":"2024-03-26T05:07:29.539375+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:58.963801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.963801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":92,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.542035+0000","last_change":"2024-03-26T05:05:36.409043+0000","last_active":"2024-03-26T05:07:29.542035+0000","last_peered":"2024-03-26T05:07:29.542035+0000","last_clean":"2024-03-26T05:07:29.542035+0000","last_became_active":"2024-03-26T05:05:36.408618+0000","last_became_peered":"2024-03-26T05:05:36.408618+0000","last_unstale":"2024-03-26T05:07:29.542035+0000","last_undegraded":"2024-03-26T05:07:29.542035+0000","last_fullsized":"2024-03-26T05:07:29.542035+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:10.922975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:10.922975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":101,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.518414+0000","last_change":"2024-03-26T05:06:38.336990+0000","last_active":"2024-03-26T05:07:29.518414+0000","last_peered":"2024-03-26T05:07:29.518414+0000","last_clean":"2024-03-26T05:07:29.518414+0000","last_became_active":"2024-03-26T05:05:36.411627+0000","last_became_peered":"2024-03-26T05:05:36.411627+0000","last_unstale":"2024-03-26T05:07:29.518414+0000","last_undegraded":"2024-03-26T05:07:29.518414+0000","last_fullsized":"2024-03-26T05:07:29.518414+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:38.336853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:38.336853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:54.884679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"75'21","reported_seq":339,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.518022+0000","last_change":"2024-03-26T05:05:53.230294+0000","last_active":"2024-03-26T05:07:29.518022+0000","last_peered":"2024-03-26T05:07:29.518022+0000","last_clean":"2024-03-26T05:07:29.518022+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:07:29.518022+0000","last_undegraded":"2024-03-26T05:07:29.518022+0000","last_fullsized":"2024-03-26T05:07:29.518022+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:05:53.230145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:53.230145+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604580+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":92,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531081+0000","last_change":"2024-03-26T05:05:36.424628+0000","last_active":"2024-03-26T05:07:29.531081+0000","last_peered":"2024-03-26T05:07:29.531081+0000","last_clean":"2024-03-26T05:07:29.531081+0000","last_became_active":"2024-03-26T05:05:36.424269+0000","last_became_peered":"2024-03-26T05:05:36.424269+0000","last_unstale":"2024-03-26T05:07:29.531081+0000","last_undegraded":"2024-03-26T05:07:29.531081+0000","last_fullsized":"2024-03-26T05:07:29.531081+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:12.931045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.931045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:33.316892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":293,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.528856+0000","last_change":"2024-03-26T05:05:13.726750+0000","last_active":"2024-03-26T05:07:29.528856+0000","last_peered":"2024-03-26T05:07:29.528856+0000","last_clean":"2024-03-26T05:07:29.528856+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:07:29.528856+0000","last_undegraded":"2024-03-26T05:07:29.528856+0000","last_fullsized":"2024-03-26T05:07:29.528856+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:13.726612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:13.726612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673363+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":77,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.521564+0000","last_change":"2024-03-26T05:06:06.962791+0000","last_active":"2024-03-26T05:07:29.521564+0000","last_peered":"2024-03-26T05:07:29.521564+0000","last_clean":"2024-03-26T05:07:29.521564+0000","last_became_active":"2024-03-26T05:06:06.518987+0000","last_became_peered":"2024-03-26T05:06:06.518987+0000","last_unstale":"2024-03-26T05:07:29.521564+0000","last_undegraded":"2024-03-26T05:07:29.521564+0000","last_fullsized":"2024-03-26T05:07:29.521564+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:06.962665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:06.962665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.234162+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"75'11","reported_seq":313,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.517784+0000","last_change":"2024-03-26T05:05:44.172448+0000","last_active":"2024-03-26T05:07:29.517784+0000","last_peered":"2024-03-26T05:07:29.517784+0000","last_clean":"2024-03-26T05:07:29.517784+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:07:29.517784+0000","last_undegraded":"2024-03-26T05:07:29.517784+0000","last_fullsized":"2024-03-26T05:07:29.517784+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-26T05:05:44.172311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:44.172311+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604264+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.518376+0000","last_change":"2024-03-26T05:06:51.272075+0000","last_active":"2024-03-26T05:07:29.518376+0000","last_peered":"2024-03-26T05:07:29.518376+0000","last_clean":"2024-03-26T05:07:29.518376+0000","last_became_active":"2024-03-26T05:05:36.416850+0000","last_became_peered":"2024-03-26T05:05:36.416850+0000","last_unstale":"2024-03-26T05:07:29.518376+0000","last_undegraded":"2024-03-26T05:07:29.518376+0000","last_fullsized":"2024-03-26T05:07:29.518376+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:51.271966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:06:51.271966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:59.127606+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":294,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.532499+0000","last_change":"2024-03-26T05:06:00.105160+0000","last_active":"2024-03-26T05:07:29.532499+0000","last_peered":"2024-03-26T05:07:29.532499+0000","last_clean":"2024-03-26T05:07:29.532499+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:07:29.532499+0000","last_undegraded":"2024-03-26T05:07:29.532499+0000","last_fullsized":"2024-03-26T05:07:29.532499+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:00.104920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:00.104920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:33.316984+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.542283+0000","last_change":"2024-03-26T05:05:08.044042+0000","last_active":"2024-03-26T05:07:29.542283+0000","last_peered":"2024-03-26T05:07:29.542283+0000","last_clean":"2024-03-26T05:07:29.542283+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:07:29.542283+0000","last_undegraded":"2024-03-26T05:07:29.542283+0000","last_fullsized":"2024-03-26T05:07:29.542283+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:08.043732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:09.914470+0000","last_clean_scrub_stamp":"2024-03-26T05:05:08.043732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":331,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.521719+0000","last_change":"2024-03-26T05:06:45.225054+0000","last_active":"2024-03-26T05:07:29.521719+0000","last_peered":"2024-03-26T05:07:29.521719+0000","last_clean":"2024-03-26T05:07:29.521719+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:07:29.521719+0000","last_undegraded":"2024-03-26T05:07:29.521719+0000","last_fullsized":"2024-03-26T05:07:29.521719+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:06:45.224956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.224956+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:58.091504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":297,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.546924+0000","last_change":"2024-03-26T05:06:02.957611+0000","last_active":"2024-03-26T05:07:29.546924+0000","last_peered":"2024-03-26T05:07:29.546924+0000","last_clean":"2024-03-26T05:07:29.546924+0000","last_became_active":"2024-03-26T05:05:36.427435+0000","last_became_peered":"2024-03-26T05:05:36.427435+0000","last_unstale":"2024-03-26T05:07:29.546924+0000","last_undegraded":"2024-03-26T05:07:29.546924+0000","last_fullsized":"2024-03-26T05:07:29.546924+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:02.957488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:05.268043+0000","last_clean_scrub_stamp":"2024-03-26T05:06:02.957488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551839+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.518045+0000","last_change":"2024-03-26T05:06:48.271761+0000","last_active":"2024-03-26T05:07:29.518045+0000","last_peered":"2024-03-26T05:07:29.518045+0000","last_clean":"2024-03-26T05:07:29.518045+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:07:29.518045+0000","last_undegraded":"2024-03-26T05:07:29.518045+0000","last_fullsized":"2024-03-26T05:07:29.518045+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:48.271668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:48.271668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:15.551094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":311,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.518135+0000","last_change":"2024-03-26T05:06:00.026433+0000","last_active":"2024-03-26T05:07:29.518135+0000","last_peered":"2024-03-26T05:07:29.518135+0000","last_clean":"2024-03-26T05:07:29.518135+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:07:29.518135+0000","last_undegraded":"2024-03-26T05:07:29.518135+0000","last_fullsized":"2024-03-26T05:07:29.518135+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:00.026316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:06:00.026316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604506+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'20","reported_seq":327,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.536320+0000","last_change":"2024-03-26T05:04:07.528781+0000","last_active":"2024-03-26T05:07:29.536320+0000","last_peered":"2024-03-26T05:07:29.536320+0000","last_clean":"2024-03-26T05:07:29.536320+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:07:29.536320+0000","last_undegraded":"2024-03-26T05:07:29.536320+0000","last_fullsized":"2024-03-26T05:07:29.536320+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:04:07.528619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:07.528619+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551388+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":300,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.529835+0000","last_change":"2024-03-26T05:05:41.875836+0000","last_active":"2024-03-26T05:07:29.529835+0000","last_peered":"2024-03-26T05:07:29.529835+0000","last_clean":"2024-03-26T05:07:29.529835+0000","last_became_active":"2024-03-26T05:05:36.416944+0000","last_became_peered":"2024-03-26T05:05:36.416944+0000","last_unstale":"2024-03-26T05:07:29.529835+0000","last_undegraded":"2024-03-26T05:07:29.529835+0000","last_fullsized":"2024-03-26T05:07:29.529835+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:05:41.875743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:41.875743+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673466+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":407,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.532034+0000","last_change":"2024-03-26T05:06:53.544685+0000","last_active":"2024-03-26T05:07:29.532034+0000","last_peered":"2024-03-26T05:07:29.532034+0000","last_clean":"2024-03-26T05:07:29.532034+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:07:29.532034+0000","last_undegraded":"2024-03-26T05:07:29.532034+0000","last_fullsized":"2024-03-26T05:07:29.532034+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:06:53.544604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:53.544604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:00.641956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531331+0000","last_change":"2024-03-26T05:06:04.028509+0000","last_active":"2024-03-26T05:07:29.531331+0000","last_peered":"2024-03-26T05:07:29.531331+0000","last_clean":"2024-03-26T05:07:29.531331+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:07:29.531331+0000","last_undegraded":"2024-03-26T05:07:29.531331+0000","last_fullsized":"2024-03-26T05:07:29.531331+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:06:04.028383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:04.028383+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:33.316926+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'16","reported_seq":327,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.541023+0000","last_change":"2024-03-26T05:05:13.083706+0000","last_active":"2024-03-26T05:07:29.541023+0000","last_peered":"2024-03-26T05:07:29.541023+0000","last_clean":"2024-03-26T05:07:29.541023+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:07:29.541023+0000","last_undegraded":"2024-03-26T05:07:29.541023+0000","last_fullsized":"2024-03-26T05:07:29.541023+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:05:13.083420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:13.083420+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":296,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.532636+0000","last_change":"2024-03-26T05:05:36.403232+0000","last_active":"2024-03-26T05:07:29.532636+0000","last_peered":"2024-03-26T05:07:29.532636+0000","last_clean":"2024-03-26T05:07:29.532636+0000","last_became_active":"2024-03-26T05:05:36.402759+0000","last_became_peered":"2024-03-26T05:05:36.402759+0000","last_unstale":"2024-03-26T05:07:29.532636+0000","last_undegraded":"2024-03-26T05:07:29.532636+0000","last_fullsized":"2024-03-26T05:07:29.532636+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:55.814330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:55.814330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:33.316865+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":296,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.541294+0000","last_change":"2024-03-26T05:05:53.093643+0000","last_active":"2024-03-26T05:07:29.541294+0000","last_peered":"2024-03-26T05:07:29.541294+0000","last_clean":"2024-03-26T05:07:29.541294+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:07:29.541294+0000","last_undegraded":"2024-03-26T05:07:29.541294+0000","last_fullsized":"2024-03-26T05:07:29.541294+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:53.093539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:53.093539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":302,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.545231+0000","last_change":"2024-03-26T05:06:00.960334+0000","last_active":"2024-03-26T05:07:29.545231+0000","last_peered":"2024-03-26T05:07:29.545231+0000","last_clean":"2024-03-26T05:07:29.545231+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:07:29.545231+0000","last_undegraded":"2024-03-26T05:07:29.545231+0000","last_fullsized":"2024-03-26T05:07:29.545231+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:00.960212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:00.960212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551808+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'17","reported_seq":377,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.532066+0000","last_change":"2024-03-26T05:06:47.435312+0000","last_active":"2024-03-26T05:07:29.532066+0000","last_peered":"2024-03-26T05:07:29.532066+0000","last_clean":"2024-03-26T05:07:29.532066+0000","last_became_active":"2024-03-26T05:05:36.410009+0000","last_became_peered":"2024-03-26T05:05:36.410009+0000","last_unstale":"2024-03-26T05:07:29.532066+0000","last_undegraded":"2024-03-26T05:07:29.532066+0000","last_fullsized":"2024-03-26T05:07:29.532066+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:06:47.435205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:47.435205+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:14.788909+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":282,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.528807+0000","last_change":"2024-03-26T05:04:27.007041+0000","last_active":"2024-03-26T05:07:29.528807+0000","last_peered":"2024-03-26T05:07:29.528807+0000","last_clean":"2024-03-26T05:07:29.528807+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:07:29.528807+0000","last_undegraded":"2024-03-26T05:07:29.528807+0000","last_fullsized":"2024-03-26T05:07:29.528807+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:27.006904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:27.006904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673266+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":302,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531525+0000","last_change":"2024-03-26T05:06:42.481238+0000","last_active":"2024-03-26T05:07:29.531525+0000","last_peered":"2024-03-26T05:07:29.531525+0000","last_clean":"2024-03-26T05:07:29.531525+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:07:29.531525+0000","last_undegraded":"2024-03-26T05:07:29.531525+0000","last_fullsized":"2024-03-26T05:07:29.531525+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:42.481122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:42.481122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:08.491912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":309,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.539267+0000","last_change":"2024-03-26T05:05:59.138866+0000","last_active":"2024-03-26T05:07:29.539267+0000","last_peered":"2024-03-26T05:07:29.539267+0000","last_clean":"2024-03-26T05:07:29.539267+0000","last_became_active":"2024-03-26T05:05:36.423767+0000","last_became_peered":"2024-03-26T05:05:36.423767+0000","last_unstale":"2024-03-26T05:07:29.539267+0000","last_undegraded":"2024-03-26T05:07:29.539267+0000","last_fullsized":"2024-03-26T05:07:29.539267+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:59.138510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:59.138510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"100'6","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.545327+0000","last_change":"2024-03-26T05:06:45.224407+0000","last_active":"2024-03-26T05:07:29.545327+0000","last_peered":"2024-03-26T05:07:29.545327+0000","last_clean":"2024-03-26T05:07:29.545327+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:07:29.545327+0000","last_undegraded":"2024-03-26T05:07:29.545327+0000","last_fullsized":"2024-03-26T05:07:29.545327+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'6","last_scrub_stamp":"2024-03-26T05:06:45.224296+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.224296+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:50.976141+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":290,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.546510+0000","last_change":"2024-03-26T05:06:49.226016+0000","last_active":"2024-03-26T05:07:29.546510+0000","last_peered":"2024-03-26T05:07:29.546510+0000","last_clean":"2024-03-26T05:07:29.546510+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:07:29.546510+0000","last_undegraded":"2024-03-26T05:07:29.546510+0000","last_fullsized":"2024-03-26T05:07:29.546510+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_clean_scrub_stamp":"2024-03-26T05:06:49.225897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:16.804230+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":294,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.529114+0000","last_change":"2024-03-26T05:05:12.747692+0000","last_active":"2024-03-26T05:07:29.529114+0000","last_peered":"2024-03-26T05:07:29.529114+0000","last_clean":"2024-03-26T05:07:29.529114+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:07:29.529114+0000","last_undegraded":"2024-03-26T05:07:29.529114+0000","last_fullsized":"2024-03-26T05:07:29.529114+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:12.747554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.747554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":122,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.529744+0000","last_change":"2024-03-26T05:06:56.886716+0000","last_active":"2024-03-26T05:07:29.529744+0000","last_peered":"2024-03-26T05:07:29.529744+0000","last_clean":"2024-03-26T05:07:29.529744+0000","last_became_active":"2024-03-26T05:05:36.417290+0000","last_became_peered":"2024-03-26T05:05:36.417290+0000","last_unstale":"2024-03-26T05:07:29.529744+0000","last_undegraded":"2024-03-26T05:07:29.529744+0000","last_fullsized":"2024-03-26T05:07:29.529744+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:06:56.886632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:07.878905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'17","reported_seq":322,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.545295+0000","last_change":"2024-03-26T05:04:09.559560+0000","last_active":"2024-03-26T05:07:29.545295+0000","last_peered":"2024-03-26T05:07:29.545295+0000","last_clean":"2024-03-26T05:07:29.545295+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:07:29.545295+0000","last_undegraded":"2024-03-26T05:07:29.545295+0000","last_fullsized":"2024-03-26T05:07:29.545295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:04:09.559412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:09.559412+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551416+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.539013+0000","last_change":"2024-03-26T05:06:54.126622+0000","last_active":"2024-03-26T05:07:29.539013+0000","last_peered":"2024-03-26T05:07:29.539013+0000","last_clean":"2024-03-26T05:07:29.539013+0000","last_became_active":"2024-03-26T05:05:36.413880+0000","last_became_peered":"2024-03-26T05:05:36.413880+0000","last_unstale":"2024-03-26T05:07:29.539013+0000","last_undegraded":"2024-03-26T05:07:29.539013+0000","last_fullsized":"2024-03-26T05:07:29.539013+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:54.126517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:54.126517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:06.674857+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":286,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.546231+0000","last_change":"2024-03-26T05:05:04.767968+0000","last_active":"2024-03-26T05:07:29.546231+0000","last_peered":"2024-03-26T05:07:29.546231+0000","last_clean":"2024-03-26T05:07:29.546231+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:07:29.546231+0000","last_undegraded":"2024-03-26T05:07:29.546231+0000","last_fullsized":"2024-03-26T05:07:29.546231+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.767790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551494+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":293,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.529547+0000","last_change":"2024-03-26T05:05:04.764827+0000","last_active":"2024-03-26T05:07:29.529547+0000","last_peered":"2024-03-26T05:07:29.529547+0000","last_clean":"2024-03-26T05:07:29.529547+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:07:29.529547+0000","last_undegraded":"2024-03-26T05:07:29.529547+0000","last_fullsized":"2024-03-26T05:07:29.529547+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.764742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673335+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":315,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.533734+0000","last_change":"2024-03-26T05:05:14.064119+0000","last_active":"2024-03-26T05:07:29.533734+0000","last_peered":"2024-03-26T05:07:29.533734+0000","last_clean":"2024-03-26T05:07:29.533734+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:07:29.533734+0000","last_undegraded":"2024-03-26T05:07:29.533734+0000","last_fullsized":"2024-03-26T05:07:29.533734+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-26T05:05:14.063125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:14.063125+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:33.316766+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":289,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.545102+0000","last_change":"2024-03-26T05:05:36.426512+0000","last_active":"2024-03-26T05:07:29.545102+0000","last_peered":"2024-03-26T05:07:29.545102+0000","last_clean":"2024-03-26T05:07:29.545102+0000","last_became_active":"2024-03-26T05:05:36.425590+0000","last_became_peered":"2024-03-26T05:05:36.425590+0000","last_unstale":"2024-03-26T05:07:29.545102+0000","last_undegraded":"2024-03-26T05:07:29.545102+0000","last_fullsized":"2024-03-26T05:07:29.545102+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:58.834141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.834141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551650+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":309,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.533685+0000","last_change":"2024-03-26T05:06:42.246793+0000","last_active":"2024-03-26T05:07:29.533685+0000","last_peered":"2024-03-26T05:07:29.533685+0000","last_clean":"2024-03-26T05:07:29.533685+0000","last_became_active":"2024-03-26T05:05:36.416479+0000","last_became_peered":"2024-03-26T05:05:36.416479+0000","last_unstale":"2024-03-26T05:07:29.533685+0000","last_undegraded":"2024-03-26T05:07:29.533685+0000","last_fullsized":"2024-03-26T05:07:29.533685+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:42.246721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:06:42.246721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:43.713447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":296,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.517817+0000","last_change":"2024-03-26T05:05:42.110557+0000","last_active":"2024-03-26T05:07:29.517817+0000","last_peered":"2024-03-26T05:07:29.517817+0000","last_clean":"2024-03-26T05:07:29.517817+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:07:29.517817+0000","last_undegraded":"2024-03-26T05:07:29.517817+0000","last_fullsized":"2024-03-26T05:07:29.517817+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:42.110480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:42.110480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"95'16","reported_seq":324,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.518069+0000","last_change":"2024-03-26T05:05:41.097931+0000","last_active":"2024-03-26T05:07:29.518069+0000","last_peered":"2024-03-26T05:07:29.518069+0000","last_clean":"2024-03-26T05:07:29.518069+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:07:29.518069+0000","last_undegraded":"2024-03-26T05:07:29.518069+0000","last_fullsized":"2024-03-26T05:07:29.518069+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:05:41.097824+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:05:41.097824+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604353+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":291,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.540741+0000","last_change":"2024-03-26T05:06:04.057848+0000","last_active":"2024-03-26T05:07:29.540741+0000","last_peered":"2024-03-26T05:07:29.540741+0000","last_clean":"2024-03-26T05:07:29.540741+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:07:29.540741+0000","last_undegraded":"2024-03-26T05:07:29.540741+0000","last_fullsized":"2024-03-26T05:07:29.540741+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_clean_scrub_stamp":"2024-03-26T05:06:04.057358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"161'9","reported_seq":153,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.532351+0000","last_change":"2024-03-26T05:06:47.137171+0000","last_active":"2024-03-26T05:07:29.532351+0000","last_peered":"2024-03-26T05:07:29.532351+0000","last_clean":"2024-03-26T05:07:29.532351+0000","last_became_active":"2024-03-26T05:05:36.422468+0000","last_became_peered":"2024-03-26T05:05:36.422468+0000","last_unstale":"2024-03-26T05:07:29.532351+0000","last_undegraded":"2024-03-26T05:07:29.532351+0000","last_fullsized":"2024-03-26T05:07:29.532351+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'9","last_scrub_stamp":"2024-03-26T05:06:47.137084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:47.137084+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:47.227857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":302,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.529249+0000","last_change":"2024-03-26T05:05:08.856522+0000","last_active":"2024-03-26T05:07:29.529249+0000","last_peered":"2024-03-26T05:07:29.529249+0000","last_clean":"2024-03-26T05:07:29.529249+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:07:29.529249+0000","last_undegraded":"2024-03-26T05:07:29.529249+0000","last_fullsized":"2024-03-26T05:07:29.529249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:08.856401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:08.856401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673394+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'22","reported_seq":338,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.544127+0000","last_change":"2024-03-26T05:05:08.701087+0000","last_active":"2024-03-26T05:07:29.544127+0000","last_peered":"2024-03-26T05:07:29.544127+0000","last_clean":"2024-03-26T05:07:29.544127+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:07:29.544127+0000","last_undegraded":"2024-03-26T05:07:29.544127+0000","last_fullsized":"2024-03-26T05:07:29.544127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:05:08.700945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:08.700945+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551523+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":297,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.768601+0000","last_change":"2024-03-26T05:07:29.768641+0000","last_active":"2024-03-26T05:07:29.768601+0000","last_peered":"2024-03-26T05:07:29.768601+0000","last_clean":"2024-03-26T05:07:29.768601+0000","last_became_active":"2024-03-26T05:05:36.405705+0000","last_became_peered":"2024-03-26T05:05:36.405705+0000","last_unstale":"2024-03-26T05:07:29.768601+0000","last_undegraded":"2024-03-26T05:07:29.768601+0000","last_fullsized":"2024-03-26T05:07:29.768601+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:29.768389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:29.768389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:33.558469+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"120'9","reported_seq":392,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.544738+0000","last_change":"2024-03-26T05:06:39.209229+0000","last_active":"2024-03-26T05:07:29.544738+0000","last_peered":"2024-03-26T05:07:29.544738+0000","last_clean":"2024-03-26T05:07:29.544738+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:07:29.544738+0000","last_undegraded":"2024-03-26T05:07:29.544738+0000","last_fullsized":"2024-03-26T05:07:29.544738+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'9","last_scrub_stamp":"2024-03-26T05:06:39.209149+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:06:39.209149+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:51.462691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":312,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.539787+0000","last_change":"2024-03-26T05:06:55.105421+0000","last_active":"2024-03-26T05:07:29.539787+0000","last_peered":"2024-03-26T05:07:29.539787+0000","last_clean":"2024-03-26T05:07:29.539787+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:07:29.539787+0000","last_undegraded":"2024-03-26T05:07:29.539787+0000","last_fullsized":"2024-03-26T05:07:29.539787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:55.105291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:55.105291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:23.534351+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"98'20","reported_seq":209,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.529774+0000","last_change":"2024-03-26T05:06:49.846150+0000","last_active":"2024-03-26T05:07:29.529774+0000","last_peered":"2024-03-26T05:07:29.529774+0000","last_clean":"2024-03-26T05:07:29.529774+0000","last_became_active":"2024-03-26T05:05:36.422436+0000","last_became_peered":"2024-03-26T05:05:36.422436+0000","last_unstale":"2024-03-26T05:07:29.529774+0000","last_undegraded":"2024-03-26T05:07:29.529774+0000","last_fullsized":"2024-03-26T05:07:29.529774+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'20","last_scrub_stamp":"2024-03-26T05:06:49.846038+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:06:49.846038+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:01.733368+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":298,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.522003+0000","last_change":"2024-03-26T05:05:58.893557+0000","last_active":"2024-03-26T05:07:29.522003+0000","last_peered":"2024-03-26T05:07:29.522003+0000","last_clean":"2024-03-26T05:07:29.522003+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:07:29.522003+0000","last_undegraded":"2024-03-26T05:07:29.522003+0000","last_fullsized":"2024-03-26T05:07:29.522003+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:58.893447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:58.893447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:34.257139+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":286,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.546837+0000","last_change":"2024-03-26T05:04:59.846243+0000","last_active":"2024-03-26T05:07:29.546837+0000","last_peered":"2024-03-26T05:07:29.546837+0000","last_clean":"2024-03-26T05:07:29.546837+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:07:29.546837+0000","last_undegraded":"2024-03-26T05:07:29.546837+0000","last_fullsized":"2024-03-26T05:07:29.546837+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:59.846123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:04:59.846123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551470+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":294,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.528742+0000","last_change":"2024-03-26T05:04:15.786624+0000","last_active":"2024-03-26T05:07:29.528742+0000","last_peered":"2024-03-26T05:07:29.528742+0000","last_clean":"2024-03-26T05:07:29.528742+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:07:29.528742+0000","last_undegraded":"2024-03-26T05:07:29.528742+0000","last_fullsized":"2024-03-26T05:07:29.528742+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:15.786545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:15.786545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'21","reported_seq":339,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.541116+0000","last_change":"2024-03-26T05:05:12.090691+0000","last_active":"2024-03-26T05:07:29.541116+0000","last_peered":"2024-03-26T05:07:29.541116+0000","last_clean":"2024-03-26T05:07:29.541116+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:07:29.541116+0000","last_undegraded":"2024-03-26T05:07:29.541116+0000","last_fullsized":"2024-03-26T05:07:29.541116+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:05:12.089325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.089325+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":289,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.529498+0000","last_change":"2024-03-26T05:05:58.845334+0000","last_active":"2024-03-26T05:07:29.529498+0000","last_peered":"2024-03-26T05:07:29.529498+0000","last_clean":"2024-03-26T05:07:29.529498+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:07:29.529498+0000","last_undegraded":"2024-03-26T05:07:29.529498+0000","last_fullsized":"2024-03-26T05:07:29.529498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:58.845212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:58.845212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":92,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.536818+0000","last_change":"2024-03-26T05:05:36.416431+0000","last_active":"2024-03-26T05:07:29.536818+0000","last_peered":"2024-03-26T05:07:29.536818+0000","last_clean":"2024-03-26T05:07:29.536818+0000","last_became_active":"2024-03-26T05:05:36.416053+0000","last_became_peered":"2024-03-26T05:05:36.416053+0000","last_unstale":"2024-03-26T05:07:29.536818+0000","last_undegraded":"2024-03-26T05:07:29.536818+0000","last_fullsized":"2024-03-26T05:07:29.536818+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:09.946734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:09.946734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":327,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531271+0000","last_change":"2024-03-26T05:06:50.546347+0000","last_active":"2024-03-26T05:07:29.531271+0000","last_peered":"2024-03-26T05:07:29.531271+0000","last_clean":"2024-03-26T05:07:29.531271+0000","last_became_active":"2024-03-26T05:05:36.419068+0000","last_became_peered":"2024-03-26T05:05:36.419068+0000","last_unstale":"2024-03-26T05:07:29.531271+0000","last_undegraded":"2024-03-26T05:07:29.531271+0000","last_fullsized":"2024-03-26T05:07:29.531271+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_clean_scrub_stamp":"2024-03-26T05:06:50.546241+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:05.099002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'12","reported_seq":319,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.546967+0000","last_change":"2024-03-26T05:06:42.178592+0000","last_active":"2024-03-26T05:07:29.546967+0000","last_peered":"2024-03-26T05:07:29.546967+0000","last_clean":"2024-03-26T05:07:29.546967+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:07:29.546967+0000","last_undegraded":"2024-03-26T05:07:29.546967+0000","last_fullsized":"2024-03-26T05:07:29.546967+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:06:42.178446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:42.178446+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:59.083463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"201'3847","reported_seq":3891,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.747962+0000","last_change":"2024-03-26T05:06:42.023680+0000","last_active":"2024-03-26T05:07:29.747962+0000","last_peered":"2024-03-26T05:07:29.747962+0000","last_clean":"2024-03-26T05:07:29.747962+0000","last_became_active":"2024-03-26T05:06:42.023267+0000","last_became_peered":"2024-03-26T05:06:42.023267+0000","last_unstale":"2024-03-26T05:07:29.747962+0000","last_undegraded":"2024-03-26T05:07:29.747962+0000","last_fullsized":"2024-03-26T05:07:29.747962+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3847,"log_dups_size":0,"ondisk_log_size":3847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:42.225911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252116992,"num_objects":3847,"num_object_clones":0,"num_object_copies":11541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3847,"num_write_kb":246208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":158,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531384+0000","last_change":"2024-03-26T05:06:48.480757+0000","last_active":"2024-03-26T05:07:29.531384+0000","last_peered":"2024-03-26T05:07:29.531384+0000","last_clean":"2024-03-26T05:07:29.531384+0000","last_became_active":"2024-03-26T05:05:36.409998+0000","last_became_peered":"2024-03-26T05:05:36.409998+0000","last_unstale":"2024-03-26T05:07:29.531384+0000","last_undegraded":"2024-03-26T05:07:29.531384+0000","last_fullsized":"2024-03-26T05:07:29.531384+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:06:48.480649+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:06:48.480649+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:52.622566+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":323,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.529560+0000","last_change":"2024-03-26T05:05:06.832886+0000","last_active":"2024-03-26T05:07:29.529560+0000","last_peered":"2024-03-26T05:07:29.529560+0000","last_clean":"2024-03-26T05:07:29.529560+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:07:29.529560+0000","last_undegraded":"2024-03-26T05:07:29.529560+0000","last_fullsized":"2024-03-26T05:07:29.529560+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:05:06.832744+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:05:06.832744+0000","last_clean_scrub_stamp":"2024-03-26T05:05:06.832744+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673302+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'27","reported_seq":188,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.528947+0000","last_change":"2024-03-26T05:06:03.850352+0000","last_active":"2024-03-26T05:07:29.528947+0000","last_peered":"2024-03-26T05:07:29.528947+0000","last_clean":"2024-03-26T05:07:29.528947+0000","last_became_active":"2024-03-26T05:05:36.399795+0000","last_became_peered":"2024-03-26T05:05:36.399795+0000","last_unstale":"2024-03-26T05:07:29.528947+0000","last_undegraded":"2024-03-26T05:07:29.528947+0000","last_fullsized":"2024-03-26T05:07:29.528947+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'27","last_scrub_stamp":"2024-03-26T05:06:03.850202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:03.850202+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":388,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.528987+0000","last_change":"2024-03-26T05:05:57.829833+0000","last_active":"2024-03-26T05:07:29.528987+0000","last_peered":"2024-03-26T05:07:29.528987+0000","last_clean":"2024-03-26T05:07:29.528987+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:07:29.528987+0000","last_undegraded":"2024-03-26T05:07:29.528987+0000","last_fullsized":"2024-03-26T05:07:29.528987+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:05:57.829724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:57.829724+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"201'3992","reported_seq":4036,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.808100+0000","last_change":"2024-03-26T05:06:41.959963+0000","last_active":"2024-03-26T05:07:29.808100+0000","last_peered":"2024-03-26T05:07:29.808100+0000","last_clean":"2024-03-26T05:07:29.808100+0000","last_became_active":"2024-03-26T05:06:41.959580+0000","last_became_peered":"2024-03-26T05:06:41.959580+0000","last_unstale":"2024-03-26T05:07:29.808100+0000","last_undegraded":"2024-03-26T05:07:29.808100+0000","last_fullsized":"2024-03-26T05:07:29.808100+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:46.553390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261619712,"num_objects":3992,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3992,"num_write_kb":255488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":291,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.546417+0000","last_change":"2024-03-26T05:06:38.198437+0000","last_active":"2024-03-26T05:07:29.546417+0000","last_peered":"2024-03-26T05:07:29.546417+0000","last_clean":"2024-03-26T05:07:29.546417+0000","last_became_active":"2024-03-26T05:00:28.985917+0000","last_became_peered":"2024-03-26T05:00:28.985917+0000","last_unstale":"2024-03-26T05:07:29.546417+0000","last_undegraded":"2024-03-26T05:07:29.546417+0000","last_fullsized":"2024-03-26T05:07:29.546417+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:38.198342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:38.198342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:58.765618+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":302,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.529746+0000","last_change":"2024-03-26T05:05:53.920453+0000","last_active":"2024-03-26T05:07:29.529746+0000","last_peered":"2024-03-26T05:07:29.529746+0000","last_clean":"2024-03-26T05:07:29.529746+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:07:29.529746+0000","last_undegraded":"2024-03-26T05:07:29.529746+0000","last_fullsized":"2024-03-26T05:07:29.529746+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:53.920375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:53.920375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673500+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"188'31","reported_seq":354,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.517814+0000","last_change":"2024-03-26T05:06:50.301509+0000","last_active":"2024-03-26T05:07:29.517814+0000","last_peered":"2024-03-26T05:07:29.517814+0000","last_clean":"2024-03-26T05:07:29.517814+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:07:29.517814+0000","last_undegraded":"2024-03-26T05:07:29.517814+0000","last_fullsized":"2024-03-26T05:07:29.517814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'31","last_scrub_stamp":"2024-03-26T05:06:50.301394+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:06:50.301394+0000","last_clean_scrub_stamp":"2024-03-26T05:06:50.301394+0000","objects_scrubbed":7,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:00.240874+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.523053+0000","last_change":"2024-03-26T05:06:53.117526+0000","last_active":"2024-03-26T05:07:29.523053+0000","last_peered":"2024-03-26T05:07:29.523053+0000","last_clean":"2024-03-26T05:07:29.523053+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:07:29.523053+0000","last_undegraded":"2024-03-26T05:07:29.523053+0000","last_fullsized":"2024-03-26T05:07:29.523053+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:53.117385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.432732+0000","last_clean_scrub_stamp":"2024-03-26T05:06:53.117385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:21.618608+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"201'3966","reported_seq":4008,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:30.318606+0000","last_change":"2024-03-26T05:06:41.963528+0000","last_active":"2024-03-26T05:07:30.318606+0000","last_peered":"2024-03-26T05:07:30.318606+0000","last_clean":"2024-03-26T05:07:30.318606+0000","last_became_active":"2024-03-26T05:06:41.962275+0000","last_became_peered":"2024-03-26T05:06:41.962275+0000","last_unstale":"2024-03-26T05:07:30.318606+0000","last_undegraded":"2024-03-26T05:07:30.318606+0000","last_fullsized":"2024-03-26T05:07:30.318606+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:43.226191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259915776,"num_objects":3966,"num_object_clones":0,"num_object_copies":11898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3966,"num_write_kb":253824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":290,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.528618+0000","last_change":"2024-03-26T05:05:46.845330+0000","last_active":"2024-03-26T05:07:29.528618+0000","last_peered":"2024-03-26T05:07:29.528618+0000","last_clean":"2024-03-26T05:07:29.528618+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:07:29.528618+0000","last_undegraded":"2024-03-26T05:07:29.528618+0000","last_fullsized":"2024-03-26T05:07:29.528618+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:46.845184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:23.071667+0000","last_clean_scrub_stamp":"2024-03-26T05:05:46.845184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673515+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":309,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.520608+0000","last_change":"2024-03-26T05:05:57.929700+0000","last_active":"2024-03-26T05:07:29.520608+0000","last_peered":"2024-03-26T05:07:29.520608+0000","last_clean":"2024-03-26T05:07:29.520608+0000","last_became_active":"2024-03-26T05:05:36.410132+0000","last_became_peered":"2024-03-26T05:05:36.410132+0000","last_unstale":"2024-03-26T05:07:29.520608+0000","last_undegraded":"2024-03-26T05:07:29.520608+0000","last_fullsized":"2024-03-26T05:07:29.520608+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:57.929170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:57.929170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:34.257114+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":300,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.529774+0000","last_change":"2024-03-26T05:05:36.422622+0000","last_active":"2024-03-26T05:07:29.529774+0000","last_peered":"2024-03-26T05:07:29.529774+0000","last_clean":"2024-03-26T05:07:29.529774+0000","last_became_active":"2024-03-26T05:05:36.422084+0000","last_became_peered":"2024-03-26T05:05:36.422084+0000","last_unstale":"2024-03-26T05:07:29.529774+0000","last_undegraded":"2024-03-26T05:07:29.529774+0000","last_fullsized":"2024-03-26T05:07:29.529774+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:23.884406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:23.884406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673232+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'25","reported_seq":349,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531511+0000","last_change":"2024-03-26T05:06:45.476643+0000","last_active":"2024-03-26T05:07:29.531511+0000","last_peered":"2024-03-26T05:07:29.531511+0000","last_clean":"2024-03-26T05:07:29.531511+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:07:29.531511+0000","last_undegraded":"2024-03-26T05:07:29.531511+0000","last_fullsized":"2024-03-26T05:07:29.531511+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:06:45.476460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.476460+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:07.614363+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"201'3967","reported_seq":4011,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.962182+0000","last_change":"2024-03-26T05:06:41.961429+0000","last_active":"2024-03-26T05:07:29.962182+0000","last_peered":"2024-03-26T05:07:29.962182+0000","last_clean":"2024-03-26T05:07:29.962182+0000","last_became_active":"2024-03-26T05:06:41.961141+0000","last_became_peered":"2024-03-26T05:06:41.961141+0000","last_unstale":"2024-03-26T05:07:29.962182+0000","last_undegraded":"2024-03-26T05:07:29.962182+0000","last_fullsized":"2024-03-26T05:07:29.962182+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3967,"log_dups_size":0,"ondisk_log_size":3967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:46.766743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259981312,"num_objects":3967,"num_object_clones":0,"num_object_copies":11901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3967,"num_write_kb":253888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":292,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.540771+0000","last_change":"2024-03-26T05:05:55.120808+0000","last_active":"2024-03-26T05:07:29.540771+0000","last_peered":"2024-03-26T05:07:29.540771+0000","last_clean":"2024-03-26T05:07:29.540771+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:07:29.540771+0000","last_undegraded":"2024-03-26T05:07:29.540771+0000","last_fullsized":"2024-03-26T05:07:29.540771+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:55.120731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:55.120731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.527387+0000","last_change":"2024-03-26T05:06:51.161774+0000","last_active":"2024-03-26T05:07:29.527387+0000","last_peered":"2024-03-26T05:07:29.527387+0000","last_clean":"2024-03-26T05:07:29.527387+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:07:29.527387+0000","last_undegraded":"2024-03-26T05:07:29.527387+0000","last_fullsized":"2024-03-26T05:07:29.527387+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:51.161669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:51.161669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:04.605897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":93,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.544304+0000","last_change":"2024-03-26T05:05:36.415361+0000","last_active":"2024-03-26T05:07:29.544304+0000","last_peered":"2024-03-26T05:07:29.544304+0000","last_clean":"2024-03-26T05:07:29.544304+0000","last_became_active":"2024-03-26T05:05:36.414992+0000","last_became_peered":"2024-03-26T05:05:36.414992+0000","last_unstale":"2024-03-26T05:07:29.544304+0000","last_undegraded":"2024-03-26T05:07:29.544304+0000","last_fullsized":"2024-03-26T05:07:29.544304+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:01.867674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:01.867674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"75'15","reported_seq":328,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.542414+0000","last_change":"2024-03-26T05:05:42.260332+0000","last_active":"2024-03-26T05:07:29.542414+0000","last_peered":"2024-03-26T05:07:29.542414+0000","last_clean":"2024-03-26T05:07:29.542414+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:07:29.542414+0000","last_undegraded":"2024-03-26T05:07:29.542414+0000","last_fullsized":"2024-03-26T05:07:29.542414+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:05:42.260202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:42.260202+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"201'4033","reported_seq":4077,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:30.197794+0000","last_change":"2024-03-26T05:06:41.964200+0000","last_active":"2024-03-26T05:07:30.197794+0000","last_peered":"2024-03-26T05:07:30.197794+0000","last_clean":"2024-03-26T05:07:30.197794+0000","last_became_active":"2024-03-26T05:06:41.963415+0000","last_became_peered":"2024-03-26T05:06:41.963415+0000","last_unstale":"2024-03-26T05:07:30.197794+0000","last_undegraded":"2024-03-26T05:07:30.197794+0000","last_fullsized":"2024-03-26T05:07:30.197794+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4033,"log_dups_size":0,"ondisk_log_size":4033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:54.093502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264306688,"num_objects":4033,"num_object_clones":0,"num_object_copies":12099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4033,"num_write_kb":258112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":283,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.536973+0000","last_change":"2024-03-26T05:04:11.571539+0000","last_active":"2024-03-26T05:07:29.536973+0000","last_peered":"2024-03-26T05:07:29.536973+0000","last_clean":"2024-03-26T05:07:29.536973+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:07:29.536973+0000","last_undegraded":"2024-03-26T05:07:29.536973+0000","last_fullsized":"2024-03-26T05:07:29.536973+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:11.571425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:11.571425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":122,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.533381+0000","last_change":"2024-03-26T05:05:36.437117+0000","last_active":"2024-03-26T05:07:29.533381+0000","last_peered":"2024-03-26T05:07:29.533381+0000","last_clean":"2024-03-26T05:07:29.533381+0000","last_became_active":"2024-03-26T05:05:36.412450+0000","last_became_peered":"2024-03-26T05:05:36.412450+0000","last_unstale":"2024-03-26T05:07:29.533381+0000","last_undegraded":"2024-03-26T05:07:29.533381+0000","last_fullsized":"2024-03-26T05:07:29.533381+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:04:58.935568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.935568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:33.316667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":296,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.517856+0000","last_change":"2024-03-26T05:05:46.234486+0000","last_active":"2024-03-26T05:07:29.517856+0000","last_peered":"2024-03-26T05:07:29.517856+0000","last_clean":"2024-03-26T05:07:29.517856+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:07:29.517856+0000","last_undegraded":"2024-03-26T05:07:29.517856+0000","last_fullsized":"2024-03-26T05:07:29.517856+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:46.234374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:46.234374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604539+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'34","reported_seq":453,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531414+0000","last_change":"2024-03-26T05:06:54.128746+0000","last_active":"2024-03-26T05:07:29.531414+0000","last_peered":"2024-03-26T05:07:29.531414+0000","last_clean":"2024-03-26T05:07:29.531414+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:07:29.531414+0000","last_undegraded":"2024-03-26T05:07:29.531414+0000","last_fullsized":"2024-03-26T05:07:29.531414+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'34","last_scrub_stamp":"2024-03-26T05:06:54.128603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:54.128603+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:12.598382+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":296,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.539013+0000","last_change":"2024-03-26T05:06:05.029804+0000","last_active":"2024-03-26T05:07:29.539013+0000","last_peered":"2024-03-26T05:07:29.539013+0000","last_clean":"2024-03-26T05:07:29.539013+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:07:29.539013+0000","last_undegraded":"2024-03-26T05:07:29.539013+0000","last_fullsized":"2024-03-26T05:07:29.539013+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:05.029701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:05.029701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:34.728767+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"201'3817","reported_seq":3860,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:30.259602+0000","last_change":"2024-03-26T05:06:41.964329+0000","last_active":"2024-03-26T05:07:30.259602+0000","last_peered":"2024-03-26T05:07:30.259602+0000","last_clean":"2024-03-26T05:07:30.259602+0000","last_became_active":"2024-03-26T05:06:41.963962+0000","last_became_peered":"2024-03-26T05:06:41.963962+0000","last_unstale":"2024-03-26T05:07:30.259602+0000","last_undegraded":"2024-03-26T05:07:30.259602+0000","last_fullsized":"2024-03-26T05:07:30.259602+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3817,"log_dups_size":0,"ondisk_log_size":3817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:48.809977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250150912,"num_objects":3817,"num_object_clones":0,"num_object_copies":11451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3817,"num_write_kb":244288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":290,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.529848+0000","last_change":"2024-03-26T05:05:42.897537+0000","last_active":"2024-03-26T05:07:29.529848+0000","last_peered":"2024-03-26T05:07:29.529848+0000","last_clean":"2024-03-26T05:07:29.529848+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:07:29.529848+0000","last_undegraded":"2024-03-26T05:07:29.529848+0000","last_fullsized":"2024-03-26T05:07:29.529848+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:42.897420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:42.897420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673438+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.532997+0000","last_change":"2024-03-26T05:06:49.178919+0000","last_active":"2024-03-26T05:07:29.532997+0000","last_peered":"2024-03-26T05:07:29.532997+0000","last_clean":"2024-03-26T05:07:29.532997+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:07:29.532997+0000","last_undegraded":"2024-03-26T05:07:29.532997+0000","last_fullsized":"2024-03-26T05:07:29.532997+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:49.178823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:49.178823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:49.961816+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":320,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.520489+0000","last_change":"2024-03-26T05:06:41.303481+0000","last_active":"2024-03-26T05:07:29.520489+0000","last_peered":"2024-03-26T05:07:29.520489+0000","last_clean":"2024-03-26T05:07:29.520489+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:07:29.520489+0000","last_undegraded":"2024-03-26T05:07:29.520489+0000","last_fullsized":"2024-03-26T05:07:29.520489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:41.303395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:06:41.303395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:02.047984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'19","reported_seq":91,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.523236+0000","last_change":"2024-03-26T05:06:38.304919+0000","last_active":"2024-03-26T05:07:29.523236+0000","last_peered":"2024-03-26T05:07:29.523236+0000","last_clean":"2024-03-26T05:07:29.523236+0000","last_became_active":"2024-03-26T05:06:06.521721+0000","last_became_peered":"2024-03-26T05:06:06.521721+0000","last_unstale":"2024-03-26T05:07:29.523236+0000","last_undegraded":"2024-03-26T05:07:29.523236+0000","last_fullsized":"2024-03-26T05:07:29.523236+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-26T05:06:38.304813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:38.304813+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:45.591514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":26,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":309,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.521346+0000","last_change":"2024-03-26T05:06:02.895249+0000","last_active":"2024-03-26T05:07:29.521346+0000","last_peered":"2024-03-26T05:07:29.521346+0000","last_clean":"2024-03-26T05:07:29.521346+0000","last_became_active":"2024-03-26T05:05:36.401910+0000","last_became_peered":"2024-03-26T05:05:36.401910+0000","last_unstale":"2024-03-26T05:07:29.521346+0000","last_undegraded":"2024-03-26T05:07:29.521346+0000","last_fullsized":"2024-03-26T05:07:29.521346+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:02.895131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:02.895131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:34.257089+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"201'3958","reported_seq":4002,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.791697+0000","last_change":"2024-03-26T05:06:41.962495+0000","last_active":"2024-03-26T05:07:29.791697+0000","last_peered":"2024-03-26T05:07:29.791697+0000","last_clean":"2024-03-26T05:07:29.791697+0000","last_became_active":"2024-03-26T05:06:41.962150+0000","last_became_peered":"2024-03-26T05:06:41.962150+0000","last_unstale":"2024-03-26T05:07:29.791697+0000","last_undegraded":"2024-03-26T05:07:29.791697+0000","last_fullsized":"2024-03-26T05:07:29.791697+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3958,"log_dups_size":0,"ondisk_log_size":3958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:44.144698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259391488,"num_objects":3958,"num_object_clones":0,"num_object_copies":11874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3958,"num_write_kb":253312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":92,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.546805+0000","last_change":"2024-03-26T05:05:36.423696+0000","last_active":"2024-03-26T05:07:29.546805+0000","last_peered":"2024-03-26T05:07:29.546805+0000","last_clean":"2024-03-26T05:07:29.546805+0000","last_became_active":"2024-03-26T05:05:36.422421+0000","last_became_peered":"2024-03-26T05:05:36.422421+0000","last_unstale":"2024-03-26T05:07:29.546805+0000","last_undegraded":"2024-03-26T05:07:29.546805+0000","last_fullsized":"2024-03-26T05:07:29.546805+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_clean_scrub_stamp":"2024-03-26T05:05:03.848208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":557,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.528800+0000","last_change":"2024-03-26T05:06:45.717598+0000","last_active":"2024-03-26T05:07:29.528800+0000","last_peered":"2024-03-26T05:07:29.528800+0000","last_clean":"2024-03-26T05:07:29.528800+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:07:29.528800+0000","last_undegraded":"2024-03-26T05:07:29.528800+0000","last_fullsized":"2024-03-26T05:07:29.528800+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:06:45.717474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.717474+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:09.381246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":309,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.547102+0000","last_change":"2024-03-26T05:05:36.424898+0000","last_active":"2024-03-26T05:07:29.547102+0000","last_peered":"2024-03-26T05:07:29.547102+0000","last_clean":"2024-03-26T05:07:29.547102+0000","last_became_active":"2024-03-26T05:05:36.424565+0000","last_became_peered":"2024-03-26T05:05:36.424565+0000","last_unstale":"2024-03-26T05:07:29.547102+0000","last_undegraded":"2024-03-26T05:07:29.547102+0000","last_fullsized":"2024-03-26T05:07:29.547102+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:07.743708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:07.743708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551712+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'24","reported_seq":422,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.532068+0000","last_change":"2024-03-26T05:06:09.443609+0000","last_active":"2024-03-26T05:07:29.532068+0000","last_peered":"2024-03-26T05:07:29.532068+0000","last_clean":"2024-03-26T05:07:29.532068+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:07:29.532068+0000","last_undegraded":"2024-03-26T05:07:29.532068+0000","last_fullsized":"2024-03-26T05:07:29.532068+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'24","last_scrub_stamp":"2024-03-26T05:06:09.443473+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:06:09.443473+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:30.982419+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":308,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.669111+0000","last_change":"2024-03-26T05:07:29.669146+0000","last_active":"2024-03-26T05:07:29.669111+0000","last_peered":"2024-03-26T05:07:29.669111+0000","last_clean":"2024-03-26T05:07:29.669111+0000","last_became_active":"2024-03-26T05:05:36.401943+0000","last_became_peered":"2024-03-26T05:05:36.401943+0000","last_unstale":"2024-03-26T05:07:29.669111+0000","last_undegraded":"2024-03-26T05:07:29.669111+0000","last_fullsized":"2024-03-26T05:07:29.669111+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:29.669034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:29.669034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:31.729702+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"201'3913","reported_seq":3957,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.995502+0000","last_change":"2024-03-26T05:06:42.023633+0000","last_active":"2024-03-26T05:07:29.995502+0000","last_peered":"2024-03-26T05:07:29.995502+0000","last_clean":"2024-03-26T05:07:29.995502+0000","last_became_active":"2024-03-26T05:06:42.023345+0000","last_became_peered":"2024-03-26T05:06:42.023345+0000","last_unstale":"2024-03-26T05:07:29.995502+0000","last_undegraded":"2024-03-26T05:07:29.995502+0000","last_fullsized":"2024-03-26T05:07:29.995502+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3913,"log_dups_size":0,"ondisk_log_size":3913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:53.018647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256442368,"num_objects":3913,"num_object_clones":0,"num_object_copies":11739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3913,"num_write_kb":250432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":297,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531083+0000","last_change":"2024-03-26T05:05:36.417609+0000","last_active":"2024-03-26T05:07:29.531083+0000","last_peered":"2024-03-26T05:07:29.531083+0000","last_clean":"2024-03-26T05:07:29.531083+0000","last_became_active":"2024-03-26T05:05:36.416657+0000","last_became_peered":"2024-03-26T05:05:36.416657+0000","last_unstale":"2024-03-26T05:07:29.531083+0000","last_undegraded":"2024-03-26T05:07:29.531083+0000","last_fullsized":"2024-03-26T05:07:29.531083+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:53.760171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:53.760171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:33.316701+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":312,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.540463+0000","last_change":"2024-03-26T05:06:39.074585+0000","last_active":"2024-03-26T05:07:29.540463+0000","last_peered":"2024-03-26T05:07:29.540463+0000","last_clean":"2024-03-26T05:07:29.540463+0000","last_became_active":"2024-03-26T05:05:36.425252+0000","last_became_peered":"2024-03-26T05:05:36.425252+0000","last_unstale":"2024-03-26T05:07:29.540463+0000","last_undegraded":"2024-03-26T05:07:29.540463+0000","last_fullsized":"2024-03-26T05:07:29.540463+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:39.074501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:39.074501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:58.133016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":736,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.528753+0000","last_change":"2024-03-26T05:05:38.886524+0000","last_active":"2024-03-26T05:07:29.528753+0000","last_peered":"2024-03-26T05:07:29.528753+0000","last_clean":"2024-03-26T05:07:29.528753+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:07:29.528753+0000","last_undegraded":"2024-03-26T05:07:29.528753+0000","last_fullsized":"2024-03-26T05:07:29.528753+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:05:38.886387+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:05:38.886387+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.673484+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":421,"num_read_kb":416,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"187'18","reported_seq":364,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.540830+0000","last_change":"2024-03-26T05:06:49.027873+0000","last_active":"2024-03-26T05:07:29.540830+0000","last_peered":"2024-03-26T05:07:29.540830+0000","last_clean":"2024-03-26T05:07:29.540830+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:07:29.540830+0000","last_undegraded":"2024-03-26T05:07:29.540830+0000","last_fullsized":"2024-03-26T05:07:29.540830+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'18","last_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_deep_scrub":"187'18","last_deep_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_clean_scrub_stamp":"2024-03-26T05:06:49.027756+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:55.000771+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":100,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.543717+0000","last_change":"2024-03-26T05:06:44.270685+0000","last_active":"2024-03-26T05:07:29.543717+0000","last_peered":"2024-03-26T05:07:29.543717+0000","last_clean":"2024-03-26T05:07:29.543717+0000","last_became_active":"2024-03-26T05:05:36.406489+0000","last_became_peered":"2024-03-26T05:05:36.406489+0000","last_unstale":"2024-03-26T05:07:29.543717+0000","last_undegraded":"2024-03-26T05:07:29.543717+0000","last_fullsized":"2024-03-26T05:07:29.543717+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:44.270594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:44.270594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:52.280450+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"201'3931","reported_seq":3975,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:30.252783+0000","last_change":"2024-03-26T05:06:42.024050+0000","last_active":"2024-03-26T05:07:30.252783+0000","last_peered":"2024-03-26T05:07:30.252783+0000","last_clean":"2024-03-26T05:07:30.252783+0000","last_became_active":"2024-03-26T05:06:42.023726+0000","last_became_peered":"2024-03-26T05:06:42.023726+0000","last_unstale":"2024-03-26T05:07:30.252783+0000","last_undegraded":"2024-03-26T05:07:30.252783+0000","last_fullsized":"2024-03-26T05:07:30.252783+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3931,"log_dups_size":0,"ondisk_log_size":3931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:56.060944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257622016,"num_objects":3931,"num_object_clones":0,"num_object_copies":11793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3931,"num_write_kb":251584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":291,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531172+0000","last_change":"2024-03-26T05:06:01.073386+0000","last_active":"2024-03-26T05:07:29.531172+0000","last_peered":"2024-03-26T05:07:29.531172+0000","last_clean":"2024-03-26T05:07:29.531172+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:07:29.531172+0000","last_undegraded":"2024-03-26T05:07:29.531172+0000","last_fullsized":"2024-03-26T05:07:29.531172+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:01.073272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:06:01.073272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:33.317011+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":309,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531316+0000","last_change":"2024-03-26T05:05:36.412560+0000","last_active":"2024-03-26T05:07:29.531316+0000","last_peered":"2024-03-26T05:07:29.531316+0000","last_clean":"2024-03-26T05:07:29.531316+0000","last_became_active":"2024-03-26T05:05:36.409491+0000","last_became_peered":"2024-03-26T05:05:36.409491+0000","last_unstale":"2024-03-26T05:07:29.531316+0000","last_undegraded":"2024-03-26T05:07:29.531316+0000","last_fullsized":"2024-03-26T05:07:29.531316+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:05.892241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:05:05.892241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:33.316822+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'25","reported_seq":382,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.542177+0000","last_change":"2024-03-26T05:06:52.015549+0000","last_active":"2024-03-26T05:07:29.542177+0000","last_peered":"2024-03-26T05:07:29.542177+0000","last_clean":"2024-03-26T05:07:29.542177+0000","last_became_active":"2024-03-26T05:05:36.414330+0000","last_became_peered":"2024-03-26T05:05:36.414330+0000","last_unstale":"2024-03-26T05:07:29.542177+0000","last_undegraded":"2024-03-26T05:07:29.542177+0000","last_fullsized":"2024-03-26T05:07:29.542177+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:06:52.015444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:52.015444+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:15.721741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":300,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531566+0000","last_change":"2024-03-26T05:05:53.365085+0000","last_active":"2024-03-26T05:07:29.531566+0000","last_peered":"2024-03-26T05:07:29.531566+0000","last_clean":"2024-03-26T05:07:29.531566+0000","last_became_active":"2024-03-26T05:05:36.406443+0000","last_became_peered":"2024-03-26T05:05:36.406443+0000","last_unstale":"2024-03-26T05:07:29.531566+0000","last_undegraded":"2024-03-26T05:07:29.531566+0000","last_fullsized":"2024-03-26T05:07:29.531566+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:53.365003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:05:53.365003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:30.982450+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"201'3906","reported_seq":3949,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:30.319213+0000","last_change":"2024-03-26T05:06:42.026870+0000","last_active":"2024-03-26T05:07:30.319213+0000","last_peered":"2024-03-26T05:07:30.319213+0000","last_clean":"2024-03-26T05:07:30.319213+0000","last_became_active":"2024-03-26T05:06:42.026460+0000","last_became_peered":"2024-03-26T05:06:42.026460+0000","last_unstale":"2024-03-26T05:07:30.319213+0000","last_undegraded":"2024-03-26T05:07:30.319213+0000","last_fullsized":"2024-03-26T05:07:30.319213+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3906,"log_dups_size":0,"ondisk_log_size":3906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:03.707481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255983616,"num_objects":3906,"num_object_clones":0,"num_object_copies":11718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3906,"num_write_kb":249984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":292,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.539068+0000","last_change":"2024-03-26T05:05:52.123187+0000","last_active":"2024-03-26T05:07:29.539068+0000","last_peered":"2024-03-26T05:07:29.539068+0000","last_clean":"2024-03-26T05:07:29.539068+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:07:29.539068+0000","last_undegraded":"2024-03-26T05:07:29.539068+0000","last_fullsized":"2024-03-26T05:07:29.539068+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:52.123084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:52.123084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":93,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.536231+0000","last_change":"2024-03-26T05:05:36.419407+0000","last_active":"2024-03-26T05:07:29.536231+0000","last_peered":"2024-03-26T05:07:29.536231+0000","last_clean":"2024-03-26T05:07:29.536231+0000","last_became_active":"2024-03-26T05:05:36.418906+0000","last_became_peered":"2024-03-26T05:05:36.418906+0000","last_unstale":"2024-03-26T05:07:29.536231+0000","last_undegraded":"2024-03-26T05:07:29.536231+0000","last_fullsized":"2024-03-26T05:07:29.536231+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:07.893841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:07.893841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"75'16","reported_seq":317,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.527631+0000","last_change":"2024-03-26T05:06:47.166740+0000","last_active":"2024-03-26T05:07:29.527631+0000","last_peered":"2024-03-26T05:07:29.527631+0000","last_clean":"2024-03-26T05:07:29.527631+0000","last_became_active":"2024-03-26T05:05:36.411117+0000","last_became_peered":"2024-03-26T05:05:36.411117+0000","last_unstale":"2024-03-26T05:07:29.527631+0000","last_undegraded":"2024-03-26T05:07:29.527631+0000","last_fullsized":"2024-03-26T05:07:29.527631+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:06:47.166607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:47.166607+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:51.108028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"161'10","reported_seq":402,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.542507+0000","last_change":"2024-03-26T05:05:36.429633+0000","last_active":"2024-03-26T05:07:29.542507+0000","last_peered":"2024-03-26T05:07:29.542507+0000","last_clean":"2024-03-26T05:07:29.542507+0000","last_became_active":"2024-03-26T05:05:36.422549+0000","last_became_peered":"2024-03-26T05:05:36.422549+0000","last_unstale":"2024-03-26T05:07:29.542507+0000","last_undegraded":"2024-03-26T05:07:29.542507+0000","last_fullsized":"2024-03-26T05:07:29.542507+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:05:11.014574+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"201'3893","reported_seq":3937,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.776449+0000","last_change":"2024-03-26T05:06:42.023702+0000","last_active":"2024-03-26T05:07:29.776449+0000","last_peered":"2024-03-26T05:07:29.776449+0000","last_clean":"2024-03-26T05:07:29.776449+0000","last_became_active":"2024-03-26T05:06:42.023260+0000","last_became_peered":"2024-03-26T05:06:42.023260+0000","last_unstale":"2024-03-26T05:07:29.776449+0000","last_undegraded":"2024-03-26T05:07:29.776449+0000","last_fullsized":"2024-03-26T05:07:29.776449+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3893,"log_dups_size":0,"ondisk_log_size":3893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:42.956733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255131648,"num_objects":3893,"num_object_clones":0,"num_object_copies":11679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3893,"num_write_kb":249152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":290,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.529627+0000","last_change":"2024-03-26T05:06:43.743986+0000","last_active":"2024-03-26T05:07:29.529627+0000","last_peered":"2024-03-26T05:07:29.529627+0000","last_clean":"2024-03-26T05:07:29.529627+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:07:29.529627+0000","last_undegraded":"2024-03-26T05:07:29.529627+0000","last_fullsized":"2024-03-26T05:07:29.529627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:43.743837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:43.743837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:04.383692+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":313,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.540929+0000","last_change":"2024-03-26T05:06:45.074041+0000","last_active":"2024-03-26T05:07:29.540929+0000","last_peered":"2024-03-26T05:07:29.540929+0000","last_clean":"2024-03-26T05:07:29.540929+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:07:29.540929+0000","last_undegraded":"2024-03-26T05:07:29.540929+0000","last_fullsized":"2024-03-26T05:07:29.540929+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:06:45.073912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.073912+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:51.266877+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'20","reported_seq":357,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.517685+0000","last_change":"2024-03-26T05:06:42.311221+0000","last_active":"2024-03-26T05:07:29.517685+0000","last_peered":"2024-03-26T05:07:29.517685+0000","last_clean":"2024-03-26T05:07:29.517685+0000","last_became_active":"2024-03-26T05:05:36.405631+0000","last_became_peered":"2024-03-26T05:05:36.405631+0000","last_unstale":"2024-03-26T05:07:29.517685+0000","last_undegraded":"2024-03-26T05:07:29.517685+0000","last_fullsized":"2024-03-26T05:07:29.517685+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:06:42.311073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:42.311073+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:44.446306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":381,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.545666+0000","last_change":"2024-03-26T05:04:26.670277+0000","last_active":"2024-03-26T05:07:29.545666+0000","last_peered":"2024-03-26T05:07:29.545666+0000","last_clean":"2024-03-26T05:07:29.545666+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:07:29.545666+0000","last_undegraded":"2024-03-26T05:07:29.545666+0000","last_fullsized":"2024-03-26T05:07:29.545666+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:26.669895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:26.669895+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551445+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"201'4111","reported_seq":4155,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:30.316765+0000","last_change":"2024-03-26T05:06:41.965131+0000","last_active":"2024-03-26T05:07:30.316765+0000","last_peered":"2024-03-26T05:07:30.316765+0000","last_clean":"2024-03-26T05:07:30.316765+0000","last_became_active":"2024-03-26T05:06:41.964810+0000","last_became_peered":"2024-03-26T05:06:41.964810+0000","last_unstale":"2024-03-26T05:07:30.316765+0000","last_undegraded":"2024-03-26T05:07:30.316765+0000","last_fullsized":"2024-03-26T05:07:30.316765+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:55.356833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269418496,"num_objects":4111,"num_object_clones":0,"num_object_copies":12333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4111,"num_write_kb":263104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":292,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.517868+0000","last_change":"2024-03-26T05:05:57.123378+0000","last_active":"2024-03-26T05:07:29.517868+0000","last_peered":"2024-03-26T05:07:29.517868+0000","last_clean":"2024-03-26T05:07:29.517868+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:07:29.517868+0000","last_undegraded":"2024-03-26T05:07:29.517868+0000","last_fullsized":"2024-03-26T05:07:29.517868+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:57.123300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:57.123300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604415+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.546682+0000","last_change":"2024-03-26T05:04:54.805782+0000","last_active":"2024-03-26T05:07:29.546682+0000","last_peered":"2024-03-26T05:07:29.546682+0000","last_clean":"2024-03-26T05:07:29.546682+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:07:29.546682+0000","last_undegraded":"2024-03-26T05:07:29.546682+0000","last_fullsized":"2024-03-26T05:07:29.546682+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:54.805655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:54.805655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551626+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"96'40","reported_seq":365,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.545568+0000","last_change":"2024-03-26T05:05:13.704632+0000","last_active":"2024-03-26T05:07:29.545568+0000","last_peered":"2024-03-26T05:07:29.545568+0000","last_clean":"2024-03-26T05:07:29.545568+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:07:29.545568+0000","last_undegraded":"2024-03-26T05:07:29.545568+0000","last_fullsized":"2024-03-26T05:07:29.545568+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:05:13.704484+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:05:13.704484+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551603+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":288,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.517544+0000","last_change":"2024-03-26T05:05:47.194069+0000","last_active":"2024-03-26T05:07:29.517544+0000","last_peered":"2024-03-26T05:07:29.517544+0000","last_clean":"2024-03-26T05:07:29.517544+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:07:29.517544+0000","last_undegraded":"2024-03-26T05:07:29.517544+0000","last_fullsized":"2024-03-26T05:07:29.517544+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:47.193989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:47.193989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604443+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"201'3981","reported_seq":4025,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.998002+0000","last_change":"2024-03-26T05:06:42.023623+0000","last_active":"2024-03-26T05:07:29.998002+0000","last_peered":"2024-03-26T05:07:29.998002+0000","last_clean":"2024-03-26T05:07:29.998002+0000","last_became_active":"2024-03-26T05:06:42.023314+0000","last_became_peered":"2024-03-26T05:06:42.023314+0000","last_unstale":"2024-03-26T05:07:29.998002+0000","last_undegraded":"2024-03-26T05:07:29.998002+0000","last_fullsized":"2024-03-26T05:07:29.998002+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3981,"log_dups_size":0,"ondisk_log_size":3981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:45.090269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260898816,"num_objects":3981,"num_object_clones":0,"num_object_copies":11943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3981,"num_write_kb":254784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":92,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.532206+0000","last_change":"2024-03-26T05:05:36.411895+0000","last_active":"2024-03-26T05:07:29.532206+0000","last_peered":"2024-03-26T05:07:29.532206+0000","last_clean":"2024-03-26T05:07:29.532206+0000","last_became_active":"2024-03-26T05:05:36.411501+0000","last_became_peered":"2024-03-26T05:05:36.411501+0000","last_unstale":"2024-03-26T05:07:29.532206+0000","last_undegraded":"2024-03-26T05:07:29.532206+0000","last_fullsized":"2024-03-26T05:07:29.532206+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.885959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.885959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:33.316731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"161'9","reported_seq":401,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.518787+0000","last_change":"2024-03-26T05:06:10.114395+0000","last_active":"2024-03-26T05:07:29.518787+0000","last_peered":"2024-03-26T05:07:29.518787+0000","last_clean":"2024-03-26T05:07:29.518787+0000","last_became_active":"2024-03-26T05:05:36.408943+0000","last_became_peered":"2024-03-26T05:05:36.408943+0000","last_unstale":"2024-03-26T05:07:29.518787+0000","last_undegraded":"2024-03-26T05:07:29.518787+0000","last_fullsized":"2024-03-26T05:07:29.518787+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'9","last_scrub_stamp":"2024-03-26T05:06:10.114267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:10.114267+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.834145+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.545476+0000","last_change":"2024-03-26T05:06:48.223737+0000","last_active":"2024-03-26T05:07:29.545476+0000","last_peered":"2024-03-26T05:07:29.545476+0000","last_clean":"2024-03-26T05:07:29.545476+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:07:29.545476+0000","last_undegraded":"2024-03-26T05:07:29.545476+0000","last_fullsized":"2024-03-26T05:07:29.545476+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:48.223661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:48.223661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:03.204242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":332,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.543435+0000","last_change":"2024-03-26T05:06:57.114810+0000","last_active":"2024-03-26T05:07:29.543435+0000","last_peered":"2024-03-26T05:07:29.543435+0000","last_clean":"2024-03-26T05:07:29.543435+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:07:29.543435+0000","last_undegraded":"2024-03-26T05:07:29.543435+0000","last_fullsized":"2024-03-26T05:07:29.543435+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:06:57.113884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:57.113884+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:59.189652+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"201'3911","reported_seq":3955,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.754784+0000","last_change":"2024-03-26T05:06:41.962649+0000","last_active":"2024-03-26T05:07:29.754784+0000","last_peered":"2024-03-26T05:07:29.754784+0000","last_clean":"2024-03-26T05:07:29.754784+0000","last_became_active":"2024-03-26T05:06:41.962268+0000","last_became_peered":"2024-03-26T05:06:41.962268+0000","last_unstale":"2024-03-26T05:07:29.754784+0000","last_undegraded":"2024-03-26T05:07:29.754784+0000","last_fullsized":"2024-03-26T05:07:29.754784+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3911,"log_dups_size":0,"ondisk_log_size":3911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:02.304727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256311296,"num_objects":3911,"num_object_clones":0,"num_object_copies":11733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":250304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":289,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.543988+0000","last_change":"2024-03-26T05:05:36.413659+0000","last_active":"2024-03-26T05:07:29.543988+0000","last_peered":"2024-03-26T05:07:29.543988+0000","last_clean":"2024-03-26T05:07:29.543988+0000","last_became_active":"2024-03-26T05:05:36.413320+0000","last_became_peered":"2024-03-26T05:05:36.413320+0000","last_unstale":"2024-03-26T05:07:29.543988+0000","last_undegraded":"2024-03-26T05:07:29.543988+0000","last_fullsized":"2024-03-26T05:07:29.543988+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:25.636012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:25.636012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551291+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":100,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.518355+0000","last_change":"2024-03-26T05:06:56.446200+0000","last_active":"2024-03-26T05:07:29.518355+0000","last_peered":"2024-03-26T05:07:29.518355+0000","last_clean":"2024-03-26T05:07:29.518355+0000","last_became_active":"2024-03-26T05:05:36.404979+0000","last_became_peered":"2024-03-26T05:05:36.404979+0000","last_unstale":"2024-03-26T05:07:29.518355+0000","last_undegraded":"2024-03-26T05:07:29.518355+0000","last_fullsized":"2024-03-26T05:07:29.518355+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:56.446060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.446060+0000","last_clean_scrub_stamp":"2024-03-26T05:06:56.446060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:23.714583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":304,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.517548+0000","last_change":"2024-03-26T05:05:49.242313+0000","last_active":"2024-03-26T05:07:29.517548+0000","last_peered":"2024-03-26T05:07:29.517548+0000","last_clean":"2024-03-26T05:07:29.517548+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:07:29.517548+0000","last_undegraded":"2024-03-26T05:07:29.517548+0000","last_fullsized":"2024-03-26T05:07:29.517548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:49.242237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:49.242237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604386+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'17","reported_seq":331,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.518247+0000","last_change":"2024-03-26T05:05:56.133675+0000","last_active":"2024-03-26T05:07:29.518247+0000","last_peered":"2024-03-26T05:07:29.518247+0000","last_clean":"2024-03-26T05:07:29.518247+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:07:29.518247+0000","last_undegraded":"2024-03-26T05:07:29.518247+0000","last_fullsized":"2024-03-26T05:07:29.518247+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:05:56.133576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:56.133576+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"201'3976","reported_seq":4020,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.988135+0000","last_change":"2024-03-26T05:06:42.023927+0000","last_active":"2024-03-26T05:07:29.988135+0000","last_peered":"2024-03-26T05:07:29.988135+0000","last_clean":"2024-03-26T05:07:29.988135+0000","last_became_active":"2024-03-26T05:06:42.022709+0000","last_became_peered":"2024-03-26T05:06:42.022709+0000","last_unstale":"2024-03-26T05:07:29.988135+0000","last_undegraded":"2024-03-26T05:07:29.988135+0000","last_fullsized":"2024-03-26T05:07:29.988135+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3976,"log_dups_size":0,"ondisk_log_size":3976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:02.947465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260571136,"num_objects":3976,"num_object_clones":0,"num_object_copies":11928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3976,"num_write_kb":254464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":291,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.523450+0000","last_change":"2024-03-26T05:05:55.843217+0000","last_active":"2024-03-26T05:07:29.523450+0000","last_peered":"2024-03-26T05:07:29.523450+0000","last_clean":"2024-03-26T05:07:29.523450+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:07:29.523450+0000","last_undegraded":"2024-03-26T05:07:29.523450+0000","last_fullsized":"2024-03-26T05:07:29.523450+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:55.843133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:55.843133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:34.257061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531575+0000","last_change":"2024-03-26T05:06:02.997794+0000","last_active":"2024-03-26T05:07:29.531575+0000","last_peered":"2024-03-26T05:07:29.531575+0000","last_clean":"2024-03-26T05:07:29.531575+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:07:29.531575+0000","last_undegraded":"2024-03-26T05:07:29.531575+0000","last_fullsized":"2024-03-26T05:07:29.531575+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:02.997671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:02.997671+0000","last_clean_scrub_stamp":"2024-03-26T05:06:02.997671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:33.316954+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":316,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.531394+0000","last_change":"2024-03-26T05:06:07.426633+0000","last_active":"2024-03-26T05:07:29.531394+0000","last_peered":"2024-03-26T05:07:29.531394+0000","last_clean":"2024-03-26T05:07:29.531394+0000","last_became_active":"2024-03-26T05:06:06.521150+0000","last_became_peered":"2024-03-26T05:06:06.521150+0000","last_unstale":"2024-03-26T05:07:29.531394+0000","last_undegraded":"2024-03-26T05:07:29.531394+0000","last_fullsized":"2024-03-26T05:07:29.531394+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:07.426512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_clean_scrub_stamp":"2024-03-26T05:06:07.426512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:30.982478+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":320,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.518352+0000","last_change":"2024-03-26T05:05:04.091067+0000","last_active":"2024-03-26T05:07:29.518352+0000","last_peered":"2024-03-26T05:07:29.518352+0000","last_clean":"2024-03-26T05:07:29.518352+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:07:29.518352+0000","last_undegraded":"2024-03-26T05:07:29.518352+0000","last_fullsized":"2024-03-26T05:07:29.518352+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:05:04.090960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.090960+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604196+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"201'3890","reported_seq":3934,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:30.206487+0000","last_change":"2024-03-26T05:06:41.964494+0000","last_active":"2024-03-26T05:07:30.206487+0000","last_peered":"2024-03-26T05:07:30.206487+0000","last_clean":"2024-03-26T05:07:30.206487+0000","last_became_active":"2024-03-26T05:06:41.963907+0000","last_became_peered":"2024-03-26T05:06:41.963907+0000","last_unstale":"2024-03-26T05:07:30.206487+0000","last_undegraded":"2024-03-26T05:07:30.206487+0000","last_fullsized":"2024-03-26T05:07:30.206487+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":3890,"log_dups_size":0,"ondisk_log_size":3890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:01.870974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254935040,"num_objects":3890,"num_object_clones":0,"num_object_copies":11670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3890,"num_write_kb":248960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":292,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.518793+0000","last_change":"2024-03-26T05:06:05.019093+0000","last_active":"2024-03-26T05:07:29.518793+0000","last_peered":"2024-03-26T05:07:29.518793+0000","last_clean":"2024-03-26T05:07:29.518793+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:07:29.518793+0000","last_undegraded":"2024-03-26T05:07:29.518793+0000","last_fullsized":"2024-03-26T05:07:29.518793+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:05.018999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:05.018999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:32.604639+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"161'10","reported_seq":409,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.526752+0000","last_change":"2024-03-26T05:06:56.184462+0000","last_active":"2024-03-26T05:07:29.526752+0000","last_peered":"2024-03-26T05:07:29.526752+0000","last_clean":"2024-03-26T05:07:29.526752+0000","last_became_active":"2024-03-26T05:05:36.424309+0000","last_became_peered":"2024-03-26T05:05:36.424309+0000","last_unstale":"2024-03-26T05:07:29.526752+0000","last_undegraded":"2024-03-26T05:07:29.526752+0000","last_fullsized":"2024-03-26T05:07:29.526752+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'10","last_scrub_stamp":"2024-03-26T05:06:56.184325+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:06:56.184325+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:07.356139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":309,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.546132+0000","last_change":"2024-03-26T05:06:06.522418+0000","last_active":"2024-03-26T05:07:29.546132+0000","last_peered":"2024-03-26T05:07:29.546132+0000","last_clean":"2024-03-26T05:07:29.546132+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:07:29.546132+0000","last_undegraded":"2024-03-26T05:07:29.546132+0000","last_fullsized":"2024-03-26T05:07:29.546132+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:05.000183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:05.000183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:31.551784+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":315,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T05:07:29.517934+0000","last_change":"2024-03-26T05:06:45.298927+0000","last_active":"2024-03-26T05:07:29.517934+0000","last_peered":"2024-03-26T05:07:29.517934+0000","last_clean":"2024-03-26T05:07:29.517934+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:07:29.517934+0000","last_undegraded":"2024-03-26T05:07:29.517934+0000","last_fullsized":"2024-03-26T05:07:29.517934+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:06:45.298241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.298241+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:49.860740+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4134797312,"num_objects":63092,"num_object_clones":0,"num_object_copies":189276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63092,"num_write_kb":4037888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12397117440,"data_stored":12397117440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63092,"ondisk_log_size":63092,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":421,"num_read_kb":416,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":984,"num_read_kb":709,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":630,"ondisk_log_size":630,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005146,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1421356,"kb_used_data":1269408,"kb_used_omap":1,"kb_used_meta":151934,"kb_avail":92323796,"statfs":{"total":95995035648,"available":94539567104,"internally_reserved":0,"allocated":1299873792,"data_stored":1299477633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":155580874},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332191,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1900484,"kb_used_data":1734652,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91844668,"statfs":{"total":95995035648,"available":94048940032,"internally_reserved":0,"allocated":1776283648,"data_stored":1775914424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169802182},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1904564,"kb_used_data":1733160,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91840588,"statfs":{"total":95995035648,"available":94044762112,"internally_reserved":0,"allocated":1774755840,"data_stored":1774342170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2023216,"kb_used_data":1785192,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":91721936,"statfs":{"total":95995035648,"available":93923262464,"internally_reserved":0,"allocated":1828036608,"data_stored":1827632209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411444,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1911944,"kb_used_data":1732220,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91833208,"statfs":{"total":95995035648,"available":94037204992,"internally_reserved":0,"allocated":1773793280,"data_stored":1773423559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117102,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2510832,"kb_used_data":2250596,"kb_used_omap":1,"kb_used_meta":260222,"kb_avail":91234320,"statfs":{"total":95995035648,"available":93423943680,"internally_reserved":0,"allocated":2304610304,"data_stored":2304208907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":266467783},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920874,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1633700,"kb_used_data":1485916,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92111452,"statfs":{"total":95995035648,"available":94322126848,"internally_reserved":0,"allocated":1521577984,"data_stored":1521184710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":176093659236,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1833435136,"data_stored":1833435136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1796734976,"data_stored":1796734976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1800536064,"data_stored":1800536064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2335244288,"data_stored":2335244288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1523515392,"data_stored":1523515392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1806237696,"data_stored":1806237696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1301413888,"data_stored":1301413888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:07:34.943 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:07:34.989 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:35.039 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:35.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:35.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:35.142 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:35.183 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:35.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:35.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:35.285 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:35.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:35.335 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:35.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:35.436 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:35.451 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:07:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:07:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:35.471 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:35.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:35.572 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:35.609 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:35.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:35.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:35.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:35.742 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:35.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:35.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:35 smithi132 ceph-mon[33500]: 4.d scrub starts 2024-03-26T05:07:35.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:35 smithi132 ceph-mon[33500]: 4.d scrub ok 2024-03-26T05:07:35.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:35 smithi132 ceph-mon[33500]: pgmap v312: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 690 B/s rd, 52 MiB/s wr, 830 op/s 2024-03-26T05:07:35.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:35 smithi132 ceph-mon[33500]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:07:35.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:07:35.844 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:35 smithi080 ceph-mon[40404]: 4.d scrub starts 2024-03-26T05:07:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:35 smithi080 ceph-mon[40404]: 4.d scrub ok 2024-03-26T05:07:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:35 smithi080 ceph-mon[40404]: pgmap v312: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 690 B/s rd, 52 MiB/s wr, 830 op/s 2024-03-26T05:07:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:35 smithi080 ceph-mon[40404]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:07:35.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:07:35.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:35 smithi080 ceph-mon[29691]: 4.d scrub starts 2024-03-26T05:07:35.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:35 smithi080 ceph-mon[29691]: 4.d scrub ok 2024-03-26T05:07:35.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:35 smithi080 ceph-mon[29691]: pgmap v312: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 690 B/s rd, 52 MiB/s wr, 830 op/s 2024-03-26T05:07:35.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:35 smithi080 ceph-mon[29691]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:07:35.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:07:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:35.875 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:35.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:35.917 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 43 16 70010 69994 101.715 63.9375 0.00312927 0.00982192 2024-03-26T05:07:35.976 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:36.017 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:36.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:36.119 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:36.156 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:36.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:36.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:36.257 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:36.304 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:36.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:36.406 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:36.467 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:36.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:36.568 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:36.595 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:36.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:36.697 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:36.717 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:07:36.718 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:07:36.718 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:36.718 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:07:36.718 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:07:36.718 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[29691]: osdmap e203: 8 total, 8 up, 7 in 2024-03-26T05:07:36.718 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[29691]: osdmap e204: 8 total, 8 up, 7 in 2024-03-26T05:07:36.718 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:07:36.718 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:07:36.718 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:36.718 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:07:36.719 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:07:36.719 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[40404]: osdmap e203: 8 total, 8 up, 7 in 2024-03-26T05:07:36.719 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:36 smithi080 ceph-mon[40404]: osdmap e204: 8 total, 8 up, 7 in 2024-03-26T05:07:36.745 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:36.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:36.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:36.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:36 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:07:36.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:36 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:07:36.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:36.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:36 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:07:36.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:36 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:07:36.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:36 smithi132 ceph-mon[33500]: osdmap e203: 8 total, 8 up, 7 in 2024-03-26T05:07:36.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:36 smithi132 ceph-mon[33500]: osdmap e204: 8 total, 8 up, 7 in 2024-03-26T05:07:36.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:36.875 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:36.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:36.917 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 44 16 71017 71001 100.834 62.9375 0.00505033 0.00991221 2024-03-26T05:07:36.977 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:37.024 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:37.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:37.126 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:37.159 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:37.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:37.260 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:37.289 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:37.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:37.391 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:37.436 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:37.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:37.538 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:37.577 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:37.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:37.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:37.651 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:07:37.652 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:07:37.652 INFO:teuthology.orchestra.run.smithi080.stdout: "complaint_time": 30, 2024-03-26T05:07:37.652 INFO:teuthology.orchestra.run.smithi080.stdout: "num_blocked_ops": 0 2024-03-26T05:07:37.652 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:07:37.679 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:37.706 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:37.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:37.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:37 smithi132 ceph-mon[33500]: pgmap v314: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-26T05:07:37.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:37 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:07:37.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:37 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:07:37.808 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:37.849 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:37.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:37.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:37 smithi080 ceph-mon[29691]: pgmap v314: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-26T05:07:37.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:37 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:07:37.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:37 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:07:37.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:37 smithi080 ceph-mon[40404]: pgmap v314: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-26T05:07:37.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:37 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:07:37.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:37 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:07:37.919 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 45 16 72139 72123 100.146 70.125 0.0170678 0.00997879 2024-03-26T05:07:37.951 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:38.003 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:38.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:38.104 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:38.134 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:38.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:38.236 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:38.264 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:38.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:38.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:38.366 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:38.403 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:38.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:38.505 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:38.522 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:38.522 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:38.543 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T05:07:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:38.588 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:38.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:38.620 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:38.689 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:38.694 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:38 smithi080 ceph-mon[29691]: 5.18 scrub starts 2024-03-26T05:07:38.694 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:38 smithi080 ceph-mon[29691]: 5.18 scrub ok 2024-03-26T05:07:38.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:38 smithi080 ceph-mon[29691]: 2.2 scrub starts 2024-03-26T05:07:38.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:38 smithi080 ceph-mon[29691]: 2.2 scrub ok 2024-03-26T05:07:38.695 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:38 smithi080 ceph-mon[40404]: 5.18 scrub starts 2024-03-26T05:07:38.695 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:38 smithi080 ceph-mon[40404]: 5.18 scrub ok 2024-03-26T05:07:38.695 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:38 smithi080 ceph-mon[40404]: 2.2 scrub starts 2024-03-26T05:07:38.695 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:38 smithi080 ceph-mon[40404]: 2.2 scrub ok 2024-03-26T05:07:38.741 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:38.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:38.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:38 smithi132 ceph-mon[33500]: 5.18 scrub starts 2024-03-26T05:07:38.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:38 smithi132 ceph-mon[33500]: 5.18 scrub ok 2024-03-26T05:07:38.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:38 smithi132 ceph-mon[33500]: 2.2 scrub starts 2024-03-26T05:07:38.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:38 smithi132 ceph-mon[33500]: 2.2 scrub ok 2024-03-26T05:07:38.843 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:38.887 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:38.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:38.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:38.922 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 46 16 73221 73205 99.4387 67.625 0.0122257 0.0100511 2024-03-26T05:07:38.989 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:39.019 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:39.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:39.121 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:39.149 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:39.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:39.252 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:39.278 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:39.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:39.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:39.380 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:39.408 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:39.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:39.510 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:39.550 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:39.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:39.652 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:39.679 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:39.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:39.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:39.781 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:39.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:39 smithi132 ceph-mon[33500]: pgmap v316: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 798 B/s rd, 59 MiB/s wr, 937 op/s 2024-03-26T05:07:39.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:39 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:07:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:39 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:07:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:39 smithi132 ceph-mon[33500]: 5.3 scrub starts 2024-03-26T05:07:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:39 smithi132 ceph-mon[33500]: 5.3 scrub ok 2024-03-26T05:07:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:39 smithi132 ceph-mon[33500]: 6.f scrub starts 2024-03-26T05:07:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:39 smithi132 ceph-mon[33500]: 6.f scrub ok 2024-03-26T05:07:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:39 smithi132 ceph-mon[33500]: osdmap e205: 8 total, 8 up, 7 in 2024-03-26T05:07:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:39 smithi132 ceph-mon[33500]: 3.11 scrub starts 2024-03-26T05:07:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:39 smithi132 ceph-mon[33500]: 3.11 scrub ok 2024-03-26T05:07:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:39 smithi132 ceph-mon[33500]: 4.1b scrub starts 2024-03-26T05:07:39.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:39 smithi132 ceph-mon[33500]: 4.1b scrub ok 2024-03-26T05:07:39.823 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:39.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:39.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[29691]: pgmap v316: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 798 B/s rd, 59 MiB/s wr, 937 op/s 2024-03-26T05:07:39.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:07:39.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:07:39.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[29691]: 5.3 scrub starts 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[29691]: 5.3 scrub ok 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[29691]: 6.f scrub starts 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[29691]: 6.f scrub ok 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[29691]: osdmap e205: 8 total, 8 up, 7 in 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[29691]: 3.11 scrub starts 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[29691]: 3.11 scrub ok 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[29691]: 4.1b scrub starts 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[29691]: 4.1b scrub ok 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[40404]: pgmap v316: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 798 B/s rd, 59 MiB/s wr, 937 op/s 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[40404]: 5.3 scrub starts 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[40404]: 5.3 scrub ok 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[40404]: 6.f scrub starts 2024-03-26T05:07:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[40404]: 6.f scrub ok 2024-03-26T05:07:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[40404]: osdmap e205: 8 total, 8 up, 7 in 2024-03-26T05:07:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[40404]: 3.11 scrub starts 2024-03-26T05:07:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[40404]: 3.11 scrub ok 2024-03-26T05:07:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[40404]: 4.1b scrub starts 2024-03-26T05:07:39.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:39 smithi080 ceph-mon[40404]: 4.1b scrub ok 2024-03-26T05:07:39.919 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 47 16 74395 74379 98.8841 73.375 0.0046788 0.0101077 2024-03-26T05:07:39.926 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:39.974 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:39.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:39.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:40.076 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:40.116 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:40.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:40.218 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:40.259 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:40.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:40.361 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:40.402 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:40.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:40.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:40.535 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:40.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:40.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:40.637 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:40.684 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:40.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:40.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:40.702 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:40 smithi080 ceph-mon[40404]: 6.4 scrub starts 2024-03-26T05:07:40.702 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:40 smithi080 ceph-mon[40404]: 6.4 scrub ok 2024-03-26T05:07:40.702 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:40 smithi080 ceph-mon[40404]: 3.14 scrub starts 2024-03-26T05:07:40.702 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:40 smithi080 ceph-mon[40404]: 3.14 scrub ok 2024-03-26T05:07:40.702 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:40 smithi080 ceph-mon[40404]: pgmap v318: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-26T05:07:40.702 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:40 smithi080 ceph-mon[29691]: 6.4 scrub starts 2024-03-26T05:07:40.703 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:40 smithi080 ceph-mon[29691]: 6.4 scrub ok 2024-03-26T05:07:40.703 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:40 smithi080 ceph-mon[29691]: 3.14 scrub starts 2024-03-26T05:07:40.703 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:40 smithi080 ceph-mon[29691]: 3.14 scrub ok 2024-03-26T05:07:40.703 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:40 smithi080 ceph-mon[29691]: pgmap v318: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-26T05:07:40.787 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:40.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:40 smithi132 ceph-mon[33500]: 6.4 scrub starts 2024-03-26T05:07:40.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:40 smithi132 ceph-mon[33500]: 6.4 scrub ok 2024-03-26T05:07:40.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:40 smithi132 ceph-mon[33500]: 3.14 scrub starts 2024-03-26T05:07:40.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:40 smithi132 ceph-mon[33500]: 3.14 scrub ok 2024-03-26T05:07:40.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:40 smithi132 ceph-mon[33500]: pgmap v318: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-26T05:07:40.828 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:40.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:40.919 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 48 16 75535 75519 98.3083 71.25 0.0325017 0.0101665 2024-03-26T05:07:40.930 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:40.972 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:40.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:40.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:41.074 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:41.118 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:41.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:41.220 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:41.257 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:41.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:41.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:41.359 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:41.390 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:41.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:41.491 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:41.520 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:41.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:41.622 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:41.659 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:41.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:41.761 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:41.784 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:41.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:41.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:41.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:41 smithi132 ceph-mon[33500]: 6.18 scrub starts 2024-03-26T05:07:41.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:41 smithi132 ceph-mon[33500]: 6.18 scrub ok 2024-03-26T05:07:41.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:41 smithi132 ceph-mon[33500]: osdmap e206: 8 total, 8 up, 7 in 2024-03-26T05:07:41.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:41 smithi080 ceph-mon[40404]: 6.18 scrub starts 2024-03-26T05:07:41.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:41 smithi080 ceph-mon[40404]: 6.18 scrub ok 2024-03-26T05:07:41.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:41 smithi080 ceph-mon[40404]: osdmap e206: 8 total, 8 up, 7 in 2024-03-26T05:07:41.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:41 smithi080 ceph-mon[29691]: 6.18 scrub starts 2024-03-26T05:07:41.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:41 smithi080 ceph-mon[29691]: 6.18 scrub ok 2024-03-26T05:07:41.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:41 smithi080 ceph-mon[29691]: osdmap e206: 8 total, 8 up, 7 in 2024-03-26T05:07:41.886 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:41.913 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:41.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:41.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:41.919 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 49 16 76518 76502 97.5559 61.4375 0.0108575 0.010245 2024-03-26T05:07:42.015 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:42.044 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:42.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:42.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:42.146 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:42.181 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:42.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:42.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:42.283 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:42.308 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:42.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:42.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:42.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:42.449 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:42.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:42.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:42.551 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:42.591 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:42.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:42.692 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:42.738 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:42.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:42.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:42 smithi132 ceph-mon[33500]: pgmap v320: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 57 MiB/s wr, 912 op/s 2024-03-26T05:07:42.817 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:42 smithi080 ceph-mon[29691]: pgmap v320: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 57 MiB/s wr, 912 op/s 2024-03-26T05:07:42.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:42 smithi080 ceph-mon[40404]: pgmap v320: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 57 MiB/s wr, 912 op/s 2024-03-26T05:07:42.840 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:42.868 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:42.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:42.919 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 50 16 77600 77584 96.9572 67.625 0.00673167 0.0103085 2024-03-26T05:07:42.970 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:43.003 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:43.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:43.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:43.105 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:43.133 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:43.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:43.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:43.235 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:43.263 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:43.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:43.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:43.395 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:43.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:43.497 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:43.539 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:43.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:43.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:43.641 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:43.670 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:43.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:43.677 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:43.772 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:43.812 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:43.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:43.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:43.914 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:43.920 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 51 16 78784 78768 96.5062 74 0.0241028 0.010357 2024-03-26T05:07:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:43.960 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:43.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:44.061 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:44.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:44.110 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:44.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:44.211 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:44.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:44.266 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:44.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:44.304 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:07:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:07:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:07:44.367 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:44.409 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:44.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:44.510 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:44.536 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:44.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:44.638 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:44.668 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:44.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:44.770 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:44.816 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:44.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:44.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:44.918 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:44.920 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 52 16 79829 79813 95.9063 65.3125 0.0077256 0.0104224 2024-03-26T05:07:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:44.949 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:44.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:45.050 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:45.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:45.081 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:45.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:45.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:44 smithi080 ceph-mon[40404]: 4.13 scrub starts 2024-03-26T05:07:45.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:44 smithi080 ceph-mon[40404]: 4.13 scrub ok 2024-03-26T05:07:45.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:44 smithi080 ceph-mon[40404]: pgmap v321: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 691 B/s rd, 46 MiB/s wr, 739 op/s 2024-03-26T05:07:45.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:44 smithi080 ceph-mon[40404]: 5.6 scrub starts 2024-03-26T05:07:45.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:44 smithi080 ceph-mon[40404]: 5.6 scrub ok 2024-03-26T05:07:45.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:44 smithi080 ceph-mon[29691]: 4.13 scrub starts 2024-03-26T05:07:45.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:44 smithi080 ceph-mon[29691]: 4.13 scrub ok 2024-03-26T05:07:45.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:44 smithi080 ceph-mon[29691]: pgmap v321: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 691 B/s rd, 46 MiB/s wr, 739 op/s 2024-03-26T05:07:45.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:44 smithi080 ceph-mon[29691]: 5.6 scrub starts 2024-03-26T05:07:45.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:44 smithi080 ceph-mon[29691]: 5.6 scrub ok 2024-03-26T05:07:45.182 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:45.225 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:45.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:45.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:45.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:44 smithi132 ceph-mon[33500]: 4.13 scrub starts 2024-03-26T05:07:45.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:44 smithi132 ceph-mon[33500]: 4.13 scrub ok 2024-03-26T05:07:45.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:44 smithi132 ceph-mon[33500]: pgmap v321: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 691 B/s rd, 46 MiB/s wr, 739 op/s 2024-03-26T05:07:45.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:44 smithi132 ceph-mon[33500]: 5.6 scrub starts 2024-03-26T05:07:45.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:44 smithi132 ceph-mon[33500]: 5.6 scrub ok 2024-03-26T05:07:45.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:07:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:07:45.326 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:45.374 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:45.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:45.475 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:45.482 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:07:45.489 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:07:45.504 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:45.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:45.527 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:07:45.566 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:07:45.602 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:07:45.605 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:45.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:45.635 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:45.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:45.643 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:07:45.687 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:07:45.724 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:07:45.732 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:07:45.732 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:07:45.732 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:07:45.732 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:07:45.732 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:07:45.732 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:07:45.732 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:07:45.732 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:07:45.737 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:45.764 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:45.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:45.867 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:45.911 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:07:45.911 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:45.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:45.920 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 53 16 80905 80889 95.3656 67.25 0.0261615 0.0104802 2024-03-26T05:07:46.008 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:45 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3559068952' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:07:46.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:45 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:07:46.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:45 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:46.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:45 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1319363562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:07:46.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:45 smithi080 ceph-mon[40404]: 6.c scrub starts 2024-03-26T05:07:46.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:45 smithi080 ceph-mon[40404]: 6.c scrub ok 2024-03-26T05:07:46.009 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:45 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3559068952' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:07:46.009 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:45 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:07:46.009 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:45 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:46.009 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:45 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1319363562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:07:46.009 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:45 smithi080 ceph-mon[29691]: 6.c scrub starts 2024-03-26T05:07:46.009 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:45 smithi080 ceph-mon[29691]: 6.c scrub ok 2024-03-26T05:07:46.014 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:46.057 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:46.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:46.158 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:46.200 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:46.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:46.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:46.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:45 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3559068952' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:07:46.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:45 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:07:46.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:45 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:46.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:45 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1319363562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:07:46.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:45 smithi132 ceph-mon[33500]: 6.c scrub starts 2024-03-26T05:07:46.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:45 smithi132 ceph-mon[33500]: 6.c scrub ok 2024-03-26T05:07:46.302 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:46.343 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:46.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:46.445 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:46.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:46.484 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:46.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:46.585 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:46.614 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:46.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:46.716 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:46.755 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:46.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:46.857 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:46.905 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:46.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:46.920 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 54 15 82051 82036 94.9271 71.6875 0.0109551 0.0105284 2024-03-26T05:07:47.006 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:47.035 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:47.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:47.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[40404]: pgmap v322: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.37k op/s 2024-03-26T05:07:47.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:07:47.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[40404]: Cluster is now healthy 2024-03-26T05:07:47.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:07:47.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[40404]: osdmap e207: 8 total, 8 up, 7 in 2024-03-26T05:07:47.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[40404]: 5.b scrub starts 2024-03-26T05:07:47.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[40404]: 5.b scrub ok 2024-03-26T05:07:47.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[40404]: osdmap e208: 8 total, 8 up, 7 in 2024-03-26T05:07:47.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[40404]: 4.19 scrub starts 2024-03-26T05:07:47.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[40404]: 4.19 scrub ok 2024-03-26T05:07:47.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[29691]: pgmap v322: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.37k op/s 2024-03-26T05:07:47.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:07:47.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[29691]: Cluster is now healthy 2024-03-26T05:07:47.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:07:47.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[29691]: osdmap e207: 8 total, 8 up, 7 in 2024-03-26T05:07:47.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[29691]: 5.b scrub starts 2024-03-26T05:07:47.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[29691]: 5.b scrub ok 2024-03-26T05:07:47.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[29691]: osdmap e208: 8 total, 8 up, 7 in 2024-03-26T05:07:47.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[29691]: 4.19 scrub starts 2024-03-26T05:07:47.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:46 smithi080 ceph-mon[29691]: 4.19 scrub ok 2024-03-26T05:07:47.135 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:47.165 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:47.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:47.265 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:47.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:46 smithi132 ceph-mon[33500]: pgmap v322: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.37k op/s 2024-03-26T05:07:47.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:46 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:07:47.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:46 smithi132 ceph-mon[33500]: Cluster is now healthy 2024-03-26T05:07:47.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:46 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:07:47.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:46 smithi132 ceph-mon[33500]: osdmap e207: 8 total, 8 up, 7 in 2024-03-26T05:07:47.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:46 smithi132 ceph-mon[33500]: 5.b scrub starts 2024-03-26T05:07:47.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:46 smithi132 ceph-mon[33500]: 5.b scrub ok 2024-03-26T05:07:47.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:46 smithi132 ceph-mon[33500]: osdmap e208: 8 total, 8 up, 7 in 2024-03-26T05:07:47.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:46 smithi132 ceph-mon[33500]: 4.19 scrub starts 2024-03-26T05:07:47.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:46 smithi132 ceph-mon[33500]: 4.19 scrub ok 2024-03-26T05:07:47.310 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:47.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:47.412 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:47.465 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:47.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:47.567 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:47.596 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:47.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:47.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:47.698 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:47.729 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:47.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:47.832 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:47.865 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:47.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:47.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:47.920 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 55 16 83195 83179 94.5 71.4375 0.0296406 0.0105762 2024-03-26T05:07:47.967 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:48.001 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:48.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:48.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:48.103 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:48.133 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:48.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:48.235 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:48.270 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:48.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:47 smithi132 ceph-mon[33500]: 6.14 scrub starts 2024-03-26T05:07:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:47 smithi132 ceph-mon[33500]: 6.14 scrub ok 2024-03-26T05:07:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:47 smithi132 ceph-mon[33500]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:07:48.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:47 smithi080 ceph-mon[40404]: 6.14 scrub starts 2024-03-26T05:07:48.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:47 smithi080 ceph-mon[40404]: 6.14 scrub ok 2024-03-26T05:07:48.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:47 smithi080 ceph-mon[40404]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:07:48.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:47 smithi080 ceph-mon[29691]: 6.14 scrub starts 2024-03-26T05:07:48.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:47 smithi080 ceph-mon[29691]: 6.14 scrub ok 2024-03-26T05:07:48.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:47 smithi080 ceph-mon[29691]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:07:48.372 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:48.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:48.410 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:48.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:48.511 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:48.538 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:48.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:48.639 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:48.674 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:48.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:48.776 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:48.804 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:48.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:48.906 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:48.920 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 56 16 84253 84237 93.9932 66.125 0.00928492 0.0106342 2024-03-26T05:07:48.936 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:48.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:48.958 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:07:48.964 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:07:48.965 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:07:49.038 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:49.091 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:49.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:49.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:49.255 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:49.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:49.268 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:48 smithi080 ceph-mon[29691]: pgmap v325: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 798 B/s rd, 53 MiB/s wr, 850 op/s 2024-03-26T05:07:49.268 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:48 smithi080 ceph-mon[29691]: 6.6 scrub starts 2024-03-26T05:07:49.268 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:48 smithi080 ceph-mon[29691]: 6.6 scrub ok 2024-03-26T05:07:49.268 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:48 smithi080 ceph-mon[40404]: pgmap v325: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 798 B/s rd, 53 MiB/s wr, 850 op/s 2024-03-26T05:07:49.269 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:48 smithi080 ceph-mon[40404]: 6.6 scrub starts 2024-03-26T05:07:49.269 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:48 smithi080 ceph-mon[40404]: 6.6 scrub ok 2024-03-26T05:07:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:48 smithi132 ceph-mon[33500]: pgmap v325: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 798 B/s rd, 53 MiB/s wr, 850 op/s 2024-03-26T05:07:49.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:48 smithi132 ceph-mon[33500]: 6.6 scrub starts 2024-03-26T05:07:49.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:48 smithi132 ceph-mon[33500]: 6.6 scrub ok 2024-03-26T05:07:49.356 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:49.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:49.412 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:49.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:49.513 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:49.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:49.560 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:49.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:49.661 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:49.703 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:49.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:49.805 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:49.833 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:49.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:49.921 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 57 16 85293 85277 93.4846 65 0.00819998 0.0106922 2024-03-26T05:07:49.934 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:49.972 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:49.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:49.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:50.074 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:50.122 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:50.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:50.223 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:50.254 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:50.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:50.356 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:50.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:50.399 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:50.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:50.448 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:50.481 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:07:50.482 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:07:50.500 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:50.567 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:50.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:50.669 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:50.705 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:50.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:50.807 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:50.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:50.850 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:50.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:50.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:07:50.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:07:50.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[40404]: 8.9 scrub starts 2024-03-26T05:07:50.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:50.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:07:50.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[40404]: 3.9 scrub starts 2024-03-26T05:07:50.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[40404]: 3.9 scrub ok 2024-03-26T05:07:50.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[40404]: pgmap v326: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 899 op/s 2024-03-26T05:07:50.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[40404]: osdmap e209: 8 total, 8 up, 7 in 2024-03-26T05:07:50.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:07:50.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:07:50.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[29691]: 8.9 scrub starts 2024-03-26T05:07:50.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:50.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:07:50.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[29691]: 3.9 scrub starts 2024-03-26T05:07:50.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[29691]: 3.9 scrub ok 2024-03-26T05:07:50.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[29691]: pgmap v326: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 899 op/s 2024-03-26T05:07:50.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:50 smithi080 ceph-mon[29691]: osdmap e209: 8 total, 8 up, 7 in 2024-03-26T05:07:50.921 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 58 16 86356 86340 93.0182 66.4375 0.0108625 0.0107452 2024-03-26T05:07:50.951 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-03-26T05:07:46.431938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5245114718,"num_objects":80255,"num_object_clones":0,"num_object_copies":240765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80255,"num_whiteouts":0,"num_read":1656,"num_read_kb":1470,"num_write":80832,"num_write_kb":5124061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":1773004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74775,"ondisk_log_size":74775,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16665448,"kb_used_data":15257456,"kb_used_omap":10,"kb_used_meta":1407925,"kb_avail":639550616,"statfs":{"total":671965249536,"available":654899830784,"internally_reserved":0,"allocated":15623634944,"data_stored":15620732190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1441715307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,5,9,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":608370688,"num_objects":9283,"num_object_clones":0,"num_object_copies":27849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9283,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9283,"num_write_kb":594112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.591881"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":307,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.431942+0000","last_change":"2024-03-26T05:06:08.113386+0000","last_active":"2024-03-26T05:07:41.431942+0000","last_peered":"2024-03-26T05:07:41.431942+0000","last_clean":"2024-03-26T05:07:41.431942+0000","last_became_active":"2024-03-26T05:05:36.407230+0000","last_became_peered":"2024-03-26T05:05:36.407230+0000","last_unstale":"2024-03-26T05:07:41.431942+0000","last_undegraded":"2024-03-26T05:07:41.431942+0000","last_fullsized":"2024-03-26T05:07:41.431942+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:08.113253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:08.113253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":310,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448098+0000","last_change":"2024-03-26T05:05:36.404129+0000","last_active":"2024-03-26T05:07:41.448098+0000","last_peered":"2024-03-26T05:07:41.448098+0000","last_clean":"2024-03-26T05:07:41.448098+0000","last_became_active":"2024-03-26T05:05:36.403620+0000","last_became_peered":"2024-03-26T05:05:36.403620+0000","last_unstale":"2024-03-26T05:07:41.448098+0000","last_undegraded":"2024-03-26T05:07:41.448098+0000","last_fullsized":"2024-03-26T05:07:41.448098+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:02.872031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:02.872031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":320,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449789+0000","last_change":"2024-03-26T05:05:36.411757+0000","last_active":"2024-03-26T05:07:41.449789+0000","last_peered":"2024-03-26T05:07:41.449789+0000","last_clean":"2024-03-26T05:07:41.449789+0000","last_became_active":"2024-03-26T05:05:36.411231+0000","last_became_peered":"2024-03-26T05:05:36.411231+0000","last_unstale":"2024-03-26T05:07:41.449789+0000","last_undegraded":"2024-03-26T05:07:41.449789+0000","last_fullsized":"2024-03-26T05:07:41.449789+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.016730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.016730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'15","reported_seq":338,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.431648+0000","last_change":"2024-03-26T05:06:52.298546+0000","last_active":"2024-03-26T05:07:41.431648+0000","last_peered":"2024-03-26T05:07:41.431648+0000","last_clean":"2024-03-26T05:07:41.431648+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:07:41.431648+0000","last_undegraded":"2024-03-26T05:07:41.431648+0000","last_fullsized":"2024-03-26T05:07:41.431648+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:06:52.298418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:52.298418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:17.770238+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":292,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449064+0000","last_change":"2024-03-26T05:06:37.226644+0000","last_active":"2024-03-26T05:07:41.449064+0000","last_peered":"2024-03-26T05:07:41.449064+0000","last_clean":"2024-03-26T05:07:41.449064+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:07:41.449064+0000","last_undegraded":"2024-03-26T05:07:41.449064+0000","last_fullsized":"2024-03-26T05:07:41.449064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:37.226521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:37.226521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:50.929750+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":306,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.431591+0000","last_change":"2024-03-26T05:07:36.616676+0000","last_active":"2024-03-26T05:07:41.431591+0000","last_peered":"2024-03-26T05:07:41.431591+0000","last_clean":"2024-03-26T05:07:41.431591+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:07:41.431591+0000","last_undegraded":"2024-03-26T05:07:41.431591+0000","last_fullsized":"2024-03-26T05:07:41.431591+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:36.616569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:36.616569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:52.540577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":313,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.450918+0000","last_change":"2024-03-26T05:05:12.713135+0000","last_active":"2024-03-26T05:07:41.450918+0000","last_peered":"2024-03-26T05:07:41.450918+0000","last_clean":"2024-03-26T05:07:41.450918+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:07:41.450918+0000","last_undegraded":"2024-03-26T05:07:41.450918+0000","last_fullsized":"2024-03-26T05:07:41.450918+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.713011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'39","reported_seq":374,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.447090+0000","last_change":"2024-03-26T05:04:20.637111+0000","last_active":"2024-03-26T05:07:41.447090+0000","last_peered":"2024-03-26T05:07:41.447090+0000","last_clean":"2024-03-26T05:07:41.447090+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:07:41.447090+0000","last_undegraded":"2024-03-26T05:07:41.447090+0000","last_fullsized":"2024-03-26T05:07:41.447090+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'39","last_scrub_stamp":"2024-03-26T05:04:20.636687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:20.636687+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":310,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.450908+0000","last_change":"2024-03-26T05:07:40.028960+0000","last_active":"2024-03-26T05:07:41.450908+0000","last_peered":"2024-03-26T05:07:41.450908+0000","last_clean":"2024-03-26T05:07:41.450908+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:07:41.450908+0000","last_undegraded":"2024-03-26T05:07:41.450908+0000","last_fullsized":"2024-03-26T05:07:41.450908+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:40.028690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:40.028690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:43.963276+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":102,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449533+0000","last_change":"2024-03-26T05:05:36.409043+0000","last_active":"2024-03-26T05:07:41.449533+0000","last_peered":"2024-03-26T05:07:41.449533+0000","last_clean":"2024-03-26T05:07:41.449533+0000","last_became_active":"2024-03-26T05:05:36.408618+0000","last_became_peered":"2024-03-26T05:05:36.408618+0000","last_unstale":"2024-03-26T05:07:41.449533+0000","last_undegraded":"2024-03-26T05:07:41.449533+0000","last_fullsized":"2024-03-26T05:07:41.449533+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:10.922975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:10.922975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":111,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432871+0000","last_change":"2024-03-26T05:06:38.336990+0000","last_active":"2024-03-26T05:07:41.432871+0000","last_peered":"2024-03-26T05:07:41.432871+0000","last_clean":"2024-03-26T05:07:41.432871+0000","last_became_active":"2024-03-26T05:05:36.411627+0000","last_became_peered":"2024-03-26T05:05:36.411627+0000","last_unstale":"2024-03-26T05:07:41.432871+0000","last_undegraded":"2024-03-26T05:07:41.432871+0000","last_fullsized":"2024-03-26T05:07:41.432871+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:38.336853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:38.336853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:54.884679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"75'21","reported_seq":349,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432122+0000","last_change":"2024-03-26T05:05:53.230294+0000","last_active":"2024-03-26T05:07:41.432122+0000","last_peered":"2024-03-26T05:07:41.432122+0000","last_clean":"2024-03-26T05:07:41.432122+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:07:41.432122+0000","last_undegraded":"2024-03-26T05:07:41.432122+0000","last_fullsized":"2024-03-26T05:07:41.432122+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:05:53.230145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:53.230145+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":102,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.431898+0000","last_change":"2024-03-26T05:05:36.424628+0000","last_active":"2024-03-26T05:07:41.431898+0000","last_peered":"2024-03-26T05:07:41.431898+0000","last_clean":"2024-03-26T05:07:41.431898+0000","last_became_active":"2024-03-26T05:05:36.424269+0000","last_became_peered":"2024-03-26T05:05:36.424269+0000","last_unstale":"2024-03-26T05:07:41.431898+0000","last_undegraded":"2024-03-26T05:07:41.431898+0000","last_fullsized":"2024-03-26T05:07:41.431898+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:12.931045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.931045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":303,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.439835+0000","last_change":"2024-03-26T05:05:13.726750+0000","last_active":"2024-03-26T05:07:41.439835+0000","last_peered":"2024-03-26T05:07:41.439835+0000","last_clean":"2024-03-26T05:07:41.439835+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:07:41.439835+0000","last_undegraded":"2024-03-26T05:07:41.439835+0000","last_fullsized":"2024-03-26T05:07:41.439835+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:13.726612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:13.726612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":87,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.436379+0000","last_change":"2024-03-26T05:06:06.962791+0000","last_active":"2024-03-26T05:07:41.436379+0000","last_peered":"2024-03-26T05:07:41.436379+0000","last_clean":"2024-03-26T05:07:41.436379+0000","last_became_active":"2024-03-26T05:06:06.518987+0000","last_became_peered":"2024-03-26T05:06:06.518987+0000","last_unstale":"2024-03-26T05:07:41.436379+0000","last_undegraded":"2024-03-26T05:07:41.436379+0000","last_fullsized":"2024-03-26T05:07:41.436379+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:06.962665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:06.962665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"75'11","reported_seq":331,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.431946+0000","last_change":"2024-03-26T05:07:38.699595+0000","last_active":"2024-03-26T05:07:41.431946+0000","last_peered":"2024-03-26T05:07:41.431946+0000","last_clean":"2024-03-26T05:07:41.431946+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:07:41.431946+0000","last_undegraded":"2024-03-26T05:07:41.431946+0000","last_fullsized":"2024-03-26T05:07:41.431946+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-26T05:07:38.699415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:38.699415+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:57.390111+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":316,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432568+0000","last_change":"2024-03-26T05:06:51.272075+0000","last_active":"2024-03-26T05:07:41.432568+0000","last_peered":"2024-03-26T05:07:41.432568+0000","last_clean":"2024-03-26T05:07:41.432568+0000","last_became_active":"2024-03-26T05:05:36.416850+0000","last_became_peered":"2024-03-26T05:05:36.416850+0000","last_unstale":"2024-03-26T05:07:41.432568+0000","last_undegraded":"2024-03-26T05:07:41.432568+0000","last_fullsized":"2024-03-26T05:07:41.432568+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:51.271966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:06:51.271966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:59.127606+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.446980+0000","last_change":"2024-03-26T05:06:00.105160+0000","last_active":"2024-03-26T05:07:41.446980+0000","last_peered":"2024-03-26T05:07:41.446980+0000","last_clean":"2024-03-26T05:07:41.446980+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:07:41.446980+0000","last_undegraded":"2024-03-26T05:07:41.446980+0000","last_fullsized":"2024-03-26T05:07:41.446980+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:00.104920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:00.104920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":313,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449693+0000","last_change":"2024-03-26T05:05:08.044042+0000","last_active":"2024-03-26T05:07:41.449693+0000","last_peered":"2024-03-26T05:07:41.449693+0000","last_clean":"2024-03-26T05:07:41.449693+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:07:41.449693+0000","last_undegraded":"2024-03-26T05:07:41.449693+0000","last_fullsized":"2024-03-26T05:07:41.449693+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:08.043732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:09.914470+0000","last_clean_scrub_stamp":"2024-03-26T05:05:08.043732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":341,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.436921+0000","last_change":"2024-03-26T05:06:45.225054+0000","last_active":"2024-03-26T05:07:41.436921+0000","last_peered":"2024-03-26T05:07:41.436921+0000","last_clean":"2024-03-26T05:07:41.436921+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:07:41.436921+0000","last_undegraded":"2024-03-26T05:07:41.436921+0000","last_fullsized":"2024-03-26T05:07:41.436921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:06:45.224956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.224956+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:58.091504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":307,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.451385+0000","last_change":"2024-03-26T05:06:02.957611+0000","last_active":"2024-03-26T05:07:41.451385+0000","last_peered":"2024-03-26T05:07:41.451385+0000","last_clean":"2024-03-26T05:07:41.451385+0000","last_became_active":"2024-03-26T05:05:36.427435+0000","last_became_peered":"2024-03-26T05:05:36.427435+0000","last_unstale":"2024-03-26T05:07:41.451385+0000","last_undegraded":"2024-03-26T05:07:41.451385+0000","last_fullsized":"2024-03-26T05:07:41.451385+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:02.957488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:05.268043+0000","last_clean_scrub_stamp":"2024-03-26T05:06:02.957488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":305,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432450+0000","last_change":"2024-03-26T05:06:48.271761+0000","last_active":"2024-03-26T05:07:41.432450+0000","last_peered":"2024-03-26T05:07:41.432450+0000","last_clean":"2024-03-26T05:07:41.432450+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:07:41.432450+0000","last_undegraded":"2024-03-26T05:07:41.432450+0000","last_fullsized":"2024-03-26T05:07:41.432450+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:48.271668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:48.271668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:15.551094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":321,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432234+0000","last_change":"2024-03-26T05:06:00.026433+0000","last_active":"2024-03-26T05:07:41.432234+0000","last_peered":"2024-03-26T05:07:41.432234+0000","last_clean":"2024-03-26T05:07:41.432234+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:07:41.432234+0000","last_undegraded":"2024-03-26T05:07:41.432234+0000","last_fullsized":"2024-03-26T05:07:41.432234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:00.026316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:06:00.026316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'20","reported_seq":337,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.450573+0000","last_change":"2024-03-26T05:04:07.528781+0000","last_active":"2024-03-26T05:07:41.450573+0000","last_peered":"2024-03-26T05:07:41.450573+0000","last_clean":"2024-03-26T05:07:41.450573+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:07:41.450573+0000","last_undegraded":"2024-03-26T05:07:41.450573+0000","last_fullsized":"2024-03-26T05:07:41.450573+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:04:07.528619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:07.528619+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":310,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.440388+0000","last_change":"2024-03-26T05:05:41.875836+0000","last_active":"2024-03-26T05:07:41.440388+0000","last_peered":"2024-03-26T05:07:41.440388+0000","last_clean":"2024-03-26T05:07:41.440388+0000","last_became_active":"2024-03-26T05:05:36.416944+0000","last_became_peered":"2024-03-26T05:05:36.416944+0000","last_unstale":"2024-03-26T05:07:41.440388+0000","last_undegraded":"2024-03-26T05:07:41.440388+0000","last_fullsized":"2024-03-26T05:07:41.440388+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:05:41.875743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:41.875743+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":420,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.442221+0000","last_change":"2024-03-26T05:06:53.544685+0000","last_active":"2024-03-26T05:07:41.442221+0000","last_peered":"2024-03-26T05:07:41.442221+0000","last_clean":"2024-03-26T05:07:41.442221+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:07:41.442221+0000","last_undegraded":"2024-03-26T05:07:41.442221+0000","last_fullsized":"2024-03-26T05:07:41.442221+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:06:53.544604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:53.544604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:00.641956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":314,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448225+0000","last_change":"2024-03-26T05:06:04.028509+0000","last_active":"2024-03-26T05:07:41.448225+0000","last_peered":"2024-03-26T05:07:41.448225+0000","last_clean":"2024-03-26T05:07:41.448225+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:07:41.448225+0000","last_undegraded":"2024-03-26T05:07:41.448225+0000","last_fullsized":"2024-03-26T05:07:41.448225+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:06:04.028383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:04.028383+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'16","reported_seq":337,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.450047+0000","last_change":"2024-03-26T05:05:13.083706+0000","last_active":"2024-03-26T05:07:41.450047+0000","last_peered":"2024-03-26T05:07:41.450047+0000","last_clean":"2024-03-26T05:07:41.450047+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:07:41.450047+0000","last_undegraded":"2024-03-26T05:07:41.450047+0000","last_fullsized":"2024-03-26T05:07:41.450047+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:05:13.083420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:13.083420+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":306,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.447128+0000","last_change":"2024-03-26T05:05:36.403232+0000","last_active":"2024-03-26T05:07:41.447128+0000","last_peered":"2024-03-26T05:07:41.447128+0000","last_clean":"2024-03-26T05:07:41.447128+0000","last_became_active":"2024-03-26T05:05:36.402759+0000","last_became_peered":"2024-03-26T05:05:36.402759+0000","last_unstale":"2024-03-26T05:07:41.447128+0000","last_undegraded":"2024-03-26T05:07:41.447128+0000","last_fullsized":"2024-03-26T05:07:41.447128+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:55.814330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:55.814330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":306,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.450365+0000","last_change":"2024-03-26T05:05:53.093643+0000","last_active":"2024-03-26T05:07:41.450365+0000","last_peered":"2024-03-26T05:07:41.450365+0000","last_clean":"2024-03-26T05:07:41.450365+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:07:41.450365+0000","last_undegraded":"2024-03-26T05:07:41.450365+0000","last_fullsized":"2024-03-26T05:07:41.450365+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:53.093539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:53.093539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":313,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.433166+0000","last_change":"2024-03-26T05:06:00.960334+0000","last_active":"2024-03-26T05:07:41.433166+0000","last_peered":"2024-03-26T05:07:41.433166+0000","last_clean":"2024-03-26T05:07:41.433166+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:07:41.433166+0000","last_undegraded":"2024-03-26T05:07:41.433166+0000","last_fullsized":"2024-03-26T05:07:41.433166+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:00.960212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:00.960212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'17","reported_seq":387,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.442247+0000","last_change":"2024-03-26T05:06:47.435312+0000","last_active":"2024-03-26T05:07:41.442247+0000","last_peered":"2024-03-26T05:07:41.442247+0000","last_clean":"2024-03-26T05:07:41.442247+0000","last_became_active":"2024-03-26T05:05:36.410009+0000","last_became_peered":"2024-03-26T05:05:36.410009+0000","last_unstale":"2024-03-26T05:07:41.442247+0000","last_undegraded":"2024-03-26T05:07:41.442247+0000","last_fullsized":"2024-03-26T05:07:41.442247+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:06:47.435205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:47.435205+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:14.788909+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":292,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.439718+0000","last_change":"2024-03-26T05:04:27.007041+0000","last_active":"2024-03-26T05:07:41.439718+0000","last_peered":"2024-03-26T05:07:41.439718+0000","last_clean":"2024-03-26T05:07:41.439718+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:07:41.439718+0000","last_undegraded":"2024-03-26T05:07:41.439718+0000","last_fullsized":"2024-03-26T05:07:41.439718+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:27.006904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:27.006904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":312,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.441680+0000","last_change":"2024-03-26T05:06:42.481238+0000","last_active":"2024-03-26T05:07:41.441680+0000","last_peered":"2024-03-26T05:07:41.441680+0000","last_clean":"2024-03-26T05:07:41.441680+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:07:41.441680+0000","last_undegraded":"2024-03-26T05:07:41.441680+0000","last_fullsized":"2024-03-26T05:07:41.441680+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:42.481122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:42.481122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:08.491912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":320,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449962+0000","last_change":"2024-03-26T05:05:59.138866+0000","last_active":"2024-03-26T05:07:41.449962+0000","last_peered":"2024-03-26T05:07:41.449962+0000","last_clean":"2024-03-26T05:07:41.449962+0000","last_became_active":"2024-03-26T05:05:36.423767+0000","last_became_peered":"2024-03-26T05:05:36.423767+0000","last_unstale":"2024-03-26T05:07:41.449962+0000","last_undegraded":"2024-03-26T05:07:41.449962+0000","last_fullsized":"2024-03-26T05:07:41.449962+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:59.138510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:59.138510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"100'6","reported_seq":317,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.433237+0000","last_change":"2024-03-26T05:06:45.224407+0000","last_active":"2024-03-26T05:07:41.433237+0000","last_peered":"2024-03-26T05:07:41.433237+0000","last_clean":"2024-03-26T05:07:41.433237+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:07:41.433237+0000","last_undegraded":"2024-03-26T05:07:41.433237+0000","last_fullsized":"2024-03-26T05:07:41.433237+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'6","last_scrub_stamp":"2024-03-26T05:06:45.224296+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.224296+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:50.976141+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":301,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449230+0000","last_change":"2024-03-26T05:06:49.226016+0000","last_active":"2024-03-26T05:07:41.449230+0000","last_peered":"2024-03-26T05:07:41.449230+0000","last_clean":"2024-03-26T05:07:41.449230+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:07:41.449230+0000","last_undegraded":"2024-03-26T05:07:41.449230+0000","last_fullsized":"2024-03-26T05:07:41.449230+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_clean_scrub_stamp":"2024-03-26T05:06:49.225897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:16.804230+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.440357+0000","last_change":"2024-03-26T05:05:12.747692+0000","last_active":"2024-03-26T05:07:41.440357+0000","last_peered":"2024-03-26T05:07:41.440357+0000","last_clean":"2024-03-26T05:07:41.440357+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:07:41.440357+0000","last_undegraded":"2024-03-26T05:07:41.440357+0000","last_fullsized":"2024-03-26T05:07:41.440357+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:12.747554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.747554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":132,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.440388+0000","last_change":"2024-03-26T05:06:56.886716+0000","last_active":"2024-03-26T05:07:41.440388+0000","last_peered":"2024-03-26T05:07:41.440388+0000","last_clean":"2024-03-26T05:07:41.440388+0000","last_became_active":"2024-03-26T05:05:36.417290+0000","last_became_peered":"2024-03-26T05:05:36.417290+0000","last_unstale":"2024-03-26T05:07:41.440388+0000","last_undegraded":"2024-03-26T05:07:41.440388+0000","last_fullsized":"2024-03-26T05:07:41.440388+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:06:56.886632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:07.878905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'17","reported_seq":332,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448496+0000","last_change":"2024-03-26T05:04:09.559560+0000","last_active":"2024-03-26T05:07:41.448496+0000","last_peered":"2024-03-26T05:07:41.448496+0000","last_clean":"2024-03-26T05:07:41.448496+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:07:41.448496+0000","last_undegraded":"2024-03-26T05:07:41.448496+0000","last_fullsized":"2024-03-26T05:07:41.448496+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:04:09.559412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:04:09.559412+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":316,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.438112+0000","last_change":"2024-03-26T05:06:54.126622+0000","last_active":"2024-03-26T05:07:41.438112+0000","last_peered":"2024-03-26T05:07:41.438112+0000","last_clean":"2024-03-26T05:07:41.438112+0000","last_became_active":"2024-03-26T05:05:36.413880+0000","last_became_peered":"2024-03-26T05:05:36.413880+0000","last_unstale":"2024-03-26T05:07:41.438112+0000","last_undegraded":"2024-03-26T05:07:41.438112+0000","last_fullsized":"2024-03-26T05:07:41.438112+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:54.126517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:54.126517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:06.674857+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":297,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448965+0000","last_change":"2024-03-26T05:05:04.767968+0000","last_active":"2024-03-26T05:07:41.448965+0000","last_peered":"2024-03-26T05:07:41.448965+0000","last_clean":"2024-03-26T05:07:41.448965+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:07:41.448965+0000","last_undegraded":"2024-03-26T05:07:41.448965+0000","last_fullsized":"2024-03-26T05:07:41.448965+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.767790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.440774+0000","last_change":"2024-03-26T05:05:04.764827+0000","last_active":"2024-03-26T05:07:41.440774+0000","last_peered":"2024-03-26T05:07:41.440774+0000","last_clean":"2024-03-26T05:07:41.440774+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:07:41.440774+0000","last_undegraded":"2024-03-26T05:07:41.440774+0000","last_fullsized":"2024-03-26T05:07:41.440774+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.764742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":325,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449521+0000","last_change":"2024-03-26T05:05:14.064119+0000","last_active":"2024-03-26T05:07:41.449521+0000","last_peered":"2024-03-26T05:07:41.449521+0000","last_clean":"2024-03-26T05:07:41.449521+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:07:41.449521+0000","last_undegraded":"2024-03-26T05:07:41.449521+0000","last_fullsized":"2024-03-26T05:07:41.449521+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-26T05:05:14.063125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:14.063125+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":299,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.433075+0000","last_change":"2024-03-26T05:05:36.426512+0000","last_active":"2024-03-26T05:07:41.433075+0000","last_peered":"2024-03-26T05:07:41.433075+0000","last_clean":"2024-03-26T05:07:41.433075+0000","last_became_active":"2024-03-26T05:05:36.425590+0000","last_became_peered":"2024-03-26T05:05:36.425590+0000","last_unstale":"2024-03-26T05:07:41.433075+0000","last_undegraded":"2024-03-26T05:07:41.433075+0000","last_fullsized":"2024-03-26T05:07:41.433075+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:58.834141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.834141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":319,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449471+0000","last_change":"2024-03-26T05:06:42.246793+0000","last_active":"2024-03-26T05:07:41.449471+0000","last_peered":"2024-03-26T05:07:41.449471+0000","last_clean":"2024-03-26T05:07:41.449471+0000","last_became_active":"2024-03-26T05:05:36.416479+0000","last_became_peered":"2024-03-26T05:05:36.416479+0000","last_unstale":"2024-03-26T05:07:41.449471+0000","last_undegraded":"2024-03-26T05:07:41.449471+0000","last_fullsized":"2024-03-26T05:07:41.449471+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:42.246721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:06:42.246721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:43.713447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":314,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432304+0000","last_change":"2024-03-26T05:07:39.699241+0000","last_active":"2024-03-26T05:07:41.432304+0000","last_peered":"2024-03-26T05:07:41.432304+0000","last_clean":"2024-03-26T05:07:41.432304+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:07:41.432304+0000","last_undegraded":"2024-03-26T05:07:41.432304+0000","last_fullsized":"2024-03-26T05:07:41.432304+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:39.699122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:39.699122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:05.562610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"95'16","reported_seq":342,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:43.754203+0000","last_change":"2024-03-26T05:07:43.754242+0000","last_active":"2024-03-26T05:07:43.754203+0000","last_peered":"2024-03-26T05:07:43.754203+0000","last_clean":"2024-03-26T05:07:43.754203+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:07:43.754203+0000","last_undegraded":"2024-03-26T05:07:43.754203+0000","last_fullsized":"2024-03-26T05:07:43.754203+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:07:43.754130+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:07:43.754130+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:10.909529+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":302,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449799+0000","last_change":"2024-03-26T05:06:04.057848+0000","last_active":"2024-03-26T05:07:41.449799+0000","last_peered":"2024-03-26T05:07:41.449799+0000","last_clean":"2024-03-26T05:07:41.449799+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:07:41.449799+0000","last_undegraded":"2024-03-26T05:07:41.449799+0000","last_fullsized":"2024-03-26T05:07:41.449799+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_clean_scrub_stamp":"2024-03-26T05:06:04.057358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"161'9","reported_seq":166,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.446860+0000","last_change":"2024-03-26T05:06:47.137171+0000","last_active":"2024-03-26T05:07:41.446860+0000","last_peered":"2024-03-26T05:07:41.446860+0000","last_clean":"2024-03-26T05:07:41.446860+0000","last_became_active":"2024-03-26T05:05:36.422468+0000","last_became_peered":"2024-03-26T05:05:36.422468+0000","last_unstale":"2024-03-26T05:07:41.446860+0000","last_undegraded":"2024-03-26T05:07:41.446860+0000","last_fullsized":"2024-03-26T05:07:41.446860+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'9","last_scrub_stamp":"2024-03-26T05:06:47.137084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:47.137084+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:47.227857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":312,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.439885+0000","last_change":"2024-03-26T05:05:08.856522+0000","last_active":"2024-03-26T05:07:41.439885+0000","last_peered":"2024-03-26T05:07:41.439885+0000","last_clean":"2024-03-26T05:07:41.439885+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:07:41.439885+0000","last_undegraded":"2024-03-26T05:07:41.439885+0000","last_fullsized":"2024-03-26T05:07:41.439885+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:08.856401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:08.856401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'22","reported_seq":348,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448793+0000","last_change":"2024-03-26T05:05:08.701087+0000","last_active":"2024-03-26T05:07:41.448793+0000","last_peered":"2024-03-26T05:07:41.448793+0000","last_clean":"2024-03-26T05:07:41.448793+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:07:41.448793+0000","last_undegraded":"2024-03-26T05:07:41.448793+0000","last_fullsized":"2024-03-26T05:07:41.448793+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:05:08.700945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:08.700945+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":308,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432374+0000","last_change":"2024-03-26T05:07:29.768641+0000","last_active":"2024-03-26T05:07:41.432374+0000","last_peered":"2024-03-26T05:07:41.432374+0000","last_clean":"2024-03-26T05:07:41.432374+0000","last_became_active":"2024-03-26T05:05:36.405705+0000","last_became_peered":"2024-03-26T05:05:36.405705+0000","last_unstale":"2024-03-26T05:07:41.432374+0000","last_undegraded":"2024-03-26T05:07:41.432374+0000","last_fullsized":"2024-03-26T05:07:41.432374+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:29.768389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:29.768389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:33.558469+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"120'9","reported_seq":406,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449275+0000","last_change":"2024-03-26T05:06:39.209229+0000","last_active":"2024-03-26T05:07:41.449275+0000","last_peered":"2024-03-26T05:07:41.449275+0000","last_clean":"2024-03-26T05:07:41.449275+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:07:41.449275+0000","last_undegraded":"2024-03-26T05:07:41.449275+0000","last_fullsized":"2024-03-26T05:07:41.449275+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'9","last_scrub_stamp":"2024-03-26T05:06:39.209149+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:06:39.209149+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:51.462691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":322,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448558+0000","last_change":"2024-03-26T05:06:55.105421+0000","last_active":"2024-03-26T05:07:41.448558+0000","last_peered":"2024-03-26T05:07:41.448558+0000","last_clean":"2024-03-26T05:07:41.448558+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:07:41.448558+0000","last_undegraded":"2024-03-26T05:07:41.448558+0000","last_fullsized":"2024-03-26T05:07:41.448558+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:55.105291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:55.105291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:23.534351+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"98'20","reported_seq":219,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.440911+0000","last_change":"2024-03-26T05:06:49.846150+0000","last_active":"2024-03-26T05:07:41.440911+0000","last_peered":"2024-03-26T05:07:41.440911+0000","last_clean":"2024-03-26T05:07:41.440911+0000","last_became_active":"2024-03-26T05:05:36.422436+0000","last_became_peered":"2024-03-26T05:05:36.422436+0000","last_unstale":"2024-03-26T05:07:41.440911+0000","last_undegraded":"2024-03-26T05:07:41.440911+0000","last_fullsized":"2024-03-26T05:07:41.440911+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'20","last_scrub_stamp":"2024-03-26T05:06:49.846038+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:06:49.846038+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:01.733368+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":309,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.437188+0000","last_change":"2024-03-26T05:05:58.893557+0000","last_active":"2024-03-26T05:07:41.437188+0000","last_peered":"2024-03-26T05:07:41.437188+0000","last_clean":"2024-03-26T05:07:41.437188+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:07:41.437188+0000","last_undegraded":"2024-03-26T05:07:41.437188+0000","last_fullsized":"2024-03-26T05:07:41.437188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:58.893447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:58.893447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":297,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.451392+0000","last_change":"2024-03-26T05:04:59.846243+0000","last_active":"2024-03-26T05:07:41.451392+0000","last_peered":"2024-03-26T05:07:41.451392+0000","last_clean":"2024-03-26T05:07:41.451392+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:07:41.451392+0000","last_undegraded":"2024-03-26T05:07:41.451392+0000","last_fullsized":"2024-03-26T05:07:41.451392+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:59.846123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:04:59.846123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":312,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.439460+0000","last_change":"2024-03-26T05:07:38.695076+0000","last_active":"2024-03-26T05:07:41.439460+0000","last_peered":"2024-03-26T05:07:41.439460+0000","last_clean":"2024-03-26T05:07:41.439460+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:07:41.439460+0000","last_undegraded":"2024-03-26T05:07:41.439460+0000","last_fullsized":"2024-03-26T05:07:41.439460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:38.694961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:38.694961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:01.889901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'21","reported_seq":350,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.450133+0000","last_change":"2024-03-26T05:05:12.090691+0000","last_active":"2024-03-26T05:07:41.450133+0000","last_peered":"2024-03-26T05:07:41.450133+0000","last_clean":"2024-03-26T05:07:41.450133+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:07:41.450133+0000","last_undegraded":"2024-03-26T05:07:41.450133+0000","last_fullsized":"2024-03-26T05:07:41.450133+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:05:12.089325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.089325+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":300,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.440757+0000","last_change":"2024-03-26T05:05:58.845334+0000","last_active":"2024-03-26T05:07:41.440757+0000","last_peered":"2024-03-26T05:07:41.440757+0000","last_clean":"2024-03-26T05:07:41.440757+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:07:41.440757+0000","last_undegraded":"2024-03-26T05:07:41.440757+0000","last_fullsized":"2024-03-26T05:07:41.440757+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:58.845212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:58.845212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":102,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.451051+0000","last_change":"2024-03-26T05:05:36.416431+0000","last_active":"2024-03-26T05:07:41.451051+0000","last_peered":"2024-03-26T05:07:41.451051+0000","last_clean":"2024-03-26T05:07:41.451051+0000","last_became_active":"2024-03-26T05:05:36.416053+0000","last_became_peered":"2024-03-26T05:05:36.416053+0000","last_unstale":"2024-03-26T05:07:41.451051+0000","last_undegraded":"2024-03-26T05:07:41.451051+0000","last_fullsized":"2024-03-26T05:07:41.451051+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:09.946734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:09.946734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":337,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.441360+0000","last_change":"2024-03-26T05:06:50.546347+0000","last_active":"2024-03-26T05:07:41.441360+0000","last_peered":"2024-03-26T05:07:41.441360+0000","last_clean":"2024-03-26T05:07:41.441360+0000","last_became_active":"2024-03-26T05:05:36.419068+0000","last_became_peered":"2024-03-26T05:05:36.419068+0000","last_unstale":"2024-03-26T05:07:41.441360+0000","last_undegraded":"2024-03-26T05:07:41.441360+0000","last_fullsized":"2024-03-26T05:07:41.441360+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_clean_scrub_stamp":"2024-03-26T05:06:50.546241+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:05.099002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'12","reported_seq":330,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.451439+0000","last_change":"2024-03-26T05:06:42.178592+0000","last_active":"2024-03-26T05:07:41.451439+0000","last_peered":"2024-03-26T05:07:41.451439+0000","last_clean":"2024-03-26T05:07:41.451439+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:07:41.451439+0000","last_undegraded":"2024-03-26T05:07:41.451439+0000","last_fullsized":"2024-03-26T05:07:41.451439+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:06:42.178446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:42.178446+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:59.083463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"206'4922","reported_seq":4976,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:44.746311+0000","last_change":"2024-03-26T05:06:42.023680+0000","last_active":"2024-03-26T05:07:44.746311+0000","last_peered":"2024-03-26T05:07:44.746311+0000","last_clean":"2024-03-26T05:07:44.746311+0000","last_became_active":"2024-03-26T05:06:42.023267+0000","last_became_peered":"2024-03-26T05:06:42.023267+0000","last_unstale":"2024-03-26T05:07:44.746311+0000","last_undegraded":"2024-03-26T05:07:44.746311+0000","last_fullsized":"2024-03-26T05:07:44.746311+0000","mapping_epoch":184,"log_start":"190'300","ondisk_log_start":"190'300","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4622,"log_dups_size":0,"ondisk_log_size":4622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322568192,"num_objects":4922,"num_object_clones":0,"num_object_copies":14766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4922,"num_write_kb":315008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":168,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.441398+0000","last_change":"2024-03-26T05:06:48.480757+0000","last_active":"2024-03-26T05:07:41.441398+0000","last_peered":"2024-03-26T05:07:41.441398+0000","last_clean":"2024-03-26T05:07:41.441398+0000","last_became_active":"2024-03-26T05:05:36.409998+0000","last_became_peered":"2024-03-26T05:05:36.409998+0000","last_unstale":"2024-03-26T05:07:41.441398+0000","last_undegraded":"2024-03-26T05:07:41.441398+0000","last_fullsized":"2024-03-26T05:07:41.441398+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:06:48.480649+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:06:48.480649+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:52.622566+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":333,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.440527+0000","last_change":"2024-03-26T05:05:06.832886+0000","last_active":"2024-03-26T05:07:41.440527+0000","last_peered":"2024-03-26T05:07:41.440527+0000","last_clean":"2024-03-26T05:07:41.440527+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:07:41.440527+0000","last_undegraded":"2024-03-26T05:07:41.440527+0000","last_fullsized":"2024-03-26T05:07:41.440527+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:05:06.832744+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:05:06.832744+0000","last_clean_scrub_stamp":"2024-03-26T05:05:06.832744+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'27","reported_seq":198,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.439631+0000","last_change":"2024-03-26T05:06:03.850352+0000","last_active":"2024-03-26T05:07:41.439631+0000","last_peered":"2024-03-26T05:07:41.439631+0000","last_clean":"2024-03-26T05:07:41.439631+0000","last_became_active":"2024-03-26T05:05:36.399795+0000","last_became_peered":"2024-03-26T05:05:36.399795+0000","last_unstale":"2024-03-26T05:07:41.439631+0000","last_undegraded":"2024-03-26T05:07:41.439631+0000","last_fullsized":"2024-03-26T05:07:41.439631+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'27","last_scrub_stamp":"2024-03-26T05:06:03.850202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:03.850202+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":401,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.439696+0000","last_change":"2024-03-26T05:05:57.829833+0000","last_active":"2024-03-26T05:07:41.439696+0000","last_peered":"2024-03-26T05:07:41.439696+0000","last_clean":"2024-03-26T05:07:41.439696+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:07:41.439696+0000","last_undegraded":"2024-03-26T05:07:41.439696+0000","last_fullsized":"2024-03-26T05:07:41.439696+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:05:57.829724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:57.829724+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"206'5045","reported_seq":5099,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:44.822472+0000","last_change":"2024-03-26T05:06:41.959963+0000","last_active":"2024-03-26T05:07:44.822472+0000","last_peered":"2024-03-26T05:07:44.822472+0000","last_clean":"2024-03-26T05:07:44.822472+0000","last_became_active":"2024-03-26T05:06:41.959580+0000","last_became_peered":"2024-03-26T05:06:41.959580+0000","last_unstale":"2024-03-26T05:07:44.822472+0000","last_undegraded":"2024-03-26T05:07:44.822472+0000","last_fullsized":"2024-03-26T05:07:44.822472+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:46.553390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330629120,"num_objects":5045,"num_object_clones":0,"num_object_copies":15135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5045,"num_write_kb":322880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":301,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449152+0000","last_change":"2024-03-26T05:06:38.198437+0000","last_active":"2024-03-26T05:07:41.449152+0000","last_peered":"2024-03-26T05:07:41.449152+0000","last_clean":"2024-03-26T05:07:41.449152+0000","last_became_active":"2024-03-26T05:00:28.985917+0000","last_became_peered":"2024-03-26T05:00:28.985917+0000","last_unstale":"2024-03-26T05:07:41.449152+0000","last_undegraded":"2024-03-26T05:07:41.449152+0000","last_fullsized":"2024-03-26T05:07:41.449152+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:38.198342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:38.198342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:58.765618+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":312,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.440486+0000","last_change":"2024-03-26T05:05:53.920453+0000","last_active":"2024-03-26T05:07:41.440486+0000","last_peered":"2024-03-26T05:07:41.440486+0000","last_clean":"2024-03-26T05:07:41.440486+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:07:41.440486+0000","last_undegraded":"2024-03-26T05:07:41.440486+0000","last_fullsized":"2024-03-26T05:07:41.440486+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:53.920375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:53.920375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"188'31","reported_seq":364,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.431945+0000","last_change":"2024-03-26T05:06:50.301509+0000","last_active":"2024-03-26T05:07:41.431945+0000","last_peered":"2024-03-26T05:07:41.431945+0000","last_clean":"2024-03-26T05:07:41.431945+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:07:41.431945+0000","last_undegraded":"2024-03-26T05:07:41.431945+0000","last_fullsized":"2024-03-26T05:07:41.431945+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'31","last_scrub_stamp":"2024-03-26T05:06:50.301394+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:06:50.301394+0000","last_clean_scrub_stamp":"2024-03-26T05:06:50.301394+0000","objects_scrubbed":7,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:00.240874+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":313,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.435386+0000","last_change":"2024-03-26T05:06:53.117526+0000","last_active":"2024-03-26T05:07:41.435386+0000","last_peered":"2024-03-26T05:07:41.435386+0000","last_clean":"2024-03-26T05:07:41.435386+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:07:41.435386+0000","last_undegraded":"2024-03-26T05:07:41.435386+0000","last_fullsized":"2024-03-26T05:07:41.435386+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:53.117385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.432732+0000","last_clean_scrub_stamp":"2024-03-26T05:06:53.117385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:21.618608+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"206'5031","reported_seq":5085,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:45.320348+0000","last_change":"2024-03-26T05:06:41.963528+0000","last_active":"2024-03-26T05:07:45.320348+0000","last_peered":"2024-03-26T05:07:45.320348+0000","last_clean":"2024-03-26T05:07:45.320348+0000","last_became_active":"2024-03-26T05:06:41.962275+0000","last_became_peered":"2024-03-26T05:06:41.962275+0000","last_unstale":"2024-03-26T05:07:45.320348+0000","last_undegraded":"2024-03-26T05:07:45.320348+0000","last_fullsized":"2024-03-26T05:07:45.320348+0000","mapping_epoch":184,"log_start":"190'100","ondisk_log_start":"190'100","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4931,"log_dups_size":0,"ondisk_log_size":4931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329711616,"num_objects":5031,"num_object_clones":0,"num_object_copies":15093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5031,"num_write_kb":321984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":300,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.439402+0000","last_change":"2024-03-26T05:05:46.845330+0000","last_active":"2024-03-26T05:07:41.439402+0000","last_peered":"2024-03-26T05:07:41.439402+0000","last_clean":"2024-03-26T05:07:41.439402+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:07:41.439402+0000","last_undegraded":"2024-03-26T05:07:41.439402+0000","last_fullsized":"2024-03-26T05:07:41.439402+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:46.845184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:23.071667+0000","last_clean_scrub_stamp":"2024-03-26T05:05:46.845184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":319,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.436267+0000","last_change":"2024-03-26T05:05:57.929700+0000","last_active":"2024-03-26T05:07:41.436267+0000","last_peered":"2024-03-26T05:07:41.436267+0000","last_clean":"2024-03-26T05:07:41.436267+0000","last_became_active":"2024-03-26T05:05:36.410132+0000","last_became_peered":"2024-03-26T05:05:36.410132+0000","last_unstale":"2024-03-26T05:07:41.436267+0000","last_undegraded":"2024-03-26T05:07:41.436267+0000","last_fullsized":"2024-03-26T05:07:41.436267+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:57.929170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:57.929170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":310,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.440860+0000","last_change":"2024-03-26T05:05:36.422622+0000","last_active":"2024-03-26T05:07:41.440860+0000","last_peered":"2024-03-26T05:07:41.440860+0000","last_clean":"2024-03-26T05:07:41.440860+0000","last_became_active":"2024-03-26T05:05:36.422084+0000","last_became_peered":"2024-03-26T05:05:36.422084+0000","last_unstale":"2024-03-26T05:07:41.440860+0000","last_undegraded":"2024-03-26T05:07:41.440860+0000","last_fullsized":"2024-03-26T05:07:41.440860+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:23.884406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:23.884406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'25","reported_seq":359,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.441694+0000","last_change":"2024-03-26T05:06:45.476643+0000","last_active":"2024-03-26T05:07:41.441694+0000","last_peered":"2024-03-26T05:07:41.441694+0000","last_clean":"2024-03-26T05:07:41.441694+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:07:41.441694+0000","last_undegraded":"2024-03-26T05:07:41.441694+0000","last_fullsized":"2024-03-26T05:07:41.441694+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:06:45.476460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.476460+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:07.614363+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"206'5034","reported_seq":5088,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:45.035523+0000","last_change":"2024-03-26T05:06:41.961429+0000","last_active":"2024-03-26T05:07:45.035523+0000","last_peered":"2024-03-26T05:07:45.035523+0000","last_clean":"2024-03-26T05:07:45.035523+0000","last_became_active":"2024-03-26T05:06:41.961141+0000","last_became_peered":"2024-03-26T05:06:41.961141+0000","last_unstale":"2024-03-26T05:07:45.035523+0000","last_undegraded":"2024-03-26T05:07:45.035523+0000","last_fullsized":"2024-03-26T05:07:45.035523+0000","mapping_epoch":184,"log_start":"190'500","ondisk_log_start":"190'500","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4534,"log_dups_size":0,"ondisk_log_size":4534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:46.766743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329908224,"num_objects":5034,"num_object_clones":0,"num_object_copies":15102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":322176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":302,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449026+0000","last_change":"2024-03-26T05:05:55.120808+0000","last_active":"2024-03-26T05:07:41.449026+0000","last_peered":"2024-03-26T05:07:41.449026+0000","last_clean":"2024-03-26T05:07:41.449026+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:07:41.449026+0000","last_undegraded":"2024-03-26T05:07:41.449026+0000","last_fullsized":"2024-03-26T05:07:41.449026+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:55.120731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:55.120731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":313,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.435478+0000","last_change":"2024-03-26T05:06:51.161774+0000","last_active":"2024-03-26T05:07:41.435478+0000","last_peered":"2024-03-26T05:07:41.435478+0000","last_clean":"2024-03-26T05:07:41.435478+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:07:41.435478+0000","last_undegraded":"2024-03-26T05:07:41.435478+0000","last_fullsized":"2024-03-26T05:07:41.435478+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:51.161669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:51.161669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:04.605897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":103,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448908+0000","last_change":"2024-03-26T05:05:36.415361+0000","last_active":"2024-03-26T05:07:41.448908+0000","last_peered":"2024-03-26T05:07:41.448908+0000","last_clean":"2024-03-26T05:07:41.448908+0000","last_became_active":"2024-03-26T05:05:36.414992+0000","last_became_peered":"2024-03-26T05:05:36.414992+0000","last_unstale":"2024-03-26T05:07:41.448908+0000","last_undegraded":"2024-03-26T05:07:41.448908+0000","last_fullsized":"2024-03-26T05:07:41.448908+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:01.867674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:01.867674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"75'15","reported_seq":338,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449860+0000","last_change":"2024-03-26T05:05:42.260332+0000","last_active":"2024-03-26T05:07:41.449860+0000","last_peered":"2024-03-26T05:07:41.449860+0000","last_clean":"2024-03-26T05:07:41.449860+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:07:41.449860+0000","last_undegraded":"2024-03-26T05:07:41.449860+0000","last_fullsized":"2024-03-26T05:07:41.449860+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:05:42.260202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:42.260202+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"206'5156","reported_seq":5209,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:45.305839+0000","last_change":"2024-03-26T05:06:41.964200+0000","last_active":"2024-03-26T05:07:45.305839+0000","last_peered":"2024-03-26T05:07:45.305839+0000","last_clean":"2024-03-26T05:07:45.305839+0000","last_became_active":"2024-03-26T05:06:41.963415+0000","last_became_peered":"2024-03-26T05:06:41.963415+0000","last_unstale":"2024-03-26T05:07:45.305839+0000","last_undegraded":"2024-03-26T05:07:45.305839+0000","last_fullsized":"2024-03-26T05:07:45.305839+0000","mapping_epoch":184,"log_start":"190'900","ondisk_log_start":"190'900","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4256,"log_dups_size":0,"ondisk_log_size":4256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:54.093502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337903616,"num_objects":5156,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5156,"num_write_kb":329984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":293,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.451187+0000","last_change":"2024-03-26T05:04:11.571539+0000","last_active":"2024-03-26T05:07:41.451187+0000","last_peered":"2024-03-26T05:07:41.451187+0000","last_clean":"2024-03-26T05:07:41.451187+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:07:41.451187+0000","last_undegraded":"2024-03-26T05:07:41.451187+0000","last_fullsized":"2024-03-26T05:07:41.451187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:11.571425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:11.571425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":141,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448881+0000","last_change":"2024-03-26T05:07:37.347726+0000","last_active":"2024-03-26T05:07:41.448881+0000","last_peered":"2024-03-26T05:07:41.448881+0000","last_clean":"2024-03-26T05:07:41.448881+0000","last_became_active":"2024-03-26T05:05:36.412450+0000","last_became_peered":"2024-03-26T05:05:36.412450+0000","last_unstale":"2024-03-26T05:07:41.448881+0000","last_undegraded":"2024-03-26T05:07:41.448881+0000","last_fullsized":"2024-03-26T05:07:41.448881+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:07:37.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:07:37.347513+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:51.081154+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":306,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.431992+0000","last_change":"2024-03-26T05:05:46.234486+0000","last_active":"2024-03-26T05:07:41.431992+0000","last_peered":"2024-03-26T05:07:41.431992+0000","last_clean":"2024-03-26T05:07:41.431992+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:07:41.431992+0000","last_undegraded":"2024-03-26T05:07:41.431992+0000","last_fullsized":"2024-03-26T05:07:41.431992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:46.234374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:46.234374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'34","reported_seq":466,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448553+0000","last_change":"2024-03-26T05:06:54.128746+0000","last_active":"2024-03-26T05:07:41.448553+0000","last_peered":"2024-03-26T05:07:41.448553+0000","last_clean":"2024-03-26T05:07:41.448553+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:07:41.448553+0000","last_undegraded":"2024-03-26T05:07:41.448553+0000","last_fullsized":"2024-03-26T05:07:41.448553+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'34","last_scrub_stamp":"2024-03-26T05:06:54.128603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:54.128603+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:12.598382+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":306,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449791+0000","last_change":"2024-03-26T05:06:05.029804+0000","last_active":"2024-03-26T05:07:41.449791+0000","last_peered":"2024-03-26T05:07:41.449791+0000","last_clean":"2024-03-26T05:07:41.449791+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:07:41.449791+0000","last_undegraded":"2024-03-26T05:07:41.449791+0000","last_fullsized":"2024-03-26T05:07:41.449791+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:05.029701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:05.029701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"206'4841","reported_seq":4895,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:45.308264+0000","last_change":"2024-03-26T05:06:41.964329+0000","last_active":"2024-03-26T05:07:45.308264+0000","last_peered":"2024-03-26T05:07:45.308264+0000","last_clean":"2024-03-26T05:07:45.308264+0000","last_became_active":"2024-03-26T05:06:41.963962+0000","last_became_peered":"2024-03-26T05:06:41.963962+0000","last_unstale":"2024-03-26T05:07:45.308264+0000","last_undegraded":"2024-03-26T05:07:45.308264+0000","last_fullsized":"2024-03-26T05:07:45.308264+0000","mapping_epoch":184,"log_start":"190'600","ondisk_log_start":"190'600","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4241,"log_dups_size":0,"ondisk_log_size":4241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:48.809977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317259776,"num_objects":4841,"num_object_clones":0,"num_object_copies":14523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4841,"num_write_kb":309824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":300,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.440947+0000","last_change":"2024-03-26T05:05:42.897537+0000","last_active":"2024-03-26T05:07:41.440947+0000","last_peered":"2024-03-26T05:07:41.440947+0000","last_clean":"2024-03-26T05:07:41.440947+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:07:41.440947+0000","last_undegraded":"2024-03-26T05:07:41.440947+0000","last_fullsized":"2024-03-26T05:07:41.440947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:42.897420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:42.897420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":313,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432111+0000","last_change":"2024-03-26T05:06:49.178919+0000","last_active":"2024-03-26T05:07:41.432111+0000","last_peered":"2024-03-26T05:07:41.432111+0000","last_clean":"2024-03-26T05:07:41.432111+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:07:41.432111+0000","last_undegraded":"2024-03-26T05:07:41.432111+0000","last_fullsized":"2024-03-26T05:07:41.432111+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:49.178823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:49.178823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:49.961816+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":330,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.435992+0000","last_change":"2024-03-26T05:06:41.303481+0000","last_active":"2024-03-26T05:07:41.435992+0000","last_peered":"2024-03-26T05:07:41.435992+0000","last_clean":"2024-03-26T05:07:41.435992+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:07:41.435992+0000","last_undegraded":"2024-03-26T05:07:41.435992+0000","last_fullsized":"2024-03-26T05:07:41.435992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:41.303395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:06:41.303395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:02.047984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'19","reported_seq":101,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.436564+0000","last_change":"2024-03-26T05:06:38.304919+0000","last_active":"2024-03-26T05:07:41.436564+0000","last_peered":"2024-03-26T05:07:41.436564+0000","last_clean":"2024-03-26T05:07:41.436564+0000","last_became_active":"2024-03-26T05:06:06.521721+0000","last_became_peered":"2024-03-26T05:06:06.521721+0000","last_unstale":"2024-03-26T05:07:41.436564+0000","last_undegraded":"2024-03-26T05:07:41.436564+0000","last_fullsized":"2024-03-26T05:07:41.436564+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-26T05:06:38.304813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:38.304813+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:45.591514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":26,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":327,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:44.363735+0000","last_change":"2024-03-26T05:07:44.363788+0000","last_active":"2024-03-26T05:07:44.363735+0000","last_peered":"2024-03-26T05:07:44.363735+0000","last_clean":"2024-03-26T05:07:44.363735+0000","last_became_active":"2024-03-26T05:05:36.401910+0000","last_became_peered":"2024-03-26T05:05:36.401910+0000","last_unstale":"2024-03-26T05:07:44.363735+0000","last_undegraded":"2024-03-26T05:07:44.363735+0000","last_fullsized":"2024-03-26T05:07:44.363735+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:44.363612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:44.363612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:10.559332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"206'5061","reported_seq":5114,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:44.809469+0000","last_change":"2024-03-26T05:06:41.962495+0000","last_active":"2024-03-26T05:07:44.809469+0000","last_peered":"2024-03-26T05:07:44.809469+0000","last_clean":"2024-03-26T05:07:44.809469+0000","last_became_active":"2024-03-26T05:06:41.962150+0000","last_became_peered":"2024-03-26T05:06:41.962150+0000","last_unstale":"2024-03-26T05:07:44.809469+0000","last_undegraded":"2024-03-26T05:07:44.809469+0000","last_fullsized":"2024-03-26T05:07:44.809469+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":5061,"log_dups_size":0,"ondisk_log_size":5061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331677696,"num_objects":5061,"num_object_clones":0,"num_object_copies":15183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5061,"num_write_kb":323904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":102,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.451272+0000","last_change":"2024-03-26T05:05:36.423696+0000","last_active":"2024-03-26T05:07:41.451272+0000","last_peered":"2024-03-26T05:07:41.451272+0000","last_clean":"2024-03-26T05:07:41.451272+0000","last_became_active":"2024-03-26T05:05:36.422421+0000","last_became_peered":"2024-03-26T05:05:36.422421+0000","last_unstale":"2024-03-26T05:07:41.451272+0000","last_undegraded":"2024-03-26T05:07:41.451272+0000","last_fullsized":"2024-03-26T05:07:41.451272+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_clean_scrub_stamp":"2024-03-26T05:05:03.848208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":567,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.439524+0000","last_change":"2024-03-26T05:06:45.717598+0000","last_active":"2024-03-26T05:07:41.439524+0000","last_peered":"2024-03-26T05:07:41.439524+0000","last_clean":"2024-03-26T05:07:41.439524+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:07:41.439524+0000","last_undegraded":"2024-03-26T05:07:41.439524+0000","last_fullsized":"2024-03-26T05:07:41.439524+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:06:45.717474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.717474+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:09.381246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":319,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.433303+0000","last_change":"2024-03-26T05:05:36.424898+0000","last_active":"2024-03-26T05:07:41.433303+0000","last_peered":"2024-03-26T05:07:41.433303+0000","last_clean":"2024-03-26T05:07:41.433303+0000","last_became_active":"2024-03-26T05:05:36.424565+0000","last_became_peered":"2024-03-26T05:05:36.424565+0000","last_unstale":"2024-03-26T05:07:41.433303+0000","last_undegraded":"2024-03-26T05:07:41.433303+0000","last_fullsized":"2024-03-26T05:07:41.433303+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:07.743708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:07.743708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'24","reported_seq":443,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.442346+0000","last_change":"2024-03-26T05:07:32.005638+0000","last_active":"2024-03-26T05:07:41.442346+0000","last_peered":"2024-03-26T05:07:41.442346+0000","last_clean":"2024-03-26T05:07:41.442346+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:07:41.442346+0000","last_undegraded":"2024-03-26T05:07:41.442346+0000","last_fullsized":"2024-03-26T05:07:41.442346+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'24","last_scrub_stamp":"2024-03-26T05:07:32.005486+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:07:32.005486+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:40.945818+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":318,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.440500+0000","last_change":"2024-03-26T05:07:29.669146+0000","last_active":"2024-03-26T05:07:41.440500+0000","last_peered":"2024-03-26T05:07:41.440500+0000","last_clean":"2024-03-26T05:07:41.440500+0000","last_became_active":"2024-03-26T05:05:36.401943+0000","last_became_peered":"2024-03-26T05:05:36.401943+0000","last_unstale":"2024-03-26T05:07:41.440500+0000","last_undegraded":"2024-03-26T05:07:41.440500+0000","last_fullsized":"2024-03-26T05:07:41.440500+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:29.669034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:29.669034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:31.729702+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"206'4987","reported_seq":5041,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:45.018592+0000","last_change":"2024-03-26T05:06:42.023633+0000","last_active":"2024-03-26T05:07:45.018592+0000","last_peered":"2024-03-26T05:07:45.018592+0000","last_clean":"2024-03-26T05:07:45.018592+0000","last_became_active":"2024-03-26T05:06:42.023345+0000","last_became_peered":"2024-03-26T05:06:42.023345+0000","last_unstale":"2024-03-26T05:07:45.018592+0000","last_undegraded":"2024-03-26T05:07:45.018592+0000","last_fullsized":"2024-03-26T05:07:45.018592+0000","mapping_epoch":184,"log_start":"190'500","ondisk_log_start":"190'500","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4487,"log_dups_size":0,"ondisk_log_size":4487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:53.018647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326828032,"num_objects":4987,"num_object_clones":0,"num_object_copies":14961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4987,"num_write_kb":319168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":315,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432141+0000","last_change":"2024-03-26T05:07:39.338198+0000","last_active":"2024-03-26T05:07:41.432141+0000","last_peered":"2024-03-26T05:07:41.432141+0000","last_clean":"2024-03-26T05:07:41.432141+0000","last_became_active":"2024-03-26T05:05:36.416657+0000","last_became_peered":"2024-03-26T05:05:36.416657+0000","last_unstale":"2024-03-26T05:07:41.432141+0000","last_undegraded":"2024-03-26T05:07:41.432141+0000","last_fullsized":"2024-03-26T05:07:41.432141+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:39.338092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:39.338092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:42.282010+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":322,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448769+0000","last_change":"2024-03-26T05:06:39.074585+0000","last_active":"2024-03-26T05:07:41.448769+0000","last_peered":"2024-03-26T05:07:41.448769+0000","last_clean":"2024-03-26T05:07:41.448769+0000","last_became_active":"2024-03-26T05:05:36.425252+0000","last_became_peered":"2024-03-26T05:05:36.425252+0000","last_unstale":"2024-03-26T05:07:41.448769+0000","last_undegraded":"2024-03-26T05:07:41.448769+0000","last_fullsized":"2024-03-26T05:07:41.448769+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:39.074501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:39.074501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:58.133016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":761,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:43.677236+0000","last_change":"2024-03-26T05:05:38.886524+0000","last_active":"2024-03-26T05:07:43.677236+0000","last_peered":"2024-03-26T05:07:43.677236+0000","last_clean":"2024-03-26T05:07:43.677236+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:07:43.677236+0000","last_undegraded":"2024-03-26T05:07:43.677236+0000","last_fullsized":"2024-03-26T05:07:43.677236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:05:38.886387+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:05:38.886387+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":436,"num_read_kb":431,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"187'18","reported_seq":374,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449860+0000","last_change":"2024-03-26T05:06:49.027873+0000","last_active":"2024-03-26T05:07:41.449860+0000","last_peered":"2024-03-26T05:07:41.449860+0000","last_clean":"2024-03-26T05:07:41.449860+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:07:41.449860+0000","last_undegraded":"2024-03-26T05:07:41.449860+0000","last_fullsized":"2024-03-26T05:07:41.449860+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'18","last_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_deep_scrub":"187'18","last_deep_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_clean_scrub_stamp":"2024-03-26T05:06:49.027756+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:55.000771+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":110,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448661+0000","last_change":"2024-03-26T05:06:44.270685+0000","last_active":"2024-03-26T05:07:41.448661+0000","last_peered":"2024-03-26T05:07:41.448661+0000","last_clean":"2024-03-26T05:07:41.448661+0000","last_became_active":"2024-03-26T05:05:36.406489+0000","last_became_peered":"2024-03-26T05:05:36.406489+0000","last_unstale":"2024-03-26T05:07:41.448661+0000","last_undegraded":"2024-03-26T05:07:41.448661+0000","last_fullsized":"2024-03-26T05:07:41.448661+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:44.270594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:44.270594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:52.280450+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"206'4939","reported_seq":4992,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:45.274476+0000","last_change":"2024-03-26T05:06:42.024050+0000","last_active":"2024-03-26T05:07:45.274476+0000","last_peered":"2024-03-26T05:07:45.274476+0000","last_clean":"2024-03-26T05:07:45.274476+0000","last_became_active":"2024-03-26T05:06:42.023726+0000","last_became_peered":"2024-03-26T05:06:42.023726+0000","last_unstale":"2024-03-26T05:07:45.274476+0000","last_undegraded":"2024-03-26T05:07:45.274476+0000","last_fullsized":"2024-03-26T05:07:45.274476+0000","mapping_epoch":184,"log_start":"190'700","ondisk_log_start":"190'700","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:56.060944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323682304,"num_objects":4939,"num_object_clones":0,"num_object_copies":14817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4939,"num_write_kb":316096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":301,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.447930+0000","last_change":"2024-03-26T05:06:01.073386+0000","last_active":"2024-03-26T05:07:41.447930+0000","last_peered":"2024-03-26T05:07:41.447930+0000","last_clean":"2024-03-26T05:07:41.447930+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:07:41.447930+0000","last_undegraded":"2024-03-26T05:07:41.447930+0000","last_fullsized":"2024-03-26T05:07:41.447930+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:01.073272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:06:01.073272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":319,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448400+0000","last_change":"2024-03-26T05:05:36.412560+0000","last_active":"2024-03-26T05:07:41.448400+0000","last_peered":"2024-03-26T05:07:41.448400+0000","last_clean":"2024-03-26T05:07:41.448400+0000","last_became_active":"2024-03-26T05:05:36.409491+0000","last_became_peered":"2024-03-26T05:05:36.409491+0000","last_unstale":"2024-03-26T05:07:41.448400+0000","last_undegraded":"2024-03-26T05:07:41.448400+0000","last_fullsized":"2024-03-26T05:07:41.448400+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:05.892241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:05:05.892241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'25","reported_seq":392,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.450174+0000","last_change":"2024-03-26T05:06:52.015549+0000","last_active":"2024-03-26T05:07:41.450174+0000","last_peered":"2024-03-26T05:07:41.450174+0000","last_clean":"2024-03-26T05:07:41.450174+0000","last_became_active":"2024-03-26T05:05:36.414330+0000","last_became_peered":"2024-03-26T05:05:36.414330+0000","last_unstale":"2024-03-26T05:07:41.450174+0000","last_undegraded":"2024-03-26T05:07:41.450174+0000","last_fullsized":"2024-03-26T05:07:41.450174+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:06:52.015444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:52.015444+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:15.721741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":318,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.441805+0000","last_change":"2024-03-26T05:07:35.020138+0000","last_active":"2024-03-26T05:07:41.441805+0000","last_peered":"2024-03-26T05:07:41.441805+0000","last_clean":"2024-03-26T05:07:41.441805+0000","last_became_active":"2024-03-26T05:05:36.406443+0000","last_became_peered":"2024-03-26T05:05:36.406443+0000","last_unstale":"2024-03-26T05:07:41.441805+0000","last_undegraded":"2024-03-26T05:07:41.441805+0000","last_fullsized":"2024-03-26T05:07:41.441805+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:35.020060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:07:35.020060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:51.601374+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"206'4955","reported_seq":5009,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:45.319523+0000","last_change":"2024-03-26T05:06:42.026870+0000","last_active":"2024-03-26T05:07:45.319523+0000","last_peered":"2024-03-26T05:07:45.319523+0000","last_clean":"2024-03-26T05:07:45.319523+0000","last_became_active":"2024-03-26T05:06:42.026460+0000","last_became_peered":"2024-03-26T05:06:42.026460+0000","last_unstale":"2024-03-26T05:07:45.319523+0000","last_undegraded":"2024-03-26T05:07:45.319523+0000","last_fullsized":"2024-03-26T05:07:45.319523+0000","mapping_epoch":184,"log_start":"190'100","ondisk_log_start":"190'100","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:03.707481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324730880,"num_objects":4955,"num_object_clones":0,"num_object_copies":14865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4955,"num_write_kb":317120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":302,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.438112+0000","last_change":"2024-03-26T05:05:52.123187+0000","last_active":"2024-03-26T05:07:41.438112+0000","last_peered":"2024-03-26T05:07:41.438112+0000","last_clean":"2024-03-26T05:07:41.438112+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:07:41.438112+0000","last_undegraded":"2024-03-26T05:07:41.438112+0000","last_fullsized":"2024-03-26T05:07:41.438112+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:52.123084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:52.123084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":104,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.450524+0000","last_change":"2024-03-26T05:05:36.419407+0000","last_active":"2024-03-26T05:07:41.450524+0000","last_peered":"2024-03-26T05:07:41.450524+0000","last_clean":"2024-03-26T05:07:41.450524+0000","last_became_active":"2024-03-26T05:05:36.418906+0000","last_became_peered":"2024-03-26T05:05:36.418906+0000","last_unstale":"2024-03-26T05:07:41.450524+0000","last_undegraded":"2024-03-26T05:07:41.450524+0000","last_fullsized":"2024-03-26T05:07:41.450524+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:07.893841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:07.893841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"75'16","reported_seq":327,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.435691+0000","last_change":"2024-03-26T05:06:47.166740+0000","last_active":"2024-03-26T05:07:41.435691+0000","last_peered":"2024-03-26T05:07:41.435691+0000","last_clean":"2024-03-26T05:07:41.435691+0000","last_became_active":"2024-03-26T05:05:36.411117+0000","last_became_peered":"2024-03-26T05:05:36.411117+0000","last_unstale":"2024-03-26T05:07:41.435691+0000","last_undegraded":"2024-03-26T05:07:41.435691+0000","last_fullsized":"2024-03-26T05:07:41.435691+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:06:47.166607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:47.166607+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:51.108028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"161'10","reported_seq":423,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.450001+0000","last_change":"2024-03-26T05:07:38.023243+0000","last_active":"2024-03-26T05:07:41.450001+0000","last_peered":"2024-03-26T05:07:41.450001+0000","last_clean":"2024-03-26T05:07:41.450001+0000","last_became_active":"2024-03-26T05:05:36.422549+0000","last_became_peered":"2024-03-26T05:05:36.422549+0000","last_unstale":"2024-03-26T05:07:41.450001+0000","last_undegraded":"2024-03-26T05:07:41.450001+0000","last_fullsized":"2024-03-26T05:07:41.450001+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'10","last_scrub_stamp":"2024-03-26T05:07:38.023111+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:07:38.023111+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:51.855201+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"206'4916","reported_seq":4970,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:44.651797+0000","last_change":"2024-03-26T05:06:42.023702+0000","last_active":"2024-03-26T05:07:44.651797+0000","last_peered":"2024-03-26T05:07:44.651797+0000","last_clean":"2024-03-26T05:07:44.651797+0000","last_became_active":"2024-03-26T05:06:42.023260+0000","last_became_peered":"2024-03-26T05:06:42.023260+0000","last_unstale":"2024-03-26T05:07:44.651797+0000","last_undegraded":"2024-03-26T05:07:44.651797+0000","last_fullsized":"2024-03-26T05:07:44.651797+0000","mapping_epoch":184,"log_start":"190'100","ondisk_log_start":"190'100","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4816,"log_dups_size":0,"ondisk_log_size":4816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322174976,"num_objects":4916,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4916,"num_write_kb":314624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":300,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.440148+0000","last_change":"2024-03-26T05:06:43.743986+0000","last_active":"2024-03-26T05:07:41.440148+0000","last_peered":"2024-03-26T05:07:41.440148+0000","last_clean":"2024-03-26T05:07:41.440148+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:07:41.440148+0000","last_undegraded":"2024-03-26T05:07:41.440148+0000","last_fullsized":"2024-03-26T05:07:41.440148+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:43.743837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:43.743837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:04.383692+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":323,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449956+0000","last_change":"2024-03-26T05:06:45.074041+0000","last_active":"2024-03-26T05:07:41.449956+0000","last_peered":"2024-03-26T05:07:41.449956+0000","last_clean":"2024-03-26T05:07:41.449956+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:07:41.449956+0000","last_undegraded":"2024-03-26T05:07:41.449956+0000","last_fullsized":"2024-03-26T05:07:41.449956+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:06:45.073912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.073912+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:51.266877+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'20","reported_seq":367,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432044+0000","last_change":"2024-03-26T05:06:42.311221+0000","last_active":"2024-03-26T05:07:41.432044+0000","last_peered":"2024-03-26T05:07:41.432044+0000","last_clean":"2024-03-26T05:07:41.432044+0000","last_became_active":"2024-03-26T05:05:36.405631+0000","last_became_peered":"2024-03-26T05:05:36.405631+0000","last_unstale":"2024-03-26T05:07:41.432044+0000","last_undegraded":"2024-03-26T05:07:41.432044+0000","last_fullsized":"2024-03-26T05:07:41.432044+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:06:42.311073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:42.311073+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:44.446306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":394,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449213+0000","last_change":"2024-03-26T05:04:26.670277+0000","last_active":"2024-03-26T05:07:41.449213+0000","last_peered":"2024-03-26T05:07:41.449213+0000","last_clean":"2024-03-26T05:07:41.449213+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:07:41.449213+0000","last_undegraded":"2024-03-26T05:07:41.449213+0000","last_fullsized":"2024-03-26T05:07:41.449213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:04:26.669895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:04:26.669895+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"206'5150","reported_seq":5204,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:45.305717+0000","last_change":"2024-03-26T05:06:41.965131+0000","last_active":"2024-03-26T05:07:45.305717+0000","last_peered":"2024-03-26T05:07:45.305717+0000","last_clean":"2024-03-26T05:07:45.305717+0000","last_became_active":"2024-03-26T05:06:41.964810+0000","last_became_peered":"2024-03-26T05:06:41.964810+0000","last_unstale":"2024-03-26T05:07:45.305717+0000","last_undegraded":"2024-03-26T05:07:45.305717+0000","last_fullsized":"2024-03-26T05:07:45.305717+0000","mapping_epoch":184,"log_start":"190'300","ondisk_log_start":"190'300","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:55.356833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337510400,"num_objects":5150,"num_object_clones":0,"num_object_copies":15450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5150,"num_write_kb":329600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":302,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432036+0000","last_change":"2024-03-26T05:05:57.123378+0000","last_active":"2024-03-26T05:07:41.432036+0000","last_peered":"2024-03-26T05:07:41.432036+0000","last_clean":"2024-03-26T05:07:41.432036+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:07:41.432036+0000","last_undegraded":"2024-03-26T05:07:41.432036+0000","last_fullsized":"2024-03-26T05:07:41.432036+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:57.123300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:57.123300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":305,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.451133+0000","last_change":"2024-03-26T05:04:54.805782+0000","last_active":"2024-03-26T05:07:41.451133+0000","last_peered":"2024-03-26T05:07:41.451133+0000","last_clean":"2024-03-26T05:07:41.451133+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:07:41.451133+0000","last_undegraded":"2024-03-26T05:07:41.451133+0000","last_fullsized":"2024-03-26T05:07:41.451133+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:54.805655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:54.805655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"96'40","reported_seq":375,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.449065+0000","last_change":"2024-03-26T05:05:13.704632+0000","last_active":"2024-03-26T05:07:41.449065+0000","last_peered":"2024-03-26T05:07:41.449065+0000","last_clean":"2024-03-26T05:07:41.449065+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:07:41.449065+0000","last_undegraded":"2024-03-26T05:07:41.449065+0000","last_fullsized":"2024-03-26T05:07:41.449065+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:05:13.704484+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:05:13.704484+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":298,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.431553+0000","last_change":"2024-03-26T05:05:47.194069+0000","last_active":"2024-03-26T05:07:41.431553+0000","last_peered":"2024-03-26T05:07:41.431553+0000","last_clean":"2024-03-26T05:07:41.431553+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:07:41.431553+0000","last_undegraded":"2024-03-26T05:07:41.431553+0000","last_fullsized":"2024-03-26T05:07:41.431553+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:47.193989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:47.193989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"206'5075","reported_seq":5128,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:45.031366+0000","last_change":"2024-03-26T05:06:42.023623+0000","last_active":"2024-03-26T05:07:45.031366+0000","last_peered":"2024-03-26T05:07:45.031366+0000","last_clean":"2024-03-26T05:07:45.031366+0000","last_became_active":"2024-03-26T05:06:42.023314+0000","last_became_peered":"2024-03-26T05:06:42.023314+0000","last_unstale":"2024-03-26T05:07:45.031366+0000","last_undegraded":"2024-03-26T05:07:45.031366+0000","last_fullsized":"2024-03-26T05:07:45.031366+0000","mapping_epoch":184,"log_start":"190'500","ondisk_log_start":"190'500","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4575,"log_dups_size":0,"ondisk_log_size":4575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:45.090269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332595200,"num_objects":5075,"num_object_clones":0,"num_object_copies":15225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5075,"num_write_kb":324800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":102,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.446608+0000","last_change":"2024-03-26T05:05:36.411895+0000","last_active":"2024-03-26T05:07:41.446608+0000","last_peered":"2024-03-26T05:07:41.446608+0000","last_clean":"2024-03-26T05:07:41.446608+0000","last_became_active":"2024-03-26T05:05:36.411501+0000","last_became_peered":"2024-03-26T05:05:36.411501+0000","last_unstale":"2024-03-26T05:07:41.446608+0000","last_undegraded":"2024-03-26T05:07:41.446608+0000","last_fullsized":"2024-03-26T05:07:41.446608+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.885959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.885959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"161'9","reported_seq":414,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432671+0000","last_change":"2024-03-26T05:06:10.114395+0000","last_active":"2024-03-26T05:07:41.432671+0000","last_peered":"2024-03-26T05:07:41.432671+0000","last_clean":"2024-03-26T05:07:41.432671+0000","last_became_active":"2024-03-26T05:05:36.408943+0000","last_became_peered":"2024-03-26T05:05:36.408943+0000","last_unstale":"2024-03-26T05:07:41.432671+0000","last_undegraded":"2024-03-26T05:07:41.432671+0000","last_fullsized":"2024-03-26T05:07:41.432671+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'9","last_scrub_stamp":"2024-03-26T05:06:10.114267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:10.114267+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":313,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448955+0000","last_change":"2024-03-26T05:06:48.223737+0000","last_active":"2024-03-26T05:07:41.448955+0000","last_peered":"2024-03-26T05:07:41.448955+0000","last_clean":"2024-03-26T05:07:41.448955+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:07:41.448955+0000","last_undegraded":"2024-03-26T05:07:41.448955+0000","last_fullsized":"2024-03-26T05:07:41.448955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:48.223661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:48.223661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:03.204242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":342,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448395+0000","last_change":"2024-03-26T05:06:57.114810+0000","last_active":"2024-03-26T05:07:41.448395+0000","last_peered":"2024-03-26T05:07:41.448395+0000","last_clean":"2024-03-26T05:07:41.448395+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:07:41.448395+0000","last_undegraded":"2024-03-26T05:07:41.448395+0000","last_fullsized":"2024-03-26T05:07:41.448395+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:06:57.113884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:57.113884+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:59.189652+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"206'4932","reported_seq":4986,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:44.713906+0000","last_change":"2024-03-26T05:06:41.962649+0000","last_active":"2024-03-26T05:07:44.713906+0000","last_peered":"2024-03-26T05:07:44.713906+0000","last_clean":"2024-03-26T05:07:44.713906+0000","last_became_active":"2024-03-26T05:06:41.962268+0000","last_became_peered":"2024-03-26T05:06:41.962268+0000","last_unstale":"2024-03-26T05:07:44.713906+0000","last_undegraded":"2024-03-26T05:07:44.713906+0000","last_fullsized":"2024-03-26T05:07:44.713906+0000","mapping_epoch":184,"log_start":"190'300","ondisk_log_start":"190'300","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:02.304727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323223552,"num_objects":4932,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4932,"num_write_kb":315648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":299,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448688+0000","last_change":"2024-03-26T05:05:36.413659+0000","last_active":"2024-03-26T05:07:41.448688+0000","last_peered":"2024-03-26T05:07:41.448688+0000","last_clean":"2024-03-26T05:07:41.448688+0000","last_became_active":"2024-03-26T05:05:36.413320+0000","last_became_peered":"2024-03-26T05:05:36.413320+0000","last_unstale":"2024-03-26T05:07:41.448688+0000","last_undegraded":"2024-03-26T05:07:41.448688+0000","last_fullsized":"2024-03-26T05:07:41.448688+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:25.636012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:25.636012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":110,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432243+0000","last_change":"2024-03-26T05:06:56.446200+0000","last_active":"2024-03-26T05:07:41.432243+0000","last_peered":"2024-03-26T05:07:41.432243+0000","last_clean":"2024-03-26T05:07:41.432243+0000","last_became_active":"2024-03-26T05:05:36.404979+0000","last_became_peered":"2024-03-26T05:05:36.404979+0000","last_unstale":"2024-03-26T05:07:41.432243+0000","last_undegraded":"2024-03-26T05:07:41.432243+0000","last_fullsized":"2024-03-26T05:07:41.432243+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:56.446060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.446060+0000","last_clean_scrub_stamp":"2024-03-26T05:06:56.446060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:23.714583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":314,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.431666+0000","last_change":"2024-03-26T05:05:49.242313+0000","last_active":"2024-03-26T05:07:41.431666+0000","last_peered":"2024-03-26T05:07:41.431666+0000","last_clean":"2024-03-26T05:07:41.431666+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:07:41.431666+0000","last_undegraded":"2024-03-26T05:07:41.431666+0000","last_fullsized":"2024-03-26T05:07:41.431666+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:49.242237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:49.242237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'17","reported_seq":341,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432515+0000","last_change":"2024-03-26T05:05:56.133675+0000","last_active":"2024-03-26T05:07:41.432515+0000","last_peered":"2024-03-26T05:07:41.432515+0000","last_clean":"2024-03-26T05:07:41.432515+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:07:41.432515+0000","last_undegraded":"2024-03-26T05:07:41.432515+0000","last_fullsized":"2024-03-26T05:07:41.432515+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:05:56.133576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:56.133576+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"206'5029","reported_seq":5083,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:45.031140+0000","last_change":"2024-03-26T05:06:42.023927+0000","last_active":"2024-03-26T05:07:45.031140+0000","last_peered":"2024-03-26T05:07:45.031140+0000","last_clean":"2024-03-26T05:07:45.031140+0000","last_became_active":"2024-03-26T05:06:42.022709+0000","last_became_peered":"2024-03-26T05:06:42.022709+0000","last_unstale":"2024-03-26T05:07:45.031140+0000","last_undegraded":"2024-03-26T05:07:45.031140+0000","last_fullsized":"2024-03-26T05:07:45.031140+0000","mapping_epoch":184,"log_start":"190'500","ondisk_log_start":"190'500","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:02.947465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329580544,"num_objects":5029,"num_object_clones":0,"num_object_copies":15087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5029,"num_write_kb":321856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":309,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.436400+0000","last_change":"2024-03-26T05:07:38.496117+0000","last_active":"2024-03-26T05:07:41.436400+0000","last_peered":"2024-03-26T05:07:41.436400+0000","last_clean":"2024-03-26T05:07:41.436400+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:07:41.436400+0000","last_undegraded":"2024-03-26T05:07:41.436400+0000","last_fullsized":"2024-03-26T05:07:41.436400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:38.495883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:38.495883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:51.825055+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":305,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448671+0000","last_change":"2024-03-26T05:06:02.997794+0000","last_active":"2024-03-26T05:07:41.448671+0000","last_peered":"2024-03-26T05:07:41.448671+0000","last_clean":"2024-03-26T05:07:41.448671+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:07:41.448671+0000","last_undegraded":"2024-03-26T05:07:41.448671+0000","last_fullsized":"2024-03-26T05:07:41.448671+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:02.997671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:02.997671+0000","last_clean_scrub_stamp":"2024-03-26T05:06:02.997671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":334,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.441521+0000","last_change":"2024-03-26T05:07:35.999912+0000","last_active":"2024-03-26T05:07:41.441521+0000","last_peered":"2024-03-26T05:07:41.441521+0000","last_clean":"2024-03-26T05:07:41.441521+0000","last_became_active":"2024-03-26T05:06:06.521150+0000","last_became_peered":"2024-03-26T05:06:06.521150+0000","last_unstale":"2024-03-26T05:07:41.441521+0000","last_undegraded":"2024-03-26T05:07:41.441521+0000","last_fullsized":"2024-03-26T05:07:41.441521+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:35.999836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_clean_scrub_stamp":"2024-03-26T05:07:35.999836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:40.950957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":338,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432983+0000","last_change":"2024-03-26T05:07:33.664903+0000","last_active":"2024-03-26T05:07:41.432983+0000","last_peered":"2024-03-26T05:07:41.432983+0000","last_clean":"2024-03-26T05:07:41.432983+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:07:41.432983+0000","last_undegraded":"2024-03-26T05:07:41.432983+0000","last_fullsized":"2024-03-26T05:07:41.432983+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:07:33.664806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:33.664806+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:49.190728+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"206'4952","reported_seq":5006,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:45.284508+0000","last_change":"2024-03-26T05:06:41.964494+0000","last_active":"2024-03-26T05:07:45.284508+0000","last_peered":"2024-03-26T05:07:45.284508+0000","last_clean":"2024-03-26T05:07:45.284508+0000","last_became_active":"2024-03-26T05:06:41.963907+0000","last_became_peered":"2024-03-26T05:06:41.963907+0000","last_unstale":"2024-03-26T05:07:45.284508+0000","last_undegraded":"2024-03-26T05:07:45.284508+0000","last_fullsized":"2024-03-26T05:07:45.284508+0000","mapping_epoch":184,"log_start":"190'700","ondisk_log_start":"190'700","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4252,"log_dups_size":0,"ondisk_log_size":4252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:01.870974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324534272,"num_objects":4952,"num_object_clones":0,"num_object_copies":14856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4952,"num_write_kb":316928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":302,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432629+0000","last_change":"2024-03-26T05:06:05.019093+0000","last_active":"2024-03-26T05:07:41.432629+0000","last_peered":"2024-03-26T05:07:41.432629+0000","last_clean":"2024-03-26T05:07:41.432629+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:07:41.432629+0000","last_undegraded":"2024-03-26T05:07:41.432629+0000","last_fullsized":"2024-03-26T05:07:41.432629+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:05.018999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:05.018999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"161'10","reported_seq":422,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.437523+0000","last_change":"2024-03-26T05:06:56.184462+0000","last_active":"2024-03-26T05:07:41.437523+0000","last_peered":"2024-03-26T05:07:41.437523+0000","last_clean":"2024-03-26T05:07:41.437523+0000","last_became_active":"2024-03-26T05:05:36.424309+0000","last_became_peered":"2024-03-26T05:05:36.424309+0000","last_unstale":"2024-03-26T05:07:41.437523+0000","last_undegraded":"2024-03-26T05:07:41.437523+0000","last_fullsized":"2024-03-26T05:07:41.437523+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'10","last_scrub_stamp":"2024-03-26T05:06:56.184325+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:06:56.184325+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:07.356139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":319,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.448862+0000","last_change":"2024-03-26T05:06:06.522418+0000","last_active":"2024-03-26T05:07:41.448862+0000","last_peered":"2024-03-26T05:07:41.448862+0000","last_clean":"2024-03-26T05:07:41.448862+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:07:41.448862+0000","last_undegraded":"2024-03-26T05:07:41.448862+0000","last_fullsized":"2024-03-26T05:07:41.448862+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:05.000183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:05.000183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":325,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-03-26T05:07:41.432402+0000","last_change":"2024-03-26T05:06:45.298927+0000","last_active":"2024-03-26T05:07:41.432402+0000","last_peered":"2024-03-26T05:07:41.432402+0000","last_clean":"2024-03-26T05:07:41.432402+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:07:41.432402+0000","last_undegraded":"2024-03-26T05:07:41.432402+0000","last_fullsized":"2024-03-26T05:07:41.432402+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:06:45.298241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.298241+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:07:49.860740+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5244518400,"num_objects":80025,"num_object_clones":0,"num_object_copies":240075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80025,"num_write_kb":5121600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15724904448,"data_stored":15724904448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73925,"ondisk_log_size":73925,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":436,"num_read_kb":431,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":984,"num_read_kb":709,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":630,"ondisk_log_size":630,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005149,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1770020,"kb_used_data":1612508,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":91975132,"statfs":{"total":95995035648,"available":94182535168,"internally_reserved":0,"allocated":1651208192,"data_stored":1650791658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":161282506},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332194,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2379404,"kb_used_data":2217732,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":91365748,"statfs":{"total":95995035648,"available":93558525952,"internally_reserved":0,"allocated":2270957568,"data_stored":2270563874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165542342},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2422456,"kb_used_data":2206640,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91322696,"statfs":{"total":95995035648,"available":93514440704,"internally_reserved":0,"allocated":2259599360,"data_stored":2259161220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738497,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2489964,"kb_used_data":2260260,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":91255188,"statfs":{"total":95995035648,"available":93445312512,"internally_reserved":0,"allocated":2314506240,"data_stored":2314081466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411447,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2415564,"kb_used_data":2208068,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":91329588,"statfs":{"total":95995035648,"available":93521498112,"internally_reserved":0,"allocated":2261061632,"data_stored":2260667441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117105,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3136680,"kb_used_data":2862560,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":90608472,"statfs":{"total":95995035648,"available":92783075328,"internally_reserved":0,"allocated":2931261440,"data_stored":2930839668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":280689095},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920877,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2051360,"kb_used_data":1889688,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":91693792,"statfs":{"total":95995035648,"available":93894443008,"internally_reserved":0,"allocated":1935040512,"data_stored":1934626863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":176093659239,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2320629760,"data_stored":2320629760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2280980480,"data_stored":2280980480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2283143168,"data_stored":2283143168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2959278080,"data_stored":2959278080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1935343616,"data_stored":1935343616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2295201792,"data_stored":2295201792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1650327552,"data_stored":1650327552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:07:50.957 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T05:07:50.957 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:07:50.958 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-26T05:07:50.958 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd in 5 2024-03-26T05:07:50.958 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:51.038 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:51.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:51.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:50 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:07:51.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:50 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:07:51.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:50 smithi132 ceph-mon[33500]: 8.9 scrub starts 2024-03-26T05:07:51.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:07:51.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:07:51.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:50 smithi132 ceph-mon[33500]: 3.9 scrub starts 2024-03-26T05:07:51.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:50 smithi132 ceph-mon[33500]: 3.9 scrub ok 2024-03-26T05:07:51.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:50 smithi132 ceph-mon[33500]: pgmap v326: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 899 op/s 2024-03-26T05:07:51.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:50 smithi132 ceph-mon[33500]: osdmap e209: 8 total, 8 up, 7 in 2024-03-26T05:07:51.139 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:51.165 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:51.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:51.266 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:51.293 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:51.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:51.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:51.395 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:51.435 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:51.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:51.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:51.537 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:51.587 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:51.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:51.644 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T05:07:51.645 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:51 smithi080 ceph-mon[40404]: 8.9 scrub ok 2024-03-26T05:07:51.645 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:51 smithi080 ceph-mon[40404]: 4.1d scrub starts 2024-03-26T05:07:51.645 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:51 smithi080 ceph-mon[40404]: 4.1d scrub ok 2024-03-26T05:07:51.645 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:51 smithi080 ceph-mon[40404]: 6.2 scrub starts 2024-03-26T05:07:51.645 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:51 smithi080 ceph-mon[40404]: 6.2 scrub ok 2024-03-26T05:07:51.646 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:51 smithi080 ceph-mon[40404]: osdmap e210: 8 total, 8 up, 7 in 2024-03-26T05:07:51.646 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:51 smithi080 ceph-mon[29691]: 8.9 scrub ok 2024-03-26T05:07:51.646 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:51 smithi080 ceph-mon[29691]: 4.1d scrub starts 2024-03-26T05:07:51.646 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:51 smithi080 ceph-mon[29691]: 4.1d scrub ok 2024-03-26T05:07:51.646 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:51 smithi080 ceph-mon[29691]: 6.2 scrub starts 2024-03-26T05:07:51.646 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:51 smithi080 ceph-mon[29691]: 6.2 scrub ok 2024-03-26T05:07:51.646 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:51 smithi080 ceph-mon[29691]: osdmap e210: 8 total, 8 up, 7 in 2024-03-26T05:07:51.689 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:51.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:51.733 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:51.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:51.835 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:51.847 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:07:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:51.880 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:51.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:51.921 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 59 16 87438 87422 92.5878 67.625 0.0268489 0.0107946 2024-03-26T05:07:51.963 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:07:51.981 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:52.014 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:52.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:52.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:51 smithi132 ceph-mon[33500]: 8.9 scrub ok 2024-03-26T05:07:52.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:51 smithi132 ceph-mon[33500]: 4.1d scrub starts 2024-03-26T05:07:52.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:51 smithi132 ceph-mon[33500]: 4.1d scrub ok 2024-03-26T05:07:52.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:51 smithi132 ceph-mon[33500]: 6.2 scrub starts 2024-03-26T05:07:52.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:51 smithi132 ceph-mon[33500]: 6.2 scrub ok 2024-03-26T05:07:52.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:51 smithi132 ceph-mon[33500]: osdmap e210: 8 total, 8 up, 7 in 2024-03-26T05:07:52.117 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:52.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:52.186 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:52.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:52.287 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:52.341 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:52.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:52.397 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:52.443 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:52.493 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:52.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:52.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:52.595 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:52.643 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:52.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:52.744 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:52.760 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:52 smithi132 ceph-mon[33500]: 5.1 scrub starts 2024-03-26T05:07:52.760 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:52 smithi132 ceph-mon[33500]: 5.1 scrub ok 2024-03-26T05:07:52.760 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:52 smithi132 ceph-mon[33500]: pgmap v329: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 864 B/s rd, 57 MiB/s wr, 917 op/s 2024-03-26T05:07:52.760 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:52 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:07:52.760 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:52 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:07:52.779 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:52.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:52.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:52.881 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:52.921 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:07:52.930636+0000 min lat: 0.00124249 max lat: 0.162251 avg lat: 0.0108484 2024-03-26T05:07:52.921 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:07:52.921 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 60 16 88486 88470 92.1363 65.5 0.0104141 0.0108484 2024-03-26T05:07:52.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:52.922 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:52.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:53.023 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:53.057 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:53.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:53.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:52 smithi080 ceph-mon[40404]: 5.1 scrub starts 2024-03-26T05:07:53.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:52 smithi080 ceph-mon[40404]: 5.1 scrub ok 2024-03-26T05:07:53.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:52 smithi080 ceph-mon[40404]: pgmap v329: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 864 B/s rd, 57 MiB/s wr, 917 op/s 2024-03-26T05:07:53.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:52 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:07:53.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:52 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:07:53.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:52 smithi080 ceph-mon[29691]: 5.1 scrub starts 2024-03-26T05:07:53.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:52 smithi080 ceph-mon[29691]: 5.1 scrub ok 2024-03-26T05:07:53.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:52 smithi080 ceph-mon[29691]: pgmap v329: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 864 B/s rd, 57 MiB/s wr, 917 op/s 2024-03-26T05:07:53.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:52 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:07:53.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:52 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:07:53.158 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:53.196 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:53.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:53.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:53.339 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:53.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:53.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:53.441 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:53.479 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:53.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:53.582 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:53.642 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:53.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:53.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:53.743 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:53.748 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:53 smithi132 ceph-mon[33500]: 8.5 scrub starts 2024-03-26T05:07:53.748 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:53 smithi132 ceph-mon[33500]: 6.d deep-scrub starts 2024-03-26T05:07:53.748 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:53 smithi132 ceph-mon[33500]: 6.d deep-scrub ok 2024-03-26T05:07:53.748 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:53 smithi132 ceph-mon[33500]: 4.11 scrub starts 2024-03-26T05:07:53.748 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:53 smithi132 ceph-mon[33500]: 4.11 scrub ok 2024-03-26T05:07:53.749 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:53 smithi132 ceph-mon[33500]: osdmap e211: 8 total, 8 up, 7 in 2024-03-26T05:07:53.793 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:53.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:53.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:53 smithi080 ceph-mon[29691]: 8.5 scrub starts 2024-03-26T05:07:53.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:53 smithi080 ceph-mon[29691]: 6.d deep-scrub starts 2024-03-26T05:07:53.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:53 smithi080 ceph-mon[29691]: 6.d deep-scrub ok 2024-03-26T05:07:53.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:53 smithi080 ceph-mon[29691]: 4.11 scrub starts 2024-03-26T05:07:53.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:53 smithi080 ceph-mon[29691]: 4.11 scrub ok 2024-03-26T05:07:53.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:53 smithi080 ceph-mon[29691]: osdmap e211: 8 total, 8 up, 7 in 2024-03-26T05:07:53.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:53 smithi080 ceph-mon[40404]: 8.5 scrub starts 2024-03-26T05:07:53.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:53 smithi080 ceph-mon[40404]: 6.d deep-scrub starts 2024-03-26T05:07:53.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:53 smithi080 ceph-mon[40404]: 6.d deep-scrub ok 2024-03-26T05:07:53.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:53 smithi080 ceph-mon[40404]: 4.11 scrub starts 2024-03-26T05:07:53.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:53 smithi080 ceph-mon[40404]: 4.11 scrub ok 2024-03-26T05:07:53.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:53 smithi080 ceph-mon[40404]: osdmap e211: 8 total, 8 up, 7 in 2024-03-26T05:07:53.896 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:53.922 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 61 16 89600 89584 91.7648 69.625 0.00852668 0.0108929 2024-03-26T05:07:53.928 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:53.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:53.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:54.030 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:54.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:54.060 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:54.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:54.117 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:07:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:07:54.161 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:54.197 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:54.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:54.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:54.286 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:07:54.286 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:07:54.299 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:54.337 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:54.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:54.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:54.439 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:54.482 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:54.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:54.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:54.566 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:07:54.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:54.653 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:54.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:54.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:54.727 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 8.5 scrub ok 2024-03-26T05:07:54.727 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 6.a scrub starts 2024-03-26T05:07:54.728 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 6.a scrub ok 2024-03-26T05:07:54.728 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 3.1b deep-scrub starts 2024-03-26T05:07:54.728 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 3.1b deep-scrub ok 2024-03-26T05:07:54.728 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 4.12 scrub starts 2024-03-26T05:07:54.728 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 4.12 scrub ok 2024-03-26T05:07:54.728 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 3.1c scrub starts 2024-03-26T05:07:54.728 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 3.1c scrub ok 2024-03-26T05:07:54.728 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 5.0 scrub starts 2024-03-26T05:07:54.728 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 5.0 scrub ok 2024-03-26T05:07:54.728 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 4.e scrub starts 2024-03-26T05:07:54.728 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: 4.e scrub ok 2024-03-26T05:07:54.728 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[40404]: pgmap v331: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 57 MiB/s wr, 905 op/s 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 8.5 scrub ok 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 6.a scrub starts 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 6.a scrub ok 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 3.1b deep-scrub starts 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 3.1b deep-scrub ok 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 4.12 scrub starts 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 4.12 scrub ok 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 3.1c scrub starts 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 3.1c scrub ok 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 5.0 scrub starts 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 5.0 scrub ok 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 4.e scrub starts 2024-03-26T05:07:54.729 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: 4.e scrub ok 2024-03-26T05:07:54.730 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:54 smithi080 ceph-mon[29691]: pgmap v331: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 57 MiB/s wr, 905 op/s 2024-03-26T05:07:54.755 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:54.788 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:54.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:54.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:54.890 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:54.925 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 62 16 90574 90558 91.2659 60.875 0.0333767 0.0109514 2024-03-26T05:07:54.937 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:54.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:54.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 8.5 scrub ok 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 6.a scrub starts 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 6.a scrub ok 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 3.1b deep-scrub starts 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 3.1b deep-scrub ok 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 4.12 scrub starts 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 4.12 scrub ok 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 3.1c scrub starts 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 3.1c scrub ok 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 5.0 scrub starts 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 5.0 scrub ok 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 4.e scrub starts 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: 4.e scrub ok 2024-03-26T05:07:55.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:54 smithi132 ceph-mon[33500]: pgmap v331: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 57 MiB/s wr, 905 op/s 2024-03-26T05:07:55.039 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:55.057 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T05:07:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:55.080 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:55.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:55.181 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:55.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:55.224 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:55.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:55.292 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:07:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:07:55.324 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:55.355 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:55.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:55.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:55.372 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:07:55.456 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:55.484 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:55.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:55.586 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:55.616 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:55.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:55.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:55.679 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:07:55.717 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:55.766 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:55.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:55.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:55 smithi132 ceph-mon[33500]: 4.1e scrub starts 2024-03-26T05:07:55.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:55 smithi132 ceph-mon[33500]: 4.1e scrub ok 2024-03-26T05:07:55.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:55 smithi132 ceph-mon[33500]: 4.7 scrub starts 2024-03-26T05:07:55.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:55 smithi132 ceph-mon[33500]: 4.7 scrub ok 2024-03-26T05:07:55.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:55 smithi132 ceph-mon[33500]: 3.1a scrub starts 2024-03-26T05:07:55.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:55 smithi132 ceph-mon[33500]: 3.1a scrub ok 2024-03-26T05:07:55.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:55 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:55.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:55 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/155822470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:07:55.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:55 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:07:55.868 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:55.898 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:55.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:55.923 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 63 16 91473 91457 90.7091 56.1875 0.0256638 0.0110193 2024-03-26T05:07:55.999 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:56.049 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:56.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:56.062 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[40404]: 4.1e scrub starts 2024-03-26T05:07:56.062 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[40404]: 4.1e scrub ok 2024-03-26T05:07:56.063 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[40404]: 4.7 scrub starts 2024-03-26T05:07:56.063 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[40404]: 4.7 scrub ok 2024-03-26T05:07:56.063 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[40404]: 3.1a scrub starts 2024-03-26T05:07:56.063 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[40404]: 3.1a scrub ok 2024-03-26T05:07:56.063 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:56.063 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/155822470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:07:56.063 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:07:56.063 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[29691]: 4.1e scrub starts 2024-03-26T05:07:56.063 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[29691]: 4.1e scrub ok 2024-03-26T05:07:56.063 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[29691]: 4.7 scrub starts 2024-03-26T05:07:56.063 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[29691]: 4.7 scrub ok 2024-03-26T05:07:56.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[29691]: 3.1a scrub starts 2024-03-26T05:07:56.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[29691]: 3.1a scrub ok 2024-03-26T05:07:56.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:07:56.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/155822470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:07:56.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:55 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:07:56.151 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:56.185 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:56.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:56.287 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:56.331 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:56.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:56.432 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:56.464 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:56.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:56.565 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:56.602 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:56.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:56.704 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:56.744 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:56.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:56.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:07:56.873 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:56.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:56.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:07:56.923 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 64 16 92612 92596 90.404 71.1875 0.0205096 0.0110563 2024-03-26T05:07:56.952 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:56 smithi132 ceph-mon[33500]: 5.17 scrub starts 2024-03-26T05:07:56.952 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:56 smithi132 ceph-mon[33500]: 5.17 scrub ok 2024-03-26T05:07:56.952 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:56 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:07:56.952 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:56 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:07:56.952 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:56 smithi132 ceph-mon[33500]: osdmap e212: 8 total, 8 up, 7 in 2024-03-26T05:07:56.952 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:56 smithi132 ceph-mon[33500]: pgmap v333: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 865 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-03-26T05:07:56.975 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:56.986 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:56 smithi080 ceph-mon[29691]: 5.17 scrub starts 2024-03-26T05:07:56.986 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:56 smithi080 ceph-mon[29691]: 5.17 scrub ok 2024-03-26T05:07:56.986 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:56 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:07:56.987 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:56 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:07:56.987 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:56 smithi080 ceph-mon[29691]: osdmap e212: 8 total, 8 up, 7 in 2024-03-26T05:07:56.987 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:56 smithi080 ceph-mon[29691]: pgmap v333: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 865 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-03-26T05:07:56.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:56 smithi080 ceph-mon[40404]: 5.17 scrub starts 2024-03-26T05:07:56.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:56 smithi080 ceph-mon[40404]: 5.17 scrub ok 2024-03-26T05:07:56.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:56 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:07:56.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:56 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:07:56.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:56 smithi080 ceph-mon[40404]: osdmap e212: 8 total, 8 up, 7 in 2024-03-26T05:07:56.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:56 smithi080 ceph-mon[40404]: pgmap v333: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 865 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-03-26T05:07:57.021 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:57.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:57.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:57.123 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:57.159 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:57.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:57.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:57.260 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:57.297 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:57.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:57.400 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:57.429 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:57.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:57.485 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:07:57.485 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:07:57.532 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:57.561 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:57.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:57.662 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:57.702 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:57.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:57.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:57.804 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:57.851 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:57.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:57.923 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 65 16 93743 93727 90.1007 70.6875 0.0140113 0.0110939 2024-03-26T05:07:57.953 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:57.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:57.993 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:57.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:58.094 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:07:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:58.133 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:58.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:07:58.235 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:58.248 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T05:07:58.276 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:58.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:58.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:58.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:58.402 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:58.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:58.504 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:58.528 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:07:58.543 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:58.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:58.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:58.645 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:58.673 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:58.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:58.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:58.775 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:07:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:58.814 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:58.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:07:58.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:58 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a[29687]: 2024-03-26T05:07:58.440+0000 7f70d2504700 -1 mon.a@0(leader).osd e212 definitely_dead 0 2024-03-26T05:07:58.915 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:07:58.919 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:58 smithi132 ceph-mon[33500]: pgmap v334: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 798 B/s rd, 51 MiB/s wr, 822 op/s 2024-03-26T05:07:58.919 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1873977250' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-26T05:07:58.919 INFO:teuthology.orchestra.run.smithi080.stderr:marked in osd.5. 2024-03-26T05:07:58.923 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 66 16 94675 94659 89.6181 58.25 0.0083803 0.0111527 2024-03-26T05:07:58.962 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:58.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:07:59.064 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:59.089 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:59.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:59.191 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:59.215 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:59.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:59.237 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:58 smithi080 ceph-mon[29691]: pgmap v334: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 798 B/s rd, 51 MiB/s wr, 822 op/s 2024-03-26T05:07:59.237 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1873977250' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-26T05:07:59.238 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:58 smithi080 ceph-mon[40404]: pgmap v334: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 798 B/s rd, 51 MiB/s wr, 822 op/s 2024-03-26T05:07:59.238 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1873977250' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-26T05:07:59.316 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:59.339 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:59.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:59.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:59.343 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:07:59.441 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:07:59.466 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:07:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:59.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:07:59.568 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:59.592 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:59.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:07:59.694 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:07:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:59.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:07:59.820 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:07:59.845 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:59.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:07:59.923 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 67 16 94700 94684 88.3039 1.5625 0.00392448 0.0111552 2024-03-26T05:07:59.946 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:07:59.974 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:07:59.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:59.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:00.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:59 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1873977250' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-26T05:08:00.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:59 smithi132 ceph-mon[33500]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T05:08:00.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:07:59 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3300456588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:00.076 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:00.100 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:00.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:00.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:00.201 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:00.225 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:00.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:00.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:00.321 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:08:00.321 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:08:00.321 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18340 7.1 7:8b7e6025:::benchmark_data_smithi080_80848_object18339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.064655+0000", 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.26322624800002, 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.22265895099999999, 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18340 2024-03-26T05:08:00.322 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064655+0000", 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064655+0000", 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.323 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064656+0000", 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.424e-06 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064739+0000", 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2670999999999995e-05 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064754+0000", 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4780000000000001e-05 2024-03-26T05:08:00.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064763+0000", 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8740000000000001e-06 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064819+0000", 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6740999999999998e-05 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.325 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064924+0000", 2024-03-26T05:08:00.326 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104545 2024-03-26T05:08:00.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.326 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.326 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.065047+0000", 2024-03-26T05:08:00.326 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012353 2024-03-26T05:08:00.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.326 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.326 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.065997+0000", 2024-03-26T05:08:00.326 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00094926299999999999 2024-03-26T05:08:00.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.066103+0000", 2024-03-26T05:08:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000106575 2024-03-26T05:08:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287201+0000", 2024-03-26T05:08:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.22109801900000001 2024-03-26T05:08:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287266+0000", 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4567999999999996e-05 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287314+0000", 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7960999999999997e-05 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18358 7.1 7:8417ed37:::benchmark_data_smithi080_80848_object18357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:00.329 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.084602+0000", 2024-03-26T05:08:00.329 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.24327948799998, 2024-03-26T05:08:00.329 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20301822899999999, 2024-03-26T05:08:00.329 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.329 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.329 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.329 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.329 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.329 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18358 2024-03-26T05:08:00.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.329 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.329 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.084602+0000", 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.084602+0000", 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.084603+0000", 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2330000000000001e-06 2024-03-26T05:08:00.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.099940+0000", 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.015337491 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.099960+0000", 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.9471999999999999e-05 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.099967+0000", 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2880000000000004e-06 2024-03-26T05:08:00.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.100023+0000", 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6268999999999999e-05 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.100143+0000", 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011924700000000001 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.100270+0000", 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012700999999999999 2024-03-26T05:08:00.332 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.101482+0000", 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001212185 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.101961+0000", 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00047930800000000001 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287548+0000", 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.185586844 2024-03-26T05:08:00.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287582+0000", 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.3637000000000003e-05 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287620+0000", 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8244999999999997e-05 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18365 7.1 7:86ff0e4b:::benchmark_data_smithi080_80848_object18364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.102039+0000", 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.22584177499999, 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18571283699999999, 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18365 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.335 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102039+0000", 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102039+0000", 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102040+0000", 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.139e-06 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.336 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102104+0000", 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3891000000000002e-05 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102114+0000", 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9869999999999994e-06 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102122+0000", 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4699999999999996e-06 2024-03-26T05:08:00.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102179+0000", 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7643999999999999e-05 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102319+0000", 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013995500000000001 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102422+0000", 2024-03-26T05:08:00.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000102944 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.103247+0000", 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082511399999999995 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.103513+0000", 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000265944 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.339 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287662+0000", 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18414857300000001 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287715+0000", 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2778999999999997e-05 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287752+0000", 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7397000000000001e-05 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.340 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18384 7.1 7:85810267:::benchmark_data_smithi080_80848_object18383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.119204+0000", 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.20867723999999, 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16863742600000001, 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18384 2024-03-26T05:08:00.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119204+0000", 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119204+0000", 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.342 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119206+0000", 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.906e-06 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119341+0000", 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013563799999999999 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119355+0000", 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3217000000000001e-05 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119372+0000", 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.7275999999999999e-05 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119434+0000", 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1624000000000005e-05 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119579+0000", 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000145859 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.119714+0000", 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013468599999999999 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.121523+0000", 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0018088920000000001 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.122062+0000", 2024-03-26T05:08:00.345 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053930099999999995 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287780+0000", 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16571784000000001 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287810+0000", 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.9467000000000001e-05 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.346 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287841+0000", 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.1720000000000001e-05 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18416 7.1 7:83212609:::benchmark_data_smithi080_80848_object18415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.136847+0000", 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.19103437199999, 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.15127143200000001, 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.347 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18416 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136847+0000", 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136847+0000", 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.349 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.349 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136848+0000", 2024-03-26T05:08:00.349 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1960000000000001e-06 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136920+0000", 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2155000000000003e-05 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136933+0000", 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2717999999999999e-05 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136939+0000", 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0950000000000001e-06 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.136995+0000", 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6252999999999998e-05 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.137133+0000", 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013836400000000001 2024-03-26T05:08:00.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.137300+0000", 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000166074 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.138066+0000", 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00076619299999999997 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.138417+0000", 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035126500000000002 2024-03-26T05:08:00.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.354 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.354 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.354 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.288055+0000", 2024-03-26T05:08:00.354 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14963824100000001 2024-03-26T05:08:00.354 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.354 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.354 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.354 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.288090+0000", 2024-03-26T05:08:00.354 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.4411e-05 2024-03-26T05:08:00.354 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.354 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.354 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.288118+0000", 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.8467e-05 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18428 7.1 7:8c797d16:::benchmark_data_smithi080_80848_object18427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.154718+0000", 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.17316321499999, 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13353142200000001, 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.355 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18428 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154718+0000", 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154718+0000", 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154719+0000", 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1650000000000001e-06 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154775+0000", 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6200999999999999e-05 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154787+0000", 2024-03-26T05:08:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2072000000000001e-05 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154793+0000", 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8340000000000004e-06 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154834+0000", 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1192999999999997e-05 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.154933+0000", 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9018000000000006e-05 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.155021+0000", 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7899000000000007e-05 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.155727+0000", 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070548900000000001 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.156041+0000", 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00031450100000000002 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.288158+0000", 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13211668100000001 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.288211+0000", 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2611000000000002e-05 2024-03-26T05:08:00.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.288249+0000", 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8757999999999997e-05 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22561 7.1 7:8ddb660b:::benchmark_data_smithi080_80848_object22560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.343328+0000", 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 312.98455330000002, 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14323868000000001, 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.361 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22561 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343328+0000", 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.362 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343328+0000", 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343329+0000", 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5209999999999999e-06 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343406+0000", 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6990999999999995e-05 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343417+0000", 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.101e-05 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343426+0000", 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4719999999999995e-06 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343486+0000", 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0572999999999998e-05 2024-03-26T05:08:00.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.365 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.365 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.365 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343618+0000", 2024-03-26T05:08:00.365 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013210300000000001 2024-03-26T05:08:00.365 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.365 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.365 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.365 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.343789+0000", 2024-03-26T05:08:00.365 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017039099999999999 2024-03-26T05:08:00.365 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.365 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.365 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.366 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345355+0000", 2024-03-26T05:08:00.366 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001565821 2024-03-26T05:08:00.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.366 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.366 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.366 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.346123+0000", 2024-03-26T05:08:00.366 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00076833499999999998 2024-03-26T05:08:00.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.366 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.366 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.366 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.486489+0000", 2024-03-26T05:08:00.366 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14036646 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.486534+0000", 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4342000000000003e-05 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.486566+0000", 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2660999999999998e-05 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.367 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31393 7.1 7:8ed06db3:::benchmark_data_smithi080_80848_object31392:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.251817+0000", 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 308.076064385, 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13570653199999999, 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31393 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.368 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251817+0000", 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251817+0000", 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251818+0000", 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5200000000000001e-06 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.369 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251861+0000", 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3078000000000003e-05 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251874+0000", 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2662e-05 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251882+0000", 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9610000000000005e-06 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.251937+0000", 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4648999999999998e-05 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252055+0000", 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000118832 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252183+0000", 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012804099999999999 2024-03-26T05:08:00.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.253129+0000", 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00094583600000000003 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.255948+0000", 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0028189959999999998 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387405+0000", 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13145715899999999 2024-03-26T05:08:00.372 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387469+0000", 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3329000000000003e-05 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387523+0000", 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4469000000000003e-05 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.373 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31398 7.1 7:8f429419:::benchmark_data_smithi080_80848_object31397:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.254661+0000", 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 308.07322021599998, 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.134356052, 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31398 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.374 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254661+0000", 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254661+0000", 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254662+0000", 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0380000000000001e-06 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254706+0000", 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4140000000000001e-05 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254716+0000", 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7580000000000004e-06 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254723+0000", 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1489999999999997e-06 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.376 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254773+0000", 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.028e-05 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.254868+0000", 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4629999999999994e-05 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.255006+0000", 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000138434 2024-03-26T05:08:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.256093+0000", 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010865339999999999 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.257095+0000", 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010021159999999999 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388900+0000", 2024-03-26T05:08:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13180470799999999 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388970+0000", 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0507999999999996e-05 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.389017+0000", 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6757e-05 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31415 7.1 7:86e7a695:::benchmark_data_smithi080_80848_object31414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.263992+0000", 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 308.06388901600002, 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.125213354, 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31415 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.263992+0000", 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.263992+0000", 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.263994+0000", 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.635e-06 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.264034+0000", 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9892999999999999e-05 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.264044+0000", 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0338e-05 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.264050+0000", 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9009999999999999e-06 2024-03-26T05:08:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.264096+0000", 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6056000000000001e-05 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.264188+0000", 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2281999999999995e-05 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.264282+0000", 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4197999999999995e-05 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.265309+0000", 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010261479999999999 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.265787+0000", 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00047891299999999997 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.389074+0000", 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.123286847 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.389165+0000", 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0341999999999997e-05 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.389205+0000", 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0800999999999998e-05 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31421 7.1 7:8d6f8e09:::benchmark_data_smithi080_80848_object31420:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:00.385 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.266844+0000", 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 308.06103705200002, 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.12681271899999999, 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31421 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266844+0000", 2024-03-26T05:08:00.386 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266844+0000", 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266845+0000", 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0800000000000003e-07 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266891+0000", 2024-03-26T05:08:00.387 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5985000000000001e-05 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266901+0000", 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.004e-05 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266907+0000", 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7790000000000001e-06 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.266958+0000", 2024-03-26T05:08:00.388 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1307000000000002e-05 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.267044+0000", 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6434000000000006e-05 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.267133+0000", 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8412999999999995e-05 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268018+0000", 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088498500000000003 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383541+0000", 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.115523535 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.393560+0000", 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010018681999999999 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.393614+0000", 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3761000000000002e-05 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.393657+0000", 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2889999999999998e-05 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(client.24701.0:31421 7.1 e111/103 ondisk, result = 0)", 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.268204+0000", 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 308.05967663199999, 2024-03-26T05:08:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.115345924, 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268204+0000", 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268204+0000", 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268205+0000", 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8400000000000003e-07 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268212+0000", 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7229999999999998e-06 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268223+0000", 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1517e-05 2024-03-26T05:08:00.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.268231+0000", 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.024e-06 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383525+0000", 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.115294747 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383534+0000", 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2310000000000002e-06 2024-03-26T05:08:00.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383550+0000", 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5897999999999999e-05 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:77844 7.1 7:8057a429:::benchmark_data_smithi080_80848_object77843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:03:26.347827+0000", 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 273.98005456999999, 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11528007, 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.395 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 77844 2024-03-26T05:08:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.347827+0000", 2024-03-26T05:08:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.347827+0000", 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.347831+0000", 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2309999999999998e-06 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.347878+0000", 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6898999999999999e-05 2024-03-26T05:08:00.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.398 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.398 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350047+0000", 2024-03-26T05:08:00.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021689029999999998 2024-03-26T05:08:00.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.398 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.398 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350062+0000", 2024-03-26T05:08:00.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5418999999999999e-05 2024-03-26T05:08:00.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455338+0000", 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.105275707 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455486+0000", 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000148455 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455608+0000", 2024-03-26T05:08:00.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012162700000000001 2024-03-26T05:08:00.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.460540+0000", 2024-03-26T05:08:00.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0049318629999999999 2024-03-26T05:08:00.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.460581+0000", 2024-03-26T05:08:00.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0967000000000003e-05 2024-03-26T05:08:00.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.462999+0000", 2024-03-26T05:08:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0024186949999999998 2024-03-26T05:08:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.463063+0000", 2024-03-26T05:08:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3484000000000004e-05 2024-03-26T05:08:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.463107+0000", 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3819999999999997e-05 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:77846 7.1 7:85e9d59f:::benchmark_data_smithi080_80848_object77845:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:03:26.350086+0000", 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 273.97779516999998, 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.115705316, 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.402 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.403 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.403 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 77846 2024-03-26T05:08:00.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.403 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.403 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350086+0000", 2024-03-26T05:08:00.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.403 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350086+0000", 2024-03-26T05:08:00.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350087+0000", 2024-03-26T05:08:00.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8899999999999998e-07 2024-03-26T05:08:00.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350140+0000", 2024-03-26T05:08:00.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2763000000000003e-05 2024-03-26T05:08:00.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350151+0000", 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1508999999999999e-05 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.350158+0000", 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3019999999999997e-06 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.458040+0000", 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10788157600000001 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.458169+0000", 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012913200000000001 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.458328+0000", 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000158426 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.463157+0000", 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0048291560000000002 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.465562+0000", 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0024053759999999999 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.465639+0000", 2024-03-26T05:08:00.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7385999999999994e-05 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.465745+0000", 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000105134 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.465791+0000", 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6666999999999999e-05 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.24701.0:77840 7.4 e123/103 7:2ec32914:::benchmark_data_smithi080_80848_object77839:head v 123'4783, mlcod=122'4781)", 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:03:26.353585+0000", 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 273.97429587699997, 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10951504200000001, 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.353585+0000", 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.353585+0000", 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.353586+0000", 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.1e-07 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.353623+0000", 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7287999999999999e-05 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455155+0000", 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.101532487 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455164+0000", 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6819999999999992e-06 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455253+0000", 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8764000000000006e-05 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.455274+0000", 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.1390999999999999e-05 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.463065+0000", 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0077907289999999997 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:03:26.463100+0000", 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5391e-05 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.412 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:188388 7.1 7:89691bc8:::benchmark_data_smithi080_80848_object68889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:18.437152+0000", 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 161.89072868, 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.12578125400000001, 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 188388 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.437152+0000", 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.437152+0000", 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.437154+0000", 2024-03-26T05:08:00.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2890000000000001e-06 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.437160+0000", 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8440000000000003e-06 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.437173+0000", 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3834e-05 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.437180+0000", 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7440000000000001e-06 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.437249+0000", 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8408e-05 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.437347+0000", 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8021000000000002e-05 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.437422+0000", 2024-03-26T05:08:00.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5841999999999997e-05 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.441318+0000", 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0038953960000000002 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.441802+0000", 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00048381999999999999 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.562832+0000", 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.121030106 2024-03-26T05:08:00.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.418 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.418 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.562884+0000", 2024-03-26T05:08:00.418 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2033000000000001e-05 2024-03-26T05:08:00.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:59 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1873977250' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-26T05:08:00.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:59 smithi080 ceph-mon[40404]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T05:08:00.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:07:59 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3300456588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:00.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:59 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1873977250' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-26T05:08:00.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:59 smithi080 ceph-mon[29691]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T05:08:00.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:07:59 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3300456588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:00.420 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.420 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.562934+0000", 2024-03-26T05:08:00.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9917000000000003e-05 2024-03-26T05:08:00.420 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.420 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.420 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.420 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:188390 7.1 7:8a2abdbc:::benchmark_data_smithi080_80848_object68891:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:18.438799+0000", 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 161.889082331, 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.124323208, 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 188390 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.438799+0000", 2024-03-26T05:08:00.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.438799+0000", 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.438800+0000", 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6499999999999998e-07 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.438806+0000", 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9530000000000002e-06 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.438820+0000", 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3101999999999999e-05 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.438828+0000", 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5180000000000004e-06 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.439440+0000", 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00061184399999999995 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.439520+0000", 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9825000000000002e-05 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.439586+0000", 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5946999999999997e-05 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.441401+0000", 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0018154600000000001 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.442087+0000", 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068623499999999999 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.562997+0000", 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.120909161 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.425 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.563090+0000", 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3280000000000001e-05 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:18.563122+0000", 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2118e-05 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24356.0:1904 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94789128800384 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:08:00.426 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:35.868777+0000", 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 144.45910449799999, 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.55502071600000003, 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24356", 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 1904 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868777+0000", 2024-03-26T05:08:00.427 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868777+0000", 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868778+0000", 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0410000000000001e-06 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868783+0000", 2024-03-26T05:08:00.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0180000000000001e-06 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868793+0000", 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0161e-05 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868800+0000", 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2559999999999996e-06 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868862+0000", 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1681999999999997e-05 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op must wait for map", 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868868+0000", 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0129999999999999e-06 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.884702+0000", 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.015834014 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.884713+0000", 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.076e-05 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:36.409253+0000", 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.524540177 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for degraded object", 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:36.409736+0000", 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00048286599999999998 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:36.421137+0000", 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011401029 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:36.421240+0000", 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103022 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:36.423797+0000", 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002557677 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:30460 8.2 8:4290ec1d:::benchmark_data_smithi080_109213_object30459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:07:06.572790+0000", 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 53.755091038000003, 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11755647599999999, 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 30460 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.572790+0000", 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.572790+0000", 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.572791+0000", 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2890000000000001e-06 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.572843+0000", 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1925000000000002e-05 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.572854+0000", 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0336999999999999e-05 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.572859+0000", 2024-03-26T05:08:00.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.682e-06 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.572921+0000", 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1236999999999997e-05 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.573034+0000", 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000113002 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.573161+0000", 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012793200000000001 2024-03-26T05:08:00.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.574417+0000", 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0012554700000000001 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.576493+0000", 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020759089999999999 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.690262+0000", 2024-03-26T05:08:00.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11376942600000001 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.690307+0000", 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5160000000000001e-05 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:06.690347+0000", 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9106999999999997e-05 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:32014 8.2 8:4f4f7d1e:::benchmark_data_smithi080_109213_object32013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:07:07.727424+0000", 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 52.600456770000001, 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11454641, 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 32014 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:00.439 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.727424+0000", 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.727424+0000", 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.727426+0000", 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3790000000000001e-06 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:00.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.727464+0000", 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8238000000000002e-05 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.727475+0000", 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.133e-05 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.727481+0000", 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2319999999999998e-06 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:00.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.727533+0000", 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1755000000000002e-05 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.727667+0000", 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013425000000000001 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.727848+0000", 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000180903 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.729132+0000", 2024-03-26T05:08:00.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0012832220000000001 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.838656+0000", 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.109524645 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.841862+0000", 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0032058339999999999 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:00.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.841920+0000", 2024-03-26T05:08:00.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8278e-05 2024-03-26T05:08:00.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:00.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:00.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:00.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:07.841971+0000", 2024-03-26T05:08:00.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0343999999999998e-05 2024-03-26T05:08:00.444 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.444 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.444 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.444 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:00.444 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:00.444 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:08:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:00.446 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:00.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:00.447 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:08:00.448 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:08:00.448 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:08:00.448 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:08:00.448 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:08:00.448 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:08:00.448 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:08:00.448 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:08:00.448 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:08:00.448 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:08:00.448 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:08:00.448 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:08:00.448 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:08:00.449 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:08:00.449 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:08:00.449 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:08:00.547 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:00.571 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:00.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:00.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:00.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:00.696 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:00.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:00.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:00.820 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:00.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:00.922 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:00.923 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 68 16 94700 94684 87.0055 0 - 0.0111552 2024-03-26T05:08:00.951 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:00.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:01.053 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:01.077 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:01.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:01.158 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:00 smithi132 ceph-mon[33500]: pgmap v336: 148 pgs: 3 unknown, 31 peering, 114 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T05:08:01.158 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:00 smithi132 ceph-mon[33500]: Health check failed: Reduced data availability: 13 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-03-26T05:08:01.158 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:00 smithi132 ceph-mon[33500]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T05:08:01.179 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:01.205 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:01.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:01.216 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T05:08:01.307 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:01.350 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:01.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:00 smithi080 ceph-mon[29691]: pgmap v336: 148 pgs: 3 unknown, 31 peering, 114 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T05:08:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:00 smithi080 ceph-mon[29691]: Health check failed: Reduced data availability: 13 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-03-26T05:08:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:00 smithi080 ceph-mon[29691]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T05:08:01.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:00 smithi080 ceph-mon[40404]: pgmap v336: 148 pgs: 3 unknown, 31 peering, 114 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T05:08:01.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:00 smithi080 ceph-mon[40404]: Health check failed: Reduced data availability: 13 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-03-26T05:08:01.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:00 smithi080 ceph-mon[40404]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T05:08:01.451 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:01.495 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:01.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:01.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:01.597 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:01.598 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:08:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:01.639 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:01.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:01.740 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:01.776 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:01.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:01.878 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:01.918 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:01.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:01.924 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 69 16 95423 95407 86.3995 22.5938 0.0145694 0.0115683 2024-03-26T05:08:02.021 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:02.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:01 smithi132 ceph-mon[33500]: 8.d starting backfill to osd.5 from (0'0,0'0] MAX to 212'5851 2024-03-26T05:08:02.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:01 smithi132 ceph-mon[33500]: 8.8 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5985 2024-03-26T05:08:02.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:01 smithi132 ceph-mon[33500]: 8.9 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5988 2024-03-26T05:08:02.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:01 smithi132 ceph-mon[33500]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T05:08:02.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:01 smithi132 ceph-mon[33500]: 8.2 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5871 2024-03-26T05:08:02.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:01 smithi132 ceph-mon[33500]: 8.4 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5890 2024-03-26T05:08:02.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:01 smithi132 ceph-mon[33500]: 8.3 starting backfill to osd.5 from (0'0,0'0] MAX to 212'6000 2024-03-26T05:08:02.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:01 smithi132 ceph-mon[33500]: 8.a starting backfill to osd.5 from (0'0,0'0] MAX to 212'5926 2024-03-26T05:08:02.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:01 smithi132 ceph-mon[33500]: 8.0 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5826 2024-03-26T05:08:02.072 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:02.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:02.145 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-26T05:08:02.173 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:02.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:02.210 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:02.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:02.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[29691]: 8.d starting backfill to osd.5 from (0'0,0'0] MAX to 212'5851 2024-03-26T05:08:02.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[29691]: 8.8 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5985 2024-03-26T05:08:02.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[29691]: 8.9 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5988 2024-03-26T05:08:02.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[29691]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T05:08:02.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[29691]: 8.2 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5871 2024-03-26T05:08:02.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[29691]: 8.4 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5890 2024-03-26T05:08:02.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[29691]: 8.3 starting backfill to osd.5 from (0'0,0'0] MAX to 212'6000 2024-03-26T05:08:02.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[29691]: 8.a starting backfill to osd.5 from (0'0,0'0] MAX to 212'5926 2024-03-26T05:08:02.212 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[29691]: 8.0 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5826 2024-03-26T05:08:02.213 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[40404]: 8.d starting backfill to osd.5 from (0'0,0'0] MAX to 212'5851 2024-03-26T05:08:02.213 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[40404]: 8.8 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5985 2024-03-26T05:08:02.213 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[40404]: 8.9 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5988 2024-03-26T05:08:02.213 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[40404]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T05:08:02.213 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[40404]: 8.2 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5871 2024-03-26T05:08:02.213 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[40404]: 8.4 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5890 2024-03-26T05:08:02.213 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[40404]: 8.3 starting backfill to osd.5 from (0'0,0'0] MAX to 212'6000 2024-03-26T05:08:02.213 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[40404]: 8.a starting backfill to osd.5 from (0'0,0'0] MAX to 212'5926 2024-03-26T05:08:02.213 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:01 smithi080 ceph-mon[40404]: 8.0 starting backfill to osd.5 from (0'0,0'0] MAX to 212'5826 2024-03-26T05:08:02.311 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:02.348 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:02.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:02.450 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:02.493 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:02.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:02.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:02.619 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:02.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:02.625 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:02.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:02.726 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:02.762 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:02.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:02.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:02.864 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:02.908 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:02.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:02.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:02.924 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 70 16 96423 96407 86.058 62.5 0.0162922 0.0116146 2024-03-26T05:08:03.010 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:03.049 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:03.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:03.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:03.151 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:03.199 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:03.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:03.301 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:03.344 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:03.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:03.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:03.446 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:03.494 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:03.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:03.597 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:03.639 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:03.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:03.740 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:03.756 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:03 smithi132 ceph-mon[33500]: pgmap v339: 148 pgs: 6 remapped+peering, 31 peering, 111 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 41 MiB/s wr, 660 op/s; 67 B/s, 0 objects/s recovering 2024-03-26T05:08:03.768 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:03.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:03.811 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:03 smithi080 ceph-mon[40404]: pgmap v339: 148 pgs: 6 remapped+peering, 31 peering, 111 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 41 MiB/s wr, 660 op/s; 67 B/s, 0 objects/s recovering 2024-03-26T05:08:03.811 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:03 smithi080 ceph-mon[29691]: pgmap v339: 148 pgs: 6 remapped+peering, 31 peering, 111 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 41 MiB/s wr, 660 op/s; 67 B/s, 0 objects/s recovering 2024-03-26T05:08:03.870 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:03.914 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:03.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:03.924 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 71 15 97291 97276 85.6109 54.3125 0.0106646 0.0116754 2024-03-26T05:08:04.017 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:04.054 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:04.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:04.117 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:08:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:08:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:08:04.157 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:04.182 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:04.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:04.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:04.266 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:08:04.266 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:08:04.266 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.266 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.15093.0:97598 8.0 e216/214 8:06a79bee:::benchmark_data_smithi080_109213_object97597:head v 216'6022, mlcod=216'6019)", 2024-03-26T05:08:04.266 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:08:04.270928+0000", 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0026773249999999999, 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0028222339999999999, 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.270928+0000", 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.270928+0000", 2024-03-26T05:08:04.267 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.270929+0000", 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9999999999999997e-07 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.270937+0000", 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9820000000000008e-06 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.270945+0000", 2024-03-26T05:08:04.268 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6470000000000004e-06 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.270951+0000", 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2269999999999998e-06 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.271056+0000", 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104114 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:04.269 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.271076+0000", 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.9984e-05 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.15093.0:97599 8.9 e216/214 8:9a2d11f2:::benchmark_data_smithi080_109213_object97598:head v 216'6176, mlcod=216'6174)", 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:08:04.272829+0000", 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.00077655300000000003, 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001000821, 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:08:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.272829+0000", 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.272829+0000", 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.272830+0000", 2024-03-26T05:08:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0299999999999998e-07 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.272845+0000", 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5056e-05 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.272857+0000", 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1942e-05 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.272862+0000", 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6230000000000002e-06 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.272958+0000", 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5685000000000003e-05 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.272981+0000", 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.3271000000000001e-05 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(8.a 216/214 [PushOp(8:50fc88a6:::benchmark_data_smithi080_109213_object96629:head, version: 215'6042, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:50fc88a6:::benchmark_data_smithi080_109213_object96629:head@215'6042, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T05:08:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:08:04.273629+0000", 2024-03-26T05:08:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 4294967295.9999762, 2024-03-26T05:08:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00030851700000000001, 2024-03-26T05:08:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:08:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273629+0000", 2024-03-26T05:08:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273629+0000", 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273630+0000", 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1600000000000001e-07 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273735+0000", 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104724 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273755+0000", 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.0564000000000002e-05 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273773+0000", 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.7555e-05 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273871+0000", 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8713000000000002e-05 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "pg_backfill(progress 8.a e 216/216 lb 8:50fc5f0d:::benchmark_data_smithi080_109213_object11:head)", 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:08:04.273801+0000", 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 4294967295.9998045, 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00019185599999999999, 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "queued for pg", 2024-03-26T05:08:04.277 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273801+0000", 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273801+0000", 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:04.278 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273801+0000", 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9999999999999998e-07 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273808+0000", 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8950000000000001e-06 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273816+0000", 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4590000000000001e-06 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:08:04.273821+0000", 2024-03-26T05:08:04.280 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.541e-06 2024-03-26T05:08:04.280 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:04.280 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:04.280 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:04.280 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:04.280 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-03-26T05:08:04.280 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 4 2024-03-26T05:08:04.280 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:08:04.283 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:04.319 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:04.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:04.421 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:04.461 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:04.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:04.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:04.563 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:04.576 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:08:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:04.611 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:04.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:04.712 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:04.777 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:04.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:04.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:04.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:04 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:08:04.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:04 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:08:04.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:04 smithi132 ceph-mon[33500]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T05:08:04.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:04 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:08:04.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:04 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:08:04.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:04 smithi080 ceph-mon[40404]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T05:08:04.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:04 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:08:04.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:04 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:08:04.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:04 smithi080 ceph-mon[29691]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T05:08:04.879 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:04.920 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:04.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:04.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:04.924 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 72 16 98241 98225 85.2449 59.3125 0.037761 0.0117257 2024-03-26T05:08:05.021 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:05.055 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:05.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:05.157 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:05.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:08:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:08:05.196 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:05.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:05.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:05.229 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T05:08:05.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:05.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:05.344 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:05.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:05.445 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:05.478 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:05.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:05.555 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:05.579 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:05.580 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:08:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:05.616 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:05.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:05.660 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:05 smithi080 ceph-mon[40404]: pgmap v341: 148 pgs: 6 remapped+peering, 31 peering, 111 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 805 op/s; 82 B/s, 1 objects/s recovering 2024-03-26T05:08:05.661 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:08:05.661 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:08:05.661 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:05 smithi080 ceph-mon[29691]: pgmap v341: 148 pgs: 6 remapped+peering, 31 peering, 111 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 805 op/s; 82 B/s, 1 objects/s recovering 2024-03-26T05:08:05.661 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:08:05.661 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:08:05.718 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:05.757 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:05 smithi132 ceph-mon[33500]: pgmap v341: 148 pgs: 6 remapped+peering, 31 peering, 111 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 805 op/s; 82 B/s, 1 objects/s recovering 2024-03-26T05:08:05.757 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:08:05.757 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:08:05.772 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:05.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:05.874 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:05.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:05.919 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:05.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:05.924 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 73 16 99164 99148 84.8674 57.6875 0.011557 0.0117782 2024-03-26T05:08:06.020 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:06.063 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:06.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:06.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:06.200 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:06.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:06.302 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:06.348 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:06.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:06.450 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:06.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:06.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:06.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:06.600 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:06.631 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:06.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:06.733 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:06.777 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:06.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:06.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:06 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:06.818 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:06 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:06.818 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:06 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:06.879 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:06.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:06.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:06.925 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 74 16 100081 100065 84.495 57.3125 0.0167382 0.0118303 2024-03-26T05:08:07.013 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:07.048 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:07.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:07.146 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:08:07.146 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:08:07.146 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-26T05:08:07.146 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T05:08:07.146 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T05:08:07.147 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-03-26T05:08:07.150 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:07.226 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:07.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:07.329 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:07.394 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:07.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:07.496 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:07.510 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:08:07.544 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:07.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:07.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:07.646 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:07.694 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:08:07.694 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:08:07.694 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2024-03-26T05:08:07.694 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2024-03-26T05:08:07.694 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:08:07.704 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:07.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:07.760 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:07 smithi132 ceph-mon[33500]: pgmap v342: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 249 MiB/s wr, 3.99k op/s; 47712/296064 objects misplaced (16.115%); 15 MiB/s, 247 objects/s recovering 2024-03-26T05:08:07.761 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:07 smithi132 ceph-mon[33500]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:08:07.761 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:07 smithi132 ceph-mon[33500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 16 pgs peering) 2024-03-26T05:08:07.761 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:07 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/85329962' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:08:07.761 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:07 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:08:07.806 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:07.846 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:07.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:07.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:07 smithi080 ceph-mon[29691]: pgmap v342: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 249 MiB/s wr, 3.99k op/s; 47712/296064 objects misplaced (16.115%); 15 MiB/s, 247 objects/s recovering 2024-03-26T05:08:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:07 smithi080 ceph-mon[29691]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:08:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:07 smithi080 ceph-mon[29691]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 16 pgs peering) 2024-03-26T05:08:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:07 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/85329962' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:08:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:07 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:08:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:07 smithi080 ceph-mon[40404]: pgmap v342: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 249 MiB/s wr, 3.99k op/s; 47712/296064 objects misplaced (16.115%); 15 MiB/s, 247 objects/s recovering 2024-03-26T05:08:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:07 smithi080 ceph-mon[40404]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:08:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:07 smithi080 ceph-mon[40404]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 16 pgs peering) 2024-03-26T05:08:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:07 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/85329962' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:08:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:07 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:08:07.925 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 75 16 101036 101020 84.1642 59.6875 0.0211291 0.0118757 2024-03-26T05:08:07.948 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:07.994 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:07.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:08.095 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:08.150 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:08.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:08.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:08.155 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:08:08.199 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:08:08.237 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:08:08.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:08.264 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:08:08.283 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:08:08.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:08.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:08.329 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:08:08.374 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:08:08.393 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:08.416 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:08:08.426 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:08.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:08.455 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:08:08.464 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:08:08.464 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:08:08.464 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:08:08.464 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:08:08.464 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:08:08.464 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:08:08.464 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:08:08.464 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:08:08.528 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:08.572 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:08.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:08.668 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:08 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:08:08.668 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:08 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:08:08.668 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:08 smithi132 ceph-mon[33500]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T05:08:08.668 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:08 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:08:08.669 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:08 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:08:08.669 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:08 smithi132 ceph-mon[33500]: pgmap v344: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 740 B/s rd, 34 MiB/s wr, 546 op/s; 47712/296064 objects misplaced (16.115%); 13 MiB/s, 211 objects/s recovering 2024-03-26T05:08:08.669 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:08 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1064685794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:08.674 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:08.712 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T05:08:08.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:08.716 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:08.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:08.817 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:08.860 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:08.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:08.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:08:08.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:08:08.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[29691]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T05:08:08.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:08:08.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:08:08.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[29691]: pgmap v344: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 740 B/s rd, 34 MiB/s wr, 546 op/s; 47712/296064 objects misplaced (16.115%); 13 MiB/s, 211 objects/s recovering 2024-03-26T05:08:08.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1064685794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:08.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:08:08.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:08:08.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[40404]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T05:08:08.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:08:08.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:08:08.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[40404]: pgmap v344: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 740 B/s rd, 34 MiB/s wr, 546 op/s; 47712/296064 objects misplaced (16.115%); 13 MiB/s, 211 objects/s recovering 2024-03-26T05:08:08.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:08 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1064685794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:08.925 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 76 16 101942 101926 83.8018 56.625 0.0202829 0.0119279 2024-03-26T05:08:08.962 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:09.003 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:09.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:09.104 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:09.147 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:09.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:09.248 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:09.292 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:09.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:09.394 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:09.439 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:09.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:09.540 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:09.581 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:09.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:09.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:09.686 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:09.709 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:08:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:09.732 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:09.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:09.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:09 smithi132 ceph-mon[33500]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T05:08:09.833 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:09.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:09 smithi080 ceph-mon[29691]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T05:08:09.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:09 smithi080 ceph-mon[40404]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T05:08:09.876 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:09.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:09.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:09.925 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 77 16 102944 102928 83.5268 62.625 0.0157596 0.0119663 2024-03-26T05:08:09.978 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:10.015 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:10.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:10.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:10.117 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:10.153 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:10.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:10.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:10.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:10.291 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:10.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:10.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:10.393 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:10.425 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:10.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:10.527 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:10.563 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:10.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:10.610 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:08:10.664 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:10.718 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:10.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:10.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:10 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:08:10.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:10 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:08:10.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:10 smithi132 ceph-mon[33500]: pgmap v346: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 45 MiB/s wr, 727 op/s; 48284/298689 objects misplaced (16.165%); 14 MiB/s, 228 objects/s recovering 2024-03-26T05:08:10.819 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:10.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:10 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:08:10.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:10 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:08:10.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:10 smithi080 ceph-mon[40404]: pgmap v346: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 45 MiB/s wr, 727 op/s; 48284/298689 objects misplaced (16.165%); 14 MiB/s, 228 objects/s recovering 2024-03-26T05:08:10.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:10 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:08:10.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:10 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:08:10.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:10 smithi080 ceph-mon[29691]: pgmap v346: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 45 MiB/s wr, 727 op/s; 48284/298689 objects misplaced (16.165%); 14 MiB/s, 228 objects/s recovering 2024-03-26T05:08:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:10.873 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:10.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:10.925 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 78 16 103914 103898 83.2331 60.625 0.0251876 0.0120091 2024-03-26T05:08:10.974 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:11.002 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:11.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:11.052 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:11.103 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:11.132 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:11.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:11.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:11.234 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:11.240 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:08:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:11.279 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:11.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:11.381 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:11.411 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:11.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:11.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:11.513 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:11.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:11.579 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:11.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:11.681 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:11.716 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:11.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:11.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:11 smithi132 ceph-mon[33500]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T05:08:11.817 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:11.855 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:11.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:11.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:11 smithi080 ceph-mon[29691]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T05:08:11.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:11 smithi080 ceph-mon[40404]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T05:08:11.925 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 79 16 104822 104806 82.8979 56.75 0.0142516 0.0120583 2024-03-26T05:08:11.958 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:11.994 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:11.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:12.095 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:12.125 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:12.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:12.226 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:12.260 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:12.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:12.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:12.363 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:12.395 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:12.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:12.497 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:12.527 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:12.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:12.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:12.629 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:12.653 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:12.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:12.755 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:12.777 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:12 smithi132 ceph-mon[33500]: pgmap v348: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 790 op/s; 48282/310284 objects misplaced (15.561%); 18 MiB/s, 283 objects/s recovering 2024-03-26T05:08:12.777 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:12 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:08:12.801 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:12.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:12.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:12.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:12 smithi080 ceph-mon[40404]: pgmap v348: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 790 op/s; 48282/310284 objects misplaced (15.561%); 18 MiB/s, 283 objects/s recovering 2024-03-26T05:08:12.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:12 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:08:12.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:12 smithi080 ceph-mon[29691]: pgmap v348: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 790 op/s; 48282/310284 objects misplaced (15.561%); 18 MiB/s, 283 objects/s recovering 2024-03-26T05:08:12.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:12 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:08:12.876 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:08:12.876 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T05:08:12.876 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:08:12.903 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:12.925 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:08:12.935095+0000 min lat: 0.00124249 max lat: 2.32797 avg lat: 0.0121225 2024-03-26T05:08:12.925 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:08:12.925 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 80 16 105587 105571 82.4593 47.8125 0.017748 0.0121225 2024-03-26T05:08:12.955 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:12.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:13.057 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:13.114 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:13.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:13.215 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:13.254 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:13.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:13.355 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:13.396 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:13.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:13.497 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:13.539 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:13.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:13.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:13.565 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:08:13.565 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:08:13.641 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:13.717 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:13.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:13.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:13 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:08:13.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:13 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:08:13.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:13 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:08:13.818 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:13.821 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:13 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:08:13.821 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:13 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:08:13.821 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:13 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:08:13.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:13 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:08:13.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:13 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:08:13.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:13 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:08:13.849 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:13.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:13.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:13.925 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 81 16 106463 106447 82.1172 54.75 0.00979853 0.0121733 2024-03-26T05:08:13.950 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:13.983 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:13.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:13.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:14.085 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:14.116 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:14.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:14.117 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:08:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:08:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:08:14.218 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:14.257 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:14.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:14.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:14.321 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:14.359 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:14.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:14.389 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:14.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:14.491 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:14.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:14.548 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:14.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:14.650 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:14.700 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:14.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:14.780 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:14 smithi080 ceph-mon[40404]: pgmap v349: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 806 B/s rd, 47 MiB/s wr, 747 op/s; 48282/310284 objects misplaced (15.561%); 17 MiB/s, 267 objects/s recovering 2024-03-26T05:08:14.781 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:14 smithi080 ceph-mon[29691]: pgmap v349: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 806 B/s rd, 47 MiB/s wr, 747 op/s; 48282/310284 objects misplaced (15.561%); 17 MiB/s, 267 objects/s recovering 2024-03-26T05:08:14.802 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:14.831 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:14.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:14.925 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 82 16 107300 107284 81.7537 52.3125 0.0253824 0.0122268 2024-03-26T05:08:14.932 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:14.961 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:14.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:14.989 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:14 smithi132 ceph-mon[33500]: pgmap v349: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 806 B/s rd, 47 MiB/s wr, 747 op/s; 48282/310284 objects misplaced (15.561%); 17 MiB/s, 267 objects/s recovering 2024-03-26T05:08:15.063 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:15.102 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:15.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:15.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:15.204 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:15.246 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:15.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:15.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:15.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:08:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:08:15.349 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:15.388 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:15.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:15.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:15.490 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:15.532 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:15.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:15.584 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:08:15.633 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:15.668 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:15.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:15.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:15.686 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:15 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1448012941' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:08:15.686 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:15 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:08:15.687 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:15 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:15.735 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T05:08:15.770 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:15.814 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:15.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:15.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:15 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1448012941' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:08:15.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:15 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:08:15.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:15 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:15 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1448012941' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:08:15.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:15 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:08:15.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:15 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:15.915 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:15.926 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 83 16 108217 108201 81.4592 57.3125 0.00772248 0.012271 2024-03-26T05:08:15.955 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:15.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:16.057 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:16.102 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:16.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:16.174 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:08:16.203 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:16.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:16.240 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:16.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:16.341 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:16.379 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:16.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:16.480 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:16.519 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:16.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:16.621 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:16.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:16.664 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:16.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:16.766 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:16.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:16.795 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:16.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:16.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:16 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:08:16.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:16 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:08:16.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:16 smithi080 ceph-mon[40404]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T05:08:16.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:16 smithi080 ceph-mon[40404]: pgmap v351: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 48609/323397 objects misplaced (15.031%); 29 MiB/s, 457 objects/s recovering 2024-03-26T05:08:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:16 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:08:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:16 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:08:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:16 smithi080 ceph-mon[29691]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T05:08:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:16 smithi080 ceph-mon[29691]: pgmap v351: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 48609/323397 objects misplaced (15.031%); 29 MiB/s, 457 objects/s recovering 2024-03-26T05:08:16.896 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:16.926 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 84 16 109113 109097 81.1561 56 0.0130924 0.0123169 2024-03-26T05:08:16.935 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:16.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:17.037 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:16 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:08:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:16 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:08:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:16 smithi132 ceph-mon[33500]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T05:08:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:16 smithi132 ceph-mon[33500]: pgmap v351: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 48609/323397 objects misplaced (15.031%); 29 MiB/s, 457 objects/s recovering 2024-03-26T05:08:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:17.061 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:17.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:17.162 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:17.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:17.201 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:17.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:17.302 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:17.341 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:17.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:17.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:17.443 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:17.479 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:17.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:17.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:17.581 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:17.619 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:17.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:17.720 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:17.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:17.761 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:17.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:17.862 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:17.900 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:17.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:17.926 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 85 16 110026 110010 80.8724 57.0625 0.0110782 0.0123606 2024-03-26T05:08:18.002 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:18.049 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:18.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:18.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:18.151 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:18.180 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:08:18.180 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:08:18.193 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:18.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:18.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:18.295 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:18.378 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:18.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:18.480 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:18.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:18.516 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:18.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:18.617 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:18.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:18.656 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:18.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:18.757 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:18.800 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:18.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:18.902 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:18.926 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 86 16 110989 110973 80.6318 60.1875 0.0206969 0.0123969 2024-03-26T05:08:18.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T05:08:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:18.945 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:18.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:19.046 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:19.093 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:19.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:19.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:18 smithi080 ceph-mon[40404]: pgmap v352: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s; 48609/323397 objects misplaced (15.031%); 26 MiB/s, 418 objects/s recovering 2024-03-26T05:08:19.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:18 smithi080 ceph-mon[29691]: pgmap v352: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s; 48609/323397 objects misplaced (15.031%); 26 MiB/s, 418 objects/s recovering 2024-03-26T05:08:19.195 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:19.235 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:19.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:19.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:18 smithi132 ceph-mon[33500]: pgmap v352: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s; 48609/323397 objects misplaced (15.031%); 26 MiB/s, 418 objects/s recovering 2024-03-26T05:08:19.304 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:08:19.337 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:19.378 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:19.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:19.480 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:19.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:19.521 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:19.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:19.622 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:19.661 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:19.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:19.762 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:19.803 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:19.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:19.905 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:19.926 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 87 16 111924 111908 80.3767 58.4375 0.0139614 0.0124365 2024-03-26T05:08:19.942 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:19.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:19.978 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:08:20.019 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:08:20.031 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:08:20.032 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:08:20.032 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-26T05:08:20.032 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T05:08:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:19 smithi132 ceph-mon[33500]: 3.d scrub starts 2024-03-26T05:08:20.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:19 smithi132 ceph-mon[33500]: 3.d scrub ok 2024-03-26T05:08:20.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:08:20.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:08:20.044 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:20.062 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:08:20.070 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:19 smithi080 ceph-mon[29691]: 3.d scrub starts 2024-03-26T05:08:20.070 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:19 smithi080 ceph-mon[29691]: 3.d scrub ok 2024-03-26T05:08:20.070 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:08:20.070 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:08:20.071 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:19 smithi080 ceph-mon[40404]: 3.d scrub starts 2024-03-26T05:08:20.071 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:19 smithi080 ceph-mon[40404]: 3.d scrub ok 2024-03-26T05:08:20.071 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:08:20.071 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:08:20.075 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:20.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:20.107 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:08:20.146 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:08:20.177 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:20.201 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:08:20.227 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:20.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:20.242 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:08:20.283 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:08:20.298 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:08:20.298 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:08:20.299 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:08:20.299 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:08:20.299 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:08:20.299 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:08:20.299 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:08:20.299 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:08:20.329 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:20.376 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:20.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:20.478 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:20.516 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:20.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:20.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:20.618 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:20.657 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:20.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:20.759 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:20.803 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:20.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:20.905 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:20.926 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 88 16 112779 112763 80.0705 53.4375 0.0034758 0.0124797 2024-03-26T05:08:20.928 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:20.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:20.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:21.030 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:21.065 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:21.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:21.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:21.129 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:20 smithi132 ceph-mon[33500]: pgmap v353: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 35 MiB/s wr, 561 op/s; 49165/325938 objects misplaced (15.084%); 11 MiB/s, 177 objects/s recovering 2024-03-26T05:08:21.130 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:20 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3113648111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:21.130 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:20 smithi132 ceph-mon[33500]: 6.12 scrub starts 2024-03-26T05:08:21.130 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:20 smithi132 ceph-mon[33500]: 6.12 scrub ok 2024-03-26T05:08:21.166 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:21.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:21.209 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:21.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:21.280 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:08:21.310 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:21.316 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:08:21.316 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:08:21.316 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2024-03-26T05:08:21.317 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2024-03-26T05:08:21.317 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:08:21.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:20 smithi080 ceph-mon[29691]: pgmap v353: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 35 MiB/s wr, 561 op/s; 49165/325938 objects misplaced (15.084%); 11 MiB/s, 177 objects/s recovering 2024-03-26T05:08:21.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:20 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3113648111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:21.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:20 smithi080 ceph-mon[29691]: 6.12 scrub starts 2024-03-26T05:08:21.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:20 smithi080 ceph-mon[29691]: 6.12 scrub ok 2024-03-26T05:08:21.323 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:20 smithi080 ceph-mon[40404]: pgmap v353: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 35 MiB/s wr, 561 op/s; 49165/325938 objects misplaced (15.084%); 11 MiB/s, 177 objects/s recovering 2024-03-26T05:08:21.323 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:20 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3113648111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:21.324 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:20 smithi080 ceph-mon[40404]: 6.12 scrub starts 2024-03-26T05:08:21.324 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:20 smithi080 ceph-mon[40404]: 6.12 scrub ok 2024-03-26T05:08:21.342 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:21.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:21.444 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:21.471 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:21.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:21.574 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:21.636 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:21.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:21.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:21.739 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:21.781 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:21.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:21.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:21.883 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:21.923 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:21.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:21.926 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 89 16 113670 113654 79.7965 55.6875 0.0168098 0.0125267 2024-03-26T05:08:22.025 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:22.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T05:08:22.054 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:22.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:22.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:22.098 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:22.156 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:22.188 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:22.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:22.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:22.290 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:22.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:22.338 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:22.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:22.422 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:08:22.440 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:22.488 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:22.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:22.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:22.591 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:22.623 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:22.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:22.723 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:22.765 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:22.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:22.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:22.867 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:22.899 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:22.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:22.926 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Total time run: 90.0108 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Total writes made: 114514 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Write size: 65536 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Object size: 65536 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Bandwidth (MB/sec): 79.5141 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev Bandwidth: 32.7801 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max bandwidth (MB/sec): 193.75 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min bandwidth (MB/sec): 0 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Average IOPS: 1272 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev IOPS: 524.492 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max IOPS: 3100 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min IOPS: 0 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Average Latency(s): 0.0125723 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev Latency(s): 0.0285265 2024-03-26T05:08:22.927 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max latency(s): 2.32797 2024-03-26T05:08:22.928 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min latency(s): 0.00124249 2024-03-26T05:08:22.932 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T05:08:23.002 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:23.037 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:23.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:23.139 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:23.178 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:23.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:23.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:23.317 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:23.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:23.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:23.418 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:23.464 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:23.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:23.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:23.567 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:23.604 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:23.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:23.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:23.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:23 smithi080 ceph-mon[29691]: pgmap v354: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s; 48997/337053 objects misplaced (14.537%); 21 MiB/s, 331 objects/s recovering 2024-03-26T05:08:23.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:23 smithi080 ceph-mon[29691]: 3.f scrub starts 2024-03-26T05:08:23.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:23 smithi080 ceph-mon[29691]: 3.f scrub ok 2024-03-26T05:08:23.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:23 smithi080 ceph-mon[40404]: pgmap v354: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s; 48997/337053 objects misplaced (14.537%); 21 MiB/s, 331 objects/s recovering 2024-03-26T05:08:23.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:23 smithi080 ceph-mon[40404]: 3.f scrub starts 2024-03-26T05:08:23.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:23 smithi080 ceph-mon[40404]: 3.f scrub ok 2024-03-26T05:08:23.706 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:23.744 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:23.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:23.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:23.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:23 smithi132 ceph-mon[33500]: pgmap v354: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s; 48997/337053 objects misplaced (14.537%); 21 MiB/s, 331 objects/s recovering 2024-03-26T05:08:23.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:23 smithi132 ceph-mon[33500]: 3.f scrub starts 2024-03-26T05:08:23.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:23 smithi132 ceph-mon[33500]: 3.f scrub ok 2024-03-26T05:08:23.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:23.874 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:23.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:23.975 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:24.014 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:24.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:24.046 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:08:24.115 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:24.150 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:24.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:24.252 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:24.285 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:24.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:24.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:24.367 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:08:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:08:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:08:24.386 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:24.421 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:24.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:24.432 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:08:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:08:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22566 7.c 7:39b9bfaa:::benchmark_data_smithi080_80848_object22565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.344062+0000", 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 337.09339192300001, 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14012134700000001, 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22566 2024-03-26T05:08:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344062+0000", 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344062+0000", 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344063+0000", 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5899999999999995e-07 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344391+0000", 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00032832899999999998 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344402+0000", 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0961999999999999e-05 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344410+0000", 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8080000000000005e-06 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344468+0000", 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7772999999999997e-05 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344596+0000", 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012822500000000001 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.344723+0000", 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012701499999999999 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.346541+0000", 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001818442 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.346698+0000", 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000156188 2024-03-26T05:08:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.484093+0000", 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13739572 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.484138+0000", 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4437000000000002e-05 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.484183+0000", 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5589e-05 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22569 7.c 7:350aed00:::benchmark_data_smithi080_80848_object22568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.345174+0000", 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 337.092279383, 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13912730000000001, 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22569 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345174+0000", 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345174+0000", 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345175+0000", 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5799999999999998e-07 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345207+0000", 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.1272000000000001e-05 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345217+0000", 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9480000000000003e-06 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345223+0000", 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9129999999999997e-06 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345625+0000", 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040126199999999999 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345710+0000", 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5051999999999998e-05 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.345853+0000", 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000143602 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.346703+0000", 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084936400000000002 2024-03-26T05:08:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.346871+0000", 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016788400000000001 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.484228+0000", 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13735725600000001 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.484264+0000", 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6599e-05 2024-03-26T05:08:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.484302+0000", 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7289999999999997e-05 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:22574 7.7 7:e024eb0a:::benchmark_data_smithi080_80848_object22573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:47.347763+0000", 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 337.08969054900001, 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14220207000000001, 2024-03-26T05:08:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 22574 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347763+0000", 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347763+0000", 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347764+0000", 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2799999999999995e-07 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347850+0000", 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5749999999999997e-05 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347864+0000", 2024-03-26T05:08:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4154999999999999e-05 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347871+0000", 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4340000000000003e-06 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.347935+0000", 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3559999999999995e-05 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.348029+0000", 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3696999999999994e-05 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.348119+0000", 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0000000000000006e-05 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.349313+0000", 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011938999999999999 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.349659+0000", 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034683400000000002 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.489846+0000", 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.140186273 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.489910+0000", 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4615000000000004e-05 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:47.489965+0000", 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5024e-05 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31380 7.d 7:bbfbc35c:::benchmark_data_smithi080_80848_object31379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.249199+0000", 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.18825462900003, 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13902161499999999, 2024-03-26T05:08:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31380 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249199+0000", 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249199+0000", 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249200+0000", 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8899999999999998e-07 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249321+0000", 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120458 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249333+0000", 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2082000000000001e-05 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249338+0000", 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5790000000000003e-06 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249394+0000", 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5948e-05 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249535+0000", 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000141225 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249671+0000", 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000135594 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.385763+0000", 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13609241499999999 2024-03-26T05:08:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388062+0000", 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002299054 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388115+0000", 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2466999999999998e-05 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388175+0000", 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0009000000000001e-05 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388221+0000", 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5895e-05 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31387 7.d 7:b9b9bc8e:::benchmark_data_smithi080_80848_object31386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.249860+0000", 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.18759405600002, 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14199305400000001, 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31387 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249860+0000", 2024-03-26T05:08:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249860+0000", 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249861+0000", 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4700000000000001e-07 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249943+0000", 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2377000000000001e-05 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249953+0000", 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8660000000000001e-06 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249961+0000", 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4740000000000005e-06 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250798+0000", 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00083652499999999996 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.382512+0000", 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.131714359 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.382690+0000", 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017777799999999999 2024-03-26T05:08:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388005+0000", 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053151559999999997 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391615+0000", 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.003609929 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391752+0000", 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013692399999999999 2024-03-26T05:08:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391811+0000", 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9286999999999997e-05 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391853+0000", 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1631999999999997e-05 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31388 7.d 7:b6deb006:::benchmark_data_smithi080_80848_object31387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.249986+0000", 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.18746826699999, 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14203564299999999, 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31388 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249986+0000", 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249986+0000", 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.249986+0000", 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3799999999999997e-07 2024-03-26T05:08:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250070+0000", 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3956999999999996e-05 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250078+0000", 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0299999999999994e-06 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250083+0000", 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.442e-06 2024-03-26T05:08:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384264+0000", 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13418155200000001 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384385+0000", 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012047100000000001 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384584+0000", 2024-03-26T05:08:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00019931699999999999 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388074+0000", 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00349016 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391670+0000", 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0035954469999999999 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391939+0000", 2024-03-26T05:08:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000269466 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391985+0000", 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6331000000000001e-05 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.392021+0000", 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5831999999999998e-05 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:31394 7.d 7:bacf0b5e:::benchmark_data_smithi080_80848_object31393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.250540+0000", 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.18691381999997, 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14170165000000001, 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 31394 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250540+0000", 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250540+0000", 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250541+0000", 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0500000000000002e-07 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250698+0000", 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015699799999999999 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250709+0000", 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0939e-05 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.250721+0000", 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2531000000000001e-05 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.385794+0000", 2024-03-26T05:08:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13507309000000001 2024-03-26T05:08:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.385910+0000", 2024-03-26T05:08:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115531 2024-03-26T05:08:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.386052+0000", 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014174899999999999 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.391713+0000", 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.005661831 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.392060+0000", 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000346998 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.392154+0000", 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3131e-05 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.392202+0000", 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8689000000000001e-05 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.392242+0000", 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9357999999999998e-05 2024-03-26T05:08:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.24701.0:31392 7.b e111/103 7:d4d557cd:::benchmark_data_smithi080_80848_object31391:head v 111'1944, mlcod=111'1941)", 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.252059+0000", 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.18539474300002, 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13566467500000001, 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252059+0000", 2024-03-26T05:08:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252059+0000", 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252060+0000", 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9199999999999999e-07 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252111+0000", 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1381000000000002e-05 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384561+0000", 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13244936700000001 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384572+0000", 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0983e-05 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384626+0000", 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4063000000000001e-05 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384648+0000", 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.2167999999999998e-05 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387664+0000", 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00301623 2024-03-26T05:08:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387724+0000", 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9490999999999997e-05 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.24701.0:31393 7.1 e111/103 7:8ed06db3:::benchmark_data_smithi080_80848_object31392:head v 111'1956, mlcod=111'1954)", 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.252281+0000", 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.185173244, 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13488787599999999, 2024-03-26T05:08:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252281+0000", 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252281+0000", 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252281+0000", 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8400000000000004e-07 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252301+0000", 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.9760999999999999e-05 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383132+0000", 2024-03-26T05:08:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13083125600000001 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383147+0000", 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4747e-05 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383219+0000", 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1748999999999994e-05 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.383258+0000", 2024-03-26T05:08:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8977999999999998e-05 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387124+0000", 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0038662129999999999 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.387168+0000", 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4487999999999998e-05 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(client.24701.0:31380 7.d e111/103 ondisk, result = 0)", 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:52.252755+0000", 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.18469868099999, 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13531982300000001, 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252755+0000", 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252755+0000", 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252756+0000", 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3000000000000001e-07 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.252758+0000", 2024-03-26T05:08:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.6759999999999999e-06 2024-03-26T05:08:24.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384584+0000", 2024-03-26T05:08:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.131825893 2024-03-26T05:08:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.384592+0000", 2024-03-26T05:08:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8320000000000006e-06 2024-03-26T05:08:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388049+0000", 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0034569150000000001 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388056+0000", 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2069999999999998e-06 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:52.388075+0000", 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.8770000000000002e-05 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:122222 7.7 7:e926a959:::benchmark_data_smithi080_80848_object2723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:11.511588+0000", 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 252.925865504, 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13703875800000001, 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 122222 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511588+0000", 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511588+0000", 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511590+0000", 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.392e-06 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511595+0000", 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.772e-06 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511610+0000", 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4817000000000001e-05 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511619+0000", 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3850000000000002e-06 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511679+0000", 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9858000000000002e-05 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511808+0000", 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012915100000000001 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.511953+0000", 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000144878 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.514360+0000", 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0024071679999999999 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.566701+0000", 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.052341721000000001 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.648533+0000", 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.081831423 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.648577+0000", 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3704e-05 2024-03-26T05:08:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:11.648627+0000", 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0488999999999997e-05 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:157212 7.d 7:b26765b3:::benchmark_data_smithi080_80848_object37713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:46.726199+0000", 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 217.711254822, 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.12664176299999999, 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 157212 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.726199+0000", 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.726199+0000", 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.726201+0000", 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5629999999999999e-06 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.726207+0000", 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8390000000000001e-06 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.726222+0000", 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4562000000000001e-05 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.726231+0000", 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6969999999999999e-06 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.727331+0000", 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011001489999999999 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.727455+0000", 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000123843 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.727569+0000", 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011413000000000001 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.730942+0000", 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0033731759999999999 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.733268+0000", 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0023263250000000002 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.852771+0000", 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.119502763 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.852808+0000", 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7085000000000002e-05 2024-03-26T05:08:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.852841+0000", 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2631e-05 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:157220 7.7 7:e3aaf5eb:::benchmark_data_smithi080_80848_object37721:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:46.732130+0000", 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 217.70532336700001, 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.128509334, 2024-03-26T05:08:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 157220 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732130+0000", 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732130+0000", 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732132+0000", 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.784e-06 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732137+0000", 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9830000000000004e-06 2024-03-26T05:08:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732148+0000", 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0933e-05 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732155+0000", 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9469999999999997e-06 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732237+0000", 2024-03-26T05:08:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.1747000000000002e-05 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732372+0000", 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000134697 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732509+0000", 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013783099999999999 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.739488+0000", 2024-03-26T05:08:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0069783199999999997 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.859410+0000", 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11992253899999999 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.860516+0000", 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011055870000000001 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.860589+0000", 2024-03-26T05:08:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3193000000000002e-05 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.860640+0000", 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0773000000000003e-05 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:157222 7.c 7:34243d50:::benchmark_data_smithi080_80848_object37723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:46.732599+0000", 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 217.704855251, 2024-03-26T05:08:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.12243174699999999, 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 157222 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732599+0000", 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732599+0000", 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732599+0000", 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5199999999999996e-07 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732603+0000", 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8659999999999999e-06 2024-03-26T05:08:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732611+0000", 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5449999999999998e-06 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732617+0000", 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5200000000000003e-06 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732663+0000", 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5843000000000001e-05 2024-03-26T05:08:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732759+0000", 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6314e-05 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.732858+0000", 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8486999999999994e-05 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.736572+0000", 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0037138869999999999 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.737677+0000", 2024-03-26T05:08:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011053110000000001 2024-03-26T05:08:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.854932+0000", 2024-03-26T05:08:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.117254436 2024-03-26T05:08:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.854985+0000", 2024-03-26T05:08:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3235000000000001e-05 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.855030+0000", 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5550999999999998e-05 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:157223 7.7 7:ea38fec5:::benchmark_data_smithi080_80848_object37724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:04:46.734236+0000", 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 217.703217373, 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126551473, 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 157223 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.734236+0000", 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.734236+0000", 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.734238+0000", 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3549999999999999e-06 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.734243+0000", 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7029999999999997e-06 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.734253+0000", 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0710000000000001e-05 2024-03-26T05:08:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.734260+0000", 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2339999999999997e-06 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.736485+0000", 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0022243499999999999 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.736612+0000", 2024-03-26T05:08:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012725099999999999 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.736733+0000", 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000121206 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.739564+0000", 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0028307889999999998 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.859455+0000", 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.119890754 2024-03-26T05:08:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.860687+0000", 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0012317739999999999 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.860743+0000", 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6700999999999998e-05 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:04:46.860788+0000", 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4645999999999999e-05 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:186914 7.d 7:b07c0e48:::benchmark_data_smithi080_80848_object67415:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:16.759339+0000", 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 187.67811490899999, 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13061483300000001, 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 186914 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759339+0000", 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759339+0000", 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759340+0000", 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4869999999999999e-06 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759346+0000", 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0820000000000004e-06 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759363+0000", 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6432000000000001e-05 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759371+0000", 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.439e-06 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759437+0000", 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5537000000000006e-05 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759562+0000", 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012494399999999999 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.759685+0000", 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012313600000000001 2024-03-26T05:08:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.766825+0000", 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0071396130000000004 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.769012+0000", 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002187446 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.889841+0000", 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.120828515 2024-03-26T05:08:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.889895+0000", 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4341e-05 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.889954+0000", 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8860999999999998e-05 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:186918 7.d 7:b8fb59a9:::benchmark_data_smithi080_80848_object67419:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:16.763188+0000", 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 187.674265846, 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.136308077, 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 186918 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763188+0000", 2024-03-26T05:08:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763188+0000", 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763189+0000", 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2720000000000001e-06 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763194+0000", 2024-03-26T05:08:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8280000000000002e-06 2024-03-26T05:08:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763206+0000", 2024-03-26T05:08:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1960000000000001e-05 2024-03-26T05:08:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763214+0000", 2024-03-26T05:08:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9589999999999995e-06 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763287+0000", 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3464e-05 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763436+0000", 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000148677 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.763564+0000", 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012827199999999999 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.766835+0000", 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.003270102 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.769152+0000", 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0023173080000000001 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.899215+0000", 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.13006367099999999 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.899456+0000", 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00024002800000000001 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:16.899496+0000", 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0535999999999999e-05 2024-03-26T05:08:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:32986 8.9 8:98b2f8a6:::benchmark_data_smithi080_109213_object32985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:07:08.578643+0000", 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 75.858810554000002, 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.120419665, 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:08:24.533 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 32986 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.578643+0000", 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.578643+0000", 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.534 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.578644+0000", 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.012e-06 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.578706+0000", 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1408999999999994e-05 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.578724+0000", 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.8515000000000001e-05 2024-03-26T05:08:24.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.578732+0000", 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9529999999999999e-06 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.578783+0000", 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0559999999999997e-05 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.578897+0000", 2024-03-26T05:08:24.536 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000114287 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.579034+0000", 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000137255 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580241+0000", 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0012069909999999999 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.537 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.695744+0000", 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.115502517 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.698954+0000", 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0032098790000000001 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.699017+0000", 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3114999999999995e-05 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.538 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.699063+0000", 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6171999999999998e-05 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:32989 8.9 8:921d1a31:::benchmark_data_smithi080_109213_object32988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:07:08.580773+0000", 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 75.856680326000003, 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11977368200000001, 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:08:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 32989 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580773+0000", 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580773+0000", 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580774+0000", 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.61e-07 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580841+0000", 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6685000000000003e-05 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580851+0000", 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7180000000000008e-06 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580856+0000", 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7239999999999997e-06 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580905+0000", 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8281000000000001e-05 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.581045+0000", 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000140054 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.581136+0000", 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.1494000000000002e-05 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.582638+0000", 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001501474 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.695788+0000", 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.113150179 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.700446+0000", 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0046577609999999998 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.700504+0000", 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7979000000000002e-05 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.700547+0000", 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3671999999999998e-05 2024-03-26T05:08:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:32990 8.9 8:90003f14:::benchmark_data_smithi080_109213_object32989:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:07:08.580873+0000", 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 75.856581200999997, 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11983304, 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 32990 2024-03-26T05:08:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580873+0000", 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580873+0000", 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580873+0000", 2024-03-26T05:08:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2599999999999998e-07 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580944+0000", 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0811999999999999e-05 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580953+0000", 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6339999999999996e-06 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:08:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.580958+0000", 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.887e-06 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.581849+0000", 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00089049099999999998 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.581968+0000", 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119475 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:08:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.582053+0000", 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4637999999999998e-05 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.582742+0000", 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068937499999999999 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.695838+0000", 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11309567299999999 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.700595+0000", 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0047570310000000001 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.700671+0000", 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5451000000000007e-05 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:07:08.700706+0000", 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5147000000000001e-05 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:08:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:08:24.551 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:08:24.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:24.584 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:24.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:24.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:24.686 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:24.722 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:24.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:24.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:24.744 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:24 smithi080 ceph-mon[29691]: 3.15 scrub starts 2024-03-26T05:08:24.744 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:24 smithi080 ceph-mon[29691]: 3.15 scrub ok 2024-03-26T05:08:24.745 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:24 smithi080 ceph-mon[40404]: 3.15 scrub starts 2024-03-26T05:08:24.745 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:24 smithi080 ceph-mon[40404]: 3.15 scrub ok 2024-03-26T05:08:24.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:24 smithi132 ceph-mon[33500]: 3.15 scrub starts 2024-03-26T05:08:24.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:24 smithi132 ceph-mon[33500]: 3.15 scrub ok 2024-03-26T05:08:24.823 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:24.853 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:24.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:24.955 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:24.981 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:24.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:25.083 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:25.127 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:25.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:25.228 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:25.264 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:25.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:25.311 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:25.366 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:25.427 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:25.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:25.455 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:08:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:08:25.455 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:25 smithi132 ceph-mon[33500]: pgmap v355: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s; 48997/337053 objects misplaced (14.537%); 21 MiB/s, 331 objects/s recovering 2024-03-26T05:08:25.455 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:25 smithi132 ceph-mon[33500]: 5.1a scrub starts 2024-03-26T05:08:25.455 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:25 smithi132 ceph-mon[33500]: 5.1a scrub ok 2024-03-26T05:08:25.529 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:25.571 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:25.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:25.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:25.673 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:25.708 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:25.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:25.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:25.810 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:25.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:25 smithi080 ceph-mon[40404]: pgmap v355: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s; 48997/337053 objects misplaced (14.537%); 21 MiB/s, 331 objects/s recovering 2024-03-26T05:08:25.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:25 smithi080 ceph-mon[40404]: 5.1a scrub starts 2024-03-26T05:08:25.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:25 smithi080 ceph-mon[40404]: 5.1a scrub ok 2024-03-26T05:08:25.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:25 smithi080 ceph-mon[29691]: pgmap v355: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s; 48997/337053 objects misplaced (14.537%); 21 MiB/s, 331 objects/s recovering 2024-03-26T05:08:25.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:25 smithi080 ceph-mon[29691]: 5.1a scrub starts 2024-03-26T05:08:25.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:25 smithi080 ceph-mon[29691]: 5.1a scrub ok 2024-03-26T05:08:25.858 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:25.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:25.960 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:26.004 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:26.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:26.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:26.106 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:26.147 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:26.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:26.197 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T05:08:26.249 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:26.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:26.284 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:26.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:26.385 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:26.427 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:26.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:26.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:26.467 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:26 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:26.529 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:26.570 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:26.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:26.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:26.581 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:08:26.672 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:26.715 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:26.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:26.734 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:26 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:26.735 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:26 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:26.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:26.856 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:26.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:26.957 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:26.994 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:26.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:27.096 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:27.128 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:27.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:27.160 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:27.220 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:08:27.220 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:08:27.220 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 220, 2024-03-26T05:08:27.220 INFO:teuthology.orchestra.run.smithi080.stdout: "fsid": "dd09e9cc-eb2c-11ee-95d0-87774f69a715", 2024-03-26T05:08:27.220 INFO:teuthology.orchestra.run.smithi080.stdout: "created": "2024-03-26T04:54:51.106954+0000", 2024-03-26T05:08:27.220 INFO:teuthology.orchestra.run.smithi080.stdout: "modified": "2024-03-26T05:08:15.568802+0000", 2024-03-26T05:08:27.220 INFO:teuthology.orchestra.run.smithi080.stdout: "last_up_change": "2024-03-26T05:00:08.801055+0000", 2024-03-26T05:08:27.220 INFO:teuthology.orchestra.run.smithi080.stdout: "last_in_change": "2024-03-26T05:07:58.901005+0000", 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_num": 5804032, 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_set": [ 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "nodeep-scrub", 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "pglog_hardlimit", 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "purged_snapdirs", 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "recovery_deletes", 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "sortbitwise" 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_version": 18, 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T05:08:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_snapshot": "", 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_max": 8, 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "max_osd": 8, 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "require_min_compat_client": "luminous", 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "min_compat_client": "luminous", 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "require_osd_release": "squid", 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "allow_crimson": false, 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "pools": [ 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 1, 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": ".mgr", 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T04:58:06.876819+0000", 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:08:27.222 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 1, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 1, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 1, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 1, 2024-03-26T05:08:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 1, 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "22", 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:08:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:08:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:08:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:08:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:08:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:08:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:08:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:08:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:08:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:08:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:08:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:08:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:08:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:08:27.229 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:08:27.229 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:08:27.229 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:08:27.229 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:08:27.229 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:08:27.229 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:08:27.230 INFO:teuthology.orchestra.run.smithi080.stdout: "options": { 2024-03-26T05:08:27.230 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_max": 32, 2024-03-26T05:08:27.230 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_min": 1 2024-03-26T05:08:27.230 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.230 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:08:27.230 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": {} 2024-03-26T05:08:27.230 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.230 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:08:27.230 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:08:27.230 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 7.8899998664855957, 2024-03-26T05:08:27.230 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 7.8899998664855957, 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.37999999523162842, 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 3, 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 3, 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 2, 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "datapool", 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:20.705417+0000", 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 8193, 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:08:27.231 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 3, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 3, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 3, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 3, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 3, 2024-03-26T05:08:27.232 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "71", 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:08:27.233 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 2, 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 71, 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:08:27.234 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:08:27.235 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd": {} 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.6500000953674316, 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.6500000953674316, 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-26T05:08:27.236 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 3, 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": ".rgw.root", 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:21.150368+0000", 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:08:27.237 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:08:27.238 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "69", 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:08:27.239 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:08:27.240 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:08:27.241 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.5, 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.5, 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:08:27.242 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.5, 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.5, 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 4, 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.log", 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:23.638065+0000", 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:08:27.243 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:08:27.244 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "71", 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:08:27.245 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:08:27.246 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:08:27.247 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.25, 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.25, 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:08:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2.25, 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2.25, 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 5, 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.control", 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:25.181000+0000", 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:08:27.249 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:08:27.250 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "73", 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:08:27.251 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:08:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:08:27.253 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:08:27.253 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:08:27.253 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:08:27.253 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:08:27.253 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:08:27.253 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:08:27.253 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.253 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:08:27.253 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:08:27.253 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:08:27.253 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:08:27.253 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:08:27.254 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.25, 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.25, 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.25, 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.25, 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 6, 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.meta", 2024-03-26T05:08:27.255 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:27.249998+0000", 2024-03-26T05:08:27.256 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:08:27.256 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:27.257 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:08:27.257 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:08:27.257 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:08:27.257 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:08:27.257 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:08:27.257 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:08:27.257 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:08:27.258 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:08:27.258 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:08:27.258 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:08:27.258 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:08:27.258 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:08:27.258 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:08:27.258 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:08:27.258 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:08:27.258 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:08:27.258 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:08:27.258 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:08:27.258 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "75", 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:08:27.259 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:08:27.260 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:08:27.261 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: "options": { 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_bias": 4 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:08:27.262 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.75, 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.75, 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.75, 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.75, 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 8, 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "unique_pool_1", 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:06:40.545739+0000", 2024-03-26T05:08:27.263 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 16, 2024-03-26T05:08:27.264 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 16, 2024-03-26T05:08:27.265 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 16, 2024-03-26T05:08:27.265 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 16, 2024-03-26T05:08:27.265 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 16, 2024-03-26T05:08:27.265 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:08:27.265 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:08:27.265 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:08:27.265 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:08:27.265 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:08:27.265 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:08:27.265 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:08:27.265 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.265 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "189", 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:08:27.266 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.267 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: "rados": {} 2024-03-26T05:08:27.268 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2, 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2, 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2, 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2, 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.269 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 0, 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366", 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 8, 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 214, 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.270 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6802", 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6803", 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:08:27.271 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6804", 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6805", 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:08:27.272 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6808", 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6809", 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.273 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6806", 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6807", 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.274 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6803/1840686622", 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6805/1840686622", 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6809/1840686622", 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6807/1840686622", 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 1, 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23", 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:08:27.275 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 13, 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 214, 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6810", 2024-03-26T05:08:27.276 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:08:27.277 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.277 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.277 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.277 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6811", 2024-03-26T05:08:27.277 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:08:27.277 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.277 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.277 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.277 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:08:27.277 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.277 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.278 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.278 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6812", 2024-03-26T05:08:27.278 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:08:27.278 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.278 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.278 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.278 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6813", 2024-03-26T05:08:27.278 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:08:27.278 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.278 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.278 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.278 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6816", 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6817", 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.279 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.280 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:08:27.280 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.280 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.280 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.280 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6814", 2024-03-26T05:08:27.280 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:08:27.280 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.280 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.280 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.280 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6815", 2024-03-26T05:08:27.280 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:08:27.280 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6811/1726833996", 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6813/1726833996", 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6817/1726833996", 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6815/1726833996", 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.281 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 2, 2024-03-26T05:08:27.282 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "536bec9b-9636-4e67-b452-ed1a55c22ab6", 2024-03-26T05:08:27.282 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:08:27.282 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:08:27.282 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:08:27.282 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:08:27.282 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:08:27.282 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:08:27.282 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 18, 2024-03-26T05:08:27.282 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 214, 2024-03-26T05:08:27.282 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:08:27.282 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:08:27.282 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6818", 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6819", 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.283 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6820", 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6821", 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.284 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6824", 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6825", 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.285 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6822", 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6823", 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.286 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6819/128713757", 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6821/128713757", 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6825/128713757", 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6823/128713757", 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 3, 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "640cfdb6-b9c5-446d-8312-504ae12c7818", 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:08:27.287 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 27, 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 214, 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6826", 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:08:27.288 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6827", 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6828", 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6829", 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6832", 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.290 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6833", 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6830", 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.291 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6831", 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6827/418771441", 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6829/418771441", 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6833/418771441", 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6831/418771441", 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:08:27.292 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.293 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.293 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.293 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 4, 2024-03-26T05:08:27.293 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639", 2024-03-26T05:08:27.293 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:08:27.293 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:08:27.293 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:08:27.293 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:08:27.293 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:08:27.293 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:08:27.293 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 33, 2024-03-26T05:08:27.293 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 214, 2024-03-26T05:08:27.294 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:08:27.294 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:08:27.294 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:08:27.295 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.295 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.295 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.295 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6800", 2024-03-26T05:08:27.295 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:08:27.295 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.295 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.295 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.295 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6801", 2024-03-26T05:08:27.295 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:08:27.295 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.295 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6802", 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6803", 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.296 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6806", 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6807", 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.297 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6804", 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6805", 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.298 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6801/1441071423", 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6803/1441071423", 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6807/1441071423", 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6805/1441071423", 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 5, 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc", 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:08:27.299 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 41, 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 213, 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6808", 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6809", 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6810", 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.301 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6811", 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6814", 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:08:27.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6815", 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6812", 2024-03-26T05:08:27.303 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6813", 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6809/794504625", 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6811/794504625", 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6815/794504625", 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6813/794504625", 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:08:27.304 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 6, 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5", 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:08:27.305 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 51, 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 214, 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6816", 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6817", 2024-03-26T05:08:27.306 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6818", 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6819", 2024-03-26T05:08:27.307 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6822", 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6823", 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.308 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6820", 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6821", 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.309 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6817/2209078972", 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6819/2209078972", 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6823/2209078972", 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6821/2209078972", 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 7, 2024-03-26T05:08:27.310 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d", 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 61, 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 213, 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.311 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6824", 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6825", 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.312 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6826", 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6827", 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.313 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6830", 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6831", 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:08:27.314 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6828", 2024-03-26T05:08:27.315 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:08:27.315 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.315 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.315 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:08:27.315 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6829", 2024-03-26T05:08:27.315 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:08:27.315 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.315 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.315 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.315 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6825/3655388282", 2024-03-26T05:08:27.315 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6827/3655388282", 2024-03-26T05:08:27.315 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6831/3655388282", 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6829/3655388282", 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_xinfo": [ 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 0, 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:08:27.316 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:08:09.691814+0000", 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 1, 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:07:25.126039+0000", 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 2, 2024-03-26T05:08:27.317 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:08:07.802805+0000", 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 3, 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:08:27.318 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:07:38.008650+0000", 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 4, 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:07:52.273398+0000", 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.319 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 5, 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:08:03.237904+0000", 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 6, 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:08:27.320 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:07:35.421427+0000", 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 7, 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:07:48.977413+0000", 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap": [ 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.1", 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: 4, 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: 5 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap_items": [ 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.a", 2024-03-26T05:08:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 4, 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 0 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.b", 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 4, 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 1 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.1c", 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 4, 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.2", 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:08:27.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.7", 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.325 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.1f", 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap_primaries": [], 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_temp": [ 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.326 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "8.0", 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: 3, 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: 1, 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: 4 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "8.2", 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: 4, 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: 3 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "8.3", 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: 1, 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: 6, 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: 2 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "8.4", 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: 2, 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: 4, 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: 6 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.328 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "8.8", 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: 3, 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: 7 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "8.9", 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: 6, 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:08:27.329 INFO:teuthology.orchestra.run.smithi080.stdout: 3 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "8.a", 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: 1, 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: 2, 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: 3 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "8.d", 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:08:27.330 INFO:teuthology.orchestra.run.smithi080.stdout: 2, 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: 7, 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: 6 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_temp": [], 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: "blocklist": { 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3992539385": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/2770889532": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/2466284312": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/490056571": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1986135300": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3796611170": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:08:27.331 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1993138634": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1832498181": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/2466284312": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1650870847": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/1418683843": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3147301987": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/2770889532": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/571905941": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1451012560": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1235125152": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/1418683843": "2024-03-27T05:01:03.689739+0000" 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "range_blocklist": {}, 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profiles": { 2024-03-26T05:08:27.332 INFO:teuthology.orchestra.run.smithi080.stdout: "default": { 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: "crush-failure-domain": "osd", 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: "k": "2", 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: "m": "1", 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: "plugin": "jerasure", 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: "technique": "reed_sol_van" 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps_queue": [], 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: "new_removed_snaps": [], 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: "new_purged_snaps": [], 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_node_flags": {}, 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: "device_class_flags": {}, 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode": { 2024-03-26T05:08:27.333 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode_enabled": false, 2024-03-26T05:08:27.334 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_bucket_count": 0, 2024-03-26T05:08:27.334 INFO:teuthology.orchestra.run.smithi080.stdout: "degraded_stretch_mode": 0, 2024-03-26T05:08:27.334 INFO:teuthology.orchestra.run.smithi080.stdout: "recovering_stretch_mode": 0, 2024-03-26T05:08:27.334 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode_bucket": 0 2024-03-26T05:08:27.334 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:27.334 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:08:27.342 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:27.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:27.444 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:27.479 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:27.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:27.580 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:27.620 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:27.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:27.722 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:27.761 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:27.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:27.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:27.785 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:27 smithi132 ceph-mon[33500]: pgmap v356: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 860 op/s; 47609/338565 objects misplaced (14.062%); 20 MiB/s, 326 objects/s recovering 2024-03-26T05:08:27.785 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:27 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3999301081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:08:27.863 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:27.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:27 smithi080 ceph-mon[29691]: pgmap v356: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 860 op/s; 47609/338565 objects misplaced (14.062%); 20 MiB/s, 326 objects/s recovering 2024-03-26T05:08:27.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:27 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3999301081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:08:27.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:27 smithi080 ceph-mon[40404]: pgmap v356: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 860 op/s; 47609/338565 objects misplaced (14.062%); 20 MiB/s, 326 objects/s recovering 2024-03-26T05:08:27.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:27 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3999301081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:08:27.901 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:27.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:28.004 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:28.036 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:28.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:28.138 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:28.183 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:28.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:28.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:28.284 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:28.317 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:28.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:28.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:28.419 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:28.454 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:28.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:28.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:28.556 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:28.587 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:28.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:28.676 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:28 smithi132 ceph-mon[33500]: 3.13 scrub starts 2024-03-26T05:08:28.676 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:28 smithi132 ceph-mon[33500]: 3.13 scrub ok 2024-03-26T05:08:28.676 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:08:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:08:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2024-03-26T05:08:28.676 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:08:28.690 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:28.727 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:28.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:28.829 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:28.855 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:28 smithi080 ceph-mon[40404]: 3.13 scrub starts 2024-03-26T05:08:28.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:28 smithi080 ceph-mon[40404]: 3.13 scrub ok 2024-03-26T05:08:28.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:28 smithi080 ceph-mon[29691]: 3.13 scrub starts 2024-03-26T05:08:28.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:28 smithi080 ceph-mon[29691]: 3.13 scrub ok 2024-03-26T05:08:28.871 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:28.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:28.909 INFO:tasks.thrashosds.thrasher:j is {'epoch': 220, 'fsid': 'dd09e9cc-eb2c-11ee-95d0-87774f69a715', 'created': '2024-03-26T04:54:51.106954+0000', 'modified': '2024-03-26T05:08:15.568802+0000', 'last_up_change': '2024-03-26T05:00:08.801055+0000', 'last_in_change': '2024-03-26T05:07:58.901005+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T04:58:06.876819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T05:00:20.705417+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T05:00:21.150368+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T05:00:23.638065+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T05:00:25.181000+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T05:00:27.249998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-26T05:06:40.545739+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '35a52544-0b0b-47d6-8d6a-571c48ceb366', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6802', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6803', 'nonce': 1840686622}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6804', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6805', 'nonce': 1840686622}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6808', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6809', 'nonce': 1840686622}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6806', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6807', 'nonce': 1840686622}]}, 'public_addr': '172.21.15.80:6803/1840686622', 'cluster_addr': '172.21.15.80:6805/1840686622', 'heartbeat_back_addr': '172.21.15.80:6809/1840686622', 'heartbeat_front_addr': '172.21.15.80:6807/1840686622', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efa5ea33-4f11-49b7-a3f3-f60d5329db23', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6810', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6811', 'nonce': 1726833996}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6812', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6813', 'nonce': 1726833996}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6816', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6817', 'nonce': 1726833996}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6814', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6815', 'nonce': 1726833996}]}, 'public_addr': '172.21.15.80:6811/1726833996', 'cluster_addr': '172.21.15.80:6813/1726833996', 'heartbeat_back_addr': '172.21.15.80:6817/1726833996', 'heartbeat_front_addr': '172.21.15.80:6815/1726833996', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '536bec9b-9636-4e67-b452-ed1a55c22ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6818', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6819', 'nonce': 128713757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6820', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6821', 'nonce': 128713757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6824', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6825', 'nonce': 128713757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6822', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6823', 'nonce': 128713757}]}, 'public_addr': '172.21.15.80:6819/128713757', 'cluster_addr': '172.21.15.80:6821/128713757', 'heartbeat_back_addr': '172.21.15.80:6825/128713757', 'heartbeat_front_addr': '172.21.15.80:6823/128713757', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '640cfdb6-b9c5-446d-8312-504ae12c7818', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6826', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6827', 'nonce': 418771441}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6828', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6829', 'nonce': 418771441}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6832', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6833', 'nonce': 418771441}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6830', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6831', 'nonce': 418771441}]}, 'public_addr': '172.21.15.80:6827/418771441', 'cluster_addr': '172.21.15.80:6829/418771441', 'heartbeat_back_addr': '172.21.15.80:6833/418771441', 'heartbeat_front_addr': '172.21.15.80:6831/418771441', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0cc7822a-ed73-4c79-a30d-1e151ddd6639', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6800', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6801', 'nonce': 1441071423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6802', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6803', 'nonce': 1441071423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6806', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6807', 'nonce': 1441071423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6804', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6805', 'nonce': 1441071423}]}, 'public_addr': '172.21.15.132:6801/1441071423', 'cluster_addr': '172.21.15.132:6803/1441071423', 'heartbeat_back_addr': '172.21.15.132:6807/1441071423', 'heartbeat_front_addr': '172.21.15.132:6805/1441071423', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 213, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6808', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6809', 'nonce': 794504625}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6810', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6811', 'nonce': 794504625}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6814', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6815', 'nonce': 794504625}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6812', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6813', 'nonce': 794504625}]}, 'public_addr': '172.21.15.132:6809/794504625', 'cluster_addr': '172.21.15.132:6811/794504625', 'heartbeat_back_addr': '172.21.15.132:6815/794504625', 'heartbeat_front_addr': '172.21.15.132:6813/794504625', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2970bbab-11b2-4d1b-b57e-037bdac089b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6816', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6817', 'nonce': 2209078972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6818', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6819', 'nonce': 2209078972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6822', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6823', 'nonce': 2209078972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6820', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6821', 'nonce': 2209078972}]}, 'public_addr': '172.21.15.132:6817/2209078972', 'cluster_addr': '172.21.15.132:6819/2209078972', 'heartbeat_back_addr': '172.21.15.132:6823/2209078972', 'heartbeat_front_addr': '172.21.15.132:6821/2209078972', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '12f4b466-28f9-469b-8fd0-65b91c6dc22d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 213, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6824', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6825', 'nonce': 3655388282}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6826', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6827', 'nonce': 3655388282}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6830', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6831', 'nonce': 3655388282}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6828', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6829', 'nonce': 3655388282}]}, 'public_addr': '172.21.15.132:6825/3655388282', 'cluster_addr': '172.21.15.132:6827/3655388282', 'heartbeat_back_addr': '172.21.15.132:6831/3655388282', 'heartbeat_front_addr': '172.21.15.132:6829/3655388282', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:08:09.691814+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:07:25.126039+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:08:07.802805+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:07:38.008650+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:07:52.273398+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:08:03.237904+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:07:35.421427+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:07:48.977413+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.1', 'osds': [4, 0, 5]}], 'pg_upmap_items': [{'pgid': '3.a', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '3.b', 'mappings': [{'from': 4, 'to': 1}]}, {'pgid': '3.1c', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.0', 'osds': [3, 1, 4]}, {'pgid': '8.2', 'osds': [4, 0, 3]}, {'pgid': '8.3', 'osds': [1, 6, 2]}, {'pgid': '8.4', 'osds': [2, 4, 6]}, {'pgid': '8.8', 'osds': [0, 3, 7]}, {'pgid': '8.9', 'osds': [6, 0, 3]}, {'pgid': '8.a', 'osds': [1, 2, 3]}, {'pgid': '8.d', 'osds': [2, 7, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.80:0/3992539385': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/2770889532': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:6801/2466284312': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/490056571': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1986135300': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/3796611170': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1993138634': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1832498181': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/2466284312': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1650870847': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6801/1418683843': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:0/3147301987': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6801/2770889532': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/571905941': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1451012560': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1235125152': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/1418683843': '2024-03-27T05:01:03.689739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T05:08:28.909 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:08:28.973 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:29.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:29.005 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:29.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:29.106 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:29.171 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:29.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:29.273 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:29.304 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:29.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:29.406 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:29.439 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:29.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:29.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:29.541 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:29.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T05:08:29.574 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:29.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:29.676 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:29.710 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:29.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:29.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:29 smithi132 ceph-mon[33500]: pgmap v357: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 938 B/s rd, 36 MiB/s wr, 737 op/s; 47609/338565 objects misplaced (14.062%); 17 MiB/s, 279 objects/s recovering 2024-03-26T05:08:29.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:29 smithi132 ceph-mon[33500]: 5.10 scrub starts 2024-03-26T05:08:29.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:29 smithi132 ceph-mon[33500]: 5.10 scrub ok 2024-03-26T05:08:29.812 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:29.845 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:29 smithi080 ceph-mon[29691]: pgmap v357: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 938 B/s rd, 36 MiB/s wr, 737 op/s; 47609/338565 objects misplaced (14.062%); 17 MiB/s, 279 objects/s recovering 2024-03-26T05:08:29.845 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:29 smithi080 ceph-mon[29691]: 5.10 scrub starts 2024-03-26T05:08:29.845 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:29 smithi080 ceph-mon[29691]: 5.10 scrub ok 2024-03-26T05:08:29.845 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:29 smithi080 ceph-mon[40404]: pgmap v357: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 938 B/s rd, 36 MiB/s wr, 737 op/s; 47609/338565 objects misplaced (14.062%); 17 MiB/s, 279 objects/s recovering 2024-03-26T05:08:29.845 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:29 smithi080 ceph-mon[40404]: 5.10 scrub starts 2024-03-26T05:08:29.846 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:29 smithi080 ceph-mon[40404]: 5.10 scrub ok 2024-03-26T05:08:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:29.859 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:29.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:29.947 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:08:29.960 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:30.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:30.005 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:30.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:30.106 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:30.136 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:30.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:30.238 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:30.271 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:30.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:30.372 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:30.406 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:30.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:30.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:30.509 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:30.551 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:30.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:30.653 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:30.690 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:30.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:30.792 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:30.835 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:30.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:30.870 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:30.937 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:30.966 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:30.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:31.068 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:31.123 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:31.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:31.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:31.225 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:31.268 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:31.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:31.369 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:31.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:31.423 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:31.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:31.524 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:31.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:31.572 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:31.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:31.642 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:31 smithi080 ceph-mon[29691]: pgmap v358: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 36 MiB/s wr, 807 op/s; 47061/336042 objects misplaced (14.004%); 17 MiB/s, 279 objects/s recovering 2024-03-26T05:08:31.643 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:31 smithi080 ceph-mon[40404]: pgmap v358: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 36 MiB/s wr, 807 op/s; 47061/336042 objects misplaced (14.004%); 17 MiB/s, 279 objects/s recovering 2024-03-26T05:08:31.673 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:31.709 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:31.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:31.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:31.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:31 smithi132 ceph-mon[33500]: pgmap v358: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 36 MiB/s wr, 807 op/s; 47061/336042 objects misplaced (14.004%); 17 MiB/s, 279 objects/s recovering 2024-03-26T05:08:31.811 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:31.849 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:31.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:31.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:31.952 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:31.992 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:31.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:32.094 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:32.135 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:32.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:32.236 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:32.285 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:32.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:32.386 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:32.421 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:32.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:32.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:32.434 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:08:32.479 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:08:32.522 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:32.526 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:08:32.537 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:08:32.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:32.571 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:32.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:32.583 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:08:32.627 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:08:32.672 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:32.680 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:08:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:32.715 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:32.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:32.723 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:08:32.764 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:08:32.764 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:32 smithi080 ceph-mon[40404]: 6.5 scrub starts 2024-03-26T05:08:32.764 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:32 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1964865336' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:08:32.764 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:32 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:08:32.765 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:32 smithi080 ceph-mon[40404]: 6.5 scrub ok 2024-03-26T05:08:32.765 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:32 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1464934647' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:32.765 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:32 smithi080 ceph-mon[29691]: 6.5 scrub starts 2024-03-26T05:08:32.765 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:32 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1964865336' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:08:32.765 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:32 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:08:32.765 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:32 smithi080 ceph-mon[29691]: 6.5 scrub ok 2024-03-26T05:08:32.765 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:32 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1464934647' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:32.774 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:08:32.774 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:08:32.774 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:08:32.775 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:08:32.775 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:08:32.775 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:08:32.775 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:08:32.775 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:08:32.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:32 smithi132 ceph-mon[33500]: 6.5 scrub starts 2024-03-26T05:08:32.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:32 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1964865336' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:08:32.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:32 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:08:32.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:32 smithi132 ceph-mon[33500]: 6.5 scrub ok 2024-03-26T05:08:32.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:32 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1464934647' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:32.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:32.850 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:32.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:32.951 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:32.980 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:32.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:32.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:33.082 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:33.111 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:33.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:33.213 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:33.246 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:33.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:33.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:33.386 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:33.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:33.478 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:08:33.478 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:08:33.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:33.524 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:33.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:33.625 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:33.658 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:33.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:33.760 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:33 smithi132 ceph-mon[33500]: pgmap v359: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 937 B/s rd, 32 MiB/s wr, 1.06k op/s; 43978/324483 objects misplaced (13.553%); 26 MiB/s, 420 objects/s recovering 2024-03-26T05:08:33.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:33 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:08:33.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:33 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:08:33.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:33 smithi132 ceph-mon[33500]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T05:08:33.794 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:33.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:33.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:33.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:33 smithi080 ceph-mon[29691]: pgmap v359: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 937 B/s rd, 32 MiB/s wr, 1.06k op/s; 43978/324483 objects misplaced (13.553%); 26 MiB/s, 420 objects/s recovering 2024-03-26T05:08:33.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:33 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:08:33.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:33 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:08:33.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:33 smithi080 ceph-mon[29691]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T05:08:33.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:33 smithi080 ceph-mon[40404]: pgmap v359: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 937 B/s rd, 32 MiB/s wr, 1.06k op/s; 43978/324483 objects misplaced (13.553%); 26 MiB/s, 420 objects/s recovering 2024-03-26T05:08:33.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:33 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:08:33.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:33 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:08:33.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:33 smithi080 ceph-mon[40404]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T05:08:33.896 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:33.927 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:33.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:34.029 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:34.072 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:34.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:34.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:34.174 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:34.198 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:34.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:34.299 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:34.301 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T05:08:34.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:08:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:08:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:08:34.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:34.377 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:34.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:34.478 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:34.516 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:34.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:34.617 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:34.657 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:34.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:34.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:34.726 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:08:34.758 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:34.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:34 smithi132 ceph-mon[33500]: 2.0 scrub starts 2024-03-26T05:08:34.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:34 smithi132 ceph-mon[33500]: 2.0 scrub ok 2024-03-26T05:08:34.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:34 smithi132 ceph-mon[33500]: pgmap v361: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 MiB/s wr, 897 op/s; 43978/324483 objects misplaced (13.553%); 21 MiB/s, 337 objects/s recovering 2024-03-26T05:08:34.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:34.813 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:34.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:34.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:34 smithi080 ceph-mon[29691]: 2.0 scrub starts 2024-03-26T05:08:34.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:34 smithi080 ceph-mon[29691]: 2.0 scrub ok 2024-03-26T05:08:34.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:34 smithi080 ceph-mon[29691]: pgmap v361: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 MiB/s wr, 897 op/s; 43978/324483 objects misplaced (13.553%); 21 MiB/s, 337 objects/s recovering 2024-03-26T05:08:34.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:34 smithi080 ceph-mon[40404]: 2.0 scrub starts 2024-03-26T05:08:34.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:34 smithi080 ceph-mon[40404]: 2.0 scrub ok 2024-03-26T05:08:34.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:34 smithi080 ceph-mon[40404]: pgmap v361: 148 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 MiB/s wr, 897 op/s; 43978/324483 objects misplaced (13.553%); 21 MiB/s, 337 objects/s recovering 2024-03-26T05:08:34.914 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:34.947 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:34.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:34.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:35.049 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:35.089 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:35.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:35.191 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:35.231 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:35.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:35.332 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:35.366 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:35.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:35.467 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:35.491 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:35.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:35.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:08:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:08:35.592 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:35.623 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:35.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:35.724 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:35.765 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:35.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:35.866 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:35.910 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:35.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:36.011 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:36.041 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:36.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:36.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:35 smithi132 ceph-mon[33500]: 6.1c scrub starts 2024-03-26T05:08:36.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:35 smithi132 ceph-mon[33500]: 6.1c scrub ok 2024-03-26T05:08:36.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:08:36.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:08:36.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:35 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:36.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:35 smithi080 ceph-mon[40404]: 6.1c scrub starts 2024-03-26T05:08:36.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:35 smithi080 ceph-mon[40404]: 6.1c scrub ok 2024-03-26T05:08:36.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:08:36.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:08:36.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:35 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:36.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:35 smithi080 ceph-mon[29691]: 6.1c scrub starts 2024-03-26T05:08:36.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:35 smithi080 ceph-mon[29691]: 6.1c scrub ok 2024-03-26T05:08:36.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:08:36.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:08:36.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:35 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:36.143 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:36.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:36.176 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:36.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:36.277 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:36.311 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:36.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:36.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:36.412 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:36.437 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:36.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:36.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:36.539 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:36.581 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:36.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:36.683 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:36.714 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:36.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:36.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:36.842 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:36.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:36.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:36.946 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:36.992 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:36.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:36.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:37.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:36 smithi132 ceph-mon[33500]: pgmap v362: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 929 op/s; 40479/310689 objects misplaced (13.029%); 21 MiB/s, 337 objects/s recovering 2024-03-26T05:08:37.094 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:37.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:36 smithi080 ceph-mon[29691]: pgmap v362: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 929 op/s; 40479/310689 objects misplaced (13.029%); 21 MiB/s, 337 objects/s recovering 2024-03-26T05:08:37.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:36 smithi080 ceph-mon[40404]: pgmap v362: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 929 op/s; 40479/310689 objects misplaced (13.029%); 21 MiB/s, 337 objects/s recovering 2024-03-26T05:08:37.118 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:37.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:37.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:37.219 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:37.265 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:37.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:37.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:37.362 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:08:37.368 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:37.430 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:37.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:37.531 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:37.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:37.589 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:37.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:37.690 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:37.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:37.724 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:37.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:37.825 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:37.862 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:37.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:37.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:38.004 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:38.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:38.014 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:08:38.014 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:08:38.106 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:38.146 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:38.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:38.247 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:38.273 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:38.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:38.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:38.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:37 smithi132 ceph-mon[33500]: 3.e scrub starts 2024-03-26T05:08:38.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:37 smithi132 ceph-mon[33500]: 3.e scrub ok 2024-03-26T05:08:38.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:37 smithi080 ceph-mon[29691]: 3.e scrub starts 2024-03-26T05:08:38.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:37 smithi080 ceph-mon[29691]: 3.e scrub ok 2024-03-26T05:08:38.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:37 smithi080 ceph-mon[40404]: 3.e scrub starts 2024-03-26T05:08:38.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:37 smithi080 ceph-mon[40404]: 3.e scrub ok 2024-03-26T05:08:38.375 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:38.415 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:38.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:38.517 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:38.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:38.547 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:38.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:38.648 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:38.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:38.685 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:38.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:38.767 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T05:08:38.786 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:38.833 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:38.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:38.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:38.935 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:38.995 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:38.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:39.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:38 smithi132 ceph-mon[33500]: pgmap v363: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 929 op/s; 40479/310689 objects misplaced (13.029%); 21 MiB/s, 337 objects/s recovering 2024-03-26T05:08:39.096 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:38 smithi080 ceph-mon[40404]: pgmap v363: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 929 op/s; 40479/310689 objects misplaced (13.029%); 21 MiB/s, 337 objects/s recovering 2024-03-26T05:08:39.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:38 smithi080 ceph-mon[29691]: pgmap v363: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 929 op/s; 40479/310689 objects misplaced (13.029%); 21 MiB/s, 337 objects/s recovering 2024-03-26T05:08:39.121 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:39.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:39.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:39.223 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:39.259 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:39.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:39.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:39.361 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:39.406 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:39.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:39.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:39.508 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:39.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:39.550 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:39.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:39.651 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:39.691 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:39.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:39.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:39.792 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:39.824 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:39.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:39.925 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:39.976 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:39.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:40.078 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:40.121 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:40.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:40.223 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:40.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:39 smithi080 ceph-mon[29691]: 6.8 deep-scrub starts 2024-03-26T05:08:40.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:39 smithi080 ceph-mon[29691]: 6.8 deep-scrub ok 2024-03-26T05:08:40.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:39 smithi080 ceph-mon[40404]: 6.8 deep-scrub starts 2024-03-26T05:08:40.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:39 smithi080 ceph-mon[40404]: 6.8 deep-scrub ok 2024-03-26T05:08:40.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:40.272 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:40.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:39 smithi132 ceph-mon[33500]: 6.8 deep-scrub starts 2024-03-26T05:08:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:39 smithi132 ceph-mon[33500]: 6.8 deep-scrub ok 2024-03-26T05:08:40.366 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:40.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:40.399 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:40.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:40.449 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:40.501 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:40.556 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:40.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:40.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:40.658 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:40.688 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:40.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:40.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:40.749 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:08:40.762 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:08:40.790 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:40.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:40.835 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:40.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:40.936 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:40.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:40.987 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:40.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:41.087 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:41.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:40 smithi080 ceph-mon[40404]: pgmap v364: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 871 op/s; 40217/309903 objects misplaced (12.977%); 21 MiB/s, 337 objects/s recovering 2024-03-26T05:08:41.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:40 smithi080 ceph-mon[40404]: 6.15 scrub starts 2024-03-26T05:08:41.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:40 smithi080 ceph-mon[40404]: 6.15 scrub ok 2024-03-26T05:08:41.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:40 smithi080 ceph-mon[29691]: pgmap v364: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 871 op/s; 40217/309903 objects misplaced (12.977%); 21 MiB/s, 337 objects/s recovering 2024-03-26T05:08:41.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:40 smithi080 ceph-mon[29691]: 6.15 scrub starts 2024-03-26T05:08:41.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:40 smithi080 ceph-mon[29691]: 6.15 scrub ok 2024-03-26T05:08:41.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:41.128 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:41.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:41.229 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:41.271 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:41.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:41.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:40 smithi132 ceph-mon[33500]: pgmap v364: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 871 op/s; 40217/309903 objects misplaced (12.977%); 21 MiB/s, 337 objects/s recovering 2024-03-26T05:08:41.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:40 smithi132 ceph-mon[33500]: 6.15 scrub starts 2024-03-26T05:08:41.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:40 smithi132 ceph-mon[33500]: 6.15 scrub ok 2024-03-26T05:08:41.373 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:41.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:41.412 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:41.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:41.513 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:41.556 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:41.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:41.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:41.658 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:41.705 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:41.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:41.739 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:08:41.806 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:41.854 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:41.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:41.956 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:41.984 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:41.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:42.080 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-03-26T05:08:39.866760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6755391862,"num_objects":103301,"num_object_clones":0,"num_object_copies":309903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40217,"num_objects_unfound":0,"num_objects_dirty":103301,"num_whiteouts":0,"num_read":1712,"num_read_kb":1526,"num_write":126766,"num_write_kb":7331358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11641,"num_bytes_recovered":756356046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86913,"ondisk_log_size":86913,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":452,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22460808,"kb_used_data":20566152,"kb_used_omap":12,"kb_used_meta":1894451,"kb_avail":727500408,"statfs":{"total":767960285184,"available":744960417792,"internally_reserved":0,"allocated":21059739648,"data_stored":21056918018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1939918384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-571015168,"num_objects":-8713,"num_object_clones":0,"num_object_copies":-26139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6844,"num_objects_unfound":0,"num_objects_dirty":-8713,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8713,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3379,"num_bytes_recovered":221446144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004615"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":344,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530404+0000","last_change":"2024-03-26T05:07:59.996779+0000","last_active":"2024-03-26T05:08:32.530404+0000","last_peered":"2024-03-26T05:08:32.530404+0000","last_clean":"2024-03-26T05:08:32.530404+0000","last_became_active":"2024-03-26T05:07:59.996436+0000","last_became_peered":"2024-03-26T05:07:59.996436+0000","last_unstale":"2024-03-26T05:08:32.530404+0000","last_undegraded":"2024-03-26T05:08:32.530404+0000","last_fullsized":"2024-03-26T05:08:32.530404+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:08.113253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:08.113253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":347,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529854+0000","last_change":"2024-03-26T05:07:59.997398+0000","last_active":"2024-03-26T05:08:32.529854+0000","last_peered":"2024-03-26T05:08:32.529854+0000","last_clean":"2024-03-26T05:08:32.529854+0000","last_became_active":"2024-03-26T05:07:59.997089+0000","last_became_peered":"2024-03-26T05:07:59.997089+0000","last_unstale":"2024-03-26T05:08:32.529854+0000","last_undegraded":"2024-03-26T05:08:32.529854+0000","last_fullsized":"2024-03-26T05:08:32.529854+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:02.872031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:02.872031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":356,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.541474+0000","last_change":"2024-03-26T05:07:59.995053+0000","last_active":"2024-03-26T05:08:32.541474+0000","last_peered":"2024-03-26T05:08:32.541474+0000","last_clean":"2024-03-26T05:08:32.541474+0000","last_became_active":"2024-03-26T05:07:59.994431+0000","last_became_peered":"2024-03-26T05:07:59.994431+0000","last_unstale":"2024-03-26T05:08:32.541474+0000","last_undegraded":"2024-03-26T05:08:32.541474+0000","last_fullsized":"2024-03-26T05:08:32.541474+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.016730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.016730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'15","reported_seq":368,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529167+0000","last_change":"2024-03-26T05:06:52.298546+0000","last_active":"2024-03-26T05:08:32.529167+0000","last_peered":"2024-03-26T05:08:32.529167+0000","last_clean":"2024-03-26T05:08:32.529167+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:08:32.529167+0000","last_undegraded":"2024-03-26T05:08:32.529167+0000","last_fullsized":"2024-03-26T05:08:32.529167+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:06:52.298418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:52.298418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":323,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.539173+0000","last_change":"2024-03-26T05:06:37.226644+0000","last_active":"2024-03-26T05:08:32.539173+0000","last_peered":"2024-03-26T05:08:32.539173+0000","last_clean":"2024-03-26T05:08:32.539173+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:08:32.539173+0000","last_undegraded":"2024-03-26T05:08:32.539173+0000","last_fullsized":"2024-03-26T05:08:32.539173+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:37.226521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:37.226521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":336,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529014+0000","last_change":"2024-03-26T05:07:36.616676+0000","last_active":"2024-03-26T05:08:32.529014+0000","last_peered":"2024-03-26T05:08:32.529014+0000","last_clean":"2024-03-26T05:08:32.529014+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:08:32.529014+0000","last_undegraded":"2024-03-26T05:08:32.529014+0000","last_fullsized":"2024-03-26T05:08:32.529014+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:36.616569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:36.616569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:52.540577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":344,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.536697+0000","last_change":"2024-03-26T05:05:12.713135+0000","last_active":"2024-03-26T05:08:32.536697+0000","last_peered":"2024-03-26T05:08:32.536697+0000","last_clean":"2024-03-26T05:08:32.536697+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:08:32.536697+0000","last_undegraded":"2024-03-26T05:08:32.536697+0000","last_fullsized":"2024-03-26T05:08:32.536697+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.713011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'39","reported_seq":412,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.537793+0000","last_change":"2024-03-26T05:07:46.642266+0000","last_active":"2024-03-26T05:08:32.537793+0000","last_peered":"2024-03-26T05:08:32.537793+0000","last_clean":"2024-03-26T05:08:32.537793+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:08:32.537793+0000","last_undegraded":"2024-03-26T05:08:32.537793+0000","last_fullsized":"2024-03-26T05:08:32.537793+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'39","last_scrub_stamp":"2024-03-26T05:07:46.641260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:46.641260+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:16.431846+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":340,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.540638+0000","last_change":"2024-03-26T05:07:40.028960+0000","last_active":"2024-03-26T05:08:32.540638+0000","last_peered":"2024-03-26T05:08:32.540638+0000","last_clean":"2024-03-26T05:08:32.540638+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:08:32.540638+0000","last_undegraded":"2024-03-26T05:08:32.540638+0000","last_fullsized":"2024-03-26T05:08:32.540638+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:40.028690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:40.028690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:43.963276+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.524234+0000","last_change":"2024-03-26T05:08:00.000523+0000","last_active":"2024-03-26T05:08:32.524234+0000","last_peered":"2024-03-26T05:08:32.524234+0000","last_clean":"2024-03-26T05:08:32.524234+0000","last_became_active":"2024-03-26T05:08:00.000128+0000","last_became_peered":"2024-03-26T05:08:00.000128+0000","last_unstale":"2024-03-26T05:08:32.524234+0000","last_undegraded":"2024-03-26T05:08:32.524234+0000","last_fullsized":"2024-03-26T05:08:32.524234+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:10.922975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:10.922975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.525259+0000","last_change":"2024-03-26T05:07:59.947882+0000","last_active":"2024-03-26T05:08:32.525259+0000","last_peered":"2024-03-26T05:08:32.525259+0000","last_clean":"2024-03-26T05:08:32.525259+0000","last_became_active":"2024-03-26T05:07:59.947557+0000","last_became_peered":"2024-03-26T05:07:59.947557+0000","last_unstale":"2024-03-26T05:08:32.525259+0000","last_undegraded":"2024-03-26T05:08:32.525259+0000","last_fullsized":"2024-03-26T05:08:32.525259+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:38.336853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:38.336853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'21","reported_seq":380,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530844+0000","last_change":"2024-03-26T05:05:53.230294+0000","last_active":"2024-03-26T05:08:32.530844+0000","last_peered":"2024-03-26T05:08:32.530844+0000","last_clean":"2024-03-26T05:08:32.530844+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:08:32.530844+0000","last_undegraded":"2024-03-26T05:08:32.530844+0000","last_fullsized":"2024-03-26T05:08:32.530844+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:05:53.230145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:53.230145+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":25,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.524716+0000","last_change":"2024-03-26T05:07:59.999227+0000","last_active":"2024-03-26T05:08:32.524716+0000","last_peered":"2024-03-26T05:08:32.524716+0000","last_clean":"2024-03-26T05:08:32.524716+0000","last_became_active":"2024-03-26T05:07:59.998777+0000","last_became_peered":"2024-03-26T05:07:59.998777+0000","last_unstale":"2024-03-26T05:08:32.524716+0000","last_undegraded":"2024-03-26T05:08:32.524716+0000","last_fullsized":"2024-03-26T05:08:32.524716+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:12.931045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.931045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":342,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532208+0000","last_change":"2024-03-26T05:08:24.706805+0000","last_active":"2024-03-26T05:08:32.532208+0000","last_peered":"2024-03-26T05:08:32.532208+0000","last_clean":"2024-03-26T05:08:32.532208+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:08:32.532208+0000","last_undegraded":"2024-03-26T05:08:32.532208+0000","last_fullsized":"2024-03-26T05:08:32.532208+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:08:24.706613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:08:24.706613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:33.541120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.524654+0000","last_change":"2024-03-26T05:08:00.011593+0000","last_active":"2024-03-26T05:08:32.524654+0000","last_peered":"2024-03-26T05:08:32.524654+0000","last_clean":"2024-03-26T05:08:32.524654+0000","last_became_active":"2024-03-26T05:08:00.011262+0000","last_became_peered":"2024-03-26T05:08:00.011262+0000","last_unstale":"2024-03-26T05:08:32.524654+0000","last_undegraded":"2024-03-26T05:08:32.524654+0000","last_fullsized":"2024-03-26T05:08:32.524654+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:53.514177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.514177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:22.242447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'11","reported_seq":362,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529297+0000","last_change":"2024-03-26T05:07:38.699595+0000","last_active":"2024-03-26T05:08:32.529297+0000","last_peered":"2024-03-26T05:08:32.529297+0000","last_clean":"2024-03-26T05:08:32.529297+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:08:32.529297+0000","last_undegraded":"2024-03-26T05:08:32.529297+0000","last_fullsized":"2024-03-26T05:08:32.529297+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-26T05:07:38.699415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:38.699415+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:57.390111+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":353,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530306+0000","last_change":"2024-03-26T05:07:59.943198+0000","last_active":"2024-03-26T05:08:32.530306+0000","last_peered":"2024-03-26T05:08:32.530306+0000","last_clean":"2024-03-26T05:08:32.530306+0000","last_became_active":"2024-03-26T05:07:59.942829+0000","last_became_peered":"2024-03-26T05:07:59.942829+0000","last_unstale":"2024-03-26T05:08:32.530306+0000","last_undegraded":"2024-03-26T05:08:32.530306+0000","last_fullsized":"2024-03-26T05:08:32.530306+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:51.271966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:06:51.271966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":335,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530165+0000","last_change":"2024-03-26T05:06:00.105160+0000","last_active":"2024-03-26T05:08:32.530165+0000","last_peered":"2024-03-26T05:08:32.530165+0000","last_clean":"2024-03-26T05:08:32.530165+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:08:32.530165+0000","last_undegraded":"2024-03-26T05:08:32.530165+0000","last_fullsized":"2024-03-26T05:08:32.530165+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:00.104920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:00.104920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":352,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.540998+0000","last_change":"2024-03-26T05:07:53.149194+0000","last_active":"2024-03-26T05:08:32.540998+0000","last_peered":"2024-03-26T05:08:32.540998+0000","last_clean":"2024-03-26T05:08:32.540998+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:08:32.540998+0000","last_undegraded":"2024-03-26T05:08:32.540998+0000","last_fullsized":"2024-03-26T05:08:32.540998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.149071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:03.971674+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":372,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.540432+0000","last_change":"2024-03-26T05:06:45.225054+0000","last_active":"2024-03-26T05:08:32.540432+0000","last_peered":"2024-03-26T05:08:32.540432+0000","last_clean":"2024-03-26T05:08:32.540432+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:08:32.540432+0000","last_undegraded":"2024-03-26T05:08:32.540432+0000","last_fullsized":"2024-03-26T05:08:32.540432+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:06:45.224956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.224956+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":343,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.543076+0000","last_change":"2024-03-26T05:07:59.960265+0000","last_active":"2024-03-26T05:08:32.543076+0000","last_peered":"2024-03-26T05:08:32.543076+0000","last_clean":"2024-03-26T05:08:32.543076+0000","last_became_active":"2024-03-26T05:07:59.959705+0000","last_became_peered":"2024-03-26T05:07:59.959705+0000","last_unstale":"2024-03-26T05:08:32.543076+0000","last_undegraded":"2024-03-26T05:08:32.543076+0000","last_fullsized":"2024-03-26T05:08:32.543076+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:02.957488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:05.268043+0000","last_clean_scrub_stamp":"2024-03-26T05:06:02.957488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":336,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530780+0000","last_change":"2024-03-26T05:06:48.271761+0000","last_active":"2024-03-26T05:08:32.530780+0000","last_peered":"2024-03-26T05:08:32.530780+0000","last_clean":"2024-03-26T05:08:32.530780+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:08:32.530780+0000","last_undegraded":"2024-03-26T05:08:32.530780+0000","last_fullsized":"2024-03-26T05:08:32.530780+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:48.271668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:48.271668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":360,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530947+0000","last_change":"2024-03-26T05:07:54.756429+0000","last_active":"2024-03-26T05:08:32.530947+0000","last_peered":"2024-03-26T05:08:32.530947+0000","last_clean":"2024-03-26T05:08:32.530947+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:08:32.530947+0000","last_undegraded":"2024-03-26T05:08:32.530947+0000","last_fullsized":"2024-03-26T05:08:32.530947+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:54.756315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:07:54.756315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:59.548740+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'20","reported_seq":376,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.536715+0000","last_change":"2024-03-26T05:07:50.662703+0000","last_active":"2024-03-26T05:08:32.536715+0000","last_peered":"2024-03-26T05:08:32.536715+0000","last_clean":"2024-03-26T05:08:32.536715+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:08:32.536715+0000","last_undegraded":"2024-03-26T05:08:32.536715+0000","last_fullsized":"2024-03-26T05:08:32.536715+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:07:50.662564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:50.662564+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:07.686794+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":359,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:34.685656+0000","last_change":"2024-03-26T05:08:34.685699+0000","last_active":"2024-03-26T05:08:34.685656+0000","last_peered":"2024-03-26T05:08:34.685656+0000","last_clean":"2024-03-26T05:08:34.685656+0000","last_became_active":"2024-03-26T05:07:59.955125+0000","last_became_peered":"2024-03-26T05:07:59.955125+0000","last_unstale":"2024-03-26T05:08:34.685656+0000","last_undegraded":"2024-03-26T05:08:34.685656+0000","last_fullsized":"2024-03-26T05:08:34.685656+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:08:34.685577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:08:34.685577+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:52.403193+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":460,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.531439+0000","last_change":"2024-03-26T05:06:53.544685+0000","last_active":"2024-03-26T05:08:32.531439+0000","last_peered":"2024-03-26T05:08:32.531439+0000","last_clean":"2024-03-26T05:08:32.531439+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:08:32.531439+0000","last_undegraded":"2024-03-26T05:08:32.531439+0000","last_fullsized":"2024-03-26T05:08:32.531439+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:06:53.544604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:53.544604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":344,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529943+0000","last_change":"2024-03-26T05:06:04.028509+0000","last_active":"2024-03-26T05:08:32.529943+0000","last_peered":"2024-03-26T05:08:32.529943+0000","last_clean":"2024-03-26T05:08:32.529943+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:08:32.529943+0000","last_undegraded":"2024-03-26T05:08:32.529943+0000","last_fullsized":"2024-03-26T05:08:32.529943+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:06:04.028383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:04.028383+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'16","reported_seq":376,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.541229+0000","last_change":"2024-03-26T05:07:54.153459+0000","last_active":"2024-03-26T05:08:32.541229+0000","last_peered":"2024-03-26T05:08:32.541229+0000","last_clean":"2024-03-26T05:08:32.541229+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:08:32.541229+0000","last_undegraded":"2024-03-26T05:08:32.541229+0000","last_fullsized":"2024-03-26T05:08:32.541229+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:07:54.153319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:54.153319+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:18.535891+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":343,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530305+0000","last_change":"2024-03-26T05:07:59.995908+0000","last_active":"2024-03-26T05:08:32.530305+0000","last_peered":"2024-03-26T05:08:32.530305+0000","last_clean":"2024-03-26T05:08:32.530305+0000","last_became_active":"2024-03-26T05:07:59.995261+0000","last_became_peered":"2024-03-26T05:07:59.995261+0000","last_unstale":"2024-03-26T05:08:32.530305+0000","last_undegraded":"2024-03-26T05:08:32.530305+0000","last_fullsized":"2024-03-26T05:08:32.530305+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:55.814330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:04:55.814330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":337,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.541932+0000","last_change":"2024-03-26T05:05:53.093643+0000","last_active":"2024-03-26T05:08:32.541932+0000","last_peered":"2024-03-26T05:08:32.541932+0000","last_clean":"2024-03-26T05:08:32.541932+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:08:32.541932+0000","last_undegraded":"2024-03-26T05:08:32.541932+0000","last_fullsized":"2024-03-26T05:08:32.541932+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:53.093539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:53.093539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":343,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.537571+0000","last_change":"2024-03-26T05:06:00.960334+0000","last_active":"2024-03-26T05:08:32.537571+0000","last_peered":"2024-03-26T05:08:32.537571+0000","last_clean":"2024-03-26T05:08:32.537571+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:08:32.537571+0000","last_undegraded":"2024-03-26T05:08:32.537571+0000","last_fullsized":"2024-03-26T05:08:32.537571+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:00.960212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:00.960212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'17","reported_seq":437,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.531184+0000","last_change":"2024-03-26T05:07:59.971416+0000","last_active":"2024-03-26T05:08:32.531184+0000","last_peered":"2024-03-26T05:08:32.531184+0000","last_clean":"2024-03-26T05:08:32.531184+0000","last_became_active":"2024-03-26T05:07:59.962007+0000","last_became_peered":"2024-03-26T05:07:59.962007+0000","last_unstale":"2024-03-26T05:08:32.531184+0000","last_undegraded":"2024-03-26T05:08:32.531184+0000","last_fullsized":"2024-03-26T05:08:32.531184+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:06:47.435205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:47.435205+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":330,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532218+0000","last_change":"2024-03-26T05:08:20.675986+0000","last_active":"2024-03-26T05:08:32.532218+0000","last_peered":"2024-03-26T05:08:32.532218+0000","last_clean":"2024-03-26T05:08:32.532218+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:08:32.532218+0000","last_undegraded":"2024-03-26T05:08:32.532218+0000","last_fullsized":"2024-03-26T05:08:32.532218+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:08:20.675869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:08:20.675869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:38.198526+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":342,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532710+0000","last_change":"2024-03-26T05:06:42.481238+0000","last_active":"2024-03-26T05:08:32.532710+0000","last_peered":"2024-03-26T05:08:32.532710+0000","last_clean":"2024-03-26T05:08:32.532710+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:08:32.532710+0000","last_undegraded":"2024-03-26T05:08:32.532710+0000","last_fullsized":"2024-03-26T05:08:32.532710+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:42.481122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:42.481122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":356,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.542428+0000","last_change":"2024-03-26T05:07:59.994755+0000","last_active":"2024-03-26T05:08:32.542428+0000","last_peered":"2024-03-26T05:08:32.542428+0000","last_clean":"2024-03-26T05:08:32.542428+0000","last_became_active":"2024-03-26T05:07:59.993996+0000","last_became_peered":"2024-03-26T05:07:59.993996+0000","last_unstale":"2024-03-26T05:08:32.542428+0000","last_undegraded":"2024-03-26T05:08:32.542428+0000","last_fullsized":"2024-03-26T05:08:32.542428+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:59.138510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:59.138510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"100'6","reported_seq":347,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.537667+0000","last_change":"2024-03-26T05:06:45.224407+0000","last_active":"2024-03-26T05:08:32.537667+0000","last_peered":"2024-03-26T05:08:32.537667+0000","last_clean":"2024-03-26T05:08:32.537667+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:08:32.537667+0000","last_undegraded":"2024-03-26T05:08:32.537667+0000","last_fullsized":"2024-03-26T05:08:32.537667+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'6","last_scrub_stamp":"2024-03-26T05:06:45.224296+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.224296+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":331,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.539317+0000","last_change":"2024-03-26T05:06:49.226016+0000","last_active":"2024-03-26T05:08:32.539317+0000","last_peered":"2024-03-26T05:08:32.539317+0000","last_clean":"2024-03-26T05:08:32.539317+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:08:32.539317+0000","last_undegraded":"2024-03-26T05:08:32.539317+0000","last_fullsized":"2024-03-26T05:08:32.539317+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_clean_scrub_stamp":"2024-03-26T05:06:49.225897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":342,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532068+0000","last_change":"2024-03-26T05:08:28.622488+0000","last_active":"2024-03-26T05:08:32.532068+0000","last_peered":"2024-03-26T05:08:32.532068+0000","last_clean":"2024-03-26T05:08:32.532068+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:08:32.532068+0000","last_undegraded":"2024-03-26T05:08:32.532068+0000","last_fullsized":"2024-03-26T05:08:32.532068+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:08:28.622401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:08:28.622401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:43.509369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.523913+0000","last_change":"2024-03-26T05:07:59.997817+0000","last_active":"2024-03-26T05:08:32.523913+0000","last_peered":"2024-03-26T05:08:32.523913+0000","last_clean":"2024-03-26T05:08:32.523913+0000","last_became_active":"2024-03-26T05:07:59.997439+0000","last_became_peered":"2024-03-26T05:07:59.997439+0000","last_unstale":"2024-03-26T05:08:32.523913+0000","last_undegraded":"2024-03-26T05:08:32.523913+0000","last_fullsized":"2024-03-26T05:08:32.523913+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:06:56.886632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'17","reported_seq":370,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.537689+0000","last_change":"2024-03-26T05:07:52.593239+0000","last_active":"2024-03-26T05:08:32.537689+0000","last_peered":"2024-03-26T05:08:32.537689+0000","last_clean":"2024-03-26T05:08:32.537689+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:08:32.537689+0000","last_undegraded":"2024-03-26T05:08:32.537689+0000","last_fullsized":"2024-03-26T05:08:32.537689+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:07:52.592943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:52.592943+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:52.776675+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":352,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.540024+0000","last_change":"2024-03-26T05:07:59.994951+0000","last_active":"2024-03-26T05:08:32.540024+0000","last_peered":"2024-03-26T05:08:32.540024+0000","last_clean":"2024-03-26T05:08:32.540024+0000","last_became_active":"2024-03-26T05:07:59.994210+0000","last_became_peered":"2024-03-26T05:07:59.994210+0000","last_unstale":"2024-03-26T05:08:32.540024+0000","last_undegraded":"2024-03-26T05:08:32.540024+0000","last_fullsized":"2024-03-26T05:08:32.540024+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:54.126517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:54.126517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":327,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.539226+0000","last_change":"2024-03-26T05:05:04.767968+0000","last_active":"2024-03-26T05:08:32.539226+0000","last_peered":"2024-03-26T05:08:32.539226+0000","last_clean":"2024-03-26T05:08:32.539226+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:08:32.539226+0000","last_undegraded":"2024-03-26T05:08:32.539226+0000","last_fullsized":"2024-03-26T05:08:32.539226+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:05:04.767790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":342,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532924+0000","last_change":"2024-03-26T05:08:23.720374+0000","last_active":"2024-03-26T05:08:32.532924+0000","last_peered":"2024-03-26T05:08:32.532924+0000","last_clean":"2024-03-26T05:08:32.532924+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:08:32.532924+0000","last_undegraded":"2024-03-26T05:08:32.532924+0000","last_fullsized":"2024-03-26T05:08:32.532924+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:08:23.720185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:08:23.720185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:43.854643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":363,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.531068+0000","last_change":"2024-03-26T05:07:53.425727+0000","last_active":"2024-03-26T05:08:32.531068+0000","last_peered":"2024-03-26T05:08:32.531068+0000","last_clean":"2024-03-26T05:08:32.531068+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:08:32.531068+0000","last_undegraded":"2024-03-26T05:08:32.531068+0000","last_fullsized":"2024-03-26T05:08:32.531068+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-26T05:07:53.425472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.425472+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:12.356357+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":335,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.537415+0000","last_change":"2024-03-26T05:08:00.000061+0000","last_active":"2024-03-26T05:08:32.537415+0000","last_peered":"2024-03-26T05:08:32.537415+0000","last_clean":"2024-03-26T05:08:32.537415+0000","last_became_active":"2024-03-26T05:07:59.999703+0000","last_became_peered":"2024-03-26T05:07:59.999703+0000","last_unstale":"2024-03-26T05:08:32.537415+0000","last_undegraded":"2024-03-26T05:08:32.537415+0000","last_fullsized":"2024-03-26T05:08:32.537415+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:58.834141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:04:58.834141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":355,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530979+0000","last_change":"2024-03-26T05:08:00.003136+0000","last_active":"2024-03-26T05:08:32.530979+0000","last_peered":"2024-03-26T05:08:32.530979+0000","last_clean":"2024-03-26T05:08:32.530979+0000","last_became_active":"2024-03-26T05:08:00.001601+0000","last_became_peered":"2024-03-26T05:08:00.001601+0000","last_unstale":"2024-03-26T05:08:32.530979+0000","last_undegraded":"2024-03-26T05:08:32.530979+0000","last_fullsized":"2024-03-26T05:08:32.530979+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:42.246721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:06:42.246721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":344,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529129+0000","last_change":"2024-03-26T05:07:39.699241+0000","last_active":"2024-03-26T05:08:32.529129+0000","last_peered":"2024-03-26T05:08:32.529129+0000","last_clean":"2024-03-26T05:08:32.529129+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:08:32.529129+0000","last_undegraded":"2024-03-26T05:08:32.529129+0000","last_fullsized":"2024-03-26T05:08:32.529129+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:39.699122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:39.699122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:05.562610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"95'16","reported_seq":372,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529514+0000","last_change":"2024-03-26T05:07:43.754242+0000","last_active":"2024-03-26T05:08:32.529514+0000","last_peered":"2024-03-26T05:08:32.529514+0000","last_clean":"2024-03-26T05:08:32.529514+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:08:32.529514+0000","last_undegraded":"2024-03-26T05:08:32.529514+0000","last_fullsized":"2024-03-26T05:08:32.529514+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:07:43.754130+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:07:43.754130+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:10.909529+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":332,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.540863+0000","last_change":"2024-03-26T05:06:04.057848+0000","last_active":"2024-03-26T05:08:32.540863+0000","last_peered":"2024-03-26T05:08:32.540863+0000","last_clean":"2024-03-26T05:08:32.540863+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:08:32.540863+0000","last_undegraded":"2024-03-26T05:08:32.540863+0000","last_fullsized":"2024-03-26T05:08:32.540863+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_clean_scrub_stamp":"2024-03-26T05:06:04.057358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"216'10","reported_seq":37,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:33.491340+0000","last_change":"2024-03-26T05:08:00.009781+0000","last_active":"2024-03-26T05:08:33.491340+0000","last_peered":"2024-03-26T05:08:33.491340+0000","last_clean":"2024-03-26T05:08:33.491340+0000","last_became_active":"2024-03-26T05:08:00.005648+0000","last_became_peered":"2024-03-26T05:08:00.005648+0000","last_unstale":"2024-03-26T05:08:33.491340+0000","last_undegraded":"2024-03-26T05:08:33.491340+0000","last_fullsized":"2024-03-26T05:08:33.491340+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'9","last_scrub_stamp":"2024-03-26T05:06:47.137084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:47.137084+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":350,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532331+0000","last_change":"2024-03-26T05:08:27.671307+0000","last_active":"2024-03-26T05:08:32.532331+0000","last_peered":"2024-03-26T05:08:32.532331+0000","last_clean":"2024-03-26T05:08:32.532331+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:08:32.532331+0000","last_undegraded":"2024-03-26T05:08:32.532331+0000","last_fullsized":"2024-03-26T05:08:32.532331+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:08:27.671192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:08:27.671192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:41.778468+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'22","reported_seq":378,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.534914+0000","last_change":"2024-03-26T05:05:08.701087+0000","last_active":"2024-03-26T05:08:32.534914+0000","last_peered":"2024-03-26T05:08:32.534914+0000","last_clean":"2024-03-26T05:08:32.534914+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:08:32.534914+0000","last_undegraded":"2024-03-26T05:08:32.534914+0000","last_fullsized":"2024-03-26T05:08:32.534914+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:05:08.700945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:08.700945+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":344,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530286+0000","last_change":"2024-03-26T05:07:59.942867+0000","last_active":"2024-03-26T05:08:32.530286+0000","last_peered":"2024-03-26T05:08:32.530286+0000","last_clean":"2024-03-26T05:08:32.530286+0000","last_became_active":"2024-03-26T05:07:59.942512+0000","last_became_peered":"2024-03-26T05:07:59.942512+0000","last_unstale":"2024-03-26T05:08:32.530286+0000","last_undegraded":"2024-03-26T05:08:32.530286+0000","last_fullsized":"2024-03-26T05:08:32.530286+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:29.768389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:29.768389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:58.516907+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"120'9","reported_seq":446,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.535462+0000","last_change":"2024-03-26T05:06:39.209229+0000","last_active":"2024-03-26T05:08:32.535462+0000","last_peered":"2024-03-26T05:08:32.535462+0000","last_clean":"2024-03-26T05:08:32.535462+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:08:32.535462+0000","last_undegraded":"2024-03-26T05:08:32.535462+0000","last_fullsized":"2024-03-26T05:08:32.535462+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'9","last_scrub_stamp":"2024-03-26T05:06:39.209149+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:06:39.209149+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":352,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.540467+0000","last_change":"2024-03-26T05:06:55.105421+0000","last_active":"2024-03-26T05:08:32.540467+0000","last_peered":"2024-03-26T05:08:32.540467+0000","last_clean":"2024-03-26T05:08:32.540467+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:08:32.540467+0000","last_undegraded":"2024-03-26T05:08:32.540467+0000","last_fullsized":"2024-03-26T05:08:32.540467+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:55.105291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:55.105291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"98'20","reported_seq":37,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.524707+0000","last_change":"2024-03-26T05:07:59.970828+0000","last_active":"2024-03-26T05:08:32.524707+0000","last_peered":"2024-03-26T05:08:32.524707+0000","last_clean":"2024-03-26T05:08:32.524707+0000","last_became_active":"2024-03-26T05:07:59.960346+0000","last_became_peered":"2024-03-26T05:07:59.960346+0000","last_unstale":"2024-03-26T05:08:32.524707+0000","last_undegraded":"2024-03-26T05:08:32.524707+0000","last_fullsized":"2024-03-26T05:08:32.524707+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'20","last_scrub_stamp":"2024-03-26T05:06:49.846038+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:06:49.846038+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":347,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.540707+0000","last_change":"2024-03-26T05:07:47.397785+0000","last_active":"2024-03-26T05:08:32.540707+0000","last_peered":"2024-03-26T05:08:32.540707+0000","last_clean":"2024-03-26T05:08:32.540707+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:08:32.540707+0000","last_undegraded":"2024-03-26T05:08:32.540707+0000","last_fullsized":"2024-03-26T05:08:32.540707+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:47.397576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:47.397576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:48.713565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":335,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.542990+0000","last_change":"2024-03-26T05:07:55.590013+0000","last_active":"2024-03-26T05:08:32.542990+0000","last_peered":"2024-03-26T05:08:32.542990+0000","last_clean":"2024-03-26T05:08:32.542990+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:08:32.542990+0000","last_undegraded":"2024-03-26T05:08:32.542990+0000","last_fullsized":"2024-03-26T05:08:32.542990+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:55.589886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:07:55.589886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:03.886426+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":342,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532005+0000","last_change":"2024-03-26T05:07:38.695076+0000","last_active":"2024-03-26T05:08:32.532005+0000","last_peered":"2024-03-26T05:08:32.532005+0000","last_clean":"2024-03-26T05:08:32.532005+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:08:32.532005+0000","last_undegraded":"2024-03-26T05:08:32.532005+0000","last_fullsized":"2024-03-26T05:08:32.532005+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:38.694961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:38.694961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:01.889901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'21","reported_seq":380,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.541727+0000","last_change":"2024-03-26T05:05:12.090691+0000","last_active":"2024-03-26T05:08:32.541727+0000","last_peered":"2024-03-26T05:08:32.541727+0000","last_clean":"2024-03-26T05:08:32.541727+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:08:32.541727+0000","last_undegraded":"2024-03-26T05:08:32.541727+0000","last_fullsized":"2024-03-26T05:08:32.541727+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:05:12.089325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:12.089325+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":330,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532498+0000","last_change":"2024-03-26T05:05:58.845334+0000","last_active":"2024-03-26T05:08:32.532498+0000","last_peered":"2024-03-26T05:08:32.532498+0000","last_clean":"2024-03-26T05:08:32.532498+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:08:32.532498+0000","last_undegraded":"2024-03-26T05:08:32.532498+0000","last_fullsized":"2024-03-26T05:08:32.532498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:58.845212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:58.845212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.523820+0000","last_change":"2024-03-26T05:07:59.997795+0000","last_active":"2024-03-26T05:08:32.523820+0000","last_peered":"2024-03-26T05:08:32.523820+0000","last_clean":"2024-03-26T05:08:32.523820+0000","last_became_active":"2024-03-26T05:07:59.997434+0000","last_became_peered":"2024-03-26T05:07:59.997434+0000","last_unstale":"2024-03-26T05:08:32.523820+0000","last_undegraded":"2024-03-26T05:08:32.523820+0000","last_fullsized":"2024-03-26T05:08:32.523820+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:09.946734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:09.946734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":377,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530264+0000","last_change":"2024-03-26T05:08:00.002668+0000","last_active":"2024-03-26T05:08:32.530264+0000","last_peered":"2024-03-26T05:08:32.530264+0000","last_clean":"2024-03-26T05:08:32.530264+0000","last_became_active":"2024-03-26T05:07:59.992217+0000","last_became_peered":"2024-03-26T05:07:59.992217+0000","last_unstale":"2024-03-26T05:08:32.530264+0000","last_undegraded":"2024-03-26T05:08:32.530264+0000","last_fullsized":"2024-03-26T05:08:32.530264+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_clean_scrub_stamp":"2024-03-26T05:06:50.546241+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'12","reported_seq":360,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.543168+0000","last_change":"2024-03-26T05:06:42.178592+0000","last_active":"2024-03-26T05:08:32.543168+0000","last_peered":"2024-03-26T05:08:32.543168+0000","last_clean":"2024-03-26T05:08:32.543168+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:08:32.543168+0000","last_undegraded":"2024-03-26T05:08:32.543168+0000","last_fullsized":"2024-03-26T05:08:32.543168+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-26T05:06:42.178446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:42.178446+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"221'7830","reported_seq":7918,"reported_epoch":221,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-26T05:08:35.152699+0000","last_change":"2024-03-26T05:08:01.231964+0000","last_active":"2024-03-26T05:08:35.152699+0000","last_peered":"2024-03-26T05:08:35.152699+0000","last_clean":"2024-03-26T05:07:58.856211+0000","last_became_active":"2024-03-26T05:08:01.224161+0000","last_became_peered":"2024-03-26T05:08:01.224161+0000","last_unstale":"2024-03-26T05:08:35.152699+0000","last_undegraded":"2024-03-26T05:08:35.152699+0000","last_fullsized":"2024-03-26T05:08:35.152699+0000","mapping_epoch":214,"log_start":"191'1400","ondisk_log_start":"191'1400","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":6430,"log_dups_size":0,"ondisk_log_size":6430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422182912,"num_objects":6442,"num_object_clones":0,"num_object_copies":19326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6442,"num_objects_unfound":0,"num_objects_dirty":6442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7830,"num_write_kb":456704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[2,4,6],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4,6","objects":6442}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":29,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.525672+0000","last_change":"2024-03-26T05:08:00.007340+0000","last_active":"2024-03-26T05:08:32.525672+0000","last_peered":"2024-03-26T05:08:32.525672+0000","last_clean":"2024-03-26T05:08:32.525672+0000","last_became_active":"2024-03-26T05:08:00.000166+0000","last_became_peered":"2024-03-26T05:08:00.000166+0000","last_unstale":"2024-03-26T05:08:32.525672+0000","last_undegraded":"2024-03-26T05:08:32.525672+0000","last_fullsized":"2024-03-26T05:08:32.525672+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:07:53.088449+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.088449+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:20.726261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":371,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532783+0000","last_change":"2024-03-26T05:08:22.719022+0000","last_active":"2024-03-26T05:08:32.532783+0000","last_peered":"2024-03-26T05:08:32.532783+0000","last_clean":"2024-03-26T05:08:32.532783+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:08:32.532783+0000","last_undegraded":"2024-03-26T05:08:32.532783+0000","last_fullsized":"2024-03-26T05:08:32.532783+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:08:22.718767+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:05:06.832744+0000","last_clean_scrub_stamp":"2024-03-26T05:08:22.718767+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:42.372811+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'27","reported_seq":45,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.525177+0000","last_change":"2024-03-26T05:07:59.966410+0000","last_active":"2024-03-26T05:08:32.525177+0000","last_peered":"2024-03-26T05:08:32.525177+0000","last_clean":"2024-03-26T05:08:32.525177+0000","last_became_active":"2024-03-26T05:07:59.947478+0000","last_became_peered":"2024-03-26T05:07:59.947478+0000","last_unstale":"2024-03-26T05:08:32.525177+0000","last_undegraded":"2024-03-26T05:08:32.525177+0000","last_fullsized":"2024-03-26T05:08:32.525177+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'27","last_scrub_stamp":"2024-03-26T05:06:03.850202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:03.850202+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":442,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:33.491331+0000","last_change":"2024-03-26T05:05:57.829833+0000","last_active":"2024-03-26T05:08:33.491331+0000","last_peered":"2024-03-26T05:08:33.491331+0000","last_clean":"2024-03-26T05:08:33.491331+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:08:33.491331+0000","last_undegraded":"2024-03-26T05:08:33.491331+0000","last_fullsized":"2024-03-26T05:08:33.491331+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:05:57.829724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:05:57.829724+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"221'7970","reported_seq":8115,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:34.856235+0000","last_change":"2024-03-26T05:07:53.002642+0000","last_active":"2024-03-26T05:08:34.856235+0000","last_peered":"2024-03-26T05:08:34.856235+0000","last_clean":"2024-03-26T05:08:34.856235+0000","last_became_active":"2024-03-26T05:06:41.959580+0000","last_became_peered":"2024-03-26T05:06:41.959580+0000","last_unstale":"2024-03-26T05:08:34.856235+0000","last_undegraded":"2024-03-26T05:08:34.856235+0000","last_fullsized":"2024-03-26T05:08:34.856235+0000","mapping_epoch":184,"log_start":"192'1900","ondisk_log_start":"192'1900","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'5617","last_scrub_stamp":"2024-03-26T05:07:53.002409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.002409+0000","objects_scrubbed":5582,"log_size":6070,"log_dups_size":0,"ondisk_log_size":6070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:21.587055+0000","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427950080,"num_objects":6530,"num_object_clones":0,"num_object_copies":19590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7970,"num_write_kb":464000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":331,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.539270+0000","last_change":"2024-03-26T05:06:38.198437+0000","last_active":"2024-03-26T05:08:32.539270+0000","last_peered":"2024-03-26T05:08:32.539270+0000","last_clean":"2024-03-26T05:08:32.539270+0000","last_became_active":"2024-03-26T05:00:28.985917+0000","last_became_peered":"2024-03-26T05:00:28.985917+0000","last_unstale":"2024-03-26T05:08:32.539270+0000","last_undegraded":"2024-03-26T05:08:32.539270+0000","last_fullsized":"2024-03-26T05:08:32.539270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:38.198342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:38.198342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":350,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:37.706317+0000","last_change":"2024-03-26T05:08:37.706361+0000","last_active":"2024-03-26T05:08:37.706317+0000","last_peered":"2024-03-26T05:08:37.706317+0000","last_clean":"2024-03-26T05:08:37.706317+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:08:37.706317+0000","last_undegraded":"2024-03-26T05:08:37.706317+0000","last_fullsized":"2024-03-26T05:08:37.706317+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:08:37.704138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:08:37.704138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:07.264376+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"188'31","reported_seq":394,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529608+0000","last_change":"2024-03-26T05:06:50.301509+0000","last_active":"2024-03-26T05:08:32.529608+0000","last_peered":"2024-03-26T05:08:32.529608+0000","last_clean":"2024-03-26T05:08:32.529608+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:08:32.529608+0000","last_undegraded":"2024-03-26T05:08:32.529608+0000","last_fullsized":"2024-03-26T05:08:32.529608+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'31","last_scrub_stamp":"2024-03-26T05:06:50.301394+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:06:50.301394+0000","last_clean_scrub_stamp":"2024-03-26T05:06:50.301394+0000","objects_scrubbed":7,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":343,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.528039+0000","last_change":"2024-03-26T05:06:53.117526+0000","last_active":"2024-03-26T05:08:32.528039+0000","last_peered":"2024-03-26T05:08:32.528039+0000","last_clean":"2024-03-26T05:08:32.528039+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:08:32.528039+0000","last_undegraded":"2024-03-26T05:08:32.528039+0000","last_fullsized":"2024-03-26T05:08:32.528039+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:53.117385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.432732+0000","last_clean_scrub_stamp":"2024-03-26T05:06:53.117385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"221'7802","reported_seq":7886,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:35.423615+0000","last_change":"2024-03-26T05:06:41.963528+0000","last_active":"2024-03-26T05:08:35.423615+0000","last_peered":"2024-03-26T05:08:35.423615+0000","last_clean":"2024-03-26T05:08:35.423615+0000","last_became_active":"2024-03-26T05:06:41.962275+0000","last_became_peered":"2024-03-26T05:06:41.962275+0000","last_unstale":"2024-03-26T05:08:35.423615+0000","last_undegraded":"2024-03-26T05:08:35.423615+0000","last_fullsized":"2024-03-26T05:08:35.423615+0000","mapping_epoch":184,"log_start":"193'2200","ondisk_log_start":"193'2200","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417988608,"num_objects":6378,"num_object_clones":0,"num_object_copies":19134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7802,"num_write_kb":453760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":338,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:39.681596+0000","last_change":"2024-03-26T05:08:39.681640+0000","last_active":"2024-03-26T05:08:39.681596+0000","last_peered":"2024-03-26T05:08:39.681596+0000","last_clean":"2024-03-26T05:08:39.681596+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:08:39.681596+0000","last_undegraded":"2024-03-26T05:08:39.681596+0000","last_fullsized":"2024-03-26T05:08:39.681596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_clean_scrub_stamp":"2024-03-26T05:08:39.681463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:48.296075+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":363,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.533611+0000","last_change":"2024-03-26T05:08:00.000387+0000","last_active":"2024-03-26T05:08:32.533611+0000","last_peered":"2024-03-26T05:08:32.533611+0000","last_clean":"2024-03-26T05:08:32.533611+0000","last_became_active":"2024-03-26T05:08:00.000009+0000","last_became_peered":"2024-03-26T05:08:00.000009+0000","last_unstale":"2024-03-26T05:08:32.533611+0000","last_undegraded":"2024-03-26T05:08:32.533611+0000","last_fullsized":"2024-03-26T05:08:32.533611+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:46.384777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:46.384777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:06.002390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":354,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.533029+0000","last_change":"2024-03-26T05:08:18.727445+0000","last_active":"2024-03-26T05:08:32.533029+0000","last_peered":"2024-03-26T05:08:32.533029+0000","last_clean":"2024-03-26T05:08:32.533029+0000","last_became_active":"2024-03-26T05:07:59.945688+0000","last_became_peered":"2024-03-26T05:07:59.945688+0000","last_unstale":"2024-03-26T05:08:32.533029+0000","last_undegraded":"2024-03-26T05:08:32.533029+0000","last_fullsized":"2024-03-26T05:08:32.533029+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:08:18.727127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:08:18.727127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:36.748889+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'25","reported_seq":389,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.531008+0000","last_change":"2024-03-26T05:06:45.476643+0000","last_active":"2024-03-26T05:08:32.531008+0000","last_peered":"2024-03-26T05:08:32.531008+0000","last_clean":"2024-03-26T05:08:32.531008+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:08:32.531008+0000","last_undegraded":"2024-03-26T05:08:32.531008+0000","last_fullsized":"2024-03-26T05:08:32.531008+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:06:45.476460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.476460+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"221'7856","reported_seq":7940,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:35.136625+0000","last_change":"2024-03-26T05:06:41.961429+0000","last_active":"2024-03-26T05:08:35.136625+0000","last_peered":"2024-03-26T05:08:35.136625+0000","last_clean":"2024-03-26T05:08:35.136625+0000","last_became_active":"2024-03-26T05:06:41.961141+0000","last_became_peered":"2024-03-26T05:06:41.961141+0000","last_unstale":"2024-03-26T05:08:35.136625+0000","last_undegraded":"2024-03-26T05:08:35.136625+0000","last_fullsized":"2024-03-26T05:08:35.136625+0000","mapping_epoch":184,"log_start":"196'3000","ondisk_log_start":"196'3000","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424148992,"num_objects":6472,"num_object_clones":0,"num_object_copies":19416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7856,"num_write_kb":458496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":332,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.542111+0000","last_change":"2024-03-26T05:05:55.120808+0000","last_active":"2024-03-26T05:08:32.542111+0000","last_peered":"2024-03-26T05:08:32.542111+0000","last_clean":"2024-03-26T05:08:32.542111+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:08:32.542111+0000","last_undegraded":"2024-03-26T05:08:32.542111+0000","last_fullsized":"2024-03-26T05:08:32.542111+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:55.120731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:55.120731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":343,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.533806+0000","last_change":"2024-03-26T05:06:51.161774+0000","last_active":"2024-03-26T05:08:32.533806+0000","last_peered":"2024-03-26T05:08:32.533806+0000","last_clean":"2024-03-26T05:08:32.533806+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:08:32.533806+0000","last_undegraded":"2024-03-26T05:08:32.533806+0000","last_fullsized":"2024-03-26T05:08:32.533806+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:51.161669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:51.161669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.524564+0000","last_change":"2024-03-26T05:07:59.960272+0000","last_active":"2024-03-26T05:08:32.524564+0000","last_peered":"2024-03-26T05:08:32.524564+0000","last_clean":"2024-03-26T05:08:32.524564+0000","last_became_active":"2024-03-26T05:07:59.959978+0000","last_became_peered":"2024-03-26T05:07:59.959978+0000","last_unstale":"2024-03-26T05:08:32.524564+0000","last_undegraded":"2024-03-26T05:08:32.524564+0000","last_fullsized":"2024-03-26T05:08:32.524564+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:01.867674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:01.867674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'15","reported_seq":368,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.541145+0000","last_change":"2024-03-26T05:05:42.260332+0000","last_active":"2024-03-26T05:08:32.541145+0000","last_peered":"2024-03-26T05:08:32.541145+0000","last_clean":"2024-03-26T05:08:32.541145+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:08:32.541145+0000","last_undegraded":"2024-03-26T05:08:32.541145+0000","last_fullsized":"2024-03-26T05:08:32.541145+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:05:42.260202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:05:42.260202+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"221'7968","reported_seq":16240,"reported_epoch":221,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T05:08:35.576936+0000","last_change":"2024-03-26T05:08:01.216724+0000","last_active":"2024-03-26T05:08:35.576936+0000","last_peered":"2024-03-26T05:08:35.576936+0000","last_clean":"2024-03-26T05:07:58.893035+0000","last_became_active":"2024-03-26T05:08:01.210408+0000","last_became_peered":"2024-03-26T05:08:01.210408+0000","last_unstale":"2024-03-26T05:08:35.576936+0000","last_undegraded":"2024-03-26T05:08:35.576936+0000","last_fullsized":"2024-03-26T05:08:35.576936+0000","mapping_epoch":214,"log_start":"194'2400","ondisk_log_start":"194'2400","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426770432,"num_objects":6512,"num_object_clones":0,"num_object_copies":19536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2543,"num_objects_unfound":0,"num_objects_dirty":6512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7968,"num_write_kb":463360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4093,"num_bytes_recovered":268238848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":6512}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":331,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.536959+0000","last_change":"2024-03-26T05:07:48.635847+0000","last_active":"2024-03-26T05:08:32.536959+0000","last_peered":"2024-03-26T05:08:32.536959+0000","last_clean":"2024-03-26T05:08:32.536959+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:08:32.536959+0000","last_undegraded":"2024-03-26T05:08:32.536959+0000","last_fullsized":"2024-03-26T05:08:32.536959+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:48.635728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:48.635728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:11.079741+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":29,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.525721+0000","last_change":"2024-03-26T05:08:00.004448+0000","last_active":"2024-03-26T05:08:32.525721+0000","last_peered":"2024-03-26T05:08:32.525721+0000","last_clean":"2024-03-26T05:08:32.525721+0000","last_became_active":"2024-03-26T05:07:59.998511+0000","last_became_peered":"2024-03-26T05:07:59.998511+0000","last_unstale":"2024-03-26T05:08:32.525721+0000","last_undegraded":"2024-03-26T05:08:32.525721+0000","last_fullsized":"2024-03-26T05:08:32.525721+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:07:37.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:07:37.347513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:41.156229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":336,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529386+0000","last_change":"2024-03-26T05:05:46.234486+0000","last_active":"2024-03-26T05:08:32.529386+0000","last_peered":"2024-03-26T05:08:32.529386+0000","last_clean":"2024-03-26T05:08:32.529386+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:08:32.529386+0000","last_undegraded":"2024-03-26T05:08:32.529386+0000","last_fullsized":"2024-03-26T05:08:32.529386+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:46.234374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:46.234374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'34","reported_seq":506,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529708+0000","last_change":"2024-03-26T05:06:54.128746+0000","last_active":"2024-03-26T05:08:32.529708+0000","last_peered":"2024-03-26T05:08:32.529708+0000","last_clean":"2024-03-26T05:08:32.529708+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:08:32.529708+0000","last_undegraded":"2024-03-26T05:08:32.529708+0000","last_fullsized":"2024-03-26T05:08:32.529708+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'34","last_scrub_stamp":"2024-03-26T05:06:54.128603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:54.128603+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":336,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.540694+0000","last_change":"2024-03-26T05:06:05.029804+0000","last_active":"2024-03-26T05:08:32.540694+0000","last_peered":"2024-03-26T05:08:32.540694+0000","last_clean":"2024-03-26T05:08:32.540694+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:08:32.540694+0000","last_undegraded":"2024-03-26T05:08:32.540694+0000","last_fullsized":"2024-03-26T05:08:32.540694+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:05.029701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:05.029701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"221'7678","reported_seq":7762,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:35.434361+0000","last_change":"2024-03-26T05:06:41.964329+0000","last_active":"2024-03-26T05:08:35.434361+0000","last_peered":"2024-03-26T05:08:35.434361+0000","last_clean":"2024-03-26T05:08:35.434361+0000","last_became_active":"2024-03-26T05:06:41.963962+0000","last_became_peered":"2024-03-26T05:06:41.963962+0000","last_unstale":"2024-03-26T05:08:35.434361+0000","last_undegraded":"2024-03-26T05:08:35.434361+0000","last_fullsized":"2024-03-26T05:08:35.434361+0000","mapping_epoch":184,"log_start":"196'2900","ondisk_log_start":"196'2900","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413794304,"num_objects":6314,"num_object_clones":0,"num_object_copies":18942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7678,"num_write_kb":447744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":338,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.533118+0000","last_change":"2024-03-26T05:08:31.662655+0000","last_active":"2024-03-26T05:08:32.533118+0000","last_peered":"2024-03-26T05:08:32.533118+0000","last_clean":"2024-03-26T05:08:32.533118+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:08:32.533118+0000","last_undegraded":"2024-03-26T05:08:32.533118+0000","last_fullsized":"2024-03-26T05:08:32.533118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:08:31.662549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:08:31.662549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:36.141851+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":344,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.528914+0000","last_change":"2024-03-26T05:06:49.178919+0000","last_active":"2024-03-26T05:08:32.528914+0000","last_peered":"2024-03-26T05:08:32.528914+0000","last_clean":"2024-03-26T05:08:32.528914+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:08:32.528914+0000","last_undegraded":"2024-03-26T05:08:32.528914+0000","last_fullsized":"2024-03-26T05:08:32.528914+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:49.178823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:49.178823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":360,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.527927+0000","last_change":"2024-03-26T05:06:41.303481+0000","last_active":"2024-03-26T05:08:32.527927+0000","last_peered":"2024-03-26T05:08:32.527927+0000","last_clean":"2024-03-26T05:08:32.527927+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:08:32.527927+0000","last_undegraded":"2024-03-26T05:08:32.527927+0000","last_fullsized":"2024-03-26T05:08:32.527927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:41.303395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:06:41.303395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'19","reported_seq":160,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.533187+0000","last_change":"2024-03-26T05:08:00.019227+0000","last_active":"2024-03-26T05:08:32.533187+0000","last_peered":"2024-03-26T05:08:32.533187+0000","last_clean":"2024-03-26T05:08:32.533187+0000","last_became_active":"2024-03-26T05:08:00.001398+0000","last_became_peered":"2024-03-26T05:08:00.001398+0000","last_unstale":"2024-03-26T05:08:32.533187+0000","last_undegraded":"2024-03-26T05:08:32.533187+0000","last_fullsized":"2024-03-26T05:08:32.533187+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-26T05:07:54.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:54.529843+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:58.519685+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":26,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":363,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.531688+0000","last_change":"2024-03-26T05:07:59.955942+0000","last_active":"2024-03-26T05:08:32.531688+0000","last_peered":"2024-03-26T05:08:32.531688+0000","last_clean":"2024-03-26T05:08:32.531688+0000","last_became_active":"2024-03-26T05:07:59.955456+0000","last_became_peered":"2024-03-26T05:07:59.955456+0000","last_unstale":"2024-03-26T05:08:32.531688+0000","last_undegraded":"2024-03-26T05:08:32.531688+0000","last_fullsized":"2024-03-26T05:08:32.531688+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:44.363612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:44.363612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:50.872899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"221'7840","reported_seq":7993,"reported_epoch":221,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-26T05:08:34.841684+0000","last_change":"2024-03-26T05:08:01.239534+0000","last_active":"2024-03-26T05:08:34.841684+0000","last_peered":"2024-03-26T05:08:34.841684+0000","last_clean":"2024-03-26T05:07:58.873926+0000","last_became_active":"2024-03-26T05:08:01.233806+0000","last_became_peered":"2024-03-26T05:08:01.233806+0000","last_unstale":"2024-03-26T05:08:34.841684+0000","last_undegraded":"2024-03-26T05:08:34.841684+0000","last_fullsized":"2024-03-26T05:08:34.841684+0000","mapping_epoch":214,"log_start":"192'1800","ondisk_log_start":"192'1800","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'5439","last_scrub_stamp":"2024-03-26T05:07:50.296391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:07:50.296391+0000","objects_scrubbed":5392,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429391872,"num_objects":6552,"num_object_clones":0,"num_object_copies":19656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6552,"num_objects_unfound":0,"num_objects_dirty":6552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7840,"num_write_kb":460544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,0,3],"avail_no_missing":["6","0","3"],"object_location_counts":[{"shards":"0,3,6","objects":6552}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":25,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.524452+0000","last_change":"2024-03-26T05:07:59.960360+0000","last_active":"2024-03-26T05:08:32.524452+0000","last_peered":"2024-03-26T05:08:32.524452+0000","last_clean":"2024-03-26T05:08:32.524452+0000","last_became_active":"2024-03-26T05:07:59.959853+0000","last_became_peered":"2024-03-26T05:07:59.959853+0000","last_unstale":"2024-03-26T05:08:32.524452+0000","last_undegraded":"2024-03-26T05:08:32.524452+0000","last_fullsized":"2024-03-26T05:08:32.524452+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_clean_scrub_stamp":"2024-03-26T05:05:03.848208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":598,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.531931+0000","last_change":"2024-03-26T05:06:45.717598+0000","last_active":"2024-03-26T05:08:32.531931+0000","last_peered":"2024-03-26T05:08:32.531931+0000","last_clean":"2024-03-26T05:08:32.531931+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:08:32.531931+0000","last_undegraded":"2024-03-26T05:08:32.531931+0000","last_fullsized":"2024-03-26T05:08:32.531931+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:06:45.717474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.717474+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":355,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.535171+0000","last_change":"2024-03-26T05:07:59.960407+0000","last_active":"2024-03-26T05:08:32.535171+0000","last_peered":"2024-03-26T05:08:32.535171+0000","last_clean":"2024-03-26T05:08:32.535171+0000","last_became_active":"2024-03-26T05:07:59.960061+0000","last_became_peered":"2024-03-26T05:07:59.960061+0000","last_unstale":"2024-03-26T05:08:32.535171+0000","last_undegraded":"2024-03-26T05:08:32.535171+0000","last_fullsized":"2024-03-26T05:08:32.535171+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:07.743708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:07.743708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'24","reported_seq":483,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.531806+0000","last_change":"2024-03-26T05:07:32.005638+0000","last_active":"2024-03-26T05:08:32.531806+0000","last_peered":"2024-03-26T05:08:32.531806+0000","last_clean":"2024-03-26T05:08:32.531806+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:08:32.531806+0000","last_undegraded":"2024-03-26T05:08:32.531806+0000","last_fullsized":"2024-03-26T05:08:32.531806+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'24","last_scrub_stamp":"2024-03-26T05:07:32.005486+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:07:32.005486+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:40.945818+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":354,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532694+0000","last_change":"2024-03-26T05:07:59.957053+0000","last_active":"2024-03-26T05:08:32.532694+0000","last_peered":"2024-03-26T05:08:32.532694+0000","last_clean":"2024-03-26T05:08:32.532694+0000","last_became_active":"2024-03-26T05:07:59.956080+0000","last_became_peered":"2024-03-26T05:07:59.956080+0000","last_unstale":"2024-03-26T05:08:32.532694+0000","last_undegraded":"2024-03-26T05:08:32.532694+0000","last_fullsized":"2024-03-26T05:08:32.532694+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:29.669034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:29.669034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:47.793919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"221'7764","reported_seq":15900,"reported_epoch":221,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T05:08:35.185014+0000","last_change":"2024-03-26T05:08:01.241681+0000","last_active":"2024-03-26T05:08:35.185014+0000","last_peered":"2024-03-26T05:08:35.185014+0000","last_clean":"2024-03-26T05:07:58.913602+0000","last_became_active":"2024-03-26T05:08:01.226563+0000","last_became_peered":"2024-03-26T05:08:01.226563+0000","last_unstale":"2024-03-26T05:08:35.185014+0000","last_undegraded":"2024-03-26T05:08:35.185014+0000","last_fullsized":"2024-03-26T05:08:35.185014+0000","mapping_epoch":214,"log_start":"196'2900","ondisk_log_start":"196'2900","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423100416,"num_objects":6456,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2498,"num_objects_unfound":0,"num_objects_dirty":6456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7764,"num_write_kb":455040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4023,"num_bytes_recovered":263651328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":6456}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":351,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529374+0000","last_change":"2024-03-26T05:08:00.002985+0000","last_active":"2024-03-26T05:08:32.529374+0000","last_peered":"2024-03-26T05:08:32.529374+0000","last_clean":"2024-03-26T05:08:32.529374+0000","last_became_active":"2024-03-26T05:08:00.001657+0000","last_became_peered":"2024-03-26T05:08:00.001657+0000","last_unstale":"2024-03-26T05:08:32.529374+0000","last_undegraded":"2024-03-26T05:08:32.529374+0000","last_fullsized":"2024-03-26T05:08:32.529374+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:39.338092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:39.338092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:40.968270+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":240,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529477+0000","last_change":"2024-03-26T05:07:59.992807+0000","last_active":"2024-03-26T05:08:32.529477+0000","last_peered":"2024-03-26T05:08:32.529477+0000","last_clean":"2024-03-26T05:08:32.529477+0000","last_became_active":"2024-03-26T05:07:59.992489+0000","last_became_peered":"2024-03-26T05:07:59.992489+0000","last_unstale":"2024-03-26T05:08:32.529477+0000","last_undegraded":"2024-03-26T05:08:32.529477+0000","last_fullsized":"2024-03-26T05:08:32.529477+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:39.074501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:39.074501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":855,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:38.752258+0000","last_change":"2024-03-26T05:08:33.643027+0000","last_active":"2024-03-26T05:08:38.752258+0000","last_peered":"2024-03-26T05:08:38.752258+0000","last_clean":"2024-03-26T05:08:38.752258+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:08:38.752258+0000","last_undegraded":"2024-03-26T05:08:38.752258+0000","last_fullsized":"2024-03-26T05:08:38.752258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:08:33.642796+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:08:33.642796+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:45.983969+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":491,"num_read_kb":486,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"187'18","reported_seq":404,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.540960+0000","last_change":"2024-03-26T05:06:49.027873+0000","last_active":"2024-03-26T05:08:32.540960+0000","last_peered":"2024-03-26T05:08:32.540960+0000","last_clean":"2024-03-26T05:08:32.540960+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:08:32.540960+0000","last_undegraded":"2024-03-26T05:08:32.540960+0000","last_fullsized":"2024-03-26T05:08:32.540960+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'18","last_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_deep_scrub":"187'18","last_deep_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_clean_scrub_stamp":"2024-03-26T05:06:49.027756+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.524379+0000","last_change":"2024-03-26T05:08:00.000428+0000","last_active":"2024-03-26T05:08:32.524379+0000","last_peered":"2024-03-26T05:08:32.524379+0000","last_clean":"2024-03-26T05:08:32.524379+0000","last_became_active":"2024-03-26T05:07:59.999990+0000","last_became_peered":"2024-03-26T05:07:59.999990+0000","last_unstale":"2024-03-26T05:08:32.524379+0000","last_undegraded":"2024-03-26T05:08:32.524379+0000","last_fullsized":"2024-03-26T05:08:32.524379+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:44.270594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:44.270594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"221'7834","reported_seq":7916,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:35.446785+0000","last_change":"2024-03-26T05:06:42.024050+0000","last_active":"2024-03-26T05:08:35.446785+0000","last_peered":"2024-03-26T05:08:35.446785+0000","last_clean":"2024-03-26T05:08:35.446785+0000","last_became_active":"2024-03-26T05:06:42.023726+0000","last_became_peered":"2024-03-26T05:06:42.023726+0000","last_unstale":"2024-03-26T05:08:35.446785+0000","last_undegraded":"2024-03-26T05:08:35.446785+0000","last_fullsized":"2024-03-26T05:08:35.446785+0000","mapping_epoch":184,"log_start":"196'3100","ondisk_log_start":"196'3100","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417595392,"num_objects":6372,"num_object_clones":0,"num_object_copies":19116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7834,"num_write_kb":454592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":331,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529759+0000","last_change":"2024-03-26T05:06:01.073386+0000","last_active":"2024-03-26T05:08:32.529759+0000","last_peered":"2024-03-26T05:08:32.529759+0000","last_clean":"2024-03-26T05:08:32.529759+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:08:32.529759+0000","last_undegraded":"2024-03-26T05:08:32.529759+0000","last_fullsized":"2024-03-26T05:08:32.529759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:01.073272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:06:01.073272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":355,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529576+0000","last_change":"2024-03-26T05:08:00.004675+0000","last_active":"2024-03-26T05:08:32.529576+0000","last_peered":"2024-03-26T05:08:32.529576+0000","last_clean":"2024-03-26T05:08:32.529576+0000","last_became_active":"2024-03-26T05:08:00.003099+0000","last_became_peered":"2024-03-26T05:08:00.003099+0000","last_unstale":"2024-03-26T05:08:32.529576+0000","last_undegraded":"2024-03-26T05:08:32.529576+0000","last_fullsized":"2024-03-26T05:08:32.529576+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:05.892241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:05:05.892241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'25","reported_seq":448,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.540891+0000","last_change":"2024-03-26T05:08:00.021144+0000","last_active":"2024-03-26T05:08:32.540891+0000","last_peered":"2024-03-26T05:08:32.540891+0000","last_clean":"2024-03-26T05:08:32.540891+0000","last_became_active":"2024-03-26T05:07:59.994226+0000","last_became_peered":"2024-03-26T05:07:59.994226+0000","last_unstale":"2024-03-26T05:08:32.540891+0000","last_undegraded":"2024-03-26T05:08:32.540891+0000","last_fullsized":"2024-03-26T05:08:32.540891+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:06:52.015444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:52.015444+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":354,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.531098+0000","last_change":"2024-03-26T05:07:59.991824+0000","last_active":"2024-03-26T05:08:32.531098+0000","last_peered":"2024-03-26T05:08:32.531098+0000","last_clean":"2024-03-26T05:08:32.531098+0000","last_became_active":"2024-03-26T05:07:59.991484+0000","last_became_peered":"2024-03-26T05:07:59.991484+0000","last_unstale":"2024-03-26T05:08:32.531098+0000","last_undegraded":"2024-03-26T05:08:32.531098+0000","last_fullsized":"2024-03-26T05:08:32.531098+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:35.020060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:07:35.020060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:40.014473+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"221'7857","reported_seq":7941,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:35.476161+0000","last_change":"2024-03-26T05:06:42.026870+0000","last_active":"2024-03-26T05:08:35.476161+0000","last_peered":"2024-03-26T05:08:35.476161+0000","last_clean":"2024-03-26T05:08:35.476161+0000","last_became_active":"2024-03-26T05:06:42.026460+0000","last_became_peered":"2024-03-26T05:06:42.026460+0000","last_unstale":"2024-03-26T05:08:35.476161+0000","last_undegraded":"2024-03-26T05:08:35.476161+0000","last_fullsized":"2024-03-26T05:08:35.476161+0000","mapping_epoch":184,"log_start":"193'2300","ondisk_log_start":"193'2300","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418709504,"num_objects":6389,"num_object_clones":0,"num_object_copies":19167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":455872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":333,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.540332+0000","last_change":"2024-03-26T05:05:52.123187+0000","last_active":"2024-03-26T05:08:32.540332+0000","last_peered":"2024-03-26T05:08:32.540332+0000","last_clean":"2024-03-26T05:08:32.540332+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:08:32.540332+0000","last_undegraded":"2024-03-26T05:08:32.540332+0000","last_fullsized":"2024-03-26T05:08:32.540332+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:52.123084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:05:52.123084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.524797+0000","last_change":"2024-03-26T05:07:59.961574+0000","last_active":"2024-03-26T05:08:32.524797+0000","last_peered":"2024-03-26T05:08:32.524797+0000","last_clean":"2024-03-26T05:08:32.524797+0000","last_became_active":"2024-03-26T05:07:59.960768+0000","last_became_peered":"2024-03-26T05:07:59.960768+0000","last_unstale":"2024-03-26T05:08:32.524797+0000","last_undegraded":"2024-03-26T05:08:32.524797+0000","last_fullsized":"2024-03-26T05:08:32.524797+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:05:07.893841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:05:07.893841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'16","reported_seq":377,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.534027+0000","last_change":"2024-03-26T05:07:59.970786+0000","last_active":"2024-03-26T05:08:32.534027+0000","last_peered":"2024-03-26T05:08:32.534027+0000","last_clean":"2024-03-26T05:08:32.534027+0000","last_became_active":"2024-03-26T05:07:59.960351+0000","last_became_peered":"2024-03-26T05:07:59.960351+0000","last_unstale":"2024-03-26T05:08:32.534027+0000","last_undegraded":"2024-03-26T05:08:32.534027+0000","last_fullsized":"2024-03-26T05:08:32.534027+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:06:47.166607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:47.166607+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"216'11","reported_seq":473,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.541302+0000","last_change":"2024-03-26T05:08:00.008273+0000","last_active":"2024-03-26T05:08:32.541302+0000","last_peered":"2024-03-26T05:08:32.541302+0000","last_clean":"2024-03-26T05:08:32.541302+0000","last_became_active":"2024-03-26T05:07:59.994682+0000","last_became_peered":"2024-03-26T05:07:59.994682+0000","last_unstale":"2024-03-26T05:08:32.541302+0000","last_undegraded":"2024-03-26T05:08:32.541302+0000","last_fullsized":"2024-03-26T05:08:32.541302+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'10","last_scrub_stamp":"2024-03-26T05:07:38.023111+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:07:38.023111+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:59.777944+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"221'7831","reported_seq":7926,"reported_epoch":221,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T05:08:35.168129+0000","last_change":"2024-03-26T05:08:31.239812+0000","last_active":"2024-03-26T05:08:35.168129+0000","last_peered":"2024-03-26T05:08:35.168129+0000","last_clean":"2024-03-26T05:07:58.904622+0000","last_became_active":"2024-03-26T05:08:01.222510+0000","last_became_peered":"2024-03-26T05:08:01.222510+0000","last_unstale":"2024-03-26T05:08:35.168129+0000","last_undegraded":"2024-03-26T05:08:35.168129+0000","last_fullsized":"2024-03-26T05:08:35.168129+0000","mapping_epoch":214,"log_start":"191'1400","ondisk_log_start":"191'1400","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":6431,"log_dups_size":0,"ondisk_log_size":6431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413270040,"num_objects":6307,"num_object_clones":0,"num_object_copies":18921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6307,"num_objects_unfound":0,"num_objects_dirty":6307,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7831,"num_write_kb":452353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[2,7,6],"avail_no_missing":["2","6","7"],"object_location_counts":[{"shards":"2,6,7","objects":6307}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":330,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532587+0000","last_change":"2024-03-26T05:06:43.743986+0000","last_active":"2024-03-26T05:08:32.532587+0000","last_peered":"2024-03-26T05:08:32.532587+0000","last_clean":"2024-03-26T05:08:32.532587+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:08:32.532587+0000","last_undegraded":"2024-03-26T05:08:32.532587+0000","last_fullsized":"2024-03-26T05:08:32.532587+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:43.743837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:43.743837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":353,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.541060+0000","last_change":"2024-03-26T05:06:45.074041+0000","last_active":"2024-03-26T05:08:32.541060+0000","last_peered":"2024-03-26T05:08:32.541060+0000","last_clean":"2024-03-26T05:08:32.541060+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:08:32.541060+0000","last_undegraded":"2024-03-26T05:08:32.541060+0000","last_fullsized":"2024-03-26T05:08:32.541060+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:06:45.073912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.073912+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'20","reported_seq":421,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529340+0000","last_change":"2024-03-26T05:07:59.958885+0000","last_active":"2024-03-26T05:08:32.529340+0000","last_peered":"2024-03-26T05:08:32.529340+0000","last_clean":"2024-03-26T05:08:32.529340+0000","last_became_active":"2024-03-26T05:07:59.942510+0000","last_became_peered":"2024-03-26T05:07:59.942510+0000","last_unstale":"2024-03-26T05:08:32.529340+0000","last_undegraded":"2024-03-26T05:08:32.529340+0000","last_fullsized":"2024-03-26T05:08:32.529340+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:06:42.311073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:42.311073+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":443,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:33.505114+0000","last_change":"2024-03-26T05:07:53.583159+0000","last_active":"2024-03-26T05:08:33.505114+0000","last_peered":"2024-03-26T05:08:33.505114+0000","last_clean":"2024-03-26T05:08:33.505114+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:08:33.505114+0000","last_undegraded":"2024-03-26T05:08:33.505114+0000","last_fullsized":"2024-03-26T05:08:33.505114+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:07:53.583016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.583016+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:10.191268+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"221'8044","reported_seq":8128,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:35.451861+0000","last_change":"2024-03-26T05:06:41.965131+0000","last_active":"2024-03-26T05:08:35.451861+0000","last_peered":"2024-03-26T05:08:35.451861+0000","last_clean":"2024-03-26T05:08:35.451861+0000","last_became_active":"2024-03-26T05:06:41.964810+0000","last_became_peered":"2024-03-26T05:06:41.964810+0000","last_unstale":"2024-03-26T05:08:35.451861+0000","last_undegraded":"2024-03-26T05:08:35.451861+0000","last_fullsized":"2024-03-26T05:08:35.451861+0000","mapping_epoch":184,"log_start":"194'2400","ondisk_log_start":"194'2400","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":5644,"log_dups_size":0,"ondisk_log_size":5644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435552256,"num_objects":6646,"num_object_clones":0,"num_object_copies":19938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8044,"num_write_kb":470080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":340,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530538+0000","last_change":"2024-03-26T05:07:50.818124+0000","last_active":"2024-03-26T05:08:32.530538+0000","last_peered":"2024-03-26T05:08:32.530538+0000","last_clean":"2024-03-26T05:08:32.530538+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:08:32.530538+0000","last_undegraded":"2024-03-26T05:08:32.530538+0000","last_fullsized":"2024-03-26T05:08:32.530538+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:50.818000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:50.818000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:17.505115+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":335,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.542771+0000","last_change":"2024-03-26T05:04:54.805782+0000","last_active":"2024-03-26T05:08:32.542771+0000","last_peered":"2024-03-26T05:08:32.542771+0000","last_clean":"2024-03-26T05:08:32.542771+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:08:32.542771+0000","last_undegraded":"2024-03-26T05:08:32.542771+0000","last_fullsized":"2024-03-26T05:08:32.542771+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:04:54.805655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:04:54.805655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"96'40","reported_seq":406,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.537864+0000","last_change":"2024-03-26T05:05:13.704632+0000","last_active":"2024-03-26T05:08:32.537864+0000","last_peered":"2024-03-26T05:08:32.537864+0000","last_clean":"2024-03-26T05:08:32.537864+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:08:32.537864+0000","last_undegraded":"2024-03-26T05:08:32.537864+0000","last_fullsized":"2024-03-26T05:08:32.537864+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:05:13.704484+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:05:13.704484+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":336,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529117+0000","last_change":"2024-03-26T05:07:51.790409+0000","last_active":"2024-03-26T05:08:32.529117+0000","last_peered":"2024-03-26T05:08:32.529117+0000","last_clean":"2024-03-26T05:08:32.529117+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:08:32.529117+0000","last_undegraded":"2024-03-26T05:08:32.529117+0000","last_fullsized":"2024-03-26T05:08:32.529117+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:51.790277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:51.790277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:57.763766+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"221'7979","reported_seq":8072,"reported_epoch":221,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-03-26T05:08:35.103992+0000","last_change":"2024-03-26T05:08:31.256113+0000","last_active":"2024-03-26T05:08:35.103992+0000","last_peered":"2024-03-26T05:08:35.103992+0000","last_clean":"2024-03-26T05:07:58.919958+0000","last_became_active":"2024-03-26T05:08:01.237374+0000","last_became_peered":"2024-03-26T05:08:01.237374+0000","last_unstale":"2024-03-26T05:08:35.103992+0000","last_undegraded":"2024-03-26T05:08:35.103992+0000","last_fullsized":"2024-03-26T05:08:35.103992+0000","mapping_epoch":214,"log_start":"196'3100","ondisk_log_start":"196'3100","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4879,"log_dups_size":0,"ondisk_log_size":4879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430112768,"num_objects":6563,"num_object_clones":0,"num_object_copies":19689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6563,"num_objects_unfound":0,"num_objects_dirty":6563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7979,"num_write_kb":465344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,6,2],"avail_no_missing":["1","2","6"],"object_location_counts":[{"shards":"1,2,6","objects":6563}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":25,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.523979+0000","last_change":"2024-03-26T05:08:00.006006+0000","last_active":"2024-03-26T05:08:32.523979+0000","last_peered":"2024-03-26T05:08:32.523979+0000","last_clean":"2024-03-26T05:08:32.523979+0000","last_became_active":"2024-03-26T05:08:00.005676+0000","last_became_peered":"2024-03-26T05:08:00.005676+0000","last_unstale":"2024-03-26T05:08:32.523979+0000","last_undegraded":"2024-03-26T05:08:32.523979+0000","last_fullsized":"2024-03-26T05:08:32.523979+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_clean_scrub_stamp":"2024-03-26T05:07:52.465619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:06.467059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"216'10","reported_seq":465,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:33.491303+0000","last_change":"2024-03-26T05:08:00.009563+0000","last_active":"2024-03-26T05:08:33.491303+0000","last_peered":"2024-03-26T05:08:33.491303+0000","last_clean":"2024-03-26T05:08:33.491303+0000","last_became_active":"2024-03-26T05:07:59.997429+0000","last_became_peered":"2024-03-26T05:07:59.997429+0000","last_unstale":"2024-03-26T05:08:33.491303+0000","last_undegraded":"2024-03-26T05:08:33.491303+0000","last_fullsized":"2024-03-26T05:08:33.491303+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'9","last_scrub_stamp":"2024-03-26T05:06:10.114267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:06:10.114267+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":343,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.537766+0000","last_change":"2024-03-26T05:06:48.223737+0000","last_active":"2024-03-26T05:08:32.537766+0000","last_peered":"2024-03-26T05:08:32.537766+0000","last_clean":"2024-03-26T05:08:32.537766+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:08:32.537766+0000","last_undegraded":"2024-03-26T05:08:32.537766+0000","last_fullsized":"2024-03-26T05:08:32.537766+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:48.223661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:48.223661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":372,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.531382+0000","last_change":"2024-03-26T05:06:57.114810+0000","last_active":"2024-03-26T05:08:32.531382+0000","last_peered":"2024-03-26T05:08:32.531382+0000","last_clean":"2024-03-26T05:08:32.531382+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:08:32.531382+0000","last_undegraded":"2024-03-26T05:08:32.531382+0000","last_fullsized":"2024-03-26T05:08:32.531382+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:06:57.113884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:57.113884+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"221'7966","reported_seq":8054,"reported_epoch":221,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-26T05:08:39.733601+0000","last_change":"2024-03-26T05:08:01.243493+0000","last_active":"2024-03-26T05:08:39.733601+0000","last_peered":"2024-03-26T05:08:39.733601+0000","last_clean":"2024-03-26T05:07:58.913423+0000","last_became_active":"2024-03-26T05:08:01.235567+0000","last_became_peered":"2024-03-26T05:08:01.235567+0000","last_unstale":"2024-03-26T05:08:39.733601+0000","last_undegraded":"2024-03-26T05:08:39.733601+0000","last_fullsized":"2024-03-26T05:08:39.733601+0000","mapping_epoch":214,"log_start":"196'3000","ondisk_log_start":"196'3000","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4966,"log_dups_size":0,"ondisk_log_size":4966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404226048,"num_objects":6168,"num_object_clones":0,"num_object_copies":18504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6168,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7966,"num_write_kb":452288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,3],"avail_no_missing":["4","0","3"],"object_location_counts":[{"shards":"0,3,4","objects":6168}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":343,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.534730+0000","last_change":"2024-03-26T05:07:59.958531+0000","last_active":"2024-03-26T05:08:32.534730+0000","last_peered":"2024-03-26T05:08:32.534730+0000","last_clean":"2024-03-26T05:08:32.534730+0000","last_became_active":"2024-03-26T05:07:59.957995+0000","last_became_peered":"2024-03-26T05:07:59.957995+0000","last_unstale":"2024-03-26T05:08:32.534730+0000","last_undegraded":"2024-03-26T05:08:32.534730+0000","last_fullsized":"2024-03-26T05:08:32.534730+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:45.571452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:45.571452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:50.198392+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.524297+0000","last_change":"2024-03-26T05:07:59.948077+0000","last_active":"2024-03-26T05:08:32.524297+0000","last_peered":"2024-03-26T05:08:32.524297+0000","last_clean":"2024-03-26T05:08:32.524297+0000","last_became_active":"2024-03-26T05:07:59.947747+0000","last_became_peered":"2024-03-26T05:07:59.947747+0000","last_unstale":"2024-03-26T05:08:32.524297+0000","last_undegraded":"2024-03-26T05:08:32.524297+0000","last_fullsized":"2024-03-26T05:08:32.524297+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:56.446060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.446060+0000","last_clean_scrub_stamp":"2024-03-26T05:06:56.446060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":352,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529203+0000","last_change":"2024-03-26T05:07:49.831449+0000","last_active":"2024-03-26T05:08:32.529203+0000","last_peered":"2024-03-26T05:08:32.529203+0000","last_clean":"2024-03-26T05:08:32.529203+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:08:32.529203+0000","last_undegraded":"2024-03-26T05:08:32.529203+0000","last_fullsized":"2024-03-26T05:08:32.529203+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:49.831323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:49.831323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:19.249915+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'17","reported_seq":379,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530205+0000","last_change":"2024-03-26T05:07:53.755669+0000","last_active":"2024-03-26T05:08:32.530205+0000","last_peered":"2024-03-26T05:08:32.530205+0000","last_clean":"2024-03-26T05:08:32.530205+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:08:32.530205+0000","last_undegraded":"2024-03-26T05:08:32.530205+0000","last_fullsized":"2024-03-26T05:08:32.530205+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:07:53.755536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.755536+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:09:00.492337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"221'7933","reported_seq":8016,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:35.188686+0000","last_change":"2024-03-26T05:06:42.023927+0000","last_active":"2024-03-26T05:08:35.188686+0000","last_peered":"2024-03-26T05:08:35.188686+0000","last_clean":"2024-03-26T05:08:35.188686+0000","last_became_active":"2024-03-26T05:06:42.022709+0000","last_became_peered":"2024-03-26T05:06:42.022709+0000","last_unstale":"2024-03-26T05:08:35.188686+0000","last_undegraded":"2024-03-26T05:08:35.188686+0000","last_fullsized":"2024-03-26T05:08:35.188686+0000","mapping_epoch":184,"log_start":"196'3000","ondisk_log_start":"196'3000","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426967040,"num_objects":6515,"num_object_clones":0,"num_object_copies":19545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":462336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":339,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532928+0000","last_change":"2024-03-26T05:07:38.496117+0000","last_active":"2024-03-26T05:08:32.532928+0000","last_peered":"2024-03-26T05:08:32.532928+0000","last_clean":"2024-03-26T05:08:32.532928+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:08:32.532928+0000","last_undegraded":"2024-03-26T05:08:32.532928+0000","last_fullsized":"2024-03-26T05:08:32.532928+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:38.495883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:38.495883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:51.825055+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":335,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.529823+0000","last_change":"2024-03-26T05:06:02.997794+0000","last_active":"2024-03-26T05:08:32.529823+0000","last_peered":"2024-03-26T05:08:32.529823+0000","last_clean":"2024-03-26T05:08:32.529823+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:08:32.529823+0000","last_undegraded":"2024-03-26T05:08:32.529823+0000","last_fullsized":"2024-03-26T05:08:32.529823+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:02.997671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:02.997671+0000","last_clean_scrub_stamp":"2024-03-26T05:06:02.997671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":364,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530853+0000","last_change":"2024-03-26T05:07:35.999912+0000","last_active":"2024-03-26T05:08:32.530853+0000","last_peered":"2024-03-26T05:08:32.530853+0000","last_clean":"2024-03-26T05:08:32.530853+0000","last_became_active":"2024-03-26T05:06:06.521150+0000","last_became_peered":"2024-03-26T05:06:06.521150+0000","last_unstale":"2024-03-26T05:08:32.530853+0000","last_undegraded":"2024-03-26T05:08:32.530853+0000","last_fullsized":"2024-03-26T05:08:32.530853+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:35.999836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:28.386172+0000","last_clean_scrub_stamp":"2024-03-26T05:07:35.999836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:40.950957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":368,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530926+0000","last_change":"2024-03-26T05:07:33.664903+0000","last_active":"2024-03-26T05:08:32.530926+0000","last_peered":"2024-03-26T05:08:32.530926+0000","last_clean":"2024-03-26T05:08:32.530926+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:08:32.530926+0000","last_undegraded":"2024-03-26T05:08:32.530926+0000","last_fullsized":"2024-03-26T05:08:32.530926+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:07:33.664806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:33.664806+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:08:49.190728+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"221'7807","reported_seq":14697,"reported_epoch":221,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T05:08:35.446858+0000","last_change":"2024-03-26T05:08:01.222315+0000","last_active":"2024-03-26T05:08:35.446858+0000","last_peered":"2024-03-26T05:08:35.446858+0000","last_clean":"2024-03-26T05:07:58.925634+0000","last_became_active":"2024-03-26T05:08:01.210221+0000","last_became_peered":"2024-03-26T05:08:01.210221+0000","last_unstale":"2024-03-26T05:08:35.446858+0000","last_undegraded":"2024-03-26T05:08:35.446858+0000","last_fullsized":"2024-03-26T05:08:35.446858+0000","mapping_epoch":214,"log_start":"196'3100","ondisk_log_start":"196'3100","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:40.933999+0000","last_clean_scrub_stamp":"2024-03-26T05:06:40.933999+0000","objects_scrubbed":0,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423034880,"num_objects":6455,"num_object_clones":0,"num_object_copies":19365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3144,"num_objects_unfound":0,"num_objects_dirty":6455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7807,"num_write_kb":456384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3398,"num_bytes_recovered":222691328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":6455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":332,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530398+0000","last_change":"2024-03-26T05:06:05.019093+0000","last_active":"2024-03-26T05:08:32.530398+0000","last_peered":"2024-03-26T05:08:32.530398+0000","last_clean":"2024-03-26T05:08:32.530398+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:08:32.530398+0000","last_undegraded":"2024-03-26T05:08:32.530398+0000","last_fullsized":"2024-03-26T05:08:32.530398+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:05.018999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:06:05.018999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"216'11","reported_seq":472,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.532567+0000","last_change":"2024-03-26T05:07:59.964946+0000","last_active":"2024-03-26T05:08:32.532567+0000","last_peered":"2024-03-26T05:08:32.532567+0000","last_clean":"2024-03-26T05:08:32.532567+0000","last_became_active":"2024-03-26T05:07:59.960519+0000","last_became_peered":"2024-03-26T05:07:59.960519+0000","last_unstale":"2024-03-26T05:08:32.532567+0000","last_undegraded":"2024-03-26T05:08:32.532567+0000","last_fullsized":"2024-03-26T05:08:32.532567+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'10","last_scrub_stamp":"2024-03-26T05:06:56.184325+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:06:56.184325+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":349,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.539079+0000","last_change":"2024-03-26T05:06:06.522418+0000","last_active":"2024-03-26T05:08:32.539079+0000","last_peered":"2024-03-26T05:08:32.539079+0000","last_clean":"2024-03-26T05:08:32.539079+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:08:32.539079+0000","last_undegraded":"2024-03-26T05:08:32.539079+0000","last_fullsized":"2024-03-26T05:08:32.539079+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:05.000183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:06:05.000183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":355,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-26T05:08:32.530680+0000","last_change":"2024-03-26T05:06:45.298927+0000","last_active":"2024-03-26T05:08:32.530680+0000","last_peered":"2024-03-26T05:08:32.530680+0000","last_clean":"2024-03-26T05:08:32.530680+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:08:32.530680+0000","last_undegraded":"2024-03-26T05:08:32.530680+0000","last_fullsized":"2024-03-26T05:08:32.530680+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:06:45.298241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:06:45.298241+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6754795544,"num_objects":103071,"num_object_clones":0,"num_object_copies":309213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40217,"num_objects_unfound":0,"num_objects_dirty":103071,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125959,"num_write_kb":7328897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11514,"num_bytes_recovered":754581504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20960784384,"data_stored":20960772168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86059,"ondisk_log_size":86059,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":491,"num_read_kb":486,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":984,"num_read_kb":709,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":1870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":630,"ondisk_log_size":630,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005160,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2176012,"kb_used_data":2004612,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91569140,"statfs":{"total":95995035648,"available":93766799360,"internally_reserved":0,"allocated":2052722688,"data_stored":2052368016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":175503818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332204,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3137340,"kb_used_data":2907636,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":90607812,"statfs":{"total":95995035648,"available":92782399488,"internally_reserved":0,"allocated":2977419264,"data_stored":2977079460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":235207110},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574982,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3182428,"kb_used_data":2912468,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":90562724,"statfs":{"total":95995035648,"available":92736229376,"internally_reserved":0,"allocated":2982367232,"data_stored":2981995026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738507,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3176532,"kb_used_data":2910732,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":90568620,"statfs":{"total":95995035648,"available":92742266880,"internally_reserved":0,"allocated":2980589568,"data_stored":2980226833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411457,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3171388,"kb_used_data":2901428,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":90573764,"statfs":{"total":95995035648,"available":92747534336,"internally_reserved":0,"allocated":2971062272,"data_stored":2970722068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117115,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4079376,"kb_used_data":3751048,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":89665776,"statfs":{"total":95995035648,"available":91817754624,"internally_reserved":0,"allocated":3841073152,"data_stored":3840712780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":336132551},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920888,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2574476,"kb_used_data":2390532,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91170676,"statfs":{"total":95995035648,"available":93358772224,"internally_reserved":0,"allocated":2447904768,"data_stored":2447549081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659250,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":963256,"kb_used_data":787696,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92781896,"statfs":{"total":95995035648,"available":95008661504,"internally_reserved":0,"allocated":806600704,"data_stored":806264754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2969763840,"data_stored":2969763840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2961768448,"data_stored":2961768448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2954039296,"data_stored":2954035224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3807641600,"data_stored":3807641600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2439315456,"data_stored":2439315456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":806813696,"data_stored":806813696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2972585984,"data_stored":2972581912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2048856064,"data_stored":2048851992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:08:42.086 INFO:tasks.thrashosds.thrasher:Setting 6.b to [7, 4, 2, 0, 1, 3] 2024-03-26T05:08:42.086 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.b', '7', '4', '2', '0', '1', '3'] 2024-03-26T05:08:42.086 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pg-upmap-items 6.b 7 4 2 0 1 3 2024-03-26T05:08:42.086 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:42.133 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:42.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:42.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:42.235 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:42.286 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:42.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:42.387 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:42.436 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:42.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:42.538 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:42.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:42.593 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:42.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:42.694 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:42.730 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:42.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:42.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:42 smithi132 ceph-mon[33500]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:08:42.831 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:42.865 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:42.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:42.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:42 smithi080 ceph-mon[40404]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:08:42.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:42 smithi080 ceph-mon[29691]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:08:42.968 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:43.002 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:43.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:43.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:43.104 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:43.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:43.149 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:43.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:43.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:43.285 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:43.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:43.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:43.387 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:43.416 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:43.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:43.518 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:43.553 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:43.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:43.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:43.654 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:43.682 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:43.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:43.785 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:43.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:43 smithi132 ceph-mon[33500]: pgmap v365: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 859 op/s; 37242/298722 objects misplaced (12.467%); 21 MiB/s, 336 objects/s recovering 2024-03-26T05:08:43.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:43 smithi132 ceph-mon[33500]: 5.9 scrub starts 2024-03-26T05:08:43.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:43 smithi132 ceph-mon[33500]: 5.9 scrub ok 2024-03-26T05:08:43.823 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:43.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:43.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:43 smithi080 ceph-mon[40404]: pgmap v365: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 859 op/s; 37242/298722 objects misplaced (12.467%); 21 MiB/s, 336 objects/s recovering 2024-03-26T05:08:43.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:43 smithi080 ceph-mon[40404]: 5.9 scrub starts 2024-03-26T05:08:43.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:43 smithi080 ceph-mon[40404]: 5.9 scrub ok 2024-03-26T05:08:43.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:43 smithi080 ceph-mon[29691]: pgmap v365: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 859 op/s; 37242/298722 objects misplaced (12.467%); 21 MiB/s, 336 objects/s recovering 2024-03-26T05:08:43.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:43 smithi080 ceph-mon[29691]: 5.9 scrub starts 2024-03-26T05:08:43.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:43 smithi080 ceph-mon[29691]: 5.9 scrub ok 2024-03-26T05:08:43.925 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:43.968 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:43.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:44.069 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:44.102 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:44.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:44.203 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:44.246 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:44.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:44.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:44.367 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:08:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:08:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:08:44.395 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:44.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:44.496 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:44.523 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:44.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:44.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:44.570 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:44.580 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:44.624 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:44.683 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:44.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:44.785 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:44.825 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:44.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:44.928 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:44.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:45.065 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:45.097 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:45.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:45.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:45.199 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:45.239 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:45.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:45.343 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:45.370 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:45.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:45.471 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:08:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:08:45.472 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:45.502 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:45.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:45.605 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:45.637 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:45.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:45.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:45.738 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:45.775 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:45.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:45.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:45.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:45 smithi132 ceph-mon[33500]: pgmap v366: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 902 B/s rd, 0 B/s wr, 757 op/s; 37242/298722 objects misplaced (12.467%); 19 MiB/s, 297 objects/s recovering 2024-03-26T05:08:45.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:45 smithi132 ceph-mon[33500]: 6.3 scrub starts 2024-03-26T05:08:45.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:45 smithi132 ceph-mon[33500]: 6.3 scrub ok 2024-03-26T05:08:45.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:45 smithi080 ceph-mon[40404]: pgmap v366: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 902 B/s rd, 0 B/s wr, 757 op/s; 37242/298722 objects misplaced (12.467%); 19 MiB/s, 297 objects/s recovering 2024-03-26T05:08:45.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:45 smithi080 ceph-mon[40404]: 6.3 scrub starts 2024-03-26T05:08:45.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:45 smithi080 ceph-mon[40404]: 6.3 scrub ok 2024-03-26T05:08:45.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:45 smithi080 ceph-mon[29691]: pgmap v366: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 902 B/s rd, 0 B/s wr, 757 op/s; 37242/298722 objects misplaced (12.467%); 19 MiB/s, 297 objects/s recovering 2024-03-26T05:08:45.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:45 smithi080 ceph-mon[29691]: 6.3 scrub starts 2024-03-26T05:08:45.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:45 smithi080 ceph-mon[29691]: 6.3 scrub ok 2024-03-26T05:08:45.877 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:45.925 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:45.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:46.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:46.056 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:46.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:46.158 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:46.204 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:46.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:46.306 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:46.334 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:46.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:46.352 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:08:46.392 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:08:46.429 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:08:46.437 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:46.478 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:08:46.483 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:46.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:46.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:46.484 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:46 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:46.515 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:08:46.567 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:08:46.585 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:46.612 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:08:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:46.613 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:46.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15093.0:135532 8.9 e221/214 8:99caf04e:::benchmark_data_smithi080_109213_object21015:head v 221'8496, mlcod=221'8494)", 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:46.658005+0000", 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0090728670000000001, 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0091045180000000007, 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658005+0000", 2024-03-26T05:08:46.659 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658005+0000", 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658006+0000", 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5499999999999997e-07 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658013+0000", 2024-03-26T05:08:46.660 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1339999999999998e-06 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658029+0000", 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6016999999999999e-05 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658037+0000", 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2090000000000005e-06 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658671+0000", 2024-03-26T05:08:46.661 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00063377799999999999 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658703+0000", 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2737000000000002e-05 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:135534 8.c 8:30da69d0:::benchmark_data_smithi080_109213_object21017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:46.658704+0000", 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0083734570000000008, 2024-03-26T05:08:46.662 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0084741339999999995, 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 135534 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658704+0000", 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:08:46.663 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658704+0000", 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658706+0000", 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.344e-06 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658710+0000", 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7469999999999996e-06 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.664 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658722+0000", 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.148e-05 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658730+0000", 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0700000000000007e-06 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658783+0000", 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3672999999999999e-05 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.665 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.658910+0000", 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012637600000000001 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.659027+0000", 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117179 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.662529+0000", 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0035023939999999998 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.666 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.662568+0000", 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8890000000000002e-05 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:135536 8.8 8:193634e7:::benchmark_data_smithi080_109213_object21019:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:46.664699+0000", 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0023788709999999999, 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0025673470000000002, 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:08:46.667 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 135536 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.664699+0000", 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.664699+0000", 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:08:46.668 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.664700+0000", 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3289999999999999e-06 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.664705+0000", 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2329999999999997e-06 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.664717+0000", 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1602e-05 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.669 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.664725+0000", 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7360000000000001e-06 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.665236+0000", 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00051138400000000004 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.665364+0000", 2024-03-26T05:08:46.670 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012845200000000001 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.665502+0000", 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013746900000000001 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15093.0:135541 8.9 e221/214)", 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:46.666089+0000", 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.00098824599999999992, 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0012442099999999999, 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:08:46.671 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "queued for pg", 2024-03-26T05:08:46.672 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:08:46.672 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.672 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:08:46.672 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.666089+0000", 2024-03-26T05:08:46.672 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:08:46.672 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.672 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.672 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:08:46.672 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.666089+0000", 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.666090+0000", 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7499999999999999e-07 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.666096+0000", 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1399999999999997e-06 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:08:46.673 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.666107+0000", 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0944000000000001e-05 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.666114+0000", 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0600000000000002e-06 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:135530 8.e 8:709d275e:::benchmark_data_smithi080_109213_object21013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:46.655715+0000", 2024-03-26T05:08:46.674 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.011362494000000001, 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.011670296, 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 135530 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.655715+0000", 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.675 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.655715+0000", 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.655717+0000", 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.773e-06 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.655723+0000", 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.562e-06 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.676 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.655738+0000", 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4885999999999999e-05 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.655748+0000", 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1989999999999995e-06 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.655810+0000", 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2198000000000003e-05 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.677 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.655943+0000", 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000133211 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,4", 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.656152+0000", 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000209485 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.661731+0000", 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055787750000000002 2024-03-26T05:08:46.678 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:08:46.679 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:08:46.679 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:08:46.679 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:46.666913+0000", 2024-03-26T05:08:46.679 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051818610000000003 2024-03-26T05:08:46.679 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:46.679 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:08:46.679 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:46.679 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:08:46.679 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:08:46.679 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 5 2024-03-26T05:08:46.679 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:08:46.679 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:46 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1916834560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:46.680 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:46 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:46.680 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:46 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1916834560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:46.680 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:08:46.680 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:08:46.680 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:08:46.680 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:08:46.681 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:08:46.681 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:08:46.681 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:08:46.681 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:08:46.681 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:08:46.714 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:46.757 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:46.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:46.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:46.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:46 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:46.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:46 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1916834560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:08:46.859 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:46.886 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:46.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:46.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:46.987 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:47.021 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:47.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:47.121 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:47.155 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:47.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:47.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:47.257 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:47.298 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:47.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:47.400 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:47.441 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:47.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:47.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:47.576 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:47.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:47.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:47.714 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:47.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:47.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:47 smithi132 ceph-mon[33500]: pgmap v367: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 33795/285108 objects misplaced (11.853%); 27 MiB/s, 432 objects/s recovering 2024-03-26T05:08:47.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:47 smithi132 ceph-mon[33500]: 1.0 scrub starts 2024-03-26T05:08:47.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:47 smithi132 ceph-mon[33500]: 1.0 scrub ok 2024-03-26T05:08:47.816 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:47.863 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:47.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:47.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:47 smithi080 ceph-mon[40404]: pgmap v367: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 33795/285108 objects misplaced (11.853%); 27 MiB/s, 432 objects/s recovering 2024-03-26T05:08:47.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:47 smithi080 ceph-mon[40404]: 1.0 scrub starts 2024-03-26T05:08:47.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:47 smithi080 ceph-mon[40404]: 1.0 scrub ok 2024-03-26T05:08:47.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:47 smithi080 ceph-mon[29691]: pgmap v367: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 33795/285108 objects misplaced (11.853%); 27 MiB/s, 432 objects/s recovering 2024-03-26T05:08:47.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:47 smithi080 ceph-mon[29691]: 1.0 scrub starts 2024-03-26T05:08:47.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:47 smithi080 ceph-mon[29691]: 1.0 scrub ok 2024-03-26T05:08:47.965 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:48.004 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:48.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:48.106 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:48.136 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:48.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:48.238 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:48.279 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:48.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:48.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:48.380 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:48.436 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:48.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:48.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:48.538 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:48.572 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:48.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:48.674 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:48.705 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:48.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:48.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:48.836 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:48.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:48.938 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:48.971 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:48.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:48.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:49.073 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:49.124 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:49.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:49.226 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:49.265 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:49.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:49.367 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:49.391 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:49.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:49.493 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:49.536 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:49.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:49.637 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:49.681 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:49.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:49.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:49.782 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:49.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:49 smithi132 ceph-mon[33500]: pgmap v368: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 711 op/s; 33795/285108 objects misplaced (11.853%); 18 MiB/s, 292 objects/s recovering 2024-03-26T05:08:49.825 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:49.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:49.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:49 smithi080 ceph-mon[40404]: pgmap v368: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 711 op/s; 33795/285108 objects misplaced (11.853%); 18 MiB/s, 292 objects/s recovering 2024-03-26T05:08:49.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:49 smithi080 ceph-mon[29691]: pgmap v368: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 711 op/s; 33795/285108 objects misplaced (11.853%); 18 MiB/s, 292 objects/s recovering 2024-03-26T05:08:49.927 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:49.954 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:49.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:49.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:50.056 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:50.084 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:50.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:50.186 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:50.227 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:50.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:50.328 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:50.355 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:50.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:50.457 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:50.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:50.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:50.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:50.601 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:50.635 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:50.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:50.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:50.737 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:50.761 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:50.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:50.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:50.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:08:50.863 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:50.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:08:50.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:08:50.894 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:50.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:50.996 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:51.020 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:51.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:51.121 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:51.147 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:51.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:51.249 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:51.279 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:51.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:51.381 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:51.406 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:51.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:51.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:51.508 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:51.550 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:51.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:51.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:51.652 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:51.676 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:51.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:51.777 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:51.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:51 smithi132 ceph-mon[33500]: pgmap v369: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 737 op/s; 33486/284181 objects misplaced (11.783%); 18 MiB/s, 292 objects/s recovering 2024-03-26T05:08:51.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:51 smithi132 ceph-mon[33500]: 5.4 scrub starts 2024-03-26T05:08:51.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:51 smithi132 ceph-mon[33500]: 5.4 scrub ok 2024-03-26T05:08:51.821 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:51.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:51.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:51 smithi080 ceph-mon[40404]: pgmap v369: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 737 op/s; 33486/284181 objects misplaced (11.783%); 18 MiB/s, 292 objects/s recovering 2024-03-26T05:08:51.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:51 smithi080 ceph-mon[40404]: 5.4 scrub starts 2024-03-26T05:08:51.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:51 smithi080 ceph-mon[40404]: 5.4 scrub ok 2024-03-26T05:08:51.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:51 smithi080 ceph-mon[29691]: pgmap v369: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 737 op/s; 33486/284181 objects misplaced (11.783%); 18 MiB/s, 292 objects/s recovering 2024-03-26T05:08:51.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:51 smithi080 ceph-mon[29691]: 5.4 scrub starts 2024-03-26T05:08:51.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:51 smithi080 ceph-mon[29691]: 5.4 scrub ok 2024-03-26T05:08:51.924 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:51.955 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:51.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:52.057 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:52.100 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:52.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:52.202 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:52.239 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:52.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:52.340 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:52.368 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:52.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:52.469 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:52.503 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:52.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:52.605 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:52.636 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:52.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:52.738 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:52.768 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:52.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:52.870 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:52.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:52.906 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:52.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:53.007 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:53.034 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:53.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:53.135 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:53.170 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:53.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:53.271 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:53.302 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:53.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:53.403 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:53.444 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:53.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:53.546 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:53.595 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:53.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:53.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:53.697 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:53.710 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:53 smithi080 ceph-mon[40404]: pgmap v370: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 30428/270918 objects misplaced (11.231%); 27 MiB/s, 436 objects/s recovering 2024-03-26T05:08:53.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:53 smithi080 ceph-mon[29691]: pgmap v370: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 30428/270918 objects misplaced (11.231%); 27 MiB/s, 436 objects/s recovering 2024-03-26T05:08:53.724 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:53.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:53.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:53 smithi132 ceph-mon[33500]: pgmap v370: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 30428/270918 objects misplaced (11.231%); 27 MiB/s, 436 objects/s recovering 2024-03-26T05:08:53.826 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:53.872 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:53.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:53.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:53.973 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:54.038 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:54.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:54.039 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T05:08:54.139 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:54.185 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:54.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:54.287 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:54.314 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:54.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:54.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:54.368 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:08:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:08:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:08:54.415 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:54.466 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:54.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:54.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:54.513 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:54.555 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:08:54.561 INFO:teuthology.orchestra.run.smithi080.stderr:set 6.b pg_upmap_items mapping to [7->4,2->0,1->3] 2024-03-26T05:08:54.569 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:54.596 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:54.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:54.698 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:54.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:54.735 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:54.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:54.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:54 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1843392596' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:08:54.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:54 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:08:54.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:54 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1643970922' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 4, 2, 0, 1, 3]}]: dispatch 2024-03-26T05:08:54.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:54 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 4, 2, 0, 1, 3]}]: dispatch 2024-03-26T05:08:54.838 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:54.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:54 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1843392596' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:08:54.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:54 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:08:54.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:54 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1643970922' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 4, 2, 0, 1, 3]}]: dispatch 2024-03-26T05:08:54.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:54 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 4, 2, 0, 1, 3]}]: dispatch 2024-03-26T05:08:54.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:54 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1843392596' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:08:54.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:54 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:08:54.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:54 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1643970922' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 4, 2, 0, 1, 3]}]: dispatch 2024-03-26T05:08:54.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:54 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 4, 2, 0, 1, 3]}]: dispatch 2024-03-26T05:08:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:54.869 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:54.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:54.970 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:54.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:54.998 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:54.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:55.099 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:55.125 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:55.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:55.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:55.227 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:55.254 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:55.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:55.355 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:55.381 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:55.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:55.469 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:08:55.482 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:55.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:08:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:08:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:55.543 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:55.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:55.644 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:55.677 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:55.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:55.778 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:55.810 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:55.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:55.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[29691]: pgmap v371: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 772 op/s; 30428/270918 objects misplaced (11.231%); 18 MiB/s, 295 objects/s recovering 2024-03-26T05:08:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:08:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:08:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 4, 2, 0, 1, 3]}]': finished 2024-03-26T05:08:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[29691]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T05:08:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:08:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:08:55.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[40404]: pgmap v371: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 772 op/s; 30428/270918 objects misplaced (11.231%); 18 MiB/s, 295 objects/s recovering 2024-03-26T05:08:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:08:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:08:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 4, 2, 0, 1, 3]}]': finished 2024-03-26T05:08:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[40404]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T05:08:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:08:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:08:55.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:55 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:55.913 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:55.940 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:55.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:55.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:56.042 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:56.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:55 smithi132 ceph-mon[33500]: pgmap v371: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 772 op/s; 30428/270918 objects misplaced (11.231%); 18 MiB/s, 295 objects/s recovering 2024-03-26T05:08:56.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:55 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:08:56.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:55 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:08:56.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:55 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 4, 2, 0, 1, 3]}]': finished 2024-03-26T05:08:56.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:55 smithi132 ceph-mon[33500]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T05:08:56.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:55 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:08:56.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:55 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:08:56.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:55 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:08:56.069 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:56.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:56.171 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:56.199 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:56.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:56.301 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:56.327 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:56.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:56.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:56.429 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:56.459 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:56.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:56.560 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:56.598 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:56.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:56.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:56.700 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:56.725 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:56.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:56.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:56.827 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:56.855 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:56.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:56.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:56.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:08:56.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:08:56.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:08:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:08:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[40404]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T05:08:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[40404]: pgmap v374: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 139 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 988 op/s; 27621/261402 objects misplaced (10.566%); 26 MiB/s, 420 objects/s recovering 2024-03-26T05:08:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[40404]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T05:08:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:08:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:08:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:08:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:08:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[29691]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T05:08:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[29691]: pgmap v374: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 139 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 988 op/s; 27621/261402 objects misplaced (10.566%); 26 MiB/s, 420 objects/s recovering 2024-03-26T05:08:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:56 smithi080 ceph-mon[29691]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T05:08:56.957 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:56.985 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:56.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:56.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:56 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:08:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:56 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:08:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:56 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:08:57.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:56 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:08:57.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:56 smithi132 ceph-mon[33500]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T05:08:57.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:56 smithi132 ceph-mon[33500]: pgmap v374: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 139 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 988 op/s; 27621/261402 objects misplaced (10.566%); 26 MiB/s, 420 objects/s recovering 2024-03-26T05:08:57.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:56 smithi132 ceph-mon[33500]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T05:08:57.087 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:57.112 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:57.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:57.214 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:57.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:57.243 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:57.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:57.343 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:57.368 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:57.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:57.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:57.470 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:57.501 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:57.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:57.574 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:08:57.602 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:57.667 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:08:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:57.672 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:57.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:57.773 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:08:57.813 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:57.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:57.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:08:57.915 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:57.949 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:57.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:58.051 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:58.096 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:58.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:58.197 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:58.237 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:58.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:58.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:58.338 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:08:58.377 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:58.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:08:58.479 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:58.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:58.515 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:58.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:58.616 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:58.656 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:58.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:58.757 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:58.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:58 smithi132 ceph-mon[33500]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T05:08:58.797 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:58.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:58.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:58.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:58 smithi080 ceph-mon[29691]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T05:08:58.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:58 smithi080 ceph-mon[40404]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T05:08:58.899 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:58.938 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:58.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:58.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:59.040 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:08:59.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:59.075 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:59.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:08:59.176 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:08:59.209 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:59.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:59.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:08:59.312 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:08:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:59.352 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:59.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:08:59.453 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:08:59.486 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:08:59.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:59.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:08:59.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:08:59.626 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:59.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:08:59.727 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:59.761 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:59.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:59.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:59 smithi132 ceph-mon[33500]: pgmap v377: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 139 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 794 op/s; 27621/261402 objects misplaced (10.566%); 26 MiB/s, 409 objects/s recovering 2024-03-26T05:08:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:59 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:08:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:59 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:08:59.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:08:59 smithi132 ceph-mon[33500]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T05:08:59.863 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:08:59.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:59 smithi080 ceph-mon[40404]: pgmap v377: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 139 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 794 op/s; 27621/261402 objects misplaced (10.566%); 26 MiB/s, 409 objects/s recovering 2024-03-26T05:08:59.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:59 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:08:59.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:59 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:08:59.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:08:59 smithi080 ceph-mon[40404]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T05:08:59.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:59 smithi080 ceph-mon[29691]: pgmap v377: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 139 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 794 op/s; 27621/261402 objects misplaced (10.566%); 26 MiB/s, 409 objects/s recovering 2024-03-26T05:08:59.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:59 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:08:59.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:59 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:08:59.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:08:59 smithi080 ceph-mon[29691]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T05:08:59.897 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:08:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:59.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:08:59.999 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:00.035 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:00.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:00.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:00.136 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:00.169 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:00.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:00.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:00.270 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:00.302 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:00.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:00.404 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:00.441 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:00.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:00.542 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:00.568 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:00.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:00.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:00.671 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:00.710 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:00.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:00.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:00.811 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:00.849 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:00.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:00.951 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:00.988 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:00.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:01.090 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:01.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:01.132 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:01.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:01.233 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:01.242 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:09:01.243 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 0 1 2024-03-26T05:09:01.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:01.273 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:01.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:01.374 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:01.404 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:01.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:01.506 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:01.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:01.561 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:01.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:01.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:01 smithi080 ceph-mon[29691]: pgmap v379: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 260 op/s; 25904/259740 objects misplaced (9.973%); 22 MiB/s, 355 objects/s recovering 2024-03-26T05:09:01.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:01 smithi080 ceph-mon[40404]: pgmap v379: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 260 op/s; 25904/259740 objects misplaced (9.973%); 22 MiB/s, 355 objects/s recovering 2024-03-26T05:09:01.662 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:01.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:01.695 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:01.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:01.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:01 smithi132 ceph-mon[33500]: pgmap v379: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 260 op/s; 25904/259740 objects misplaced (9.973%); 22 MiB/s, 355 objects/s recovering 2024-03-26T05:09:01.796 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:01.831 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:01.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:01.933 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:01.972 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:01.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:02.074 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:02.115 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:02.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:02.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:02.217 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:02.248 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:02.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:02.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:02.349 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:02.383 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:02.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:02.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:02.485 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:02.515 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:02.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:02.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:02.617 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:02.651 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:02.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:02.753 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:02.791 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:02.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:02.792 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:09:02.792 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:09:02.792 INFO:teuthology.orchestra.run.smithi080.stdout: "complaint_time": 30, 2024-03-26T05:09:02.793 INFO:teuthology.orchestra.run.smithi080.stdout: "num_blocked_ops": 0 2024-03-26T05:09:02.793 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:09:02.893 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:02.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:02.934 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:02.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:03.036 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:03.074 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:03.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:03.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:03.176 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:03.208 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:03.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:03.310 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:03.342 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:03.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:03.389 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:03.416 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:03.430 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:09:03.444 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:03.463 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:09:03.477 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:03.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:03.506 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:09:03.546 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:09:03.579 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:03.587 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:09:03.612 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:03.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:03.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:03.630 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:09:03.675 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:09:03.675 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:03 smithi080 ceph-mon[29691]: pgmap v380: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 573 op/s; 21596/251088 objects misplaced (8.601%); 54 MiB/s, 866 objects/s recovering 2024-03-26T05:09:03.676 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:03 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2497165487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:09:03.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:03 smithi080 ceph-mon[40404]: pgmap v380: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 573 op/s; 21596/251088 objects misplaced (8.601%); 54 MiB/s, 866 objects/s recovering 2024-03-26T05:09:03.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:03 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2497165487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:09:03.714 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:03.717 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:09:03.722 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:09:03.722 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:09:03.722 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:09:03.723 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:09:03.723 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:09:03.723 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:09:03.723 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:09:03.723 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:09:03.750 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:03.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:03.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:03.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:03 smithi132 ceph-mon[33500]: pgmap v380: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 573 op/s; 21596/251088 objects misplaced (8.601%); 54 MiB/s, 866 objects/s recovering 2024-03-26T05:09:03.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:03 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2497165487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:09:03.852 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:03.889 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:03.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:03.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:03.962 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T05:09:03.991 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:04.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:04.032 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:04.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:04.133 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:04.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:04.185 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:04.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:04.275 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:09:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:09:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:09:04.286 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:04.324 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:04.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:04.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:04.426 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:04.470 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:04.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:04.571 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:04.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:04.607 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:04.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:04.708 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:04.748 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:04.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:04.850 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:04.888 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:04.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:04.990 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:05.028 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:05.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:05.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:05.130 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:05.172 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:05.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:05.273 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:05.311 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:05.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:05.413 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:05.453 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:05.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:05.500 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:09:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:09:05.556 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:05.593 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:05.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:05.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:05.694 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:05.730 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:05.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:05.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:05.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:05 smithi132 ceph-mon[33500]: pgmap v381: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 463 op/s; 21596/251088 objects misplaced (8.601%); 44 MiB/s, 700 objects/s recovering 2024-03-26T05:09:05.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:09:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:09:05.832 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:05.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:05 smithi080 ceph-mon[29691]: pgmap v381: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 463 op/s; 21596/251088 objects misplaced (8.601%); 44 MiB/s, 700 objects/s recovering 2024-03-26T05:09:05.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:09:05.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:09:05.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:05 smithi080 ceph-mon[40404]: pgmap v381: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 463 op/s; 21596/251088 objects misplaced (8.601%); 44 MiB/s, 700 objects/s recovering 2024-03-26T05:09:05.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:09:05.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:09:05.868 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:05.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:05.970 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:06.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:06.007 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:06.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:06.108 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:06.145 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:06.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:06.247 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:06.290 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:06.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:06.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:06.392 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:06.433 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:06.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:06.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:06.570 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:06.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:06.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:06.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:06.709 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:06.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:06.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:06 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:06.810 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:06.844 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:06.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:06.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:06 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:06.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:06 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:06.907 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:09:06.946 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:06.979 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:06.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:06.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:07.081 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:07.117 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:07.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:07.218 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:07.262 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:07.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:07.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:07.364 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:07.396 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:07.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:07.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:07.498 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:07.529 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:07.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:07.631 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:07.667 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:07.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:07.769 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:07.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:07 smithi132 ceph-mon[33500]: pgmap v382: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 17745/235881 objects misplaced (7.523%); 58 MiB/s, 933 objects/s recovering 2024-03-26T05:09:07.810 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:07.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:07.851 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:07 smithi080 ceph-mon[29691]: pgmap v382: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 17745/235881 objects misplaced (7.523%); 58 MiB/s, 933 objects/s recovering 2024-03-26T05:09:07.852 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:07 smithi080 ceph-mon[40404]: pgmap v382: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 17745/235881 objects misplaced (7.523%); 58 MiB/s, 933 objects/s recovering 2024-03-26T05:09:07.912 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:07.950 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:07.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:08.051 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:08.100 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:08.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:08.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:08.201 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:08.243 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:08.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:08.345 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:08.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:08.385 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:08.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:08.486 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:08.530 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:08.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:08.632 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:08.666 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:08.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:08.768 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:08.806 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:08.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:08.907 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:08.908 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:08.968 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:08.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:08.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:09.070 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:09.101 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:09.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:09.203 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:09.258 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:09.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:09.360 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:09.397 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:09.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:09.498 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:09.534 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:09.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:09.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:09 smithi080 ceph-mon[29691]: pgmap v383: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 851 op/s; 17745/235881 objects misplaced (7.523%); 49 MiB/s, 785 objects/s recovering 2024-03-26T05:09:09.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:09 smithi080 ceph-mon[40404]: pgmap v383: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 851 op/s; 17745/235881 objects misplaced (7.523%); 49 MiB/s, 785 objects/s recovering 2024-03-26T05:09:09.636 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:09.670 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:09.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:09.772 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:09.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:09 smithi132 ceph-mon[33500]: pgmap v383: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 851 op/s; 17745/235881 objects misplaced (7.523%); 49 MiB/s, 785 objects/s recovering 2024-03-26T05:09:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:09.799 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:09.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:09.900 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:09.939 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:09.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:10.040 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:10.077 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:10.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:10.178 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:10.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:10.220 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:10.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:10.254 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:10.321 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:10.389 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:10.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:10.490 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:10.524 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:10.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:10.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:10.551 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:09:10.626 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:10.660 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:10.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:10.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:10.762 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:10.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:10 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/628964320' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:09:10.801 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:10.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:10.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:10 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/628964320' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:09:10.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:10 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/628964320' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:09:10.903 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:10.953 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:10.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:11.055 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:11.097 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:11.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:11.198 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:11.235 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:11.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:11.337 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:11.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:11.372 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:11.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:11.473 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:11.526 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:11.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:11.628 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:11.666 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:11.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:11.721 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:11 smithi080 ceph-mon[40404]: pgmap v384: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 805 op/s; 16517/233880 objects misplaced (7.062%); 49 MiB/s, 781 objects/s recovering 2024-03-26T05:09:11.721 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:11 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:09:11.721 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:11 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/628964320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:09:11.721 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:11 smithi080 ceph-mon[40404]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T05:09:11.721 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:11 smithi080 ceph-mon[29691]: pgmap v384: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 805 op/s; 16517/233880 objects misplaced (7.062%); 49 MiB/s, 781 objects/s recovering 2024-03-26T05:09:11.721 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:11 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:09:11.722 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:11 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/628964320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:09:11.722 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:11 smithi080 ceph-mon[29691]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T05:09:11.769 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:11.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:11 smithi132 ceph-mon[33500]: pgmap v384: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 805 op/s; 16517/233880 objects misplaced (7.062%); 49 MiB/s, 781 objects/s recovering 2024-03-26T05:09:11.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:11 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:09:11.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:11 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/628964320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:09:11.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:11 smithi132 ceph-mon[33500]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T05:09:11.807 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:11.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:11.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:11.909 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:11.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:11.946 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:11.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:12.047 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:12.089 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:12.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:12.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:12.191 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:12.231 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:12.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:12.332 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:12.373 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:12.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:12.475 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:12.509 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:12.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:12.611 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:12.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:12.644 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:12.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:12.745 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:12.784 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:12.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:12.886 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:12.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:12.923 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:12.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:13.024 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:13.058 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:13.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:13.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:13.201 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:13.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:13.303 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:13.351 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:13.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:13.454 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:13.488 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:13.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:13.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:13.505 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:09:13.568 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.0 to 1 (10000) 2024-03-26T05:09:13.590 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:13.648 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:13.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:13.749 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:13.789 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:13.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:13.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:13 smithi132 ceph-mon[33500]: pgmap v386: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 14030/220272 objects misplaced (6.369%); 33 MiB/s, 527 objects/s recovering 2024-03-26T05:09:13.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:13 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1820697767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:09:13.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:13 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:09:13.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:13 smithi080 ceph-mon[29691]: pgmap v386: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 14030/220272 objects misplaced (6.369%); 33 MiB/s, 527 objects/s recovering 2024-03-26T05:09:13.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:13 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1820697767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:09:13.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:13 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:09:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:13 smithi080 ceph-mon[40404]: pgmap v386: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 14030/220272 objects misplaced (6.369%); 33 MiB/s, 527 objects/s recovering 2024-03-26T05:09:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:13 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1820697767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:09:13.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:13 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:09:13.891 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:13.931 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:13.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:14.032 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:14.080 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:14.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:14.181 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:14.235 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:14.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:14.295 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:09:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:09:14.337 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:14.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:14.394 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:14.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:14.495 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:14.532 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:14.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:14.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:14.633 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:14.663 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:14.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:14.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:14.765 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:14.800 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:14.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:14.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:09:14.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-mon[40404]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T05:09:14.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:09:14.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-mon[40404]: pgmap v388: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 650 op/s; 14030/220272 objects misplaced (6.369%); 21 MiB/s, 328 objects/s recovering 2024-03-26T05:09:14.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:09:14.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:09:14.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:09:14.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-mon[29691]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T05:09:14.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:09:14.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-mon[29691]: pgmap v388: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 650 op/s; 14030/220272 objects misplaced (6.369%); 21 MiB/s, 328 objects/s recovering 2024-03-26T05:09:14.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:09:14.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:14 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:09:14.902 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:14.939 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:14.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:15.041 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:14 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:09:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:14 smithi132 ceph-mon[33500]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T05:09:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:14 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:09:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:14 smithi132 ceph-mon[33500]: pgmap v388: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 650 op/s; 14030/220272 objects misplaced (6.369%); 21 MiB/s, 328 objects/s recovering 2024-03-26T05:09:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:14 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:09:15.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:14 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:09:15.076 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:15.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:15.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:15.178 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:15.212 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:15.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:15.314 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:15.352 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:15.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:15.407 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T05:09:15.453 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:15.490 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:15.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:15.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:15.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:09:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:09:15.592 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:15.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:15 smithi080 ceph-mon[40404]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T05:09:15.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:15 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:15.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:15 smithi080 ceph-mon[29691]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T05:09:15.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:15 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:15.628 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:15.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:15.730 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:15.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:15.774 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:15.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:15.875 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:15.911 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:15.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:16.013 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:16.041 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:16.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:16.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:16.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:15 smithi132 ceph-mon[33500]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T05:09:16.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:15 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:16.143 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:16.170 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:16.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:16.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:16.272 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:16.298 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:16.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:16.301 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:09:16.330 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:09:16.353 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:09:16.359 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:09:16.359 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:09:16.380 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:09:16.400 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:16.409 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:09:16.430 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:16.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:16.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:16.441 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:09:16.472 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:09:16.498 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:09:16.504 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:09:16.504 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:09:16.504 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:09:16.504 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:09:16.504 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:09:16.504 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:09:16.504 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:09:16.504 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:09:16.534 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:16.561 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:16.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:16.663 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:16.695 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:16.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:16.797 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:16.824 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:16.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:16.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[40404]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T05:09:16.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[40404]: pgmap v391: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 959 B/s rd, 0 B/s wr, 1.14k op/s; 10402/205158 objects misplaced (5.070%); 33 MiB/s, 520 objects/s recovering 2024-03-26T05:09:16.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:09:16.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:09:16.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:09:16.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1245482230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:09:16.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[40404]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T05:09:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[29691]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T05:09:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[29691]: pgmap v391: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 959 B/s rd, 0 B/s wr, 1.14k op/s; 10402/205158 objects misplaced (5.070%); 33 MiB/s, 520 objects/s recovering 2024-03-26T05:09:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:09:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:09:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:09:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1245482230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:09:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:16 smithi080 ceph-mon[29691]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T05:09:16.926 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:16.953 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:16.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:16.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:16 smithi132 ceph-mon[33500]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T05:09:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:16 smithi132 ceph-mon[33500]: pgmap v391: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 959 B/s rd, 0 B/s wr, 1.14k op/s; 10402/205158 objects misplaced (5.070%); 33 MiB/s, 520 objects/s recovering 2024-03-26T05:09:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:16 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:09:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:16 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:09:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:16 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:09:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:16 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1245482230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:09:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:16 smithi132 ceph-mon[33500]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T05:09:17.055 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:17.074 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:17.081 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:17.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:17.110 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T05:09:17.182 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:17.211 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:17.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:17.313 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:17.338 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:17.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:17.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:17.440 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:17.465 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:17.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:17.567 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:17.601 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:17.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:17.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:17.703 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:17.749 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:17.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:17.851 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:17.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:17.883 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:17.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:17.984 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:18.025 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:18.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:18.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:18.127 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:18.165 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:18.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:18.266 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:18.296 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:18.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:18.398 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:18.431 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:18.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:18.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:18.533 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:18.558 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:18.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:18.659 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:18.698 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:18.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:18.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:18 smithi132 ceph-mon[33500]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T05:09:18.799 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:18.840 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:18.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:18.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:18 smithi080 ceph-mon[29691]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T05:09:18.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:18 smithi080 ceph-mon[40404]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T05:09:18.940 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:18.981 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:18.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:19.082 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:19.106 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:19.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:19.207 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:19.245 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:19.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:19.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:19.347 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:19.391 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:19.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:19.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:19.493 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:19.531 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:19.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:19.634 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:19.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:19.675 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:19.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:19.736 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:19 smithi080 ceph-mon[29691]: pgmap v394: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 10402/205158 objects misplaced (5.070%); 43 MiB/s, 694 objects/s recovering 2024-03-26T05:09:19.736 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:19 smithi080 ceph-mon[40404]: pgmap v394: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 10402/205158 objects misplaced (5.070%); 43 MiB/s, 694 objects/s recovering 2024-03-26T05:09:19.776 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:19.808 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:19 smithi132 ceph-mon[33500]: pgmap v394: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 10402/205158 objects misplaced (5.070%); 43 MiB/s, 694 objects/s recovering 2024-03-26T05:09:19.844 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:19.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:19.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:19.946 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:19.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:19.984 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:19.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:20.086 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:20.119 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:20.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:20.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:20.221 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:20.258 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:20.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:20.360 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:20.387 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:20.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:20.489 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:20.524 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:20.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:20.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:20.626 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:20.665 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:20.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:20.768 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:20.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:20 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:09:20.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:20 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:09:20.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:20 smithi132 ceph-mon[33500]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T05:09:20.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:09:20.797 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:20.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:20.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:20.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:20 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:09:20.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:20 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:09:20.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:20 smithi080 ceph-mon[29691]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T05:09:20.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:09:20.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:20 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:09:20.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:20 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:09:20.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:20 smithi080 ceph-mon[40404]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T05:09:20.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:09:20.899 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:20.923 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:20.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:21.024 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:21.059 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:21.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:21.160 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:21.192 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:21.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:21.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:21.294 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:21.332 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:21.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:21.434 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:21.467 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:21.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:21.569 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:21.601 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:21.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:21.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:21.703 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:21.736 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:21.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:21 smithi132 ceph-mon[33500]: pgmap v396: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 144 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 328 op/s; 9938/203754 objects misplaced (4.877%); 6.7 MiB/s, 106 objects/s recovering 2024-03-26T05:09:21.839 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:21.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:21 smithi080 ceph-mon[29691]: pgmap v396: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 144 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 328 op/s; 9938/203754 objects misplaced (4.877%); 6.7 MiB/s, 106 objects/s recovering 2024-03-26T05:09:21.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:21 smithi080 ceph-mon[40404]: pgmap v396: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 144 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 328 op/s; 9938/203754 objects misplaced (4.877%); 6.7 MiB/s, 106 objects/s recovering 2024-03-26T05:09:21.877 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:21.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:21.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:21.979 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:22.017 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:22.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:22.099 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:22.119 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:22.191 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:22.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:22.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:22.293 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:22.325 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:22.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:22.426 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:22.470 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:22.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:22.571 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:22.606 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:22.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:22.708 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:22.737 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:22.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:22.838 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:22.863 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:22.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:22.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:22 smithi080 ceph-mon[29691]: pgmap v397: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 590 op/s; 6385/194538 objects misplaced (3.282%); 37 MiB/s, 599 objects/s recovering 2024-03-26T05:09:22.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:22 smithi080 ceph-mon[29691]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T05:09:22.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:22 smithi080 ceph-mon[29691]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T05:09:22.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:22 smithi080 ceph-mon[40404]: pgmap v397: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 590 op/s; 6385/194538 objects misplaced (3.282%); 37 MiB/s, 599 objects/s recovering 2024-03-26T05:09:22.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:22 smithi080 ceph-mon[40404]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T05:09:22.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:22 smithi080 ceph-mon[40404]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T05:09:22.914 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:22.965 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:22.992 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:22.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:22 smithi132 ceph-mon[33500]: pgmap v397: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 590 op/s; 6385/194538 objects misplaced (3.282%); 37 MiB/s, 599 objects/s recovering 2024-03-26T05:09:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:22 smithi132 ceph-mon[33500]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T05:09:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:22 smithi132 ceph-mon[33500]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T05:09:23.093 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:23.118 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:23.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:23.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:23.220 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:23.247 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:23.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:23.349 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:23.376 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:23.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:23.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:23.478 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:23.505 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:23.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:23.607 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:23.641 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:23.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:23.647 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:09:23.742 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:23.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:23.773 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:23.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:23.873 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:23.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:23.920 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:23.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:24.021 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:24.070 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:24.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:24.172 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:24.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:24.209 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:24.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:24.310 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:24.338 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:24.339 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:09:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:09:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:09:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:24.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:24.440 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:24.466 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:24.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:24.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:24.531 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:09:24.568 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:24.593 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:24.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:24.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:24.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:24 smithi080 ceph-mon[40404]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T05:09:24.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:24 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1428660983' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:09:24.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:24 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:09:24.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:24 smithi080 ceph-mon[29691]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T05:09:24.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:24 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1428660983' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:09:24.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:24 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:09:24.695 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:24.723 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:24.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:24.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:24 smithi132 ceph-mon[33500]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T05:09:24.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1428660983' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:09:24.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:24 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:09:24.826 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:24.863 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:24.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:24.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:24.965 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:24.996 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:24.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:25.097 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:25.121 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:25.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:25.222 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:25.256 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:25.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:25.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:25.358 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:25.398 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:25.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:25.500 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:25.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:09:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:09:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:25.526 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:25.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:25.626 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:25.657 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:25.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:25.759 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:25.789 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:25.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:25.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:25 smithi132 ceph-mon[33500]: pgmap v400: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 590 op/s; 6385/194538 objects misplaced (3.282%); 37 MiB/s, 599 objects/s recovering 2024-03-26T05:09:25.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:25 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:09:25.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:25 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:09:25.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:25 smithi132 ceph-mon[33500]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T05:09:25.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:25 smithi132 ceph-mon[33500]: 3.1f scrub starts 2024-03-26T05:09:25.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:25 smithi132 ceph-mon[33500]: 3.1f scrub ok 2024-03-26T05:09:25.855 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:25 smithi080 ceph-mon[40404]: pgmap v400: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 590 op/s; 6385/194538 objects misplaced (3.282%); 37 MiB/s, 599 objects/s recovering 2024-03-26T05:09:25.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:25 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:09:25.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:25 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:09:25.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:25 smithi080 ceph-mon[40404]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T05:09:25.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:25 smithi080 ceph-mon[40404]: 3.1f scrub starts 2024-03-26T05:09:25.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:25 smithi080 ceph-mon[40404]: 3.1f scrub ok 2024-03-26T05:09:25.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:25 smithi080 ceph-mon[29691]: pgmap v400: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 590 op/s; 6385/194538 objects misplaced (3.282%); 37 MiB/s, 599 objects/s recovering 2024-03-26T05:09:25.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:25 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:09:25.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:25 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:09:25.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:25 smithi080 ceph-mon[29691]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T05:09:25.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:25 smithi080 ceph-mon[29691]: 3.1f scrub starts 2024-03-26T05:09:25.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:25 smithi080 ceph-mon[29691]: 3.1f scrub ok 2024-03-26T05:09:25.891 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:25.932 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:25.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:25.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:26.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:26.071 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:26.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:26.172 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:26.207 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:26.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:26.308 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:26.349 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:26.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:26.450 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:26.488 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:26.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:26.590 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:26.614 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:26.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:26.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:26.716 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:26.747 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:26.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:26.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:26 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:26.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:26 smithi132 ceph-mon[33500]: 4.16 scrub starts 2024-03-26T05:09:26.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:26 smithi132 ceph-mon[33500]: 4.16 scrub ok 2024-03-26T05:09:26.849 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:26.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:26 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:26.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:26 smithi080 ceph-mon[40404]: 4.16 scrub starts 2024-03-26T05:09:26.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:26 smithi080 ceph-mon[40404]: 4.16 scrub ok 2024-03-26T05:09:26.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:26 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:26.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:26 smithi080 ceph-mon[29691]: 4.16 scrub starts 2024-03-26T05:09:26.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:26 smithi080 ceph-mon[29691]: 4.16 scrub ok 2024-03-26T05:09:26.877 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:26.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:26.979 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:27.011 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:27.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:27.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:27.147 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:27.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:27.249 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:27.282 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:27.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:27.385 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:27.432 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:27.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:27.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:27.534 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:27.578 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:27.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:27.680 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:27.735 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:27.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:27.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:27 smithi132 ceph-mon[33500]: pgmap v402: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 3842/182166 objects misplaced (2.109%); 55 MiB/s, 886 objects/s recovering 2024-03-26T05:09:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:27 smithi132 ceph-mon[33500]: 5.13 scrub starts 2024-03-26T05:09:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:27 smithi132 ceph-mon[33500]: 5.13 scrub ok 2024-03-26T05:09:27.838 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:27.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:27 smithi080 ceph-mon[40404]: pgmap v402: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 3842/182166 objects misplaced (2.109%); 55 MiB/s, 886 objects/s recovering 2024-03-26T05:09:27.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:27 smithi080 ceph-mon[40404]: 5.13 scrub starts 2024-03-26T05:09:27.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:27 smithi080 ceph-mon[40404]: 5.13 scrub ok 2024-03-26T05:09:27.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:27 smithi080 ceph-mon[29691]: pgmap v402: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 3842/182166 objects misplaced (2.109%); 55 MiB/s, 886 objects/s recovering 2024-03-26T05:09:27.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:27 smithi080 ceph-mon[29691]: 5.13 scrub starts 2024-03-26T05:09:27.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:27 smithi080 ceph-mon[29691]: 5.13 scrub ok 2024-03-26T05:09:27.874 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:27.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:27.976 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:28.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:28.013 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:28.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:28.114 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:28.151 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:28.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:28.253 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:28.345 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:28.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:28.447 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:28.477 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:28.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:28.579 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:28.620 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:28.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:28.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:28.721 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:28.753 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:28.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:28.784 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:28 smithi080 ceph-mon[40404]: 4.b scrub starts 2024-03-26T05:09:28.784 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:28 smithi080 ceph-mon[40404]: 4.b scrub ok 2024-03-26T05:09:28.784 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:28 smithi080 ceph-mon[40404]: pgmap v403: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 687 op/s; 3842/182166 objects misplaced (2.109%); 23 MiB/s, 363 objects/s recovering 2024-03-26T05:09:28.785 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:28 smithi080 ceph-mon[29691]: 4.b scrub starts 2024-03-26T05:09:28.785 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:28 smithi080 ceph-mon[29691]: 4.b scrub ok 2024-03-26T05:09:28.785 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:28 smithi080 ceph-mon[29691]: pgmap v403: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 687 op/s; 3842/182166 objects misplaced (2.109%); 23 MiB/s, 363 objects/s recovering 2024-03-26T05:09:28.854 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:28.895 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:28.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:28.903 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:09:28.903 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:09:28.903 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:172317 8.1 8:8a9a36c7:::benchmark_data_smithi080_109213_object57800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:09:28.902345+0000", 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0089671909999999994, 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0090003849999999996, 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 172317 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:28.904 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902345+0000", 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902345+0000", 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902345+0000", 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3500000000000002e-07 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.905 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902347+0000", 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4990000000000001e-06 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902354+0000", 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0680000000000002e-06 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902358+0000", 2024-03-26T05:09:28.906 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9559999999999996e-06 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.905469+0000", 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0031106490000000001 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.905575+0000", 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000105438 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:09:28.907 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.905682+0000", 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010730000000000001 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:172320 8.a 8:58748918:::benchmark_data_smithi080_109213_object57803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:09:28.909749+0000", 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0015624440000000001, 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001678697, 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:28.908 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 172320 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909749+0000", 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909749+0000", 2024-03-26T05:09:28.909 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909751+0000", 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6810000000000001e-06 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909757+0000", 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8170000000000004e-06 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.910 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909774+0000", 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7133999999999999e-05 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909783+0000", 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2350000000000005e-06 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909857+0000", 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3652999999999994e-05 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.911 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909992+0000", 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013493199999999999 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,5", 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.910110+0000", 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118583 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15093.0:172319 8.b e236/184 8:d44c3a03:::benchmark_data_smithi080_109213_object57802:head v 236'10497, mlcod=236'10495)", 2024-03-26T05:09:28.912 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:09:28.909824+0000", 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0014876030000000001, 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001676355, 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909824+0000", 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909824+0000", 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:28.913 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909825+0000", 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7000000000000004e-07 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909831+0000", 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0689999999999998e-06 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:28.914 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909848+0000", 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7408e-05 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909855+0000", 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5119999999999997e-06 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909918+0000", 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3328000000000001e-05 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:28.915 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.909938+0000", 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9871999999999999e-05 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:172316 8.1 8:82a02c28:::benchmark_data_smithi080_109213_object57799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:09:28.902298+0000", 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0090137670000000007, 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0092632760000000008, 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:09:28.916 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 172316 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902298+0000", 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902298+0000", 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.917 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902299+0000", 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3370000000000001e-06 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902303+0000", 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0389999999999998e-06 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902318+0000", 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4293e-05 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.918 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902326+0000", 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9070000000000007e-06 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902451+0000", 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012554299999999999 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902559+0000", 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000108037 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:28.919 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,2", 2024-03-26T05:09:28.920 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:28.902667+0000", 2024-03-26T05:09:28.920 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010784499999999999 2024-03-26T05:09:28.920 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:28.920 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:28.920 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:28.920 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:28.920 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:09:28.920 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 4 2024-03-26T05:09:28.920 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:09:28.997 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:29.030 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:29.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:29.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:29.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:28 smithi132 ceph-mon[33500]: 4.b scrub starts 2024-03-26T05:09:29.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:28 smithi132 ceph-mon[33500]: 4.b scrub ok 2024-03-26T05:09:29.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:28 smithi132 ceph-mon[33500]: pgmap v403: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 687 op/s; 3842/182166 objects misplaced (2.109%); 23 MiB/s, 363 objects/s recovering 2024-03-26T05:09:29.132 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:29.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:29.168 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:29.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:29.269 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:29.301 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:29.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:29.403 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:29.437 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:29.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:29.539 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:29.592 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:29.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:29.692 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:29.735 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:29.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:29.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:29.836 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:29.853 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:29 smithi080 ceph-mon[29691]: 3.11 scrub starts 2024-03-26T05:09:29.853 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:29 smithi080 ceph-mon[29691]: 3.11 scrub ok 2024-03-26T05:09:29.853 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:29 smithi080 ceph-mon[29691]: 6.11 scrub starts 2024-03-26T05:09:29.853 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:29 smithi080 ceph-mon[29691]: 6.11 scrub ok 2024-03-26T05:09:29.853 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:29 smithi080 ceph-mon[29691]: 6.0 scrub starts 2024-03-26T05:09:29.854 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:29 smithi080 ceph-mon[29691]: 6.0 scrub ok 2024-03-26T05:09:29.854 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:29 smithi080 ceph-mon[40404]: 3.11 scrub starts 2024-03-26T05:09:29.854 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:29 smithi080 ceph-mon[40404]: 3.11 scrub ok 2024-03-26T05:09:29.854 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:29 smithi080 ceph-mon[40404]: 6.11 scrub starts 2024-03-26T05:09:29.854 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:29 smithi080 ceph-mon[40404]: 6.11 scrub ok 2024-03-26T05:09:29.854 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:29 smithi080 ceph-mon[40404]: 6.0 scrub starts 2024-03-26T05:09:29.854 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:29 smithi080 ceph-mon[40404]: 6.0 scrub ok 2024-03-26T05:09:29.883 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:29.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:29.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:30.020 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:30.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:30.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:30.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:29 smithi132 ceph-mon[33500]: 3.11 scrub starts 2024-03-26T05:09:30.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:29 smithi132 ceph-mon[33500]: 3.11 scrub ok 2024-03-26T05:09:30.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:29 smithi132 ceph-mon[33500]: 6.11 scrub starts 2024-03-26T05:09:30.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:29 smithi132 ceph-mon[33500]: 6.11 scrub ok 2024-03-26T05:09:30.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:29 smithi132 ceph-mon[33500]: 6.0 scrub starts 2024-03-26T05:09:30.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:29 smithi132 ceph-mon[33500]: 6.0 scrub ok 2024-03-26T05:09:30.122 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:30.165 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:30.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:30.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:30.211 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T05:09:30.255 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:09:30.267 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:30.297 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:30.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:30.398 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:30.481 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:30.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:30.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:30.502 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:30.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:30.612 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:30.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:30.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:30.714 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:30.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:30.765 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:30.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:30.779 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:30 smithi080 ceph-mon[40404]: pgmap v404: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 696 op/s; 3842/179139 objects misplaced (2.145%); 18 MiB/s, 295 objects/s recovering 2024-03-26T05:09:30.779 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:30 smithi080 ceph-mon[29691]: pgmap v404: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 696 op/s; 3842/179139 objects misplaced (2.145%); 18 MiB/s, 295 objects/s recovering 2024-03-26T05:09:30.866 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:30.900 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:30.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:31.002 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:31.036 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:31.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:31.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:30 smithi132 ceph-mon[33500]: pgmap v404: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 696 op/s; 3842/179139 objects misplaced (2.145%); 18 MiB/s, 295 objects/s recovering 2024-03-26T05:09:31.139 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:31.189 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:31.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:31.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:31.290 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:31.330 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:31.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:31.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:31.432 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:31.460 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:31.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:31.562 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:31.596 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:31.597 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T05:09:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:31.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:31.698 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:31.721 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:31.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:31.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:31.823 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:31.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:31.862 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:31.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:09:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:09:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:09:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:09:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[29691]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T05:09:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[29691]: 4.14 scrub starts 2024-03-26T05:09:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[29691]: 4.14 scrub ok 2024-03-26T05:09:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2082436963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:09:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:09:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[29691]: 3.3 scrub starts 2024-03-26T05:09:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[29691]: 3.3 scrub ok 2024-03-26T05:09:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:09:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:09:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:09:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:09:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[40404]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T05:09:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[40404]: 4.14 scrub starts 2024-03-26T05:09:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[40404]: 4.14 scrub ok 2024-03-26T05:09:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2082436963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:09:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:09:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[40404]: 3.3 scrub starts 2024-03-26T05:09:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:31 smithi080 ceph-mon[40404]: 3.3 scrub ok 2024-03-26T05:09:31.963 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:32.001 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:32.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:32.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:31 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:09:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:31 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:09:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:31 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:09:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:31 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:09:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:31 smithi132 ceph-mon[33500]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T05:09:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:31 smithi132 ceph-mon[33500]: 4.14 scrub starts 2024-03-26T05:09:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:31 smithi132 ceph-mon[33500]: 4.14 scrub ok 2024-03-26T05:09:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:31 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2082436963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:09:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:31 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:09:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:31 smithi132 ceph-mon[33500]: 3.3 scrub starts 2024-03-26T05:09:32.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:31 smithi132 ceph-mon[33500]: 3.3 scrub ok 2024-03-26T05:09:32.102 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:32.131 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:32.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:32.232 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:32.280 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:32.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:32.349 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:32.381 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:32.431 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:32.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:32.532 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:32.569 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:32.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:32.670 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:32.711 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:32.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:32.811 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:32.847 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:32.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:32.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:32.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[29691]: 5.1f scrub starts 2024-03-26T05:09:32.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[29691]: 5.1f scrub ok 2024-03-26T05:09:32.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:09:32.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[29691]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T05:09:32.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[29691]: 3.17 scrub starts 2024-03-26T05:09:32.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[29691]: 3.17 scrub ok 2024-03-26T05:09:32.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:09:32.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:09:32.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[29691]: pgmap v407: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s; 2360/166260 objects misplaced (1.419%); 29 MiB/s, 461 objects/s recovering 2024-03-26T05:09:32.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[40404]: 5.1f scrub starts 2024-03-26T05:09:32.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[40404]: 5.1f scrub ok 2024-03-26T05:09:32.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:09:32.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[40404]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T05:09:32.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[40404]: 3.17 scrub starts 2024-03-26T05:09:32.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[40404]: 3.17 scrub ok 2024-03-26T05:09:32.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:09:32.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:09:32.874 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:32 smithi080 ceph-mon[40404]: pgmap v407: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s; 2360/166260 objects misplaced (1.419%); 29 MiB/s, 461 objects/s recovering 2024-03-26T05:09:32.949 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:32.983 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:32.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:33.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:32 smithi132 ceph-mon[33500]: 5.1f scrub starts 2024-03-26T05:09:33.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:32 smithi132 ceph-mon[33500]: 5.1f scrub ok 2024-03-26T05:09:33.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:32 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:09:33.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:32 smithi132 ceph-mon[33500]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T05:09:33.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:32 smithi132 ceph-mon[33500]: 3.17 scrub starts 2024-03-26T05:09:33.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:32 smithi132 ceph-mon[33500]: 3.17 scrub ok 2024-03-26T05:09:33.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:32 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:09:33.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:32 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:09:33.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:32 smithi132 ceph-mon[33500]: pgmap v407: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s; 2360/166260 objects misplaced (1.419%); 29 MiB/s, 461 objects/s recovering 2024-03-26T05:09:33.085 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:33.135 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:33.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:33.236 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:33.268 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:33.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:33.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:33.369 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:33.420 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T05:09:33.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:33.435 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:33.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:33.535 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:33.602 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:33.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:33.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:33.704 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:33.706 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:33 smithi080 ceph-mon[29691]: 3.10 scrub starts 2024-03-26T05:09:33.706 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:33 smithi080 ceph-mon[29691]: 3.10 scrub ok 2024-03-26T05:09:33.706 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:33 smithi080 ceph-mon[29691]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T05:09:33.706 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:33 smithi080 ceph-mon[29691]: 4.1a scrub starts 2024-03-26T05:09:33.706 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:33 smithi080 ceph-mon[29691]: 4.1a scrub ok 2024-03-26T05:09:33.707 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:33 smithi080 ceph-mon[40404]: 3.10 scrub starts 2024-03-26T05:09:33.707 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:33 smithi080 ceph-mon[40404]: 3.10 scrub ok 2024-03-26T05:09:33.707 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:33 smithi080 ceph-mon[40404]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T05:09:33.707 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:33 smithi080 ceph-mon[40404]: 4.1a scrub starts 2024-03-26T05:09:33.707 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:33 smithi080 ceph-mon[40404]: 4.1a scrub ok 2024-03-26T05:09:33.758 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:33.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:33.860 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:33.896 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:33.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:33.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:33 smithi132 ceph-mon[33500]: 3.10 scrub starts 2024-03-26T05:09:33.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:33 smithi132 ceph-mon[33500]: 3.10 scrub ok 2024-03-26T05:09:33.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:33 smithi132 ceph-mon[33500]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T05:09:33.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:33 smithi132 ceph-mon[33500]: 4.1a scrub starts 2024-03-26T05:09:33.944 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:33 smithi132 ceph-mon[33500]: 4.1a scrub ok 2024-03-26T05:09:33.998 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:34.051 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:34.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:34.152 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:34.200 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:34.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:34.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:34.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:34.318 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:09:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:09:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:09:34.339 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:34.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:34.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:34.443 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:34.479 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:34.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:34.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:34.581 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:34.617 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:34.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:34.661 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:09:34.661 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:09:34.661 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:09:34.661 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:09:34.661 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:09:34.661 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:09:34.661 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:09:34.661 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:09:34.662 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:09:34.662 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:09:34.662 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:09:34.662 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:09:34.662 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:09:34.662 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:09:34.662 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:09:34.662 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:09:34.673 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:34.718 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:34.757 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:34.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:34.858 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:34.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:34 smithi080 ceph-mon[40404]: pgmap v409: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 883 op/s; 2360/166260 objects misplaced (1.419%); 13 MiB/s, 202 objects/s recovering 2024-03-26T05:09:34.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:34 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1320138817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:09:34.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:34 smithi080 ceph-mon[29691]: pgmap v409: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 883 op/s; 2360/166260 objects misplaced (1.419%); 13 MiB/s, 202 objects/s recovering 2024-03-26T05:09:34.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:34 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1320138817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:09:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:34.897 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:34.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:34.958 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:34 smithi132 ceph-mon[33500]: pgmap v409: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 883 op/s; 2360/166260 objects misplaced (1.419%); 13 MiB/s, 202 objects/s recovering 2024-03-26T05:09:34.958 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:34 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1320138817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:09:34.999 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:35.041 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:35.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:35.142 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:35.178 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:35.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:35.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:35.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:35.292 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:09:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:09:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:35.316 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:35.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:35.417 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:35.455 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:35.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:35.557 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:35.594 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:35.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:35.696 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:35.730 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:35.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:35.832 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:35.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[40404]: 4.a scrub starts 2024-03-26T05:09:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[40404]: 4.a scrub ok 2024-03-26T05:09:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[40404]: 4.0 scrub starts 2024-03-26T05:09:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[40404]: 4.0 scrub ok 2024-03-26T05:09:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[40404]: 6.16 scrub starts 2024-03-26T05:09:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[40404]: 6.16 scrub ok 2024-03-26T05:09:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:09:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:09:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:35.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[29691]: 4.a scrub starts 2024-03-26T05:09:35.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[29691]: 4.a scrub ok 2024-03-26T05:09:35.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[29691]: 4.0 scrub starts 2024-03-26T05:09:35.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[29691]: 4.0 scrub ok 2024-03-26T05:09:35.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[29691]: 6.16 scrub starts 2024-03-26T05:09:35.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[29691]: 6.16 scrub ok 2024-03-26T05:09:35.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:09:35.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:09:35.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:35 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:35.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:35.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:35.971 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:36.011 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:36.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:36.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:36.133 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:35 smithi132 ceph-mon[33500]: 4.a scrub starts 2024-03-26T05:09:36.134 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:35 smithi132 ceph-mon[33500]: 4.a scrub ok 2024-03-26T05:09:36.134 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:35 smithi132 ceph-mon[33500]: 4.0 scrub starts 2024-03-26T05:09:36.134 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:35 smithi132 ceph-mon[33500]: 4.0 scrub ok 2024-03-26T05:09:36.134 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:35 smithi132 ceph-mon[33500]: 6.16 scrub starts 2024-03-26T05:09:36.134 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:35 smithi132 ceph-mon[33500]: 6.16 scrub ok 2024-03-26T05:09:36.134 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:09:36.134 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:09:36.134 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:35 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:36.156 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:36.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:36.257 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:36.285 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:36.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:36.386 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:36.419 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:36.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:36.521 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:36.553 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:36.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:36.655 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:36.696 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:36.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:36.797 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:36.842 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:36.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:36.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:36.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:36.983 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:36.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:37.085 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:37.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[29691]: 5.1a scrub starts 2024-03-26T05:09:37.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[29691]: 5.1a scrub ok 2024-03-26T05:09:37.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[29691]: 5.11 scrub starts 2024-03-26T05:09:37.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[29691]: 5.11 scrub ok 2024-03-26T05:09:37.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[29691]: pgmap v410: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.58k op/s; 1071/150714 objects misplaced (0.711%); 24 MiB/s, 388 objects/s recovering 2024-03-26T05:09:37.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[29691]: 5.e scrub starts 2024-03-26T05:09:37.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[29691]: 5.e scrub ok 2024-03-26T05:09:37.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[40404]: 5.1a scrub starts 2024-03-26T05:09:37.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[40404]: 5.1a scrub ok 2024-03-26T05:09:37.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[40404]: 5.11 scrub starts 2024-03-26T05:09:37.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[40404]: 5.11 scrub ok 2024-03-26T05:09:37.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[40404]: pgmap v410: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.58k op/s; 1071/150714 objects misplaced (0.711%); 24 MiB/s, 388 objects/s recovering 2024-03-26T05:09:37.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[40404]: 5.e scrub starts 2024-03-26T05:09:37.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:36 smithi080 ceph-mon[40404]: 5.e scrub ok 2024-03-26T05:09:37.121 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:37.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:37.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:37.223 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:37.259 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:37.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:37.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:36 smithi132 ceph-mon[33500]: 5.1a scrub starts 2024-03-26T05:09:37.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:36 smithi132 ceph-mon[33500]: 5.1a scrub ok 2024-03-26T05:09:37.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:36 smithi132 ceph-mon[33500]: 5.11 scrub starts 2024-03-26T05:09:37.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:36 smithi132 ceph-mon[33500]: 5.11 scrub ok 2024-03-26T05:09:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:36 smithi132 ceph-mon[33500]: pgmap v410: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.58k op/s; 1071/150714 objects misplaced (0.711%); 24 MiB/s, 388 objects/s recovering 2024-03-26T05:09:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:36 smithi132 ceph-mon[33500]: 5.e scrub starts 2024-03-26T05:09:37.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:36 smithi132 ceph-mon[33500]: 5.e scrub ok 2024-03-26T05:09:37.361 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:37.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:37.396 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:37.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:37.497 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:37.545 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:37.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:37.565 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:37.646 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:37.693 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:37.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:37.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:37.795 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:37.839 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:37.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:37.940 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:37.976 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:37.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:38.077 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:38.112 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:09:38.116 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:38.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:37 smithi080 ceph-mon[40404]: 6.5 scrub starts 2024-03-26T05:09:38.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:37 smithi080 ceph-mon[40404]: 6.5 scrub ok 2024-03-26T05:09:38.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:37 smithi080 ceph-mon[40404]: 5.1e scrub starts 2024-03-26T05:09:38.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:37 smithi080 ceph-mon[40404]: 5.1e scrub ok 2024-03-26T05:09:38.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:37 smithi080 ceph-mon[29691]: 6.5 scrub starts 2024-03-26T05:09:38.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:37 smithi080 ceph-mon[29691]: 6.5 scrub ok 2024-03-26T05:09:38.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:37 smithi080 ceph-mon[29691]: 5.1e scrub starts 2024-03-26T05:09:38.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:37 smithi080 ceph-mon[29691]: 5.1e scrub ok 2024-03-26T05:09:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:38.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:38.219 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:38.259 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:38.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:38.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:37 smithi132 ceph-mon[33500]: 6.5 scrub starts 2024-03-26T05:09:38.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:37 smithi132 ceph-mon[33500]: 6.5 scrub ok 2024-03-26T05:09:38.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:37 smithi132 ceph-mon[33500]: 5.1e scrub starts 2024-03-26T05:09:38.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:37 smithi132 ceph-mon[33500]: 5.1e scrub ok 2024-03-26T05:09:38.361 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:38.395 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:38.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:38.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:38.497 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:38.537 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:38.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:38.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:38.639 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:38.663 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:38.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:38.764 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:38.799 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:38.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:38.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:38.901 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:38.925 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:38.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:39.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:39.057 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:39.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:39.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[40404]: 3.d scrub starts 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[40404]: 3.d scrub ok 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[40404]: 6.9 scrub starts 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[40404]: pgmap v411: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 699 B/s rd, 0 B/s wr, 856 op/s; 1071/150714 objects misplaced (0.711%); 9.6 MiB/s, 153 objects/s recovering 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[40404]: 6.9 scrub ok 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[40404]: 6.e scrub starts 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[40404]: 6.e scrub ok 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[29691]: 3.d scrub starts 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[29691]: 3.d scrub ok 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[29691]: 6.9 scrub starts 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[29691]: pgmap v411: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 699 B/s rd, 0 B/s wr, 856 op/s; 1071/150714 objects misplaced (0.711%); 9.6 MiB/s, 153 objects/s recovering 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[29691]: 6.9 scrub ok 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[29691]: 6.e scrub starts 2024-03-26T05:09:39.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:38 smithi080 ceph-mon[29691]: 6.e scrub ok 2024-03-26T05:09:39.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:39.184 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:39.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:39.286 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:39.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:38 smithi132 ceph-mon[33500]: 3.d scrub starts 2024-03-26T05:09:39.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:38 smithi132 ceph-mon[33500]: 3.d scrub ok 2024-03-26T05:09:39.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:38 smithi132 ceph-mon[33500]: 6.9 scrub starts 2024-03-26T05:09:39.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:38 smithi132 ceph-mon[33500]: pgmap v411: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 699 B/s rd, 0 B/s wr, 856 op/s; 1071/150714 objects misplaced (0.711%); 9.6 MiB/s, 153 objects/s recovering 2024-03-26T05:09:39.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:38 smithi132 ceph-mon[33500]: 6.9 scrub ok 2024-03-26T05:09:39.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:38 smithi132 ceph-mon[33500]: 6.e scrub starts 2024-03-26T05:09:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:38 smithi132 ceph-mon[33500]: 6.e scrub ok 2024-03-26T05:09:39.309 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:39.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:39.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:39.411 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:39.434 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:39.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:39.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:39.536 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:39.559 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:39.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:39.661 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:39.686 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:39.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:39.787 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:39.811 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:39.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:39.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:39 smithi080 ceph-mon[29691]: 3.7 scrub starts 2024-03-26T05:09:39.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:39 smithi080 ceph-mon[29691]: 3.7 scrub ok 2024-03-26T05:09:39.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:39 smithi080 ceph-mon[29691]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T05:09:39.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:39 smithi080 ceph-mon[29691]: 6.1a scrub starts 2024-03-26T05:09:39.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:39 smithi080 ceph-mon[29691]: 6.1a scrub ok 2024-03-26T05:09:39.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:39 smithi080 ceph-mon[40404]: 3.7 scrub starts 2024-03-26T05:09:39.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:39 smithi080 ceph-mon[40404]: 3.7 scrub ok 2024-03-26T05:09:39.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:39 smithi080 ceph-mon[40404]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T05:09:39.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:39 smithi080 ceph-mon[40404]: 6.1a scrub starts 2024-03-26T05:09:39.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:39 smithi080 ceph-mon[40404]: 6.1a scrub ok 2024-03-26T05:09:39.913 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:39.946 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:39.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:40.048 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:40.074 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:40.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:40.176 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:40.200 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:40.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:39 smithi132 ceph-mon[33500]: 3.7 scrub starts 2024-03-26T05:09:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:39 smithi132 ceph-mon[33500]: 3.7 scrub ok 2024-03-26T05:09:40.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:39 smithi132 ceph-mon[33500]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T05:09:40.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:39 smithi132 ceph-mon[33500]: 6.1a scrub starts 2024-03-26T05:09:40.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:39 smithi132 ceph-mon[33500]: 6.1a scrub ok 2024-03-26T05:09:40.302 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:40.332 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:40.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:40.434 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:40.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:40.461 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:40.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:40.562 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:40.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:40.587 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:40.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:40.688 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:40.713 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:40.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:40.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:40.740 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:09:40.740 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:09:40.815 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:40.839 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:40.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:40.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:40.844 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:09:40.941 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:40.976 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:40.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:41.078 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:41.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[40404]: 6.12 scrub starts 2024-03-26T05:09:41.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[40404]: 6.12 scrub ok 2024-03-26T05:09:41.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[40404]: 3.2 scrub starts 2024-03-26T05:09:41.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[40404]: 3.2 scrub ok 2024-03-26T05:09:41.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[40404]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T05:09:41.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[40404]: pgmap v414: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 868 op/s; 662/147303 objects misplaced (0.449%); 13 MiB/s, 206 objects/s recovering 2024-03-26T05:09:41.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1152924536' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:09:41.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[29691]: 6.12 scrub starts 2024-03-26T05:09:41.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[29691]: 6.12 scrub ok 2024-03-26T05:09:41.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[29691]: 3.2 scrub starts 2024-03-26T05:09:41.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[29691]: 3.2 scrub ok 2024-03-26T05:09:41.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[29691]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T05:09:41.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[29691]: pgmap v414: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 868 op/s; 662/147303 objects misplaced (0.449%); 13 MiB/s, 206 objects/s recovering 2024-03-26T05:09:41.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:40 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1152924536' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:09:41.119 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:41.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:41.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:41.221 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:41.255 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:41.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:41.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:40 smithi132 ceph-mon[33500]: 6.12 scrub starts 2024-03-26T05:09:41.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:40 smithi132 ceph-mon[33500]: 6.12 scrub ok 2024-03-26T05:09:41.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:40 smithi132 ceph-mon[33500]: 3.2 scrub starts 2024-03-26T05:09:41.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:40 smithi132 ceph-mon[33500]: 3.2 scrub ok 2024-03-26T05:09:41.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:40 smithi132 ceph-mon[33500]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T05:09:41.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:40 smithi132 ceph-mon[33500]: pgmap v414: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 868 op/s; 662/147303 objects misplaced (0.449%); 13 MiB/s, 206 objects/s recovering 2024-03-26T05:09:41.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:40 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1152924536' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:09:41.357 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:41.384 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:41.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:41.486 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:41.519 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:41.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:41.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:41.620 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:41.650 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:41.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:41.751 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:41.788 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:41.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:41.890 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:41.926 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:41.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:41.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:42.028 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:42.064 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:42.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[40404]: 3.c scrub starts 2024-03-26T05:09:42.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[40404]: 3.c scrub ok 2024-03-26T05:09:42.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:09:42.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:09:42.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[40404]: 3.1e scrub starts 2024-03-26T05:09:42.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[40404]: 3.1e scrub ok 2024-03-26T05:09:42.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:09:42.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[40404]: Cluster is now healthy 2024-03-26T05:09:42.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1152924536' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:09:42.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[40404]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T05:09:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[29691]: 3.c scrub starts 2024-03-26T05:09:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[29691]: 3.c scrub ok 2024-03-26T05:09:42.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:09:42.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:09:42.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[29691]: 3.1e scrub starts 2024-03-26T05:09:42.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[29691]: 3.1e scrub ok 2024-03-26T05:09:42.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:09:42.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[29691]: Cluster is now healthy 2024-03-26T05:09:42.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1152924536' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:09:42.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:41 smithi080 ceph-mon[29691]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T05:09:42.166 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:42.209 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:42.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:42.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:41 smithi132 ceph-mon[33500]: 3.c scrub starts 2024-03-26T05:09:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:41 smithi132 ceph-mon[33500]: 3.c scrub ok 2024-03-26T05:09:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:41 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:09:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:41 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:09:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:41 smithi132 ceph-mon[33500]: 3.1e scrub starts 2024-03-26T05:09:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:41 smithi132 ceph-mon[33500]: 3.1e scrub ok 2024-03-26T05:09:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:41 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:09:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:41 smithi132 ceph-mon[33500]: Cluster is now healthy 2024-03-26T05:09:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:41 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1152924536' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:09:42.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:41 smithi132 ceph-mon[33500]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T05:09:42.310 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:42.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:42.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:42.434 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:42.463 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:42.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:42.564 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:42.596 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:42.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:42.698 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:42.734 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:42.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:42.835 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:42.872 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:42.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:42.974 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:43.010 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:43.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:43.112 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:42 smithi080 ceph-mon[29691]: 4.5 scrub starts 2024-03-26T05:09:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:42 smithi080 ceph-mon[29691]: 4.5 scrub ok 2024-03-26T05:09:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:42 smithi080 ceph-mon[29691]: 5.5 scrub starts 2024-03-26T05:09:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:42 smithi080 ceph-mon[29691]: 5.5 scrub ok 2024-03-26T05:09:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:42 smithi080 ceph-mon[29691]: pgmap v416: 148 pgs: 1 peering, 147 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 722 op/s; 11 MiB/s, 168 objects/s recovering 2024-03-26T05:09:43.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:42 smithi080 ceph-mon[40404]: 4.5 scrub starts 2024-03-26T05:09:43.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:42 smithi080 ceph-mon[40404]: 4.5 scrub ok 2024-03-26T05:09:43.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:42 smithi080 ceph-mon[40404]: 5.5 scrub starts 2024-03-26T05:09:43.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:42 smithi080 ceph-mon[40404]: 5.5 scrub ok 2024-03-26T05:09:43.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:42 smithi080 ceph-mon[40404]: pgmap v416: 148 pgs: 1 peering, 147 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 722 op/s; 11 MiB/s, 168 objects/s recovering 2024-03-26T05:09:43.146 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:43.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:43.247 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:43.281 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:43.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:43.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:43.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:42 smithi132 ceph-mon[33500]: 4.5 scrub starts 2024-03-26T05:09:43.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:42 smithi132 ceph-mon[33500]: 4.5 scrub ok 2024-03-26T05:09:43.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:42 smithi132 ceph-mon[33500]: 5.5 scrub starts 2024-03-26T05:09:43.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:42 smithi132 ceph-mon[33500]: 5.5 scrub ok 2024-03-26T05:09:43.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:42 smithi132 ceph-mon[33500]: pgmap v416: 148 pgs: 1 peering, 147 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 722 op/s; 11 MiB/s, 168 objects/s recovering 2024-03-26T05:09:43.382 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:43.419 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:43.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:43.520 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:43.556 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:43.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:43.657 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:43.689 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:43.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:43.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:43.791 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:43.824 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:43.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:43.925 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:43.949 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:43.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:44.050 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:44.085 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:44.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:44.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:43 smithi080 ceph-mon[29691]: 3.5 scrub starts 2024-03-26T05:09:44.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:43 smithi080 ceph-mon[29691]: 3.5 scrub ok 2024-03-26T05:09:44.116 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:09:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:09:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:09:44.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:43 smithi080 ceph-mon[40404]: 3.5 scrub starts 2024-03-26T05:09:44.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:43 smithi080 ceph-mon[40404]: 3.5 scrub ok 2024-03-26T05:09:44.186 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:44.220 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:44.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:44.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:43 smithi132 ceph-mon[33500]: 3.5 scrub starts 2024-03-26T05:09:44.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:43 smithi132 ceph-mon[33500]: 3.5 scrub ok 2024-03-26T05:09:44.323 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:44.359 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:44.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:44.460 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:44.514 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:44.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:44.615 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:44.655 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:44.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:44.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:44.756 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:44.790 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:44.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:44.892 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:44.923 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:44.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:44.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:45.024 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:45.065 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:45.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:45.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:44 smithi080 ceph-mon[29691]: pgmap v417: 148 pgs: 1 peering, 147 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 722 op/s; 11 MiB/s, 168 objects/s recovering 2024-03-26T05:09:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:44 smithi080 ceph-mon[40404]: pgmap v417: 148 pgs: 1 peering, 147 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 722 op/s; 11 MiB/s, 168 objects/s recovering 2024-03-26T05:09:45.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:45.203 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:45.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:45.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:45.218 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:44 smithi132 ceph-mon[33500]: pgmap v417: 148 pgs: 1 peering, 147 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 722 op/s; 11 MiB/s, 168 objects/s recovering 2024-03-26T05:09:45.304 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:45.341 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:45.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:45.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:45.442 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:45.482 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:45.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:45.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:09:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:09:45.583 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:45.626 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:45.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:45.727 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:45.768 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:45.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:45.870 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:45.909 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:45.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:46.010 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:46.048 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:46.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:46.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:45 smithi080 ceph-mon[40404]: 5.19 scrub starts 2024-03-26T05:09:46.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:45 smithi080 ceph-mon[40404]: 5.19 scrub ok 2024-03-26T05:09:46.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:45 smithi080 ceph-mon[40404]: 8.6 scrub starts 2024-03-26T05:09:46.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:45 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:45 smithi080 ceph-mon[29691]: 5.19 scrub starts 2024-03-26T05:09:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:45 smithi080 ceph-mon[29691]: 5.19 scrub ok 2024-03-26T05:09:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:45 smithi080 ceph-mon[29691]: 8.6 scrub starts 2024-03-26T05:09:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:45 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:46.150 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:46.211 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:46.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:46.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:46.234 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:45 smithi132 ceph-mon[33500]: 5.19 scrub starts 2024-03-26T05:09:46.234 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:45 smithi132 ceph-mon[33500]: 5.19 scrub ok 2024-03-26T05:09:46.234 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:45 smithi132 ceph-mon[33500]: 8.6 scrub starts 2024-03-26T05:09:46.234 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:45 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:46.312 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:46.363 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:46.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:46.420 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:46.465 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:46.505 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:46.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:46.606 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:46.645 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:46.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:46.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:46.746 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:46.786 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:46.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:46.867 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T05:09:46.888 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:46.919 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:46.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:47.020 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:47.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:47.055 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:47.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:47.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[40404]: 3.13 scrub starts 2024-03-26T05:09:47.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[40404]: 3.13 scrub ok 2024-03-26T05:09:47.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[40404]: 3.a deep-scrub starts 2024-03-26T05:09:47.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[40404]: 3.a deep-scrub ok 2024-03-26T05:09:47.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[40404]: 6.1f deep-scrub starts 2024-03-26T05:09:47.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[40404]: 8.6 scrub ok 2024-03-26T05:09:47.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[40404]: 6.1f deep-scrub ok 2024-03-26T05:09:47.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[40404]: pgmap v418: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.33k op/s; 9.0 MiB/s, 143 objects/s recovering 2024-03-26T05:09:47.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[29691]: 3.13 scrub starts 2024-03-26T05:09:47.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[29691]: 3.13 scrub ok 2024-03-26T05:09:47.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[29691]: 3.a deep-scrub starts 2024-03-26T05:09:47.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[29691]: 3.a deep-scrub ok 2024-03-26T05:09:47.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[29691]: 6.1f deep-scrub starts 2024-03-26T05:09:47.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[29691]: 8.6 scrub ok 2024-03-26T05:09:47.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[29691]: 6.1f deep-scrub ok 2024-03-26T05:09:47.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:46 smithi080 ceph-mon[29691]: pgmap v418: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.33k op/s; 9.0 MiB/s, 143 objects/s recovering 2024-03-26T05:09:47.155 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:47.187 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:47.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:47.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:47.289 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:47.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:46 smithi132 ceph-mon[33500]: 3.13 scrub starts 2024-03-26T05:09:47.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:46 smithi132 ceph-mon[33500]: 3.13 scrub ok 2024-03-26T05:09:47.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:46 smithi132 ceph-mon[33500]: 3.a deep-scrub starts 2024-03-26T05:09:47.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:46 smithi132 ceph-mon[33500]: 3.a deep-scrub ok 2024-03-26T05:09:47.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:46 smithi132 ceph-mon[33500]: 6.1f deep-scrub starts 2024-03-26T05:09:47.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:46 smithi132 ceph-mon[33500]: 8.6 scrub ok 2024-03-26T05:09:47.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:46 smithi132 ceph-mon[33500]: 6.1f deep-scrub ok 2024-03-26T05:09:47.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:46 smithi132 ceph-mon[33500]: pgmap v418: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.33k op/s; 9.0 MiB/s, 143 objects/s recovering 2024-03-26T05:09:47.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:47.325 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:47.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:47.426 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:47.462 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:47.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:47.564 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:47.595 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:09:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:47.600 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:47.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:47.701 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:47.747 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:47.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:47.848 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:47.882 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:47.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:47.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:47.890 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T05:09:47.984 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:48.020 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:48.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:48.122 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:48.183 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:48.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:48.277 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[40404]: 6.7 scrub starts 2024-03-26T05:09:48.277 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[40404]: 6.7 scrub ok 2024-03-26T05:09:48.277 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[40404]: 6.1d scrub starts 2024-03-26T05:09:48.277 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[40404]: 6.1d scrub ok 2024-03-26T05:09:48.278 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/22203889' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:09:48.278 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:09:48.278 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[40404]: 4.1 scrub starts 2024-03-26T05:09:48.278 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[40404]: 4.1 scrub ok 2024-03-26T05:09:48.278 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:09:48.278 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[40404]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T05:09:48.278 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[29691]: 6.7 scrub starts 2024-03-26T05:09:48.278 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[29691]: 6.7 scrub ok 2024-03-26T05:09:48.278 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[29691]: 6.1d scrub starts 2024-03-26T05:09:48.278 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[29691]: 6.1d scrub ok 2024-03-26T05:09:48.278 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/22203889' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:09:48.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:09:48.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[29691]: 4.1 scrub starts 2024-03-26T05:09:48.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[29691]: 4.1 scrub ok 2024-03-26T05:09:48.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:09:48.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:47 smithi080 ceph-mon[29691]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T05:09:48.285 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:47 smithi132 ceph-mon[33500]: 6.7 scrub starts 2024-03-26T05:09:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:47 smithi132 ceph-mon[33500]: 6.7 scrub ok 2024-03-26T05:09:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:47 smithi132 ceph-mon[33500]: 6.1d scrub starts 2024-03-26T05:09:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:47 smithi132 ceph-mon[33500]: 6.1d scrub ok 2024-03-26T05:09:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:47 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/22203889' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:09:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:47 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:09:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:47 smithi132 ceph-mon[33500]: 4.1 scrub starts 2024-03-26T05:09:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:47 smithi132 ceph-mon[33500]: 4.1 scrub ok 2024-03-26T05:09:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:47 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:09:48.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:47 smithi132 ceph-mon[33500]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T05:09:48.317 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:48.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:48.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:48.418 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:48.457 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:48.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:48.559 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:48.597 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:48.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:48.698 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:48.727 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:48.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:48.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:48.829 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:48.867 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:48.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:48.968 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:49.005 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:49.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:49.106 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:49.137 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:49.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:49.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:49.158 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[40404]: 4.2 scrub starts 2024-03-26T05:09:49.158 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[40404]: 4.2 scrub ok 2024-03-26T05:09:49.158 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[40404]: pgmap v420: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 4.9 MiB/s, 78 objects/s recovering 2024-03-26T05:09:49.158 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[40404]: 3.6 scrub starts 2024-03-26T05:09:49.159 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[40404]: 3.6 scrub ok 2024-03-26T05:09:49.159 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[40404]: 3.1 deep-scrub starts 2024-03-26T05:09:49.159 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[40404]: 3.1 deep-scrub ok 2024-03-26T05:09:49.159 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[29691]: 4.2 scrub starts 2024-03-26T05:09:49.159 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[29691]: 4.2 scrub ok 2024-03-26T05:09:49.159 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[29691]: pgmap v420: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 4.9 MiB/s, 78 objects/s recovering 2024-03-26T05:09:49.159 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[29691]: 3.6 scrub starts 2024-03-26T05:09:49.159 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[29691]: 3.6 scrub ok 2024-03-26T05:09:49.159 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[29691]: 3.1 deep-scrub starts 2024-03-26T05:09:49.159 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:48 smithi080 ceph-mon[29691]: 3.1 deep-scrub ok 2024-03-26T05:09:49.218 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 3 1 2024-03-26T05:09:49.231 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:49.240 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:49.266 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:49.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:48 smithi132 ceph-mon[33500]: 4.2 scrub starts 2024-03-26T05:09:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:48 smithi132 ceph-mon[33500]: 4.2 scrub ok 2024-03-26T05:09:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:48 smithi132 ceph-mon[33500]: pgmap v420: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 4.9 MiB/s, 78 objects/s recovering 2024-03-26T05:09:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:48 smithi132 ceph-mon[33500]: 3.6 scrub starts 2024-03-26T05:09:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:48 smithi132 ceph-mon[33500]: 3.6 scrub ok 2024-03-26T05:09:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:48 smithi132 ceph-mon[33500]: 3.1 deep-scrub starts 2024-03-26T05:09:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:48 smithi132 ceph-mon[33500]: 3.1 deep-scrub ok 2024-03-26T05:09:49.369 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:49.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:49.418 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:49.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:49.460 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:49.519 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:49.548 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:49.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:49.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:49.649 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:49.692 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:49.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:49.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:49.794 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:49.840 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:49.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:49.941 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:49.978 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:49.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:49.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:50.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[29691]: 6.19 scrub starts 2024-03-26T05:09:50.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[29691]: 6.19 scrub ok 2024-03-26T05:09:50.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[29691]: 4.1c scrub starts 2024-03-26T05:09:50.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[29691]: 4.1c scrub ok 2024-03-26T05:09:50.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[29691]: 3.f scrub starts 2024-03-26T05:09:50.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[29691]: 3.f scrub ok 2024-03-26T05:09:50.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[29691]: 4.c scrub starts 2024-03-26T05:09:50.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[29691]: 4.c scrub ok 2024-03-26T05:09:50.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:09:50.042 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:09:50.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[40404]: 6.19 scrub starts 2024-03-26T05:09:50.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[40404]: 6.19 scrub ok 2024-03-26T05:09:50.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[40404]: 4.1c scrub starts 2024-03-26T05:09:50.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[40404]: 4.1c scrub ok 2024-03-26T05:09:50.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[40404]: 3.f scrub starts 2024-03-26T05:09:50.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[40404]: 3.f scrub ok 2024-03-26T05:09:50.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[40404]: 4.c scrub starts 2024-03-26T05:09:50.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[40404]: 4.c scrub ok 2024-03-26T05:09:50.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:09:50.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:09:50.080 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:50.110 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:50.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:50.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:50.211 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:50.249 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:50.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:50.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:50.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:49 smithi132 ceph-mon[33500]: 6.19 scrub starts 2024-03-26T05:09:50.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:49 smithi132 ceph-mon[33500]: 6.19 scrub ok 2024-03-26T05:09:50.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:49 smithi132 ceph-mon[33500]: 4.1c scrub starts 2024-03-26T05:09:50.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:49 smithi132 ceph-mon[33500]: 4.1c scrub ok 2024-03-26T05:09:50.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:49 smithi132 ceph-mon[33500]: 3.f scrub starts 2024-03-26T05:09:50.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:49 smithi132 ceph-mon[33500]: 3.f scrub ok 2024-03-26T05:09:50.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:49 smithi132 ceph-mon[33500]: 4.c scrub starts 2024-03-26T05:09:50.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:49 smithi132 ceph-mon[33500]: 4.c scrub ok 2024-03-26T05:09:50.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:09:50.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:09:50.351 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:50.390 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:50.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:50.491 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:50.657 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:50.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:50.758 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:50.793 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:50.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:50.894 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:50.928 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:50.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:50.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:51.029 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:51.076 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:51.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:51.178 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:51.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:50 smithi080 ceph-mon[29691]: 5.10 scrub starts 2024-03-26T05:09:51.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:50 smithi080 ceph-mon[29691]: 5.10 scrub ok 2024-03-26T05:09:51.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:50 smithi080 ceph-mon[29691]: pgmap v421: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 722 op/s 2024-03-26T05:09:51.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:50 smithi080 ceph-mon[29691]: 4.6 scrub starts 2024-03-26T05:09:51.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:50 smithi080 ceph-mon[29691]: 4.6 scrub ok 2024-03-26T05:09:51.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:50 smithi080 ceph-mon[29691]: 8.2 scrub starts 2024-03-26T05:09:51.191 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:50 smithi080 ceph-mon[40404]: 5.10 scrub starts 2024-03-26T05:09:51.191 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:50 smithi080 ceph-mon[40404]: 5.10 scrub ok 2024-03-26T05:09:51.191 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:50 smithi080 ceph-mon[40404]: pgmap v421: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 722 op/s 2024-03-26T05:09:51.191 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:50 smithi080 ceph-mon[40404]: 4.6 scrub starts 2024-03-26T05:09:51.192 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:50 smithi080 ceph-mon[40404]: 4.6 scrub ok 2024-03-26T05:09:51.192 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:50 smithi080 ceph-mon[40404]: 8.2 scrub starts 2024-03-26T05:09:51.211 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:51.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:51.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:50 smithi132 ceph-mon[33500]: 5.10 scrub starts 2024-03-26T05:09:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:50 smithi132 ceph-mon[33500]: 5.10 scrub ok 2024-03-26T05:09:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:50 smithi132 ceph-mon[33500]: pgmap v421: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 722 op/s 2024-03-26T05:09:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:50 smithi132 ceph-mon[33500]: 4.6 scrub starts 2024-03-26T05:09:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:50 smithi132 ceph-mon[33500]: 4.6 scrub ok 2024-03-26T05:09:51.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:50 smithi132 ceph-mon[33500]: 8.2 scrub starts 2024-03-26T05:09:51.313 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:51.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:51.351 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:51.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:51.452 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:51.505 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:51.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:51.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:51.607 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:51.644 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:51.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:51.746 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:51.746 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:09:51.776 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:51.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:51.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:51.788 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:09:51.833 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:09:51.875 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:09:51.878 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:51.912 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:09:51.917 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:51.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:51.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:51.953 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:09:51.992 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:09:52.019 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:52.035 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:09:52.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:51 smithi080 ceph-mon[40404]: 8.9 deep-scrub starts 2024-03-26T05:09:52.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:51 smithi080 ceph-mon[40404]: 8.c scrub starts 2024-03-26T05:09:52.037 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:51 smithi080 ceph-mon[40404]: 8.2 scrub ok 2024-03-26T05:09:52.037 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:51 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1971099315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:09:52.037 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:51 smithi080 ceph-mon[29691]: 8.9 deep-scrub starts 2024-03-26T05:09:52.037 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:51 smithi080 ceph-mon[29691]: 8.c scrub starts 2024-03-26T05:09:52.037 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:51 smithi080 ceph-mon[29691]: 8.2 scrub ok 2024-03-26T05:09:52.037 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:51 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1971099315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:09:52.039 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:09:52.039 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:09:52.039 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:09:52.039 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:09:52.039 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:09:52.039 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:09:52.039 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:09:52.040 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:09:52.057 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:52.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:52.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:52.194 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:52.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:52.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:51 smithi132 ceph-mon[33500]: 8.9 deep-scrub starts 2024-03-26T05:09:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:51 smithi132 ceph-mon[33500]: 8.c scrub starts 2024-03-26T05:09:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:51 smithi132 ceph-mon[33500]: 8.2 scrub ok 2024-03-26T05:09:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:51 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1971099315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:09:52.296 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:52.328 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:52.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:52.430 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:52.463 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:52.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:52.565 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:52.600 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:52.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:52.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:52.702 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:52.737 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:52.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:52.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:52.839 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:52.877 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:52.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:52.979 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:53.013 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:53.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:53.115 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:53.153 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:53.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:53.254 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:53.290 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:53.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:52 smithi132 ceph-mon[33500]: 8.c scrub ok 2024-03-26T05:09:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:52 smithi132 ceph-mon[33500]: 8.e scrub starts 2024-03-26T05:09:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:52 smithi132 ceph-mon[33500]: pgmap v422: 148 pgs: 148 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-26T05:09:53.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:52 smithi080 ceph-mon[29691]: 8.c scrub ok 2024-03-26T05:09:53.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:52 smithi080 ceph-mon[29691]: 8.e scrub starts 2024-03-26T05:09:53.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:52 smithi080 ceph-mon[29691]: pgmap v422: 148 pgs: 148 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-26T05:09:53.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:52 smithi080 ceph-mon[40404]: 8.c scrub ok 2024-03-26T05:09:53.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:52 smithi080 ceph-mon[40404]: 8.e scrub starts 2024-03-26T05:09:53.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:52 smithi080 ceph-mon[40404]: pgmap v422: 148 pgs: 148 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-26T05:09:53.392 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:53.424 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:53.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:53.526 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:53.561 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:53.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:53.662 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:53.695 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:53.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:53.797 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:53.830 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:53.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:53.932 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:53.972 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:53.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:54.073 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:54.107 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:54.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:54.208 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:54.223 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:53 smithi080 ceph-mon[29691]: 5.16 scrub starts 2024-03-26T05:09:54.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:53 smithi080 ceph-mon[29691]: 5.16 scrub ok 2024-03-26T05:09:54.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:53 smithi080 ceph-mon[29691]: 8.e scrub ok 2024-03-26T05:09:54.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:53 smithi080 ceph-mon[29691]: 4.9 deep-scrub starts 2024-03-26T05:09:54.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:53 smithi080 ceph-mon[29691]: 4.9 deep-scrub ok 2024-03-26T05:09:54.224 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:09:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:09:54.224 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:53 smithi080 ceph-mon[40404]: 5.16 scrub starts 2024-03-26T05:09:54.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:53 smithi080 ceph-mon[40404]: 5.16 scrub ok 2024-03-26T05:09:54.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:53 smithi080 ceph-mon[40404]: 8.e scrub ok 2024-03-26T05:09:54.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:53 smithi080 ceph-mon[40404]: 4.9 deep-scrub starts 2024-03-26T05:09:54.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:53 smithi080 ceph-mon[40404]: 4.9 deep-scrub ok 2024-03-26T05:09:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:54.249 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:54.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:54.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:53 smithi132 ceph-mon[33500]: 5.16 scrub starts 2024-03-26T05:09:54.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:53 smithi132 ceph-mon[33500]: 5.16 scrub ok 2024-03-26T05:09:54.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:53 smithi132 ceph-mon[33500]: 8.e scrub ok 2024-03-26T05:09:54.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:53 smithi132 ceph-mon[33500]: 4.9 deep-scrub starts 2024-03-26T05:09:54.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:53 smithi132 ceph-mon[33500]: 4.9 deep-scrub ok 2024-03-26T05:09:54.350 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:54.387 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:54.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:54.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:54.489 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:54.531 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:54.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:54.633 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:54.661 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:54.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:54.763 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:54.797 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:54.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:54.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:54.899 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:54.933 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:54.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:55.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:55.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:55.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:55.170 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:55.203 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:55.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:55.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:55.231 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:54 smithi132 ceph-mon[33500]: 3.15 scrub starts 2024-03-26T05:09:55.231 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:54 smithi132 ceph-mon[33500]: 3.15 scrub ok 2024-03-26T05:09:55.231 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:54 smithi132 ceph-mon[33500]: pgmap v423: 148 pgs: 148 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-26T05:09:55.231 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:54 smithi132 ceph-mon[33500]: 4.3 deep-scrub starts 2024-03-26T05:09:55.231 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:54 smithi132 ceph-mon[33500]: 4.3 deep-scrub ok 2024-03-26T05:09:55.231 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:54 smithi132 ceph-mon[33500]: 5.1c scrub starts 2024-03-26T05:09:55.231 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:54 smithi132 ceph-mon[33500]: 5.1c scrub ok 2024-03-26T05:09:55.305 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:55.343 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:55.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:55.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[40404]: 3.15 scrub starts 2024-03-26T05:09:55.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[40404]: 3.15 scrub ok 2024-03-26T05:09:55.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[40404]: pgmap v423: 148 pgs: 148 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-26T05:09:55.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[40404]: 4.3 deep-scrub starts 2024-03-26T05:09:55.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[40404]: 4.3 deep-scrub ok 2024-03-26T05:09:55.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[40404]: 5.1c scrub starts 2024-03-26T05:09:55.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[40404]: 5.1c scrub ok 2024-03-26T05:09:55.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[29691]: 3.15 scrub starts 2024-03-26T05:09:55.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[29691]: 3.15 scrub ok 2024-03-26T05:09:55.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[29691]: pgmap v423: 148 pgs: 148 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-26T05:09:55.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[29691]: 4.3 deep-scrub starts 2024-03-26T05:09:55.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[29691]: 4.3 deep-scrub ok 2024-03-26T05:09:55.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[29691]: 5.1c scrub starts 2024-03-26T05:09:55.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:54 smithi080 ceph-mon[29691]: 5.1c scrub ok 2024-03-26T05:09:55.444 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:55.479 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:55.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:55.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:55.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:09:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:09:55.581 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:55.613 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:55.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:55.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:55.715 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:55.756 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:55.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:55.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:55.858 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:55.890 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:55.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:55.992 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:56.041 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:56.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:56.142 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:56.181 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:56.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:56.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:56.243 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:55 smithi080 ceph-mon[29691]: 3.4 scrub starts 2024-03-26T05:09:56.243 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:55 smithi080 ceph-mon[29691]: 3.4 scrub ok 2024-03-26T05:09:56.243 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:55 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:56.243 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:55 smithi080 ceph-mon[29691]: 4.18 scrub starts 2024-03-26T05:09:56.243 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:55 smithi080 ceph-mon[29691]: 4.18 scrub ok 2024-03-26T05:09:56.243 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:55 smithi080 ceph-mon[40404]: 3.4 scrub starts 2024-03-26T05:09:56.243 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:55 smithi080 ceph-mon[40404]: 3.4 scrub ok 2024-03-26T05:09:56.243 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:55 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:56.243 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:55 smithi080 ceph-mon[40404]: 4.18 scrub starts 2024-03-26T05:09:56.243 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:55 smithi080 ceph-mon[40404]: 4.18 scrub ok 2024-03-26T05:09:56.283 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:55 smithi132 ceph-mon[33500]: 3.4 scrub starts 2024-03-26T05:09:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:55 smithi132 ceph-mon[33500]: 3.4 scrub ok 2024-03-26T05:09:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:55 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:09:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:55 smithi132 ceph-mon[33500]: 4.18 scrub starts 2024-03-26T05:09:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:55 smithi132 ceph-mon[33500]: 4.18 scrub ok 2024-03-26T05:09:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:56.314 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:56.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:56.415 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:56.452 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:56.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:56.554 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:56.586 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:56.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:56.688 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:56.723 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:56.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:56.825 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:09:56.859 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:56.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:56.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:09:56.961 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:56.990 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:56.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:57.092 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:57.131 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:09:57.131 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206018 7.b 7:d4295363:::benchmark_data_smithi080_80848_object86519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.872555+0000", 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.588883017, 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.670752888, 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206018 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872555+0000", 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872555+0000", 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872556+0000", 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3629999999999999e-06 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872562+0000", 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5949999999999998e-06 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872577+0000", 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5183e-05 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872587+0000", 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4620000000000002e-06 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872681+0000", 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4724999999999999e-05 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.873453+0000", 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00077158499999999998 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.873565+0000", 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011171900000000001 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.900679+0000", 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.027114513 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.900691+0000", 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2386e-05 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.136 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392310+0000", 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.491618318 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392323+0000", 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2852e-05 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.543230+0000", 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1509076469999999 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.137 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.543308+0000", 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7540000000000006e-05 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206009 7.0 7:0aae643b:::benchmark_data_smithi080_80848_object86510:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.886333+0000", 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.57510482200001, 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6852875250000001, 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.138 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206009 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886333+0000", 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886333+0000", 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.139 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886335+0000", 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2790000000000001e-06 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886341+0000", 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6509999999999998e-06 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886354+0000", 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2736e-05 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.140 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886364+0000", 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0159999999999999e-05 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.887501+0000", 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0011366639999999999 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.887510+0000", 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8609999999999993e-06 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.141 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.394783+0000", 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.50727263899999997 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.394988+0000", 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020464800000000001 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.569805+0000", 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1748175009999999 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.142 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.571621+0000", 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001815386 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206029 7.0 7:0428147a:::benchmark_data_smithi080_80848_object86530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.897565+0000", 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.56387279699999, 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7163614869999999, 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.143 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206029 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897565+0000", 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897565+0000", 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.144 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897566+0000", 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.319e-06 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897569+0000", 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.7920000000000001e-06 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897578+0000", 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4840000000000004e-06 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.145 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897585+0000", 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5750000000000003e-06 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.898208+0000", 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00062269599999999995 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.898218+0000", 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0395e-05 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.146 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395017+0000", 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.496798144 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395026+0000", 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6509999999999996e-06 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.571919+0000", 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1768926829999999 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.147 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.572211+0000", 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00029205600000000001 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.572422+0000", 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021075299999999999 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591634+0000", 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.019212002999999998 2024-03-26T05:09:57.148 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596999+0000", 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0053655129999999997 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597046+0000", 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6810000000000001e-05 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613723+0000", 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016677112000000001 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.149 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613791+0000", 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8207999999999995e-05 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613927+0000", 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013529300000000001 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206032 7.b 7:db699848:::benchmark_data_smithi080_80848_object86533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:09:57.150 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.907812+0000", 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.55362549099999, 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.654457345, 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206032 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907812+0000", 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.151 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907812+0000", 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907814+0000", 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7710000000000001e-06 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907821+0000", 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0049999999999998e-06 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907845+0000", 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4151e-05 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907854+0000", 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8279999999999992e-06 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.909278+0000", 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014236120000000001 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.153 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.909289+0000", 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1054e-05 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392352+0000", 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.48306306100000002 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392361+0000", 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9190000000000005e-06 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.154 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.543561+0000", 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.151200486 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.543686+0000", 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012433600000000001 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,5,6", 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.544119+0000", 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00043299000000000001 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.155 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558280+0000", 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014161366999999999 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558330+0000", 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0148000000000002e-05 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558377+0000", 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6708000000000001e-05 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.156 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.561932+0000", 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0035554779999999999 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562152+0000", 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021959599999999999 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562270+0000", 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117835 2024-03-26T05:09:57.157 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206042 7.0 7:0cf7d89a:::benchmark_data_smithi080_80848_object86543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.917369+0000", 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.544068993, 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.696765531, 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206042 2024-03-26T05:09:57.158 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917369+0000", 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917369+0000", 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917371+0000", 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.232e-06 2024-03-26T05:09:57.159 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917379+0000", 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2479999999999996e-06 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917402+0000", 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2521999999999999e-05 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917444+0000", 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1974999999999997e-05 2024-03-26T05:09:57.160 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917648+0000", 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020391800000000001 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917662+0000", 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4052e-05 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395054+0000", 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.47739227899999997 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.161 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395062+0000", 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2689999999999999e-06 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.586328+0000", 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.191265663 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.587220+0000", 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00089233699999999999 2024-03-26T05:09:57.162 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.587549+0000", 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00032828700000000002 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597064+0000", 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0095153470000000004 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597117+0000", 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2428e-05 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.163 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613547+0000", 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016430177000000001 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613969+0000", 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00042261700000000002 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614022+0000", 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2505e-05 2024-03-26T05:09:57.164 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614134+0000", 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000112675 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206043 7.0 7:0c4ab3a4:::benchmark_data_smithi080_80848_object86544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.917482+0000", 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.54395629800001, 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6968406790000001, 2024-03-26T05:09:57.165 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206043 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917482+0000", 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917482+0000", 2024-03-26T05:09:57.166 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917482+0000", 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7299999999999996e-07 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917487+0000", 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5020000000000003e-06 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917495+0000", 2024-03-26T05:09:57.167 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3049999999999993e-06 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917501+0000", 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1340000000000001e-06 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917687+0000", 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018563299999999999 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917694+0000", 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.708e-06 2024-03-26T05:09:57.168 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395094+0000", 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.47740021100000002 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395102+0000", 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2160000000000006e-06 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591668+0000", 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1965658779999999 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.169 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591783+0000", 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011553 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591918+0000", 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000134303 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597070+0000", 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051523330000000003 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.170 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613593+0000", 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016523013 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613637+0000", 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4057999999999997e-05 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614180+0000", 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00054313899999999999 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.171 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614232+0000", 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2014e-05 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614322+0000", 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0129000000000004e-05 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206045 7.0 7:0d078525:::benchmark_data_smithi080_80848_object86546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.919727+0000", 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.54171056400003, 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.707372347, 2024-03-26T05:09:57.172 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206045 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919727+0000", 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.173 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919727+0000", 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919729+0000", 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9149999999999999e-06 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919737+0000", 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.926e-06 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.174 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919757+0000", 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0190000000000002e-05 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919767+0000", 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9899999999999992e-06 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919857+0000", 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9819000000000002e-05 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.175 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919869+0000", 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2241e-05 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395136+0000", 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.47526629399999998 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395144+0000", 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1950000000000003e-06 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.176 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.594264+0000", 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.199119764 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.594373+0000", 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010950699999999999 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.594495+0000", 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000122044 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.177 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597075+0000", 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002579951 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613680+0000", 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016604739 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614358+0000", 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00067832500000000002 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.178 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.626740+0000", 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.012381614000000001 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.626798+0000", 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8226000000000001e-05 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.627100+0000", 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00030160700000000001 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.179 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206049 7.b 7:d42fcd17:::benchmark_data_smithi080_80848_object86550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.923737+0000", 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.537700573, 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.638929509, 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206049 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.180 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923737+0000", 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923737+0000", 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923739+0000", 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9750000000000001e-06 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923747+0000", 2024-03-26T05:09:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0439999999999996e-06 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923781+0000", 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4029000000000002e-05 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923793+0000", 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1834999999999999e-05 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923882+0000", 2024-03-26T05:09:57.182 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8941000000000001e-05 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923895+0000", 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2989e-05 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392396+0000", 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.46850114399999998 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392404+0000", 2024-03-26T05:09:57.183 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9589999999999995e-06 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.554545+0000", 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.162140891 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.554683+0000", 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013746099999999999 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,5,6", 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.555704+0000", 2024-03-26T05:09:57.184 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001020927 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558652+0000", 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0029487879999999999 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558667+0000", 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4798e-05 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.185 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558710+0000", 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2973999999999999e-05 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562312+0000", 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036020919999999999 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562556+0000", 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000244057 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.186 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562667+0000", 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000110605 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206053 7.0 7:08b24a04:::benchmark_data_smithi080_80848_object86554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.926223+0000", 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.53521455999999, 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7011446530000001, 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.187 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206053 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926223+0000", 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926223+0000", 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.188 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926225+0000", 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5409999999999999e-06 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926230+0000", 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5219999999999998e-06 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926246+0000", 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5339999999999999e-05 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.189 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926255+0000", 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7110000000000001e-06 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926346+0000", 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1246e-05 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926354+0000", 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0260000000000007e-06 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.190 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395168+0000", 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.46881425399999999 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395176+0000", 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0439999999999996e-06 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597154+0000", 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2019782939999999 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597287+0000", 2024-03-26T05:09:57.191 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000132848 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597420+0000", 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000132917 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614394+0000", 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016974182000000001 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614560+0000", 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016532100000000001 2024-03-26T05:09:57.192 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.619983+0000", 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0054232200000000003 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.627148+0000", 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0071655720000000003 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.627250+0000", 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010138599999999999 2024-03-26T05:09:57.193 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.627368+0000", 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118229 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236543 7.0 7:02068409:::benchmark_data_smithi080_80848_object117044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.452286+0000", 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 223.00915146299999, 2024-03-26T05:09:57.194 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.001309335, 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236543 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452286+0000", 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.195 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452286+0000", 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452288+0000", 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4249999999999999e-06 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452293+0000", 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8969999999999999e-06 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.196 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452305+0000", 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2170999999999999e-05 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452313+0000", 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6329999999999996e-06 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452604+0000", 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000290943 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452613+0000", 2024-03-26T05:09:57.197 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0860000000000008e-06 2024-03-26T05:09:57.198 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.198 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.198 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.198 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.447196+0000", 2024-03-26T05:09:57.198 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99458339399999995 2024-03-26T05:09:57.198 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.198 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.198 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.198 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.447355+0000", 2024-03-26T05:09:57.198 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015935600000000001 2024-03-26T05:09:57.198 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.198 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.447444+0000", 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8975999999999996e-05 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.449713+0000", 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022690480000000001 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.449794+0000", 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1021999999999998e-05 2024-03-26T05:09:57.199 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453503+0000", 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0037082650000000001 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453557+0000", 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4512000000000003e-05 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453596+0000", 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8606999999999998e-05 2024-03-26T05:09:57.200 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236550 7.0 7:0e84af01:::benchmark_data_smithi080_80848_object117051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.460859+0000", 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 223.00057931800001, 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99296332399999998, 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236550 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.201 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460859+0000", 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460859+0000", 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460860+0000", 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1090000000000001e-06 2024-03-26T05:09:57.202 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460865+0000", 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5389999999999998e-06 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460881+0000", 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6152999999999999e-05 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460889+0000", 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7100000000000007e-06 2024-03-26T05:09:57.203 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460950+0000", 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1356e-05 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460959+0000", 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6209999999999993e-06 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.449848+0000", 2024-03-26T05:09:57.204 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98888869099999999 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.449965+0000", 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117137 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.450042+0000", 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7497999999999996e-05 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453686+0000", 2024-03-26T05:09:57.205 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036440470000000001 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453744+0000", 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7787000000000001e-05 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453769+0000", 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4307e-05 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453799+0000", 2024-03-26T05:09:57.206 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.0151999999999999e-05 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453822+0000", 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3217000000000001e-05 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236558 7.0 7:0d5d8d1f:::benchmark_data_smithi080_80848_object117059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.474483+0000", 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 222.98695460299999, 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97948798000000004, 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.207 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236558 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474483+0000", 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474483+0000", 2024-03-26T05:09:57.208 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474485+0000", 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.412e-06 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474488+0000", 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6160000000000002e-06 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474497+0000", 2024-03-26T05:09:57.209 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9619999999999999e-06 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474504+0000", 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4899999999999997e-06 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474602+0000", 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8004999999999994e-05 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474611+0000", 2024-03-26T05:09:57.210 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6840000000000002e-06 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.451610+0000", 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97699924800000004 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.451703+0000", 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2864999999999999e-05 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.451781+0000", 2024-03-26T05:09:57.211 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8761e-05 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453735+0000", 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0019537589999999998 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453749+0000", 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4021e-05 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453879+0000", 2024-03-26T05:09:57.212 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000129605 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453931+0000", 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1987999999999998e-05 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453971+0000", 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0564e-05 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.213 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236560 7.0 7:0afff4a7:::benchmark_data_smithi080_80848_object117061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.474521+0000", 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 222.98691686500001, 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98418719899999996, 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236560 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.214 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474521+0000", 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474521+0000", 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474521+0000", 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1300000000000001e-07 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.215 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474524+0000", 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.289e-06 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474530+0000", 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2160000000000003e-06 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474535+0000", 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.011e-06 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.216 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474632+0000", 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6506000000000001e-05 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474637+0000", 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3539999999999999e-06 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.454019+0000", 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97938261800000004 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.217 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.454151+0000", 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013146800000000001 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.454275+0000", 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000123635 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458530+0000", 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0042555229999999998 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.218 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458574+0000", 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4110999999999998e-05 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458589+0000", 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.505e-05 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458635+0000", 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5636000000000001e-05 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.219 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458708+0000", 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3368999999999995e-05 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236566 7.0 7:0070f26d:::benchmark_data_smithi080_80848_object117067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.480345+0000", 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 222.98109312400001, 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98387816699999997, 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.220 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236566 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480345+0000", 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480345+0000", 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.221 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480347+0000", 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6750000000000001e-06 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480351+0000", 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9309999999999999e-06 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480367+0000", 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5599000000000001e-05 2024-03-26T05:09:57.222 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480376+0000", 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7949999999999996e-06 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480438+0000", 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2456e-05 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480448+0000", 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4320000000000005e-06 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.223 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.456042+0000", 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97559416799999998 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.456124+0000", 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2108999999999996e-05 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.456202+0000", 2024-03-26T05:09:57.224 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8367000000000004e-05 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458755+0000", 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0025522359999999998 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458765+0000", 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0862e-05 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464122+0000", 2024-03-26T05:09:57.225 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0053569200000000003 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464175+0000", 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2556999999999998e-05 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464223+0000", 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8059999999999997e-05 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.226 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236568 7.0 7:07c96239:::benchmark_data_smithi080_80848_object117069:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.480947+0000", 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 222.98049055600001, 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98352315300000004, 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236568 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.227 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480947+0000", 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480947+0000", 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480948+0000", 2024-03-26T05:09:57.228 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0470000000000001e-06 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480953+0000", 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7589999999999997e-06 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480969+0000", 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5812e-05 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480977+0000", 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7659999999999998e-06 2024-03-26T05:09:57.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.481028+0000", 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0794999999999997e-05 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.481035+0000", 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3629999999999996e-06 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458791+0000", 2024-03-26T05:09:57.230 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97775615100000002 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458955+0000", 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000164411 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.459033+0000", 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7306999999999997e-05 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464267+0000", 2024-03-26T05:09:57.231 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0052343420000000003 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464329+0000", 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1607999999999997e-05 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464386+0000", 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7380000000000003e-05 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464431+0000", 2024-03-26T05:09:57.232 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5321999999999997e-05 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464471+0000", 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.909e-05 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236570 7.0 7:03c0468e:::benchmark_data_smithi080_80848_object117071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.482326+0000", 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 222.97911223099999, 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98740077800000003, 2024-03-26T05:09:57.233 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236570 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482326+0000", 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.234 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482326+0000", 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482327+0000", 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.522e-06 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482332+0000", 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8600000000000001e-06 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.235 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482347+0000", 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4577e-05 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482354+0000", 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7160000000000003e-06 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482406+0000", 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1476e-05 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.236 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482415+0000", 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2310000000000002e-06 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.461304+0000", 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97888937600000003 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.461434+0000", 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012907000000000001 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:09:57.237 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.461601+0000", 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000167589 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464375+0000", 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027743030000000001 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464479+0000", 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103106 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.238 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469623+0000", 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051441489999999998 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469676+0000", 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3794e-05 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469726+0000", 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0009000000000002e-05 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.239 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236654 7.0 7:028e4924:::benchmark_data_smithi080_80848_object117155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.555854+0000", 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 222.90558403599999, 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93320793999999996, 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236654 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.240 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555854+0000", 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555854+0000", 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555856+0000", 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6899999999999999e-06 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.241 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555861+0000", 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3449999999999996e-06 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555877+0000", 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5977e-05 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555885+0000", 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9959999999999993e-06 2024-03-26T05:09:57.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555943+0000", 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8131999999999998e-05 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555951+0000", 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9440000000000005e-06 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.472491+0000", 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.91653987999999997 2024-03-26T05:09:57.243 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.472604+0000", 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000113354 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.472720+0000", 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115812 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.483540+0000", 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010820412 2024-03-26T05:09:57.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.483703+0000", 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000162645 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.488951+0000", 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0052477599999999998 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489011+0000", 2024-03-26T05:09:57.245 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9954000000000002e-05 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489062+0000", 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1038999999999997e-05 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142102 8.a 8:5d371889:::benchmark_data_smithi080_109213_object27585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:54.540155+0000", 2024-03-26T05:09:57.246 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 59.921282564000002, 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.032435765, 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142102 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540155+0000", 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.247 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540155+0000", 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540157+0000", 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.505e-06 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540162+0000", 2024-03-26T05:09:57.248 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4240000000000001e-06 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540173+0000", 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1156e-05 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540181+0000", 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2230000000000001e-06 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540234+0000", 2024-03-26T05:09:57.249 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2961000000000003e-05 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540370+0000", 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000136611 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540527+0000", 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015644199999999999 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.250 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.549579+0000", 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0090522169999999996 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.550582+0000", 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010026220000000001 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.555385+0000", 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0048038550000000001 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.251 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.576371+0000", 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.020985875000000001 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.576390+0000", 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8377e-05 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.572512+0000", 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99612238600000003 2024-03-26T05:09:57.252 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.572591+0000", 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9110999999999995e-05 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142135 8.a 8:57959c30:::benchmark_data_smithi080_109213_object27618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:54.600498+0000", 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 59.860940118000002, 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.981086244, 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.253 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142135 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600498+0000", 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.254 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600498+0000", 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600499+0000", 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4529999999999999e-06 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600506+0000", 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4169999999999997e-06 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.255 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600527+0000", 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0822000000000002e-05 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600535+0000", 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9379999999999998e-06 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600632+0000", 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6347000000000004e-05 2024-03-26T05:09:57.256 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600641+0000", 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6239999999999997e-06 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.573005+0000", 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97236343199999997 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.573102+0000", 2024-03-26T05:09:57.257 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.666e-05 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,5", 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.573212+0000", 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011006999999999999 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.578723+0000", 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.005511719 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.258 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.578760+0000", 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7172000000000003e-05 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.581485+0000", 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027241370000000002 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.581539+0000", 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4712000000000001e-05 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.259 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.581584+0000", 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4740999999999997e-05 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142148 8.a 8:58078de1:::benchmark_data_smithi080_109213_object27631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:54.627220+0000", 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 59.834217834, 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95987994300000001, 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:09:57.260 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142148 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627220+0000", 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.261 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:09:57.262 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627220+0000", 2024-03-26T05:09:57.262 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:09:57.262 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.262 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.262 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:09:57.262 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627222+0000", 2024-03-26T05:09:57.262 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.392e-06 2024-03-26T05:09:57.262 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.262 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.262 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:09:57.262 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627227+0000", 2024-03-26T05:09:57.262 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9580000000000004e-06 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627248+0000", 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0429e-05 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627258+0000", 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0284e-05 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.263 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627341+0000", 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2446000000000003e-05 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627350+0000", 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8460000000000003e-06 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.575891+0000", 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.94854066800000003 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.576039+0000", 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000148193 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,5", 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.576169+0000", 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012968100000000001 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.578794+0000", 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002625334 2024-03-26T05:09:57.265 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.581412+0000", 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0026176149999999998 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.586988+0000", 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.005576335 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.587046+0000", 2024-03-26T05:09:57.266 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7314e-05 2024-03-26T05:09:57.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:09:57.267 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:09:57.267 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:09:57.267 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.587100+0000", 2024-03-26T05:09:57.267 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4447999999999998e-05 2024-03-26T05:09:57.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.267 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:09:57.267 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:09:57.267 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:09:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:57.269 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:57.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:57.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:56 smithi132 ceph-mon[33500]: 2.0 scrub starts 2024-03-26T05:09:57.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:56 smithi132 ceph-mon[33500]: 2.0 scrub ok 2024-03-26T05:09:57.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:56 smithi132 ceph-mon[33500]: 3.19 scrub starts 2024-03-26T05:09:57.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:56 smithi132 ceph-mon[33500]: 2.1 scrub starts 2024-03-26T05:09:57.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:56 smithi132 ceph-mon[33500]: 3.19 scrub ok 2024-03-26T05:09:57.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:56 smithi132 ceph-mon[33500]: 3.b scrub starts 2024-03-26T05:09:57.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:56 smithi132 ceph-mon[33500]: 3.b scrub ok 2024-03-26T05:09:57.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:56 smithi132 ceph-mon[33500]: 2.1 scrub ok 2024-03-26T05:09:57.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:56 smithi132 ceph-mon[33500]: pgmap v424: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.8 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-26T05:09:57.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:56 smithi132 ceph-mon[33500]: 6.1e scrub starts 2024-03-26T05:09:57.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:56 smithi132 ceph-mon[33500]: 6.1e scrub ok 2024-03-26T05:09:57.319 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:57.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[29691]: 2.0 scrub starts 2024-03-26T05:09:57.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[29691]: 2.0 scrub ok 2024-03-26T05:09:57.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[29691]: 3.19 scrub starts 2024-03-26T05:09:57.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[29691]: 2.1 scrub starts 2024-03-26T05:09:57.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[29691]: 3.19 scrub ok 2024-03-26T05:09:57.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[29691]: 3.b scrub starts 2024-03-26T05:09:57.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[29691]: 3.b scrub ok 2024-03-26T05:09:57.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[29691]: 2.1 scrub ok 2024-03-26T05:09:57.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[29691]: pgmap v424: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.8 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-26T05:09:57.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[29691]: 6.1e scrub starts 2024-03-26T05:09:57.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[29691]: 6.1e scrub ok 2024-03-26T05:09:57.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[40404]: 2.0 scrub starts 2024-03-26T05:09:57.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[40404]: 2.0 scrub ok 2024-03-26T05:09:57.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[40404]: 3.19 scrub starts 2024-03-26T05:09:57.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[40404]: 2.1 scrub starts 2024-03-26T05:09:57.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[40404]: 3.19 scrub ok 2024-03-26T05:09:57.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[40404]: 3.b scrub starts 2024-03-26T05:09:57.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[40404]: 3.b scrub ok 2024-03-26T05:09:57.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[40404]: 2.1 scrub ok 2024-03-26T05:09:57.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[40404]: pgmap v424: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.8 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-26T05:09:57.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[40404]: 6.1e scrub starts 2024-03-26T05:09:57.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:56 smithi080 ceph-mon[40404]: 6.1e scrub ok 2024-03-26T05:09:57.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:57.404 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:57.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:57.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:57.506 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:57.540 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:57.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:57.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:57.571 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:09:57.642 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:57.677 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:57.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:57.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:57.814 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:57.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:57.855 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T05:09:57.915 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:57.965 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:57.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:57.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:58.067 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:58.092 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:57 smithi080 ceph-mon[29691]: 5.c deep-scrub starts 2024-03-26T05:09:58.092 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:57 smithi080 ceph-mon[29691]: 5.c deep-scrub ok 2024-03-26T05:09:58.092 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:57 smithi080 ceph-mon[29691]: 3.18 scrub starts 2024-03-26T05:09:58.092 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:57 smithi080 ceph-mon[29691]: 3.18 scrub ok 2024-03-26T05:09:58.093 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:57 smithi080 ceph-mon[40404]: 5.c deep-scrub starts 2024-03-26T05:09:58.093 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:57 smithi080 ceph-mon[40404]: 5.c deep-scrub ok 2024-03-26T05:09:58.093 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:57 smithi080 ceph-mon[40404]: 3.18 scrub starts 2024-03-26T05:09:58.093 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:57 smithi080 ceph-mon[40404]: 3.18 scrub ok 2024-03-26T05:09:58.099 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:58.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:58.201 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:58.231 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:58.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:58.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:58.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:57 smithi132 ceph-mon[33500]: 5.c deep-scrub starts 2024-03-26T05:09:58.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:57 smithi132 ceph-mon[33500]: 5.c deep-scrub ok 2024-03-26T05:09:58.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:57 smithi132 ceph-mon[33500]: 3.18 scrub starts 2024-03-26T05:09:58.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:57 smithi132 ceph-mon[33500]: 3.18 scrub ok 2024-03-26T05:09:58.333 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:09:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:58.367 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:58.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:09:58.468 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:58.513 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:58.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:58.614 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:58.657 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:58.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:58.759 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:58.788 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:58.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:58.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:58.890 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:09:58.928 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:58.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:09:59.029 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:59.060 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:59.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:59.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:59.068 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:59.161 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:09:59.191 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:59.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:09:59.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[29691]: 5.1b scrub starts 2024-03-26T05:09:59.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[29691]: 5.1b scrub ok 2024-03-26T05:09:59.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[29691]: 5.1d scrub starts 2024-03-26T05:09:59.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[29691]: 5.1d scrub ok 2024-03-26T05:09:59.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[29691]: 8.5 scrub starts 2024-03-26T05:09:59.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[29691]: pgmap v425: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.8 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 920 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-26T05:09:59.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[29691]: 8.5 scrub ok 2024-03-26T05:09:59.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[29691]: 4.d scrub starts 2024-03-26T05:09:59.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[29691]: 4.d scrub ok 2024-03-26T05:09:59.226 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[40404]: 5.1b scrub starts 2024-03-26T05:09:59.226 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[40404]: 5.1b scrub ok 2024-03-26T05:09:59.226 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[40404]: 5.1d scrub starts 2024-03-26T05:09:59.226 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[40404]: 5.1d scrub ok 2024-03-26T05:09:59.226 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[40404]: 8.5 scrub starts 2024-03-26T05:09:59.226 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[40404]: pgmap v425: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.8 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 920 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-26T05:09:59.226 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[40404]: 8.5 scrub ok 2024-03-26T05:09:59.226 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[40404]: 4.d scrub starts 2024-03-26T05:09:59.226 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:58 smithi080 ceph-mon[40404]: 4.d scrub ok 2024-03-26T05:09:59.292 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:59.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:58 smithi132 ceph-mon[33500]: 5.1b scrub starts 2024-03-26T05:09:59.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:58 smithi132 ceph-mon[33500]: 5.1b scrub ok 2024-03-26T05:09:59.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:58 smithi132 ceph-mon[33500]: 5.1d scrub starts 2024-03-26T05:09:59.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:58 smithi132 ceph-mon[33500]: 5.1d scrub ok 2024-03-26T05:09:59.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:58 smithi132 ceph-mon[33500]: 8.5 scrub starts 2024-03-26T05:09:59.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:58 smithi132 ceph-mon[33500]: pgmap v425: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.8 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 920 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-26T05:09:59.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:58 smithi132 ceph-mon[33500]: 8.5 scrub ok 2024-03-26T05:09:59.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:58 smithi132 ceph-mon[33500]: 4.d scrub starts 2024-03-26T05:09:59.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:58 smithi132 ceph-mon[33500]: 4.d scrub ok 2024-03-26T05:09:59.330 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:59.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:59.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:59.432 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:09:59.472 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:59.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:09:59.574 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:09:59.606 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:59.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:59.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:09:59.677 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:09:59.707 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:09:59.750 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:09:59.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:59.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:09:59.852 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:09:59.883 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:09:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:09:59.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:09:59.969 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:09:59.985 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:00.027 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:00.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:00.129 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:00.161 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:00.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:00.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:00.263 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:00.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:59 smithi132 ceph-mon[33500]: 5.a scrub starts 2024-03-26T05:10:00.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:59 smithi132 ceph-mon[33500]: 5.a scrub ok 2024-03-26T05:10:00.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:59 smithi132 ceph-mon[33500]: 8.b scrub starts 2024-03-26T05:10:00.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:59 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2256831166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:10:00.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:09:59 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:10:00.300 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:00.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:00.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:00.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:59 smithi080 ceph-mon[40404]: 5.a scrub starts 2024-03-26T05:10:00.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:59 smithi080 ceph-mon[40404]: 5.a scrub ok 2024-03-26T05:10:00.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:59 smithi080 ceph-mon[40404]: 8.b scrub starts 2024-03-26T05:10:00.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:59 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2256831166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:10:00.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:09:59 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:10:00.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:59 smithi080 ceph-mon[29691]: 5.a scrub starts 2024-03-26T05:10:00.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:59 smithi080 ceph-mon[29691]: 5.a scrub ok 2024-03-26T05:10:00.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:59 smithi080 ceph-mon[29691]: 8.b scrub starts 2024-03-26T05:10:00.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:59 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2256831166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:10:00.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:09:59 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:10:00.401 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:00.438 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:00.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:00.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:00.539 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:00.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:00.572 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:00.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:00.673 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:00.706 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:00.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:00.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:00.808 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:00.842 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:00.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:00.943 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:00.951 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: 8.b scrub ok 2024-03-26T05:10:00.977 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:00.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:00.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:01.079 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:01.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:01.120 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:01.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:01.221 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:01.253 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:01.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:01.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:01.288 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: 8.b scrub ok 2024-03-26T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: 3.12 scrub starts 2024-03-26T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: 3.12 scrub ok 2024-03-26T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: 4.8 deep-scrub starts 2024-03-26T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: 4.8 deep-scrub ok 2024-03-26T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: 6.1b scrub starts 2024-03-26T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: 6.1b scrub ok 2024-03-26T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: pgmap v426: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-26T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-03-26T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[40404]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-26T05:10:01.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: 3.12 scrub starts 2024-03-26T05:10:01.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: 3.12 scrub ok 2024-03-26T05:10:01.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: 4.8 deep-scrub starts 2024-03-26T05:10:01.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: 4.8 deep-scrub ok 2024-03-26T05:10:01.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: 6.1b scrub starts 2024-03-26T05:10:01.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: 6.1b scrub ok 2024-03-26T05:10:01.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: pgmap v426: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-26T05:10:01.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:10:01.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:10:01.290 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T05:10:01.291 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-03-26T05:10:01.291 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:00 smithi080 ceph-mon[29691]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-26T05:10:01.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: 8.b scrub ok 2024-03-26T05:10:01.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: 3.12 scrub starts 2024-03-26T05:10:01.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: 3.12 scrub ok 2024-03-26T05:10:01.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: 4.8 deep-scrub starts 2024-03-26T05:10:01.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: 4.8 deep-scrub ok 2024-03-26T05:10:01.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: 6.1b scrub starts 2024-03-26T05:10:01.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: 6.1b scrub ok 2024-03-26T05:10:01.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: pgmap v426: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-26T05:10:01.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:10:01.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:10:01.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T05:10:01.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-03-26T05:10:01.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:00 smithi132 ceph-mon[33500]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-26T05:10:01.356 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:01.392 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:01.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:01.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:01.495 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:01.526 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:01.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:01.628 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:01.661 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:01.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:01.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:01.763 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:01.794 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:01.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:01.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:01.897 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:01.927 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:01.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:01.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:02.029 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:02.072 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:02.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:02.174 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:02.213 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:02.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:02.315 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:02.355 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:02.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:02.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:02.456 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:02.484 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:02.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:02.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:02.623 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:02.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:02.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:02.755 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:02.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:02.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:02.857 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:02.895 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:02.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:02.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:02.996 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:03.032 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:03.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:03.134 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:03.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:03.173 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:03.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:03.273 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:03.308 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:03.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:03.410 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:03.444 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:03.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:03.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:03.492 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.3 to 1 (10000) 2024-03-26T05:10:03.546 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:03.586 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:03.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:03.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:03 smithi080 ceph-mon[40404]: pgmap v428: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.5 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 988 op/s 2024-03-26T05:10:03.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:03 smithi080 ceph-mon[40404]: 8.9 deep-scrub ok 2024-03-26T05:10:03.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:03 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1548640250' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:10:03.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:03 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:10:03.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:03 smithi080 ceph-mon[29691]: pgmap v428: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.5 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 988 op/s 2024-03-26T05:10:03.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:03 smithi080 ceph-mon[29691]: 8.9 deep-scrub ok 2024-03-26T05:10:03.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:03 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1548640250' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:10:03.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:03 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:10:03.688 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:03.724 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:03.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:03.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:03.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:03 smithi132 ceph-mon[33500]: pgmap v428: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.5 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 988 op/s 2024-03-26T05:10:03.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:03 smithi132 ceph-mon[33500]: 8.9 deep-scrub ok 2024-03-26T05:10:03.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:03 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1548640250' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:10:03.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:03 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:10:03.826 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:03.854 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:03.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:03.956 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:03.992 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:03.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:04.094 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:04.123 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:04.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:04.225 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:04.259 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:04.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:04.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:04.361 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:04.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:10:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:10:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:10:04.394 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:04.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:04.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:04.496 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:04.535 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:04.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:04.637 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:04.668 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:04.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:04.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:04.770 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:04.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:04 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:10:04.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:04 smithi132 ceph-mon[33500]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T05:10:04.808 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:04.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:04.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:04.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:04 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:10:04.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:04 smithi080 ceph-mon[40404]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T05:10:04.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:04 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:10:04.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:04 smithi080 ceph-mon[29691]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T05:10:04.909 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:04.929 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T05:10:04.936 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:04.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:04.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:05.037 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:05.072 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:05.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:05.173 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:05.203 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:05.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:05.304 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:05.351 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:05.351 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:10:05.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:05.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:05.454 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:05.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:05 smithi132 ceph-mon[33500]: pgmap v430: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.5 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 605 op/s 2024-03-26T05:10:05.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:10:05.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:10:05.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:10:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:10:05.543 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:05.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:05.574 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:10:05.619 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:10:05.645 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:05.660 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:10:05.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:05.697 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:05.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:05.698 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:10:05.698 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:05 smithi080 ceph-mon[29691]: pgmap v430: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.5 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 605 op/s 2024-03-26T05:10:05.698 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:10:05.698 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:10:05.698 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:05 smithi080 ceph-mon[40404]: pgmap v430: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.5 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 605 op/s 2024-03-26T05:10:05.699 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:10:05.699 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:10:05.736 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:10:05.777 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:10:05.798 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:05.815 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:10:05.831 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:05.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:05.858 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:10:05.863 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:10:05.863 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:10:05.863 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:10:05.863 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:10:05.863 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:10:05.863 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:10:05.863 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:10:05.863 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:10:05.933 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:05.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:05.978 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:05.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:06.079 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:06.111 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:06.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:06.213 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:06.262 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:06.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:06.322 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:06.364 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:06.392 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:06.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:06.429 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:10:06.429 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:10:06.494 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:06.539 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:06.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:06.640 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:06.668 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:06.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:06.770 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:06.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:06 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:06.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:06 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1816687175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:06.809 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:06.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:06.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:06 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:06.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:06 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1816687175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:06 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:06 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1816687175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:06.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:06.944 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:06.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:06.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:07.048 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:07.080 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:07.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:07.181 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:07.220 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:07.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:07.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:07.322 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:07.361 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:07.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:07.463 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:07.497 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:07.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:07.599 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:07.632 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:07.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:07.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:07.733 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:07.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:07.778 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:07.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:07.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:07 smithi132 ceph-mon[33500]: pgmap v431: 148 pgs: 148 active+clean; 1.2 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-26T05:10:07.800 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:07 smithi080 ceph-mon[40404]: pgmap v431: 148 pgs: 148 active+clean; 1.2 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-26T05:10:07.800 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:07 smithi080 ceph-mon[29691]: pgmap v431: 148 pgs: 148 active+clean; 1.2 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-26T05:10:07.847 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T05:10:07.879 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:07.931 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:07.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:07.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:08.033 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:08.083 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:08.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:08.185 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:08.216 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:08.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:08.317 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:08.366 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:08.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:08.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:08.468 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:08.502 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:08.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:08.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:08.604 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:08.634 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:08.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:08.736 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:08.774 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:08.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:08.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:08.873 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:08.876 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:08.904 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:08.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:08.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:08.942 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:08.949 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:10:09.006 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:09.034 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:09.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:09.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:09.136 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:09.164 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:09.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:09.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:09.266 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:09.306 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:09.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:09.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:09.408 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:09.462 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:09.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:09.564 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:09.594 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:09.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:09.637 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:09 smithi080 ceph-mon[29691]: pgmap v432: 148 pgs: 148 active+clean; 1.2 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T05:10:09.638 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:09 smithi080 ceph-mon[40404]: pgmap v432: 148 pgs: 148 active+clean; 1.2 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T05:10:09.695 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:09.728 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:09.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:09.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:09 smithi132 ceph-mon[33500]: pgmap v432: 148 pgs: 148 active+clean; 1.2 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T05:10:09.830 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:09.864 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:09.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:09.965 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:10.002 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:10.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:10.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:10.104 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:10.145 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:10.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:10.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:10.256 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:10.288 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:10.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:10.390 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:10.424 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:10.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:10.526 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:10.569 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:10.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:10.671 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:10.703 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:10.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:10.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:10 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:10:10.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:10 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:10:10.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:10 smithi132 ceph-mon[33500]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T05:10:10.805 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:10.837 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:10.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:10.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:10 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:10:10.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:10 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:10:10.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:10 smithi080 ceph-mon[40404]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T05:10:10.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:10 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:10:10.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:10 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:10:10.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:10 smithi080 ceph-mon[29691]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T05:10:10.939 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:10.953 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:11.003 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:11.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:11.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:11.105 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:11.138 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:11.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:11.240 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:11.278 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:11.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:11.379 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:11.414 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:11.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:11.516 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:11.555 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:11.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:11.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:11.564 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T05:10:11.657 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:11.686 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:11.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:11.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:11.787 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:11.827 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:11.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:11.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:11.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:11 smithi080 ceph-mon[40404]: pgmap v434: 148 pgs: 148 active+clean; 1.1 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 889 op/s 2024-03-26T05:10:11.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:11 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3595451255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:10:11.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:11 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:10:11.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:11 smithi080 ceph-mon[29691]: pgmap v434: 148 pgs: 148 active+clean; 1.1 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 889 op/s 2024-03-26T05:10:11.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:11 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3595451255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:10:11.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:11 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:10:11.929 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:11.961 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:11.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:11 smithi132 ceph-mon[33500]: pgmap v434: 148 pgs: 148 active+clean; 1.1 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 889 op/s 2024-03-26T05:10:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:11 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3595451255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:10:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:11 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:10:12.064 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:12.090 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:12.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:12.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:12.228 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:12.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:12.329 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:12.369 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:12.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:12.471 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:12.507 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:12.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:12.609 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:12.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:12 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:10:12.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:12 smithi080 ceph-mon[29691]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T05:10:12.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:12 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:10:12.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:12 smithi080 ceph-mon[40404]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T05:10:12.642 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:12.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:12.744 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:12.780 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:12.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:12.882 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:12.940 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:12.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:13.041 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:12 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:10:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:12 smithi132 ceph-mon[33500]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T05:10:13.074 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:13.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:13.176 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:13.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:13.223 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:13.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:13.324 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:13.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:13.365 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:13.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:13.466 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:13.508 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:13.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:13.610 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:13.657 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:13.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:13.758 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:13.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:13.802 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:13.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:13.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:13 smithi080 ceph-mon[40404]: pgmap v436: 148 pgs: 148 active+clean; 825 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-03-26T05:10:13.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:13 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:10:13.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:13 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:10:13.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:13 smithi080 ceph-mon[40404]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T05:10:13.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:13 smithi080 ceph-mon[29691]: pgmap v436: 148 pgs: 148 active+clean; 825 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-03-26T05:10:13.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:13 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:10:13.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:13 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:10:13.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:13 smithi080 ceph-mon[29691]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T05:10:13.903 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:13.941 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:13.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:13.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:13.984 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 6 1 2024-03-26T05:10:14.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:13 smithi132 ceph-mon[33500]: pgmap v436: 148 pgs: 148 active+clean; 825 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-03-26T05:10:14.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:13 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:10:14.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:13 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:10:14.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:13 smithi132 ceph-mon[33500]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T05:10:14.042 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:14.091 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:14.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:14.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:14.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:14.217 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:10:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:10:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:10:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:14.258 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:14.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:14.359 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:14.389 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:14.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:14.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:14.491 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:14.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:14.527 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:14.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:14.593 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:10:14.628 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:14.657 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:14.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:14.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:14.759 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:14.814 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:14.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:14.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:14 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1967571035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:10:14.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:14 smithi080 ceph-mon[29691]: pgmap v438: 148 pgs: 148 active+clean; 825 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 995 op/s 2024-03-26T05:10:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:14 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1967571035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:10:14.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:14 smithi080 ceph-mon[40404]: pgmap v438: 148 pgs: 148 active+clean; 825 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 995 op/s 2024-03-26T05:10:14.916 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:14.963 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:14.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:15.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:14 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1967571035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:10:15.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:14 smithi132 ceph-mon[33500]: pgmap v438: 148 pgs: 148 active+clean; 825 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 995 op/s 2024-03-26T05:10:15.065 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:15.101 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:15.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:15.138 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:10:15.138 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:10:15.203 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:15.245 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:15.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:15.346 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:15.376 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:15.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:15.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:15.478 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:15.522 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:15.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:15.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:15.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:10:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:10:15.624 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:15.653 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:15.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:15.759 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:15.789 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:15.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:15.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:15.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:15 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:10:15.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:15 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1967571035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:10:15.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:15 smithi080 ceph-mon[40404]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T05:10:15.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:15 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:15 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:10:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:15 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1967571035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:10:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:15 smithi080 ceph-mon[29691]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T05:10:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:15 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:15.890 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:15.942 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:15.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:16.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:15 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:10:16.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:15 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1967571035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:10:16.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:15 smithi132 ceph-mon[33500]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T05:10:16.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:15 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:16.044 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:16.083 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:16.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:16.185 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:16.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:16.237 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:16.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:16.338 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:16.378 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:16.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:16.480 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:16.523 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:16.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:16.624 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:16.664 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:16.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:16.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:16 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:10:16.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:16 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:10:16.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:16 smithi080 ceph-mon[29691]: pgmap v440: 148 pgs: 148 active+clean; 451 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.75k op/s 2024-03-26T05:10:16.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:16 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:10:16.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:16 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:10:16.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:16 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:10:16.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:16 smithi080 ceph-mon[40404]: pgmap v440: 148 pgs: 148 active+clean; 451 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.75k op/s 2024-03-26T05:10:16.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:16 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:10:16.767 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:16.814 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:16.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:16.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:16.916 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:16.951 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:16.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:17.033 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T05:10:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:16 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:10:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:16 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:10:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:16 smithi132 ceph-mon[33500]: pgmap v440: 148 pgs: 148 active+clean; 451 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.75k op/s 2024-03-26T05:10:17.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:16 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:10:17.053 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:17.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:17.149 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:17.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:17.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:17.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:17.318 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:17.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:17.419 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:17.445 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:17.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:17.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:17.547 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:17.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:17.678 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:17.706 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:17.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:17.808 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:17.842 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:17.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:17.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:17 smithi080 ceph-mon[40404]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T05:10:17.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:17 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:10:17.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:17 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:10:17.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:17 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:10:17.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:17 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:10:17.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:17 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:10:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:17 smithi080 ceph-mon[29691]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T05:10:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:17 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:10:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:17 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:10:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:17 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:10:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:17 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:10:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:17 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:10:17.944 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:17.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:17.986 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:17.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:17 smithi132 ceph-mon[33500]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T05:10:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:17 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:10:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:17 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:10:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:17 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:10:18.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:17 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:10:18.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:17 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:10:18.087 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:18.118 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:18.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:18.220 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:18.256 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:18.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:18.358 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:18.396 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:18.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:18.498 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:18.528 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:18.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:18.622 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:10:18.629 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:18.652 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:18.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:18.754 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:18.784 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:18.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:18.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:18.800 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:18 smithi080 ceph-mon[40404]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T05:10:18.800 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:18 smithi080 ceph-mon[40404]: pgmap v443: 148 pgs: 148 active+clean; 451 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 960 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-26T05:10:18.800 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:18 smithi080 ceph-mon[29691]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T05:10:18.800 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:18 smithi080 ceph-mon[29691]: pgmap v443: 148 pgs: 148 active+clean; 451 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 960 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-26T05:10:18.886 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:18.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:18.931 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:18.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:19.032 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:19.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:18 smithi132 ceph-mon[33500]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T05:10:19.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:18 smithi132 ceph-mon[33500]: pgmap v443: 148 pgs: 148 active+clean; 451 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 960 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-26T05:10:19.074 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:19.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:19.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:19.175 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:19.208 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:19.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:19.310 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:19.333 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:19.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:19.438 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:19.475 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:19.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:19.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:19.577 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:19.601 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:19.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:19.702 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:19.716 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:10:19.716 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:10:19.716 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:10:19.716 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:10:19.716 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:10:19.716 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:10:19.716 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:10:19.716 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:10:19.717 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:10:19.717 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:10:19.717 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:10:19.717 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:10:19.717 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:10:19.717 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:10:19.717 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:10:19.717 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:10:19.734 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:19.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:19.835 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:19.870 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:19.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:19.972 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:20.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:20.007 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:20.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:19 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/627764087' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:20.108 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:20.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:19 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/627764087' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:20.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:19 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/627764087' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:20.141 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:20.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:20.243 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:20.272 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:20.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:20.374 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:20.413 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:20.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:20.514 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:20.552 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:20.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:20.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:20.653 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:20.682 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:20.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:20.786 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:20.825 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:20.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:20.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:20.926 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:20.951 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:20.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:10:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:20 smithi132 ceph-mon[33500]: pgmap v444: 148 pgs: 148 active+clean; 430 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-26T05:10:21.053 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:21.091 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:21.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:21.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:21.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:10:21.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:20 smithi080 ceph-mon[40404]: pgmap v444: 148 pgs: 148 active+clean; 430 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-26T05:10:21.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:10:21.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:20 smithi080 ceph-mon[29691]: pgmap v444: 148 pgs: 148 active+clean; 430 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-26T05:10:21.194 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:21.217 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:21.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:21.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:21.390 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:21.435 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:21.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:21.495 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:21.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:21.596 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:21.639 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:21.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:21.740 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:21.780 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:21.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:21.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:21.882 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:21.908 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:21.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:21.931 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Removed 114514 objects 2024-03-26T05:10:21.931 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Clean up completed and total clean up time :118.999 2024-03-26T05:10:22.010 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:22.037 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:22.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:22.138 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:22.163 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:22.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:22.264 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:22.290 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:22.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:22.391 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:22.419 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:22.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:22.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:22.521 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:22.547 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:22.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:22.648 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:22.675 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:22.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:22.776 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:22.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:22 smithi132 ceph-mon[33500]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T05:10:22.803 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:22.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:22.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:22.812 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:22 smithi080 ceph-mon[40404]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T05:10:22.812 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:22 smithi080 ceph-mon[29691]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T05:10:22.904 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:22.931 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:22.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:23.034 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:23.060 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:23.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:23.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:23.162 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:23.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:23.188 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:23.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:23.289 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:23.318 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:23.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:23.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:23.447 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:23.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:23.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:23.546 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:23.549 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:23.576 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:23.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:23.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:23.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:23 smithi080 ceph-mon[29691]: pgmap v446: 148 pgs: 148 active+clean; 108 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 915 op/s 2024-03-26T05:10:23.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:23 smithi080 ceph-mon[40404]: pgmap v446: 148 pgs: 148 active+clean; 108 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 915 op/s 2024-03-26T05:10:23.677 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:23.704 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:23.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:23.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:23 smithi132 ceph-mon[33500]: pgmap v446: 148 pgs: 148 active+clean; 108 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 915 op/s 2024-03-26T05:10:23.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:23.834 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:23.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:23.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:23.962 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:23.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:24.064 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:24.090 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:24.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:24.111 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:24.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:24.216 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:24.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:24.318 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:24.327 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:10:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:10:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:10:24.346 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:24.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:24.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:24.448 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:24.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:24.473 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:24.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:24.574 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:24.597 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:24.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:24.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:24.699 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:24.727 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:24.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:24.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:24.829 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:24.842 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:10:24.857 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:24.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:24.958 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:24.985 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:24.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:24.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:25.087 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:25.110 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:25.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:25.212 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:25.236 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:25.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:25.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:25.338 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:25.365 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:25.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:25.467 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:25.496 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:25.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:25.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:25.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:25 smithi132 ceph-mon[33500]: pgmap v447: 148 pgs: 148 active+clean; 108 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 701 B/s rd, 0 B/s wr, 752 op/s 2024-03-26T05:10:25.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:10:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:10:25.599 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:25.627 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:25.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:25.729 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:25.753 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:25.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:25.854 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:25.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:25 smithi080 ceph-mon[40404]: pgmap v447: 148 pgs: 148 active+clean; 108 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 701 B/s rd, 0 B/s wr, 752 op/s 2024-03-26T05:10:25.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:25 smithi080 ceph-mon[29691]: pgmap v447: 148 pgs: 148 active+clean; 108 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 701 B/s rd, 0 B/s wr, 752 op/s 2024-03-26T05:10:25.878 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:25.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:25.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:25.980 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:26.008 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:26.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:26.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:26.109 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:26.134 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:26.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:26.236 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:26.263 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:26.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:26.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:26.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:26.393 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:26.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:26.414 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-26T05:10:26.414 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-26T05:10:26.499 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:26.524 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:26.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:26.556 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:26 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:26.556 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:26 smithi080 ceph-mon[40404]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T05:10:26.556 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:26 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:26.556 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:26 smithi080 ceph-mon[29691]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T05:10:26.626 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:26.655 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:26.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:26.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:26.783 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:26.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:26.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:26.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:26 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:26.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:26 smithi132 ceph-mon[33500]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T05:10:26.885 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:26.909 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:26.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:27.010 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:27.038 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:27.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:27.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:27.140 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:27.165 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:27.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:27.266 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:27.294 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:27.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:27.395 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:27.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:27.420 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:27.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:27.514 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:10:27.518 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.6 to 1 (10000) 2024-03-26T05:10:27.521 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:27.550 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:27.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:27.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:27.652 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:27.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:27.677 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:27.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:27.778 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:27 smithi132 ceph-mon[33500]: pgmap v448: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 870 op/s 2024-03-26T05:10:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:27 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2929764996' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:10:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:27 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:10:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:27 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/597352167' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:10:27.802 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:27.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:27.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:27.810 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:27 smithi080 ceph-mon[40404]: pgmap v448: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 870 op/s 2024-03-26T05:10:27.811 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:27 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2929764996' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:10:27.811 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:27 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:10:27.811 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:27 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/597352167' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:10:27.811 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:27 smithi080 ceph-mon[29691]: pgmap v448: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 870 op/s 2024-03-26T05:10:27.811 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:27 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2929764996' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:10:27.811 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:27 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:10:27.811 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:27 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/597352167' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:10:27.904 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:27.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:28.034 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:28.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:28.059 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:28.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:28.159 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:28.187 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:28.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:28.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:28.289 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:28.313 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:28.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:28.334 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:28.385 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:10:28.386 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:10:28.386 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:10:28.386 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:10:28.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.386 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206041 7.e 7:7f2653d5:::benchmark_data_smithi080_80848_object86542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:10:28.386 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.917013+0000", 2024-03-26T05:10:28.386 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 292.47570204800002, 2024-03-26T05:10:28.386 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.690052211, 2024-03-26T05:10:28.386 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.386 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206041 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917013+0000", 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.387 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917013+0000", 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917015+0000", 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5099999999999999e-06 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917020+0000", 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5049999999999998e-06 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.389 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.389 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917031+0000", 2024-03-26T05:10:28.389 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0971e-05 2024-03-26T05:10:28.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.389 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.389 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.389 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917040+0000", 2024-03-26T05:10:28.390 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7690000000000002e-06 2024-03-26T05:10:28.390 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.390 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.390 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.390 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917195+0000", 2024-03-26T05:10:28.391 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015542099999999999 2024-03-26T05:10:28.391 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.391 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.391 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.391 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917210+0000", 2024-03-26T05:10:28.391 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4626e-05 2024-03-26T05:10:28.391 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.391 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.391 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.391 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.389350+0000", 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.47213961900000001 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.389365+0000", 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.556e-05 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562507+0000", 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1731419940000001 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.393 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.393 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562650+0000", 2024-03-26T05:10:28.393 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014278399999999999 2024-03-26T05:10:28.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.393 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3,5", 2024-03-26T05:10:28.393 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.563216+0000", 2024-03-26T05:10:28.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00056574299999999995 2024-03-26T05:10:28.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.394 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.394 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596626+0000", 2024-03-26T05:10:28.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.033409844000000001 2024-03-26T05:10:28.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.394 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.394 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596672+0000", 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6075999999999997e-05 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596719+0000", 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7132000000000002e-05 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.606905+0000", 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010185886 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.606964+0000", 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9617999999999997e-05 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.607065+0000", 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000101153 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206060 7.e 7:75f147c1:::benchmark_data_smithi080_80848_object86561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:10:28.396 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.940957+0000", 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 292.45175794199997, 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.666353937, 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206060 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.940957+0000", 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.398 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.940957+0000", 2024-03-26T05:10:28.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.398 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.940959+0000", 2024-03-26T05:10:28.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5400000000000001e-06 2024-03-26T05:10:28.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.940965+0000", 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.947e-06 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.940979+0000", 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4008e-05 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.940988+0000", 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1970000000000002e-06 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.399 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.941053+0000", 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4642999999999998e-05 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.941061+0000", 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5329999999999994e-06 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.389945+0000", 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.44888396000000003 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.389955+0000", 2024-03-26T05:10:28.401 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7759999999999992e-06 2024-03-26T05:10:28.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.401 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.582041+0000", 2024-03-26T05:10:28.401 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.192085727 2024-03-26T05:10:28.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.401 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.582172+0000", 2024-03-26T05:10:28.402 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013096399999999999 2024-03-26T05:10:28.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.402 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3,5", 2024-03-26T05:10:28.403 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.582308+0000", 2024-03-26T05:10:28.403 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013676500000000001 2024-03-26T05:10:28.403 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.403 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.403 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.403 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596685+0000", 2024-03-26T05:10:28.403 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014376751 2024-03-26T05:10:28.403 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.403 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596756+0000", 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1322999999999995e-05 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.607116+0000", 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010359653999999999 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.607166+0000", 2024-03-26T05:10:28.404 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9573000000000001e-05 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.607212+0000", 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6544e-05 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.607311+0000", 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9031999999999996e-05 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.405 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.406 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206066 7.e 7:7516dda5:::benchmark_data_smithi080_80848_object86567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:10:28.406 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.959389+0000", 2024-03-26T05:10:28.406 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 292.43332625199997, 2024-03-26T05:10:28.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6554198369999999, 2024-03-26T05:10:28.406 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.406 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.406 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.406 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.406 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.406 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206066 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959389+0000", 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959389+0000", 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.407 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959390+0000", 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3499999999999995e-07 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959394+0000", 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9049999999999999e-06 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959402+0000", 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7930000000000003e-06 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.408 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959408+0000", 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7150000000000003e-06 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959469+0000", 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0435e-05 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959481+0000", 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2065000000000001e-05 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.409 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.390011+0000", 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.43053056200000001 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.390020+0000", 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8130000000000002e-06 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.590994+0000", 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.200974006 2024-03-26T05:10:28.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591097+0000", 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000102939 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3,5", 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591211+0000", 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011416700000000001 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597026+0000", 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0058151940000000001 2024-03-26T05:10:28.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.598631+0000", 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001604496 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.607489+0000", 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0088579139999999997 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614504+0000", 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0070156089999999999 2024-03-26T05:10:28.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614687+0000", 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018228299999999999 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614809+0000", 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000122206 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.413 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236812 7.e 7:703f6da5:::benchmark_data_smithi080_80848_object117313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:10:28.414 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.453857+0000", 2024-03-26T05:10:28.414 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 255.93885868999999, 2024-03-26T05:10:28.414 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99701717899999998, 2024-03-26T05:10:28.414 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.414 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.414 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.414 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.414 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.414 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236812 2024-03-26T05:10:28.414 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.414 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.414 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453857+0000", 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453857+0000", 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453858+0000", 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5969999999999999e-06 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453864+0000", 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2169999999999999e-06 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453879+0000", 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4384e-05 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453888+0000", 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8170000000000005e-06 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453955+0000", 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7786000000000005e-05 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453968+0000", 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2860999999999999e-05 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.443220+0000", 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.989251518 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.417 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.443311+0000", 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0847999999999995e-05 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.443402+0000", 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1513000000000003e-05 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.445956+0000", 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0025534899999999998 2024-03-26T05:10:28.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.448724+0000", 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027681440000000002 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.450797+0000", 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0020734920000000001 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.450839+0000", 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1789000000000003e-05 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.450874+0000", 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4722999999999999e-05 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236839 7.e 7:7f30ee9a:::benchmark_data_smithi080_80848_object117340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.489276+0000", 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 255.90343907299999, 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96184769999999997, 2024-03-26T05:10:28.420 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236839 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489276+0000", 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.421 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489276+0000", 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489278+0000", 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.73e-06 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489284+0000", 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5010000000000001e-06 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489298+0000", 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3339000000000001e-05 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489307+0000", 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7219999999999994e-06 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489373+0000", 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5878999999999997e-05 2024-03-26T05:10:28.423 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489382+0000", 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.098e-06 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.446003+0000", 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95662086599999996 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.446115+0000", 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011173800000000001 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.446193+0000", 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8335000000000001e-05 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451000+0000", 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0048067680000000003 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451047+0000", 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7169999999999997e-05 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.425 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451056+0000", 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9190000000000005e-06 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451086+0000", 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.9490000000000001e-05 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451124+0000", 2024-03-26T05:10:28.426 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8145000000000002e-05 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236853 7.e 7:7a866308:::benchmark_data_smithi080_80848_object117354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.506895+0000", 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 255.88581992900001, 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.94437797099999998, 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.427 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236853 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506895+0000", 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506895+0000", 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506897+0000", 2024-03-26T05:10:28.428 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.579e-06 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506902+0000", 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8670000000000002e-06 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506914+0000", 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2329e-05 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.429 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506922+0000", 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0989999999999999e-06 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506978+0000", 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5912000000000003e-05 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506991+0000", 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3091000000000001e-05 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.448760+0000", 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.94176837199999996 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.448849+0000", 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9405000000000002e-05 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.448923+0000", 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4035000000000005e-05 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.431 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451132+0000", 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022092420000000002 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451167+0000", 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5067999999999997e-05 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451212+0000", 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4963000000000003e-05 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.432 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451253+0000", 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0608999999999997e-05 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451273+0000", 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0400000000000001e-05 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236855 7.e 7:7174933d:::benchmark_data_smithi080_80848_object117356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.507103+0000", 2024-03-26T05:10:28.433 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 255.88561187600001, 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95728903099999996, 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236855 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507103+0000", 2024-03-26T05:10:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507103+0000", 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507104+0000", 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1399999999999996e-07 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507108+0000", 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0890000000000002e-06 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507118+0000", 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8430000000000005e-06 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507125+0000", 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0609999999999998e-06 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507184+0000", 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8700999999999999e-05 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507191+0000", 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4089999999999997e-06 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451291+0000", 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.944100151 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451359+0000", 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7286999999999995e-05 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451420+0000", 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0949999999999998e-05 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464151+0000", 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.012730844 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464218+0000", 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7546999999999996e-05 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464288+0000", 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0036000000000005e-05 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464347+0000", 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8801999999999997e-05 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464392+0000", 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5497000000000002e-05 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236870 7.e 7:7a7ed7ca:::benchmark_data_smithi080_80848_object117371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.520180+0000", 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 255.872534906, 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.94435229899999995, 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236870 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520180+0000", 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520180+0000", 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520182+0000", 2024-03-26T05:10:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.468e-06 2024-03-26T05:10:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520187+0000", 2024-03-26T05:10:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0900000000000004e-06 2024-03-26T05:10:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520204+0000", 2024-03-26T05:10:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7142e-05 2024-03-26T05:10:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.443 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520213+0000", 2024-03-26T05:10:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.3049999999999998e-06 2024-03-26T05:10:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520350+0000", 2024-03-26T05:10:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000136978 2024-03-26T05:10:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520358+0000", 2024-03-26T05:10:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6490000000000008e-06 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.453989+0000", 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.933631447 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.454089+0000", 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010003 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.454198+0000", 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000108436 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464202+0000", 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010004413 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464226+0000", 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3917000000000002e-05 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464437+0000", 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000210645 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464494+0000", 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7138000000000001e-05 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464533+0000", 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8640999999999998e-05 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236882 7.e 7:74c18bc4:::benchmark_data_smithi080_80848_object117383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.532839+0000", 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 255.85987584, 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93196347400000001, 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236882 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532839+0000", 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532839+0000", 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532841+0000", 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5770000000000001e-06 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532847+0000", 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7300000000000002e-06 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532860+0000", 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2891000000000001e-05 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532868+0000", 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.259e-06 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532976+0000", 2024-03-26T05:10:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010864799999999999 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532985+0000", 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0700000000000007e-06 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.456804+0000", 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.92381993200000001 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.457000+0000", 2024-03-26T05:10:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019507699999999999 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.457116+0000", 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000116297 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464232+0000", 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0071157690000000001 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464576+0000", 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000344295 2024-03-26T05:10:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464618+0000", 2024-03-26T05:10:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2042999999999997e-05 2024-03-26T05:10:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464765+0000", 2024-03-26T05:10:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014673299999999999 2024-03-26T05:10:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464803+0000", 2024-03-26T05:10:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8152999999999999e-05 2024-03-26T05:10:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236902 7.e 7:7eec6e5e:::benchmark_data_smithi080_80848_object117403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:10:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.551741+0000", 2024-03-26T05:10:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 255.840974609, 2024-03-26T05:10:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93036452400000003, 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236902 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551741+0000", 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.456 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551741+0000", 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551742+0000", 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4920000000000001e-06 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551748+0000", 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5790000000000003e-06 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.457 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551765+0000", 2024-03-26T05:10:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7620000000000001e-05 2024-03-26T05:10:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551775+0000", 2024-03-26T05:10:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2359999999999993e-06 2024-03-26T05:10:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551846+0000", 2024-03-26T05:10:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1070999999999998e-05 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551854+0000", 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5469999999999997e-06 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.460854+0000", 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.90899986499999996 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.459 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.460962+0000", 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000107797 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.461367+0000", 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00040546499999999998 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.471563+0000", 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010195895999999999 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.460 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.471608+0000", 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4672999999999997e-05 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482013+0000", 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010404665 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482062+0000", 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9329000000000002e-05 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482105+0000", 2024-03-26T05:10:28.461 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3288999999999998e-05 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236911 7.e 7:77b891a1:::benchmark_data_smithi080_80848_object117412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.556123+0000", 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 255.83659256000001, 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.92610570000000003, 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.462 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236911 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.556123+0000", 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.556123+0000", 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.463 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.556124+0000", 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.677e-06 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.556130+0000", 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.169e-06 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.556148+0000", 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8705000000000001e-05 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.464 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.556158+0000", 2024-03-26T05:10:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0017000000000001e-05 2024-03-26T05:10:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.558482+0000", 2024-03-26T05:10:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002324156 2024-03-26T05:10:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.558495+0000", 2024-03-26T05:10:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2238e-05 2024-03-26T05:10:28.465 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464848+0000", 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.90635296700000001 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464957+0000", 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010899399999999999 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.465070+0000", 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000112959 2024-03-26T05:10:28.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.471649+0000", 2024-03-26T05:10:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0065793639999999999 2024-03-26T05:10:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.475930+0000", 2024-03-26T05:10:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0042813010000000004 2024-03-26T05:10:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.467 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482152+0000", 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0062221480000000003 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482194+0000", 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1829999999999998e-05 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482228+0000", 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4174999999999997e-05 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.468 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236941 7.e 7:756847e5:::benchmark_data_smithi080_80848_object117442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.586338+0000", 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 255.80637761899999, 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.89596953999999995, 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236941 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.469 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586338+0000", 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586338+0000", 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586339+0000", 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.685e-06 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.470 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.471 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586346+0000", 2024-03-26T05:10:28.471 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5629999999999997e-06 2024-03-26T05:10:28.471 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.471 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.471 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.471 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586365+0000", 2024-03-26T05:10:28.471 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8972999999999999e-05 2024-03-26T05:10:28.471 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.471 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.471 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.472 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586375+0000", 2024-03-26T05:10:28.472 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0319999999999999e-05 2024-03-26T05:10:28.472 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.472 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.472 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.472 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586454+0000", 2024-03-26T05:10:28.473 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9178000000000007e-05 2024-03-26T05:10:28.473 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.473 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.473 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.473 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586464+0000", 2024-03-26T05:10:28.473 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5370000000000003e-06 2024-03-26T05:10:28.473 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.468430+0000", 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.88196650799999998 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.468546+0000", 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115884 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:10:28.474 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.468660+0000", 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000113459 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.481992+0000", 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.013331989000000001 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482142+0000", 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000149932 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.475 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482234+0000", 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2281999999999995e-05 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482274+0000", 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0361000000000002e-05 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482307+0000", 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2869e-05 2024-03-26T05:10:28.476 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236962 7.e 7:7543a6bb:::benchmark_data_smithi080_80848_object117463:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.607100+0000", 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 255.78561503700001, 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.890074221, 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236962 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.477 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607100+0000", 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607100+0000", 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607102+0000", 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5540000000000001e-06 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.478 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607107+0000", 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8760000000000004e-06 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607120+0000", 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3003000000000001e-05 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607128+0000", 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9079999999999995e-06 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.479 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607185+0000", 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.749e-05 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607193+0000", 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8709999999999997e-06 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.471679+0000", 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.86448656899999998 2024-03-26T05:10:28.480 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.471781+0000", 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000101777 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.471901+0000", 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000119647 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482312+0000", 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010411521 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.481 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.482 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.482 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482350+0000", 2024-03-26T05:10:28.482 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7594e-05 2024-03-26T05:10:28.482 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.482 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.482 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.482 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.496765+0000", 2024-03-26T05:10:28.482 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014415193999999999 2024-03-26T05:10:28.482 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.482 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497123+0000", 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000357513 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497174+0000", 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1703999999999998e-05 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.483 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236970 7.e 7:73842dff:::benchmark_data_smithi080_80848_object117471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.617214+0000", 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 255.77550108400001, 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.88018738900000004, 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236970 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.484 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617214+0000", 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617214+0000", 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617216+0000", 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8390000000000001e-06 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617221+0000", 2024-03-26T05:10:28.485 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4419999999999997e-06 2024-03-26T05:10:28.486 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.486 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.486 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.486 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617235+0000", 2024-03-26T05:10:28.486 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3132999999999999e-05 2024-03-26T05:10:28.486 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.486 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.486 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.486 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617243+0000", 2024-03-26T05:10:28.486 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.755e-06 2024-03-26T05:10:28.486 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617308+0000", 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4640000000000005e-05 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617318+0000", 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8840000000000006e-06 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.475962+0000", 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.85864369799999996 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.487 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.476087+0000", 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012535899999999999 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.476209+0000", 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000122397 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482317+0000", 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0061080370000000002 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.488 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482387+0000", 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9219000000000003e-05 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497219+0000", 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014832451999999999 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497363+0000", 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014368700000000001 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.489 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497402+0000", 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8847000000000003e-05 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236991 7.e 7:79f2f371:::benchmark_data_smithi080_80848_object117492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.641178+0000", 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 255.751537015, 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.85638282899999996, 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:10:28.490 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236991 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641178+0000", 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641178+0000", 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.491 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641180+0000", 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.39e-06 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641186+0000", 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7009999999999999e-06 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641197+0000", 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.047e-05 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.492 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641204+0000", 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.759e-06 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641255+0000", 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1249999999999999e-05 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641265+0000", 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8260000000000005e-06 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.478969+0000", 2024-03-26T05:10:28.493 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.83770405299999995 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.479078+0000", 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010932700000000001 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.479192+0000", 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000114244 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.488347+0000", 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0091552439999999999 2024-03-26T05:10:28.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.491459+0000", 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0031117340000000001 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497460+0000", 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.006000521 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497518+0000", 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8733999999999997e-05 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.495 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.496 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.496 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497561+0000", 2024-03-26T05:10:28.496 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2576e-05 2024-03-26T05:10:28.496 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.496 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.496 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.496 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.496 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.496 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:94673 8.4 8:27890670:::benchmark_data_smithi080_109213_object94672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T05:10:28.496 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:59.948789+0000", 2024-03-26T05:10:28.496 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 148.44392592899999, 2024-03-26T05:10:28.496 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2924905719999999, 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 94673 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.948789+0000", 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.497 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.948789+0000", 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.948791+0000", 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7770000000000001e-06 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.948829+0000", 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7764e-05 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.498 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.948847+0000", 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.77e-05 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.948858+0000", 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1610000000000001e-05 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953737+0000", 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.004879146 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953746+0000", 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.149e-06 2024-03-26T05:10:28.499 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.227982+0000", 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.274235945 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.229412+0000", 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014291989999999999 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,5,6", 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.229601+0000", 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018986599999999999 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.500 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.240892+0000", 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.011290134 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.240913+0000", 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1382999999999998e-05 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241019+0000", 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010596500000000001 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.501 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241107+0000", 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7849000000000005e-05 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241170+0000", 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2898000000000007e-05 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241280+0000", 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011018699999999999 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.502 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:94680 8.d 8:b14d91ac:::benchmark_data_smithi080_109213_object94679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:59.950462+0000", 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 148.44225368100001, 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.286237412, 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 94680 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.503 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.950462+0000", 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.950462+0000", 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.950463+0000", 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.004e-06 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.504 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.950547+0000", 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4522000000000002e-05 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.952906+0000", 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0023584489999999999 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.952925+0000", 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9752999999999999e-05 2024-03-26T05:10:28.505 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953683+0000", 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00075799099999999996 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953695+0000", 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1571e-05 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.506 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.225855+0000", 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2721605570000001 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.227764+0000", 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001908428 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6,7", 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.227964+0000", 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000200306 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.507 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.233395+0000", 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0054312329999999997 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.235301+0000", 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001906105 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236142+0000", 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00084030800000000001 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.508 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236510+0000", 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00036848300000000002 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236573+0000", 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2756999999999996e-05 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236699+0000", 2024-03-26T05:10:28.509 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000125945 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:94684 8.4 8:22a13d96:::benchmark_data_smithi080_109213_object94683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:59.953511+0000", 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 148.43920405899999, 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2881801740000001, 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.510 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:10:28.511 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:10:28.511 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 94684 2024-03-26T05:10:28.511 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.511 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.511 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.511 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.511 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953511+0000", 2024-03-26T05:10:28.511 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.511 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.511 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.511 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.511 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953511+0000", 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953513+0000", 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.373e-06 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953551+0000", 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8426000000000001e-05 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953563+0000", 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1569999999999999e-05 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953572+0000", 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5889999999999992e-06 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953772+0000", 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000200027 2024-03-26T05:10:28.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953779+0000", 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3189999999999997e-06 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.232106+0000", 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.278326329 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.232238+0000", 2024-03-26T05:10:28.514 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013240300000000001 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,5,6", 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.232399+0000", 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000161082 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.240960+0000", 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.008560916 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.515 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.240973+0000", 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3251000000000001e-05 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241061+0000", 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7934000000000002e-05 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241532+0000", 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00047092299999999999 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.516 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241590+0000", 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8105e-05 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241691+0000", 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000100927 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:94695 8.d 8:bd3743b0:::benchmark_data_smithi080_109213_object94694:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T05:10:28.517 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:59.953586+0000", 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 148.43912956200001, 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2833069830000001, 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 94695 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953586+0000", 2024-03-26T05:10:28.518 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953586+0000", 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953586+0000", 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7999999999999995e-07 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953614+0000", 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.8221e-05 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.519 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953625+0000", 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0037000000000001e-05 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953631+0000", 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8029999999999999e-06 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.954047+0000", 2024-03-26T05:10:28.520 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00041608900000000002 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.954056+0000", 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6899999999999998e-06 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.230573+0000", 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2765164849999999 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.230722+0000", 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014921300000000001 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.521 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6,7", 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.230890+0000", 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016784600000000001 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.233965+0000", 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0030752990000000001 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.235525+0000", 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0015597969999999999 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.522 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236186+0000", 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00066131299999999996 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236736+0000", 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00055022600000000001 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236786+0000", 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0059999999999998e-05 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.523 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236893+0000", 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010632399999999999 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:94700 8.4 8:245b8f55:::benchmark_data_smithi080_109213_object94699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:59.953642+0000", 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 148.43907312799999, 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2889140180000001, 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:10:28.524 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 94700 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953642+0000", 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953642+0000", 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.525 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953643+0000", 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9699999999999996e-07 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953671+0000", 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.8181999999999999e-05 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953678+0000", 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5839999999999997e-06 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:10:28.526 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953685+0000", 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2450000000000003e-06 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953804+0000", 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000119354 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953811+0000", 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1810000000000004e-06 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.527 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.235579+0000", 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2817674969999999 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.235799+0000", 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00022075099999999999 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,5,6", 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236011+0000", 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021142099999999999 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.528 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.240979+0000", 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.004968175 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241892+0000", 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00091299600000000001 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.242287+0000", 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00039454099999999999 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.529 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.242324+0000", 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7509999999999998e-05 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.242378+0000", 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3770000000000002e-05 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.242556+0000", 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000178314 2024-03-26T05:10:28.530 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.531 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.531 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.531 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:10:28.531 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:10:28.531 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:10:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:28.554 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:28.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:28.655 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:28.683 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:28.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:28.785 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:28.790 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:28 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:10:28.790 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:28 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:10:28.790 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:28 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/597352167' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:10:28.790 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:28 smithi080 ceph-mon[40404]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T05:10:28.790 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:28 smithi080 ceph-mon[40404]: 6.8 scrub starts 2024-03-26T05:10:28.790 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:28 smithi080 ceph-mon[40404]: 6.8 scrub ok 2024-03-26T05:10:28.790 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:28 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:10:28.791 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:28 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:10:28.791 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:28 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/597352167' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:10:28.791 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:28 smithi080 ceph-mon[29691]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T05:10:28.791 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:28 smithi080 ceph-mon[29691]: 6.8 scrub starts 2024-03-26T05:10:28.791 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:28 smithi080 ceph-mon[29691]: 6.8 scrub ok 2024-03-26T05:10:28.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:28 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:10:28.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:28 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:10:28.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:28 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/597352167' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:10:28.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:28 smithi132 ceph-mon[33500]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T05:10:28.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:28 smithi132 ceph-mon[33500]: 6.8 scrub starts 2024-03-26T05:10:28.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:28 smithi132 ceph-mon[33500]: 6.8 scrub ok 2024-03-26T05:10:28.812 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:28.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:28.914 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:28.938 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:28.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:28.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:29.040 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:29.063 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:29.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:29.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:29.166 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:29.189 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:29.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:29.290 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:29.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:29.317 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:29.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:29.418 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:29.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:29.443 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:29.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:29.544 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:29.573 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:29.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:29.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:29.675 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:29.703 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:29.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:29.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:29 smithi132 ceph-mon[33500]: pgmap v451: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 267 op/s 2024-03-26T05:10:29.805 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:29.831 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:29.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:29.831 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:29 smithi080 ceph-mon[40404]: pgmap v451: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 267 op/s 2024-03-26T05:10:29.832 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:29 smithi080 ceph-mon[29691]: pgmap v451: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 267 op/s 2024-03-26T05:10:29.860 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 7 1 2024-03-26T05:10:29.932 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:29.968 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:29.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:30.069 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:30.098 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:30.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:30.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:30.200 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:30.227 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:30.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:30.328 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:30.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:30.356 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:30.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:30.457 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:30.482 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:30.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:30.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:30.584 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:30.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:30.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[40404]: 6.10 scrub starts 2024-03-26T05:10:30.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[40404]: 6.10 scrub ok 2024-03-26T05:10:30.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[40404]: 5.18 scrub starts 2024-03-26T05:10:30.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[40404]: 5.18 scrub ok 2024-03-26T05:10:30.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[40404]: 3.1d scrub starts 2024-03-26T05:10:30.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[40404]: 3.1d scrub ok 2024-03-26T05:10:30.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[40404]: 4.10 scrub starts 2024-03-26T05:10:30.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[40404]: 4.10 scrub ok 2024-03-26T05:10:30.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[40404]: 5.9 scrub starts 2024-03-26T05:10:30.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[40404]: 5.9 scrub ok 2024-03-26T05:10:30.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[29691]: 6.10 scrub starts 2024-03-26T05:10:30.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[29691]: 6.10 scrub ok 2024-03-26T05:10:30.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[29691]: 5.18 scrub starts 2024-03-26T05:10:30.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[29691]: 5.18 scrub ok 2024-03-26T05:10:30.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[29691]: 3.1d scrub starts 2024-03-26T05:10:30.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[29691]: 3.1d scrub ok 2024-03-26T05:10:30.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[29691]: 4.10 scrub starts 2024-03-26T05:10:30.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[29691]: 4.10 scrub ok 2024-03-26T05:10:30.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[29691]: 5.9 scrub starts 2024-03-26T05:10:30.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:30 smithi080 ceph-mon[29691]: 5.9 scrub ok 2024-03-26T05:10:30.710 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:30.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:30.734 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:30.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:30 smithi132 ceph-mon[33500]: 6.10 scrub starts 2024-03-26T05:10:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:30 smithi132 ceph-mon[33500]: 6.10 scrub ok 2024-03-26T05:10:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:30 smithi132 ceph-mon[33500]: 5.18 scrub starts 2024-03-26T05:10:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:30 smithi132 ceph-mon[33500]: 5.18 scrub ok 2024-03-26T05:10:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:30 smithi132 ceph-mon[33500]: 3.1d scrub starts 2024-03-26T05:10:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:30 smithi132 ceph-mon[33500]: 3.1d scrub ok 2024-03-26T05:10:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:30 smithi132 ceph-mon[33500]: 4.10 scrub starts 2024-03-26T05:10:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:30 smithi132 ceph-mon[33500]: 4.10 scrub ok 2024-03-26T05:10:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:30 smithi132 ceph-mon[33500]: 5.9 scrub starts 2024-03-26T05:10:30.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:30 smithi132 ceph-mon[33500]: 5.9 scrub ok 2024-03-26T05:10:30.835 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:30.860 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:30.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:30.961 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:30.984 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:30.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:31.087 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:31.113 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:31.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:31.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:31.156 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:10:31.175 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:10:31.194 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:10:31.213 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:10:31.215 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:31.234 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:10:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:31.239 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:31.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:31.254 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:10:31.274 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:10:31.293 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:10:31.295 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:10:31.295 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:10:31.295 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:10:31.295 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:10:31.295 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:10:31.295 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:10:31.296 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:10:31.296 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:10:31.340 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:31.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:31.366 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:31.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:31.426 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:31.467 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:31.493 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:31.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:31.594 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:31.619 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:31.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:31.720 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:31.745 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:31.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:31.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:31 smithi132 ceph-mon[33500]: pgmap v452: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 215 op/s 2024-03-26T05:10:31.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:31 smithi132 ceph-mon[33500]: 5.d scrub starts 2024-03-26T05:10:31.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:31 smithi132 ceph-mon[33500]: 5.d scrub ok 2024-03-26T05:10:31.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:31 smithi132 ceph-mon[33500]: 6.18 scrub starts 2024-03-26T05:10:31.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:31 smithi132 ceph-mon[33500]: 6.18 scrub ok 2024-03-26T05:10:31.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:31 smithi132 ceph-mon[33500]: 8.4 scrub starts 2024-03-26T05:10:31.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:31 smithi132 ceph-mon[33500]: 8.4 scrub ok 2024-03-26T05:10:31.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:31 smithi132 ceph-mon[33500]: 5.14 scrub starts 2024-03-26T05:10:31.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:31 smithi132 ceph-mon[33500]: 5.14 scrub ok 2024-03-26T05:10:31.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:31 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3693946835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:31.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:31 smithi132 ceph-mon[33500]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T05:10:31.845 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:31.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[29691]: pgmap v452: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 215 op/s 2024-03-26T05:10:31.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[29691]: 5.d scrub starts 2024-03-26T05:10:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[29691]: 5.d scrub ok 2024-03-26T05:10:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[29691]: 6.18 scrub starts 2024-03-26T05:10:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[29691]: 6.18 scrub ok 2024-03-26T05:10:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[29691]: 8.4 scrub starts 2024-03-26T05:10:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[29691]: 8.4 scrub ok 2024-03-26T05:10:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[29691]: 5.14 scrub starts 2024-03-26T05:10:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[29691]: 5.14 scrub ok 2024-03-26T05:10:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3693946835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[29691]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T05:10:31.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[40404]: pgmap v452: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 215 op/s 2024-03-26T05:10:31.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[40404]: 5.d scrub starts 2024-03-26T05:10:31.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[40404]: 5.d scrub ok 2024-03-26T05:10:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[40404]: 6.18 scrub starts 2024-03-26T05:10:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[40404]: 6.18 scrub ok 2024-03-26T05:10:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[40404]: 8.4 scrub starts 2024-03-26T05:10:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[40404]: 8.4 scrub ok 2024-03-26T05:10:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[40404]: 5.14 scrub starts 2024-03-26T05:10:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[40404]: 5.14 scrub ok 2024-03-26T05:10:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3693946835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:31.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:31 smithi080 ceph-mon[40404]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T05:10:31.869 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:31.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:31.971 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:31.992 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T05:10:31.995 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:31.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:32.097 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:32.120 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:32.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:32.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:32.222 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:32.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:32.247 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:32.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:32.347 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:32.372 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:32.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:32.473 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:32.496 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:32.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:32.598 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:32.623 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:32.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:32.710 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:10:32.725 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:32.776 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:32.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:32.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:32 smithi132 ceph-mon[33500]: 8.8 scrub starts 2024-03-26T05:10:32.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:32 smithi132 ceph-mon[33500]: 8.8 scrub ok 2024-03-26T05:10:32.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:32 smithi080 ceph-mon[29691]: 8.8 scrub starts 2024-03-26T05:10:32.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:32 smithi080 ceph-mon[29691]: 8.8 scrub ok 2024-03-26T05:10:32.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:32 smithi080 ceph-mon[40404]: 8.8 scrub starts 2024-03-26T05:10:32.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:32 smithi080 ceph-mon[40404]: 8.8 scrub ok 2024-03-26T05:10:32.878 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:32.902 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:32.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:33.004 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:33.028 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:33.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:33.129 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:33.154 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:33.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:33.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:33.279 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:33.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:33.381 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:33.405 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:33.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:33.507 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:33.534 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:33.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:33.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[29691]: pgmap v454: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-03-26T05:10:33.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[29691]: 4.1b scrub starts 2024-03-26T05:10:33.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[29691]: 4.1b scrub ok 2024-03-26T05:10:33.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[29691]: 8.d scrub starts 2024-03-26T05:10:33.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[29691]: 8.d scrub ok 2024-03-26T05:10:33.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[29691]: 6.1c scrub starts 2024-03-26T05:10:33.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[29691]: 6.1c scrub ok 2024-03-26T05:10:33.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[40404]: pgmap v454: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-03-26T05:10:33.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[40404]: 4.1b scrub starts 2024-03-26T05:10:33.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[40404]: 4.1b scrub ok 2024-03-26T05:10:33.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[40404]: 8.d scrub starts 2024-03-26T05:10:33.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[40404]: 8.d scrub ok 2024-03-26T05:10:33.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[40404]: 6.1c scrub starts 2024-03-26T05:10:33.618 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:33 smithi080 ceph-mon[40404]: 6.1c scrub ok 2024-03-26T05:10:33.636 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:33.660 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:33.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:33.762 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:33.785 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:33.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:33.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:33 smithi132 ceph-mon[33500]: pgmap v454: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-03-26T05:10:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:33 smithi132 ceph-mon[33500]: 4.1b scrub starts 2024-03-26T05:10:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:33 smithi132 ceph-mon[33500]: 4.1b scrub ok 2024-03-26T05:10:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:33 smithi132 ceph-mon[33500]: 8.d scrub starts 2024-03-26T05:10:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:33 smithi132 ceph-mon[33500]: 8.d scrub ok 2024-03-26T05:10:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:33 smithi132 ceph-mon[33500]: 6.1c scrub starts 2024-03-26T05:10:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:33 smithi132 ceph-mon[33500]: 6.1c scrub ok 2024-03-26T05:10:33.887 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:33.911 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:33.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:34.013 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:34.037 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:34.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:34.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:34.139 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:34.163 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:34.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:34.265 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:34.279 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:10:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:10:34.290 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:34.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:34.294 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:34.295 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:34.392 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:34.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:34.416 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:34.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:34.517 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:34.542 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'unique_pool_1' removed 2024-03-26T05:10:34.549 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:34.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:34.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:34.651 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:34.674 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:34.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:34.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:34.776 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:34.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:34 smithi132 ceph-mon[33500]: 5.1 scrub starts 2024-03-26T05:10:34.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:34 smithi132 ceph-mon[33500]: 5.1 scrub ok 2024-03-26T05:10:34.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:34 smithi132 ceph-mon[33500]: 4.15 scrub starts 2024-03-26T05:10:34.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:34 smithi132 ceph-mon[33500]: 4.15 scrub ok 2024-03-26T05:10:34.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:34 smithi132 ceph-mon[33500]: 6.b scrub starts 2024-03-26T05:10:34.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:34 smithi132 ceph-mon[33500]: 6.b scrub ok 2024-03-26T05:10:34.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:34 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/602687346' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:10:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:34.809 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:34.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:34.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[40404]: 5.1 scrub starts 2024-03-26T05:10:34.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[40404]: 5.1 scrub ok 2024-03-26T05:10:34.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[40404]: 4.15 scrub starts 2024-03-26T05:10:34.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[40404]: 4.15 scrub ok 2024-03-26T05:10:34.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[40404]: 6.b scrub starts 2024-03-26T05:10:34.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[40404]: 6.b scrub ok 2024-03-26T05:10:34.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/602687346' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:10:34.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[29691]: 5.1 scrub starts 2024-03-26T05:10:34.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[29691]: 5.1 scrub ok 2024-03-26T05:10:34.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[29691]: 4.15 scrub starts 2024-03-26T05:10:34.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[29691]: 4.15 scrub ok 2024-03-26T05:10:34.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[29691]: 6.b scrub starts 2024-03-26T05:10:34.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[29691]: 6.b scrub ok 2024-03-26T05:10:34.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:34 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/602687346' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:10:34.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:34.933 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:34.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:35.035 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:35.059 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:35.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:35.161 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:35.185 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:35.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:35.286 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:35.312 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:35.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:35.414 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:35.438 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:35.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:35.540 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:35.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:10:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:10:35.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:35 smithi132 ceph-mon[33500]: pgmap v455: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-03-26T05:10:35.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:35 smithi132 ceph-mon[33500]: 5.3 scrub starts 2024-03-26T05:10:35.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:35 smithi132 ceph-mon[33500]: 5.3 scrub ok 2024-03-26T05:10:35.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:35 smithi132 ceph-mon[33500]: 6.1 scrub starts 2024-03-26T05:10:35.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:35 smithi132 ceph-mon[33500]: 6.1 scrub ok 2024-03-26T05:10:35.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:35 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/602687346' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:10:35.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:35 smithi132 ceph-mon[33500]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T05:10:35.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:35 smithi132 ceph-mon[33500]: 6.3 scrub starts 2024-03-26T05:10:35.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:35 smithi132 ceph-mon[33500]: 6.3 scrub ok 2024-03-26T05:10:35.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:10:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:35.564 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:35.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:35.665 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:35.689 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:35.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:35.790 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:35.814 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:35.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:35.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:35.845 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:35.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[29691]: pgmap v455: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-03-26T05:10:35.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[29691]: 5.3 scrub starts 2024-03-26T05:10:35.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[29691]: 5.3 scrub ok 2024-03-26T05:10:35.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[29691]: 6.1 scrub starts 2024-03-26T05:10:35.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[29691]: 6.1 scrub ok 2024-03-26T05:10:35.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/602687346' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:10:35.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[29691]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T05:10:35.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[29691]: 6.3 scrub starts 2024-03-26T05:10:35.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[29691]: 6.3 scrub ok 2024-03-26T05:10:35.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:10:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[40404]: pgmap v455: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-03-26T05:10:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[40404]: 5.3 scrub starts 2024-03-26T05:10:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[40404]: 5.3 scrub ok 2024-03-26T05:10:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[40404]: 6.1 scrub starts 2024-03-26T05:10:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[40404]: 6.1 scrub ok 2024-03-26T05:10:35.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/602687346' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:10:35.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[40404]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T05:10:35.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[40404]: 6.3 scrub starts 2024-03-26T05:10:35.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[40404]: 6.3 scrub ok 2024-03-26T05:10:35.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:10:35.916 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:35.940 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:35.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:35.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:36.041 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:36.066 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:36.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:36.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:36.191 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:10:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:36.192 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:36.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:36.293 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:36.317 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:36.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:36.419 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:36.445 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:36.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:36.547 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:36.572 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:36.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:36.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:36.673 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:36.697 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:36.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:36.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:36.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:36.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:36 smithi132 ceph-mon[33500]: 6.15 scrub starts 2024-03-26T05:10:36.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:36 smithi132 ceph-mon[33500]: 6.15 scrub ok 2024-03-26T05:10:36.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:36 smithi132 ceph-mon[33500]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T05:10:36.799 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:36.825 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:36.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:36.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:36.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:36 smithi080 ceph-mon[29691]: 6.15 scrub starts 2024-03-26T05:10:36.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:36 smithi080 ceph-mon[29691]: 6.15 scrub ok 2024-03-26T05:10:36.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:36 smithi080 ceph-mon[29691]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T05:10:36.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:36.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:36 smithi080 ceph-mon[40404]: 6.15 scrub starts 2024-03-26T05:10:36.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:36 smithi080 ceph-mon[40404]: 6.15 scrub ok 2024-03-26T05:10:36.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:36 smithi080 ceph-mon[40404]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T05:10:36.926 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:36.951 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:36.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:37.045 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T05:10:37.046 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T05:10:37.046 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-26T05:10:37.047 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-26T05:10:37.053 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:37.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:37.114 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:37.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:37.215 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:37.242 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:37.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:37.343 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:37.369 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:37.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:37.471 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:37.496 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:37.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:37.505 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:37.597 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:37.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:37 smithi080 ceph-mon[29691]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 54 op/s 2024-03-26T05:10:37.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:37 smithi080 ceph-mon[29691]: 4.17 scrub starts 2024-03-26T05:10:37.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:37 smithi080 ceph-mon[29691]: 4.17 scrub ok 2024-03-26T05:10:37.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:37 smithi080 ceph-mon[29691]: 5.4 scrub starts 2024-03-26T05:10:37.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:37 smithi080 ceph-mon[29691]: 5.4 scrub ok 2024-03-26T05:10:37.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:37 smithi080 ceph-mon[40404]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 54 op/s 2024-03-26T05:10:37.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:37 smithi080 ceph-mon[40404]: 4.17 scrub starts 2024-03-26T05:10:37.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:37 smithi080 ceph-mon[40404]: 4.17 scrub ok 2024-03-26T05:10:37.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:37 smithi080 ceph-mon[40404]: 5.4 scrub starts 2024-03-26T05:10:37.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:37 smithi080 ceph-mon[40404]: 5.4 scrub ok 2024-03-26T05:10:37.623 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:37.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:37.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:37.751 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:37.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:37.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:37 smithi132 ceph-mon[33500]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 54 op/s 2024-03-26T05:10:37.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:37 smithi132 ceph-mon[33500]: 4.17 scrub starts 2024-03-26T05:10:37.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:37 smithi132 ceph-mon[33500]: 4.17 scrub ok 2024-03-26T05:10:37.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:37 smithi132 ceph-mon[33500]: 5.4 scrub starts 2024-03-26T05:10:37.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:37 smithi132 ceph-mon[33500]: 5.4 scrub ok 2024-03-26T05:10:37.852 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:37.878 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:37.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:37.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:37.980 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:38.004 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:38.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:38.106 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:38.134 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:38.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:38.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:38.236 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:38.260 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:38.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:38.361 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:38.389 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:38.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:38.490 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:38.518 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:38.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:38.619 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:38.646 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:38.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:38.747 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:38.772 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:38.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:38.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:38.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:38 smithi080 ceph-mon[29691]: 5.8 scrub starts 2024-03-26T05:10:38.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:38 smithi080 ceph-mon[29691]: 5.8 scrub ok 2024-03-26T05:10:38.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:38 smithi080 ceph-mon[40404]: 5.8 scrub starts 2024-03-26T05:10:38.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:38 smithi080 ceph-mon[40404]: 5.8 scrub ok 2024-03-26T05:10:38.873 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:38.897 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:38.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:38.999 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:39.022 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:39.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:39.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:38 smithi132 ceph-mon[33500]: 5.8 scrub starts 2024-03-26T05:10:39.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:38 smithi132 ceph-mon[33500]: 5.8 scrub ok 2024-03-26T05:10:39.124 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:39.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:39.149 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:39.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:39.250 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:39.274 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:39.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:39.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:39.376 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:39.399 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:39.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:39.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:39.501 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:39.524 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:39.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:39.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:39.551 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.7 to 1 (10000) 2024-03-26T05:10:39.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[40404]: pgmap v459: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 44 op/s 2024-03-26T05:10:39.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[40404]: 3.1b scrub starts 2024-03-26T05:10:39.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[40404]: 3.1b scrub ok 2024-03-26T05:10:39.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[40404]: 5.12 scrub starts 2024-03-26T05:10:39.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[40404]: 5.12 scrub ok 2024-03-26T05:10:39.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/628791393' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:10:39.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[40404]: 3.e scrub starts 2024-03-26T05:10:39.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[40404]: 3.e scrub ok 2024-03-26T05:10:39.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[29691]: pgmap v459: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 44 op/s 2024-03-26T05:10:39.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[29691]: 3.1b scrub starts 2024-03-26T05:10:39.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[29691]: 3.1b scrub ok 2024-03-26T05:10:39.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[29691]: 5.12 scrub starts 2024-03-26T05:10:39.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[29691]: 5.12 scrub ok 2024-03-26T05:10:39.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/628791393' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:10:39.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[29691]: 3.e scrub starts 2024-03-26T05:10:39.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:39 smithi080 ceph-mon[29691]: 3.e scrub ok 2024-03-26T05:10:39.626 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:39.650 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:39.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:39.751 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:39.776 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:39.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:39.877 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:39.901 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:39.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:39.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:40.003 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:40.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:40.028 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:40.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:40.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:39 smithi132 ceph-mon[33500]: pgmap v459: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 44 op/s 2024-03-26T05:10:40.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:39 smithi132 ceph-mon[33500]: 3.1b scrub starts 2024-03-26T05:10:40.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:39 smithi132 ceph-mon[33500]: 3.1b scrub ok 2024-03-26T05:10:40.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:39 smithi132 ceph-mon[33500]: 5.12 scrub starts 2024-03-26T05:10:40.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:39 smithi132 ceph-mon[33500]: 5.12 scrub ok 2024-03-26T05:10:40.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:39 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/628791393' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:10:40.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:39 smithi132 ceph-mon[33500]: 3.e scrub starts 2024-03-26T05:10:40.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:39 smithi132 ceph-mon[33500]: 3.e scrub ok 2024-03-26T05:10:40.128 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:40.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:40.153 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:40.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:40.254 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:40.278 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:40.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:40.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:40.379 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:40.404 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:40.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:40.505 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:40.553 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:40.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:40.559 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:10:40.654 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:40.677 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:40.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:40.736 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:10:40.736 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:10:40.736 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[40404]: 3.0 scrub starts 2024-03-26T05:10:40.737 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[40404]: 3.0 scrub ok 2024-03-26T05:10:40.737 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/628791393' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:10:40.737 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[40404]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T05:10:40.737 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:10:40.737 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:10:40.737 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[40404]: 1.0 scrub starts 2024-03-26T05:10:40.737 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[40404]: 1.0 scrub ok 2024-03-26T05:10:40.737 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[40404]: 6.17 scrub starts 2024-03-26T05:10:40.737 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3314090618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:10:40.737 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[29691]: 3.0 scrub starts 2024-03-26T05:10:40.737 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[29691]: 3.0 scrub ok 2024-03-26T05:10:40.737 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/628791393' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:10:40.738 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[29691]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T05:10:40.738 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:10:40.738 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:10:40.738 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[29691]: 1.0 scrub starts 2024-03-26T05:10:40.738 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[29691]: 1.0 scrub ok 2024-03-26T05:10:40.738 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[29691]: 6.17 scrub starts 2024-03-26T05:10:40.738 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:40 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3314090618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:10:40.779 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:40.804 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:40.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:40.905 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:40.929 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:40.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:41.031 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:41.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:40 smithi132 ceph-mon[33500]: 3.0 scrub starts 2024-03-26T05:10:41.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:40 smithi132 ceph-mon[33500]: 3.0 scrub ok 2024-03-26T05:10:41.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:40 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/628791393' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:10:41.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:40 smithi132 ceph-mon[33500]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T05:10:41.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:40 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:10:41.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:40 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:10:41.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:40 smithi132 ceph-mon[33500]: 1.0 scrub starts 2024-03-26T05:10:41.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:40 smithi132 ceph-mon[33500]: 1.0 scrub ok 2024-03-26T05:10:41.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:40 smithi132 ceph-mon[33500]: 6.17 scrub starts 2024-03-26T05:10:41.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:40 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3314090618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:10:41.054 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:41.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:41.156 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:41.179 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:41.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:41.281 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:41.305 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:41.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:41.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:41.406 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:41.434 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:41.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:41.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:41.547 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 5 1 2024-03-26T05:10:41.561 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:41.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:41.663 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:41.687 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:41.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:41.788 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:41.814 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:41.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:41.859 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[40404]: pgmap v461: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 49 KiB/s rd, 48 op/s 2024-03-26T05:10:41.859 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[40404]: 5.15 scrub starts 2024-03-26T05:10:41.859 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[40404]: 5.15 scrub ok 2024-03-26T05:10:41.859 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[40404]: 3.8 scrub starts 2024-03-26T05:10:41.859 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[40404]: 3.8 scrub ok 2024-03-26T05:10:41.859 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[40404]: 6.17 scrub ok 2024-03-26T05:10:41.859 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:10:41.860 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[40404]: Cluster is now healthy 2024-03-26T05:10:41.860 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3314090618' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:10:41.860 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[40404]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T05:10:41.860 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[29691]: pgmap v461: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 49 KiB/s rd, 48 op/s 2024-03-26T05:10:41.860 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[29691]: 5.15 scrub starts 2024-03-26T05:10:41.860 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[29691]: 5.15 scrub ok 2024-03-26T05:10:41.860 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[29691]: 3.8 scrub starts 2024-03-26T05:10:41.860 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[29691]: 3.8 scrub ok 2024-03-26T05:10:41.860 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[29691]: 6.17 scrub ok 2024-03-26T05:10:41.860 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:10:41.860 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[29691]: Cluster is now healthy 2024-03-26T05:10:41.861 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3314090618' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:10:41.861 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:41 smithi080 ceph-mon[29691]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T05:10:41.916 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:41.941 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:41.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:41.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:42.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:41 smithi132 ceph-mon[33500]: pgmap v461: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 49 KiB/s rd, 48 op/s 2024-03-26T05:10:42.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:41 smithi132 ceph-mon[33500]: 5.15 scrub starts 2024-03-26T05:10:42.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:41 smithi132 ceph-mon[33500]: 5.15 scrub ok 2024-03-26T05:10:42.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:41 smithi132 ceph-mon[33500]: 3.8 scrub starts 2024-03-26T05:10:42.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:41 smithi132 ceph-mon[33500]: 3.8 scrub ok 2024-03-26T05:10:42.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:41 smithi132 ceph-mon[33500]: 6.17 scrub ok 2024-03-26T05:10:42.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:41 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:10:42.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:41 smithi132 ceph-mon[33500]: Cluster is now healthy 2024-03-26T05:10:42.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:41 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3314090618' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:10:42.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:41 smithi132 ceph-mon[33500]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T05:10:42.043 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:42.067 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:42.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:42.169 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:42.193 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:42.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:42.293 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:42.321 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:42.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:42.423 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:42.449 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:42.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:42.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:42.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:42.579 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:42.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:42.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:42.680 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:42.709 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:42.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:42.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:42.810 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:42.834 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:42.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:42.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:42.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[40404]: 4.1e scrub starts 2024-03-26T05:10:42.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[40404]: 4.1e scrub ok 2024-03-26T05:10:42.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[40404]: 6.14 scrub starts 2024-03-26T05:10:42.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[40404]: 6.14 scrub ok 2024-03-26T05:10:42.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[40404]: 6.13 scrub starts 2024-03-26T05:10:42.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[40404]: 6.13 scrub ok 2024-03-26T05:10:42.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[40404]: 5.1a scrub starts 2024-03-26T05:10:42.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[40404]: 5.1a scrub ok 2024-03-26T05:10:42.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[29691]: 4.1e scrub starts 2024-03-26T05:10:42.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[29691]: 4.1e scrub ok 2024-03-26T05:10:42.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[29691]: 6.14 scrub starts 2024-03-26T05:10:42.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[29691]: 6.14 scrub ok 2024-03-26T05:10:42.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[29691]: 6.13 scrub starts 2024-03-26T05:10:42.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[29691]: 6.13 scrub ok 2024-03-26T05:10:42.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[29691]: 5.1a scrub starts 2024-03-26T05:10:42.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:42 smithi080 ceph-mon[29691]: 5.1a scrub ok 2024-03-26T05:10:42.899 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:42.928 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T05:10:42.937 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:42.944 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:42.966 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:42.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:43.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:42 smithi132 ceph-mon[33500]: 4.1e scrub starts 2024-03-26T05:10:43.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:42 smithi132 ceph-mon[33500]: 4.1e scrub ok 2024-03-26T05:10:43.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:42 smithi132 ceph-mon[33500]: 6.14 scrub starts 2024-03-26T05:10:43.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:42 smithi132 ceph-mon[33500]: 6.14 scrub ok 2024-03-26T05:10:43.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:42 smithi132 ceph-mon[33500]: 6.13 scrub starts 2024-03-26T05:10:43.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:42 smithi132 ceph-mon[33500]: 6.13 scrub ok 2024-03-26T05:10:43.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:42 smithi132 ceph-mon[33500]: 5.1a scrub starts 2024-03-26T05:10:43.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:42 smithi132 ceph-mon[33500]: 5.1a scrub ok 2024-03-26T05:10:43.067 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:43.094 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:43.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:43.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:43.196 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:43.222 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:43.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:43.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:43.323 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:43.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:43.348 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:43.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:43.449 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:43.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:43.474 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:43.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:43.575 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:43.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:43.599 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:43.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:43.700 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:43.727 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:43.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:43.758 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:43 smithi080 ceph-mon[29691]: pgmap v463: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:10:43.759 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:43 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:10:43.759 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:43 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:10:43.759 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:43 smithi080 ceph-mon[40404]: pgmap v463: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:10:43.759 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:43 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:10:43.759 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:43 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:10:43.759 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:10:43.759 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:10:43.759 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:10:43.760 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:10:43.760 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:10:43.760 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:10:43.778 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:10:43.798 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:10:43.799 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:10:43.799 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:10:43.799 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:10:43.800 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:10:43.800 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:10:43.800 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:10:43.800 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:10:43.800 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:10:43.828 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:43.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:43.852 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:43.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:43.953 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:43.979 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:43.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:43.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:44.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:43 smithi132 ceph-mon[33500]: pgmap v463: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:10:44.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:43 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:10:44.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:43 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:10:44.081 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:44.106 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:44.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:44.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:44.177 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:10:44.207 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:44.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:44.233 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:44.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:44.334 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:44.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:44.361 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:44.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:44.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:10:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:10:44.463 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:44.489 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:44.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:44.590 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:44.617 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:44.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:44.718 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:44.743 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:44.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:44.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: 5.6 scrub starts 2024-03-26T05:10:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: 5.6 scrub ok 2024-03-26T05:10:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: 4.16 scrub starts 2024-03-26T05:10:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: 4.16 scrub ok 2024-03-26T05:10:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: 4.f scrub starts 2024-03-26T05:10:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: 4.f scrub ok 2024-03-26T05:10:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:10:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:10:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T05:10:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3122293219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: 6.12 scrub starts 2024-03-26T05:10:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: 6.12 scrub ok 2024-03-26T05:10:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:44 smithi132 ceph-mon[33500]: pgmap v465: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:10:44.825 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: 5.6 scrub starts 2024-03-26T05:10:44.826 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: 5.6 scrub ok 2024-03-26T05:10:44.826 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: 4.16 scrub starts 2024-03-26T05:10:44.826 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: 4.16 scrub ok 2024-03-26T05:10:44.826 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: 4.f scrub starts 2024-03-26T05:10:44.826 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: 4.f scrub ok 2024-03-26T05:10:44.826 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:10:44.826 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:10:44.826 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T05:10:44.826 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3122293219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:44.826 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: 6.12 scrub starts 2024-03-26T05:10:44.826 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: 6.12 scrub ok 2024-03-26T05:10:44.827 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[40404]: pgmap v465: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:10:44.827 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: 5.6 scrub starts 2024-03-26T05:10:44.827 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: 5.6 scrub ok 2024-03-26T05:10:44.827 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: 4.16 scrub starts 2024-03-26T05:10:44.827 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: 4.16 scrub ok 2024-03-26T05:10:44.827 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: 4.f scrub starts 2024-03-26T05:10:44.827 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: 4.f scrub ok 2024-03-26T05:10:44.827 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:10:44.827 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:10:44.827 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T05:10:44.827 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3122293219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:44.827 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: 6.12 scrub starts 2024-03-26T05:10:44.828 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: 6.12 scrub ok 2024-03-26T05:10:44.828 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:44 smithi080 ceph-mon[29691]: pgmap v465: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:10:44.845 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:44.870 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:44.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:44.971 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:44.996 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:44.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:45.097 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:45.121 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:45.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:45.223 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:45.247 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:45.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:45.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:45.349 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:45.373 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:45.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:45.474 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:45.498 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:45.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:45.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:10:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:10:45.600 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:45.625 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:45.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:45.726 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:45.749 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:45.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:45.851 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:45.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[40404]: 5.7 scrub starts 2024-03-26T05:10:45.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[40404]: 5.7 scrub ok 2024-03-26T05:10:45.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[40404]: 6.c scrub starts 2024-03-26T05:10:45.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[40404]: 6.c scrub ok 2024-03-26T05:10:45.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:45.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[40404]: 3.1a scrub starts 2024-03-26T05:10:45.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[40404]: 3.1a scrub ok 2024-03-26T05:10:45.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[29691]: 5.7 scrub starts 2024-03-26T05:10:45.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[29691]: 5.7 scrub ok 2024-03-26T05:10:45.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[29691]: 6.c scrub starts 2024-03-26T05:10:45.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[29691]: 6.c scrub ok 2024-03-26T05:10:45.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:45.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[29691]: 3.1a scrub starts 2024-03-26T05:10:45.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:45 smithi080 ceph-mon[29691]: 3.1a scrub ok 2024-03-26T05:10:45.875 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:45.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:45.899 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:45.903 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:45.977 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:46.001 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:46.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:45 smithi132 ceph-mon[33500]: 5.7 scrub starts 2024-03-26T05:10:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:45 smithi132 ceph-mon[33500]: 5.7 scrub ok 2024-03-26T05:10:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:45 smithi132 ceph-mon[33500]: 6.c scrub starts 2024-03-26T05:10:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:45 smithi132 ceph-mon[33500]: 6.c scrub ok 2024-03-26T05:10:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:45 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:45 smithi132 ceph-mon[33500]: 3.1a scrub starts 2024-03-26T05:10:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:45 smithi132 ceph-mon[33500]: 3.1a scrub ok 2024-03-26T05:10:46.104 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:46.128 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:46.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:46.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:46.229 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:46.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:46.254 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:46.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:46.355 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:46.381 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:46.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:46.482 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:46.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:46.511 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:46.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:46.613 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:46.617 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:10:46.637 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:46.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:46.739 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:46.762 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:46.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:46.864 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:46.889 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:46.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:46.990 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:47.014 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:47.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:47.116 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:47.140 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:47.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:47.241 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:47.265 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:47.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:47.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:47.367 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:47.391 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:47.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:47.493 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:47.517 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:47.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:47.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:47.585 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.5 to 1 (10000) 2024-03-26T05:10:47.585 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[40404]: pgmap v466: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T05:10:47.585 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[40404]: 3.17 scrub starts 2024-03-26T05:10:47.585 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[40404]: 3.17 scrub ok 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[40404]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2423520722' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[29691]: pgmap v466: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[29691]: 3.17 scrub starts 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[29691]: 3.17 scrub ok 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[29691]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2423520722' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:10:47.586 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:47 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:10:47.618 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:47.641 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:47.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:47.743 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:47.766 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:47.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:47.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:47 smithi132 ceph-mon[33500]: pgmap v466: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T05:10:47.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:47 smithi132 ceph-mon[33500]: 3.17 scrub starts 2024-03-26T05:10:47.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:47 smithi132 ceph-mon[33500]: 3.17 scrub ok 2024-03-26T05:10:47.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:47 smithi132 ceph-mon[33500]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T05:10:47.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:47 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2423520722' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:10:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:47 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:10:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:47 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:10:47.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:47 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:10:47.868 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:47.892 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:47.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:47.993 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:48.018 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:48.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:48.119 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:48.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:48.143 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:48.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:48.244 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:48.269 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:48.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:48.371 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:48.397 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:48.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:48.499 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:48.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:48.625 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:48.648 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:48.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:48.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:48.750 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:48.775 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:48.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:48.784 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T05:10:48.784 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:10:48.849 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:48 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T05:10:48.849 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:48 smithi080 ceph-mon[29691]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T05:10:48.849 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:48 smithi080 ceph-mon[29691]: pgmap v469: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:10:48.849 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:48 smithi080 ceph-mon[29691]: 4.e scrub starts 2024-03-26T05:10:48.849 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:48 smithi080 ceph-mon[29691]: 4.e scrub ok 2024-03-26T05:10:48.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:48 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T05:10:48.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:48 smithi080 ceph-mon[40404]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T05:10:48.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:48 smithi080 ceph-mon[40404]: pgmap v469: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:10:48.850 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:48 smithi080 ceph-mon[40404]: 4.e scrub starts 2024-03-26T05:10:48.850 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:48 smithi080 ceph-mon[40404]: 4.e scrub ok 2024-03-26T05:10:48.878 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:48.902 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:48.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:49.003 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:49.028 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:49.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:48 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T05:10:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:48 smithi132 ceph-mon[33500]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T05:10:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:48 smithi132 ceph-mon[33500]: pgmap v469: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:10:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:48 smithi132 ceph-mon[33500]: 4.e scrub starts 2024-03-26T05:10:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:48 smithi132 ceph-mon[33500]: 4.e scrub ok 2024-03-26T05:10:49.129 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:49.153 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:49.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:49.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:49.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:49.279 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:49.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:49.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:49.381 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:49.409 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:49.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:49.510 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:49.535 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:49.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:49.636 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:49.659 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:49.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:49.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:49.761 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:49.784 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:49.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:49.851 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:49 smithi080 ceph-mon[40404]: 4.4 scrub starts 2024-03-26T05:10:49.851 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:49 smithi080 ceph-mon[40404]: 4.4 scrub ok 2024-03-26T05:10:49.851 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:49 smithi080 ceph-mon[40404]: 5.f scrub starts 2024-03-26T05:10:49.851 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:49 smithi080 ceph-mon[40404]: 5.f scrub ok 2024-03-26T05:10:49.852 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:49 smithi080 ceph-mon[29691]: 4.4 scrub starts 2024-03-26T05:10:49.852 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:49 smithi080 ceph-mon[29691]: 4.4 scrub ok 2024-03-26T05:10:49.852 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:49 smithi080 ceph-mon[29691]: 5.f scrub starts 2024-03-26T05:10:49.852 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:49 smithi080 ceph-mon[29691]: 5.f scrub ok 2024-03-26T05:10:49.886 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:49.908 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:49.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:49.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:50.010 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:50.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:50.034 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:50.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:50.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:49 smithi132 ceph-mon[33500]: 4.4 scrub starts 2024-03-26T05:10:50.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:49 smithi132 ceph-mon[33500]: 4.4 scrub ok 2024-03-26T05:10:50.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:49 smithi132 ceph-mon[33500]: 5.f scrub starts 2024-03-26T05:10:50.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:49 smithi132 ceph-mon[33500]: 5.f scrub ok 2024-03-26T05:10:50.135 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:50.163 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:50.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:50.265 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:50.290 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:50.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:50.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:50.392 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:50.421 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:50.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:50.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:50.523 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:50.550 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:50.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:50.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:50.607 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:10:50.652 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:50.675 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:50.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:50.777 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:50.797 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:50.805 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:50.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:50.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:50.831 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:10:50.831 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:10:50.831 INFO:teuthology.orchestra.run.smithi080.stdout: "complaint_time": 30, 2024-03-26T05:10:50.831 INFO:teuthology.orchestra.run.smithi080.stdout: "num_blocked_ops": 0 2024-03-26T05:10:50.831 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:10:50.832 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:50 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/590928628' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:10:50.832 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:50 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:10:50.832 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:50 smithi080 ceph-mon[40404]: pgmap v470: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T05:10:50.832 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:10:50.832 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:10:50.832 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:50 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/590928628' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:10:50.832 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:50 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:10:50.832 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:50 smithi080 ceph-mon[29691]: pgmap v470: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T05:10:50.833 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:10:50.833 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:10:50.907 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:50.931 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:50.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:51.033 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:51.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:50 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/590928628' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:10:51.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:50 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:10:51.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:50 smithi132 ceph-mon[33500]: pgmap v470: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T05:10:51.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:10:51.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:10:51.056 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:51.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:51.158 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:51.187 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:51.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:51.289 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:51.313 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:51.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:51.414 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:51.442 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:51.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:51.544 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:51.572 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:51.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:51.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:51.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:51 smithi080 ceph-mon[29691]: 6.16 scrub starts 2024-03-26T05:10:51.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:51 smithi080 ceph-mon[29691]: 6.16 scrub ok 2024-03-26T05:10:51.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:51 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:10:51.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:51 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:10:51.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:51 smithi080 ceph-mon[29691]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T05:10:51.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:51 smithi080 ceph-mon[40404]: 6.16 scrub starts 2024-03-26T05:10:51.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:51 smithi080 ceph-mon[40404]: 6.16 scrub ok 2024-03-26T05:10:51.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:51 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:10:51.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:51 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:10:51.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:51 smithi080 ceph-mon[40404]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T05:10:51.674 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:51.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:51.699 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:51.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:51.801 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:51.825 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:51.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:51.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:51.927 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:51.951 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:52.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:51 smithi132 ceph-mon[33500]: 6.16 scrub starts 2024-03-26T05:10:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:51 smithi132 ceph-mon[33500]: 6.16 scrub ok 2024-03-26T05:10:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:51 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:10:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:51 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:10:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:51 smithi132 ceph-mon[33500]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T05:10:52.119 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:52.147 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:52.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:52.249 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:52.274 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:52.275 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:52.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:52.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:52.376 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:52.400 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:52.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:52.502 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:52.525 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:52.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:52.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:52.627 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:52.656 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:52.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:52.737 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:52 smithi080 ceph-mon[29691]: pgmap v472: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:10:52.738 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:52 smithi080 ceph-mon[40404]: pgmap v472: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:10:52.758 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:52.764 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T05:10:52.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:52.783 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:52.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:52.885 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:52.913 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:52.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:53.014 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:53.041 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:53.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:53.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:52 smithi132 ceph-mon[33500]: pgmap v472: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:10:53.142 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:53.170 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:53.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:53.272 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:53.301 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:53.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:53.403 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:53.428 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:53.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:53.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:53.530 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:53.555 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:53.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:53.636 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:10:53.657 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:53.662 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:10:53.681 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:10:53.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:53.686 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:53.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:53.702 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:10:53.722 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:10:53.742 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:10:53.762 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:10:53.784 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:10:53.785 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:10:53.785 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:10:53.785 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:10:53.785 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:10:53.785 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:10:53.785 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:10:53.785 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:10:53.785 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:10:53.787 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:53.812 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:53.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:53.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:53 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/510851168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:53.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:53 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/510851168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:53.915 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:53.938 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:53.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:54.040 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:54.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:53 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/510851168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:10:54.067 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:54.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:54.169 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:54.197 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:54.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:54.272 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:10:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:10:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:10:54.299 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:54.325 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:54.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:54.427 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:54.455 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:54.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:54.556 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:54.580 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:54.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:54.621 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'unique_pool_2' created 2024-03-26T05:10:54.682 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:54.709 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:54.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:54.811 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:54.837 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:54.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:54.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:54 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:10:54.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:54 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:10:54.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:54 smithi080 ceph-mon[40404]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T05:10:54.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:54 smithi080 ceph-mon[40404]: pgmap v474: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-03-26T05:10:54.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:54 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2006105622' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T05:10:54.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:54 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:10:54.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:54 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:10:54.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:54 smithi080 ceph-mon[29691]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T05:10:54.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:54 smithi080 ceph-mon[29691]: pgmap v474: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-03-26T05:10:54.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:54 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2006105622' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T05:10:54.939 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:54.965 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:54.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:55.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:54 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:10:55.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:54 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:10:55.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:54 smithi132 ceph-mon[33500]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T05:10:55.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:54 smithi132 ceph-mon[33500]: pgmap v474: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-03-26T05:10:55.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:54 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2006105622' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T05:10:55.067 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:55.092 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:55.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:55.194 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:10:55.219 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:55.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:55.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:10:55.321 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:55.378 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:55.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:55.479 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:55.505 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:55.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:55.607 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:55.618 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:10:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:10:55.632 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:55.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:55.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:55.734 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:55.761 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:55.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:55.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:55.863 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:55.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:55.956 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:55 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2006105622' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-26T05:10:55.956 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:55 smithi080 ceph-mon[40404]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T05:10:55.956 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:55 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:55.956 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:55 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2006105622' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-26T05:10:55.957 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:55 smithi080 ceph-mon[29691]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T05:10:55.957 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:55 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:55.988 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:56.014 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:56.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:56.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:55 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2006105622' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-26T05:10:56.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:55 smithi132 ceph-mon[33500]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T05:10:56.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:55 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:10:56.116 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:56.140 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:56.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:56.242 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:56.266 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:56.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:56.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:56.368 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:56.393 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:56.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:56.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:56.416 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:10:56.419 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:10:56.494 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:56.519 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:56.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:56.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:56.621 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:56.650 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:56.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:56.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:56.752 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:56.776 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:56.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:56.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:56.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:56 smithi080 ceph-mon[40404]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T05:10:56.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:56 smithi080 ceph-mon[40404]: pgmap v477: 148 pgs: 13 creating+peering, 135 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-03-26T05:10:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:56 smithi080 ceph-mon[40404]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:10:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:56 smithi080 ceph-mon[40404]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T05:10:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:56 smithi080 ceph-mon[29691]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T05:10:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:56 smithi080 ceph-mon[29691]: pgmap v477: 148 pgs: 13 creating+peering, 135 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-03-26T05:10:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:56 smithi080 ceph-mon[29691]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:10:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:56 smithi080 ceph-mon[29691]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T05:10:56.877 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:56.901 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:56.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:56.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:57.002 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:57.026 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:57.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:57.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:56 smithi132 ceph-mon[33500]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T05:10:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:56 smithi132 ceph-mon[33500]: pgmap v477: 148 pgs: 13 creating+peering, 135 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-03-26T05:10:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:56 smithi132 ceph-mon[33500]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:10:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:56 smithi132 ceph-mon[33500]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T05:10:57.051 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-26T05:10:57.127 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:57.156 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:57.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:57.251 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:57.258 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:57.286 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:57.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:57.314 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:10:57.387 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:57.415 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:57.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:57.517 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:57.545 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:57.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:57.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:57.647 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:57.658 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-03-26T05:10:55.918149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":2274,"num_read_kb":2088,"num_write":1093,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1774542,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1158,"ondisk_log_size":1158,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":400,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2611004,"kb_used_data":134732,"kb_used_omap":12,"kb_used_meta":2476147,"kb_avail":747350212,"statfs":{"total":767960285184,"available":765286617088,"internally_reserved":0,"allocated":137965568,"data_stored":134468760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2535575088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.311021"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":441,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629187+0000","last_change":"2024-03-26T05:09:39.468117+0000","last_active":"2024-03-26T05:10:54.629187+0000","last_peered":"2024-03-26T05:10:54.629187+0000","last_clean":"2024-03-26T05:10:54.629187+0000","last_became_active":"2024-03-26T05:07:59.996436+0000","last_became_peered":"2024-03-26T05:07:59.996436+0000","last_unstale":"2024-03-26T05:10:54.629187+0000","last_undegraded":"2024-03-26T05:10:54.629187+0000","last_fullsized":"2024-03-26T05:10:54.629187+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:39.468037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:39.468037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":444,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634439+0000","last_change":"2024-03-26T05:09:44.530686+0000","last_active":"2024-03-26T05:10:54.634439+0000","last_peered":"2024-03-26T05:10:54.634439+0000","last_clean":"2024-03-26T05:10:54.634439+0000","last_became_active":"2024-03-26T05:07:59.997089+0000","last_became_peered":"2024-03-26T05:07:59.997089+0000","last_unstale":"2024-03-26T05:10:54.634439+0000","last_undegraded":"2024-03-26T05:10:54.634439+0000","last_fullsized":"2024-03-26T05:10:54.634439+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:44.530563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:44.530563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.884815+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":456,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.644897+0000","last_change":"2024-03-26T05:09:24.726025+0000","last_active":"2024-03-26T05:10:55.644897+0000","last_peered":"2024-03-26T05:10:55.644897+0000","last_clean":"2024-03-26T05:10:55.644897+0000","last_became_active":"2024-03-26T05:07:59.994431+0000","last_became_peered":"2024-03-26T05:07:59.994431+0000","last_unstale":"2024-03-26T05:10:55.644897+0000","last_undegraded":"2024-03-26T05:10:55.644897+0000","last_fullsized":"2024-03-26T05:10:55.644897+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:24.725911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:24.725911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:00.073176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"254'21","reported_seq":474,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629102+0000","last_change":"2024-03-26T05:09:55.417464+0000","last_active":"2024-03-26T05:10:54.629102+0000","last_peered":"2024-03-26T05:10:54.629102+0000","last_clean":"2024-03-26T05:10:54.629102+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:10:54.629102+0000","last_undegraded":"2024-03-26T05:10:54.629102+0000","last_fullsized":"2024-03-26T05:10:54.629102+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:09:55.417325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.417325+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:24.296213+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":419,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633642+0000","last_change":"2024-03-26T05:09:59.528706+0000","last_active":"2024-03-26T05:10:54.633642+0000","last_peered":"2024-03-26T05:10:54.633642+0000","last_clean":"2024-03-26T05:10:54.633642+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:10:54.633642+0000","last_undegraded":"2024-03-26T05:10:54.633642+0000","last_fullsized":"2024-03-26T05:10:54.633642+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:59.528596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:59.528596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:25.643478+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":433,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.628979+0000","last_change":"2024-03-26T05:10:29.404548+0000","last_active":"2024-03-26T05:10:54.628979+0000","last_peered":"2024-03-26T05:10:54.628979+0000","last_clean":"2024-03-26T05:10:54.628979+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:10:54.628979+0000","last_undegraded":"2024-03-26T05:10:54.628979+0000","last_fullsized":"2024-03-26T05:10:54.628979+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:29.404404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.404404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:51.272613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":440,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633566+0000","last_change":"2024-03-26T05:09:40.635212+0000","last_active":"2024-03-26T05:10:54.633566+0000","last_peered":"2024-03-26T05:10:54.633566+0000","last_clean":"2024-03-26T05:10:54.633566+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:10:54.633566+0000","last_undegraded":"2024-03-26T05:10:54.633566+0000","last_fullsized":"2024-03-26T05:10:54.633566+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:40.635129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:09:40.635129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.760848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"254'51","reported_seq":518,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635456+0000","last_change":"2024-03-26T05:07:46.642266+0000","last_active":"2024-03-26T05:10:54.635456+0000","last_peered":"2024-03-26T05:10:54.635456+0000","last_clean":"2024-03-26T05:10:54.635456+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:10:54.635456+0000","last_undegraded":"2024-03-26T05:10:54.635456+0000","last_fullsized":"2024-03-26T05:10:54.635456+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'39","last_scrub_stamp":"2024-03-26T05:07:46.641260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:46.641260+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.470839+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":439,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.645559+0000","last_change":"2024-03-26T05:10:30.211104+0000","last_active":"2024-03-26T05:10:55.645559+0000","last_peered":"2024-03-26T05:10:55.645559+0000","last_clean":"2024-03-26T05:10:55.645559+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:10:55.645559+0000","last_undegraded":"2024-03-26T05:10:55.645559+0000","last_fullsized":"2024-03-26T05:10:55.645559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:30.211007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.211007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:37.046198+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633817+0000","last_change":"2024-03-26T05:09:57.144879+0000","last_active":"2024-03-26T05:10:54.633817+0000","last_peered":"2024-03-26T05:10:54.633817+0000","last_clean":"2024-03-26T05:10:54.633817+0000","last_became_active":"2024-03-26T05:08:00.000128+0000","last_became_peered":"2024-03-26T05:08:00.000128+0000","last_unstale":"2024-03-26T05:10:54.633817+0000","last_undegraded":"2024-03-26T05:10:54.633817+0000","last_fullsized":"2024-03-26T05:10:54.633817+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:57.144806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:57.144806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:10.794551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634310+0000","last_change":"2024-03-26T05:10:29.515852+0000","last_active":"2024-03-26T05:10:54.634310+0000","last_peered":"2024-03-26T05:10:54.634310+0000","last_clean":"2024-03-26T05:10:54.634310+0000","last_became_active":"2024-03-26T05:07:59.947557+0000","last_became_peered":"2024-03-26T05:07:59.947557+0000","last_unstale":"2024-03-26T05:10:54.634310+0000","last_undegraded":"2024-03-26T05:10:54.634310+0000","last_fullsized":"2024-03-26T05:10:54.634310+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:29.515767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.515767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:52.847327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"254'27","reported_seq":486,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629367+0000","last_change":"2024-03-26T05:09:33.456052+0000","last_active":"2024-03-26T05:10:54.629367+0000","last_peered":"2024-03-26T05:10:54.629367+0000","last_clean":"2024-03-26T05:10:54.629367+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:10:54.629367+0000","last_undegraded":"2024-03-26T05:10:54.629367+0000","last_fullsized":"2024-03-26T05:10:54.629367+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:09:33.455867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:33.455867+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.632595+0000","last_change":"2024-03-26T05:09:48.186435+0000","last_active":"2024-03-26T05:10:54.632595+0000","last_peered":"2024-03-26T05:10:54.632595+0000","last_clean":"2024-03-26T05:10:54.632595+0000","last_became_active":"2024-03-26T05:07:59.998777+0000","last_became_peered":"2024-03-26T05:07:59.998777+0000","last_unstale":"2024-03-26T05:10:54.632595+0000","last_undegraded":"2024-03-26T05:10:54.632595+0000","last_fullsized":"2024-03-26T05:10:54.632595+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:48.186287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.186287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:00.482583+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":446,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634962+0000","last_change":"2024-03-26T05:10:41.797043+0000","last_active":"2024-03-26T05:10:54.634962+0000","last_peered":"2024-03-26T05:10:54.634962+0000","last_clean":"2024-03-26T05:10:54.634962+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:10:54.634962+0000","last_undegraded":"2024-03-26T05:10:54.634962+0000","last_fullsized":"2024-03-26T05:10:54.634962+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:41.796930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.796930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:56.003329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":114,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634852+0000","last_change":"2024-03-26T05:08:00.011593+0000","last_active":"2024-03-26T05:10:54.634852+0000","last_peered":"2024-03-26T05:10:54.634852+0000","last_clean":"2024-03-26T05:10:54.634852+0000","last_became_active":"2024-03-26T05:08:00.011262+0000","last_became_peered":"2024-03-26T05:08:00.011262+0000","last_unstale":"2024-03-26T05:10:54.634852+0000","last_undegraded":"2024-03-26T05:10:54.634852+0000","last_fullsized":"2024-03-26T05:10:54.634852+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:53.514177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.514177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"254'13","reported_seq":462,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629337+0000","last_change":"2024-03-26T05:10:32.425969+0000","last_active":"2024-03-26T05:10:54.629337+0000","last_peered":"2024-03-26T05:10:54.629337+0000","last_clean":"2024-03-26T05:10:54.629337+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:10:54.629337+0000","last_undegraded":"2024-03-26T05:10:54.629337+0000","last_fullsized":"2024-03-26T05:10:54.629337+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'13","last_scrub_stamp":"2024-03-26T05:10:32.425855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:32.425855+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:48.496500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.630178+0000","last_change":"2024-03-26T05:09:56.453730+0000","last_active":"2024-03-26T05:10:54.630178+0000","last_peered":"2024-03-26T05:10:54.630178+0000","last_clean":"2024-03-26T05:10:54.630178+0000","last_became_active":"2024-03-26T05:07:59.942829+0000","last_became_peered":"2024-03-26T05:07:59.942829+0000","last_unstale":"2024-03-26T05:10:54.630178+0000","last_undegraded":"2024-03-26T05:10:54.630178+0000","last_fullsized":"2024-03-26T05:10:54.630178+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:56.453608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:09:56.453608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:14.181663+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":432,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635153+0000","last_change":"2024-03-26T05:09:57.476795+0000","last_active":"2024-03-26T05:10:54.635153+0000","last_peered":"2024-03-26T05:10:54.635153+0000","last_clean":"2024-03-26T05:10:54.635153+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:10:54.635153+0000","last_undegraded":"2024-03-26T05:10:54.635153+0000","last_fullsized":"2024-03-26T05:10:54.635153+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:57.476689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:57.476689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:15.765524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":451,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.649174+0000","last_change":"2024-03-26T05:10:38.179665+0000","last_active":"2024-03-26T05:10:55.649174+0000","last_peered":"2024-03-26T05:10:55.649174+0000","last_clean":"2024-03-26T05:10:55.649174+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:10:55.649174+0000","last_undegraded":"2024-03-26T05:10:55.649174+0000","last_fullsized":"2024-03-26T05:10:55.649174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:38.179521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:10:38.179521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:45.972937+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"254'27","reported_seq":487,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634423+0000","last_change":"2024-03-26T05:09:48.440867+0000","last_active":"2024-03-26T05:10:54.634423+0000","last_peered":"2024-03-26T05:10:54.634423+0000","last_clean":"2024-03-26T05:10:54.634423+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:10:54.634423+0000","last_undegraded":"2024-03-26T05:10:54.634423+0000","last_fullsized":"2024-03-26T05:10:54.634423+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:09:48.440764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.440764+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:06.366696+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":440,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635373+0000","last_change":"2024-03-26T05:09:45.842503+0000","last_active":"2024-03-26T05:10:54.635373+0000","last_peered":"2024-03-26T05:10:54.635373+0000","last_clean":"2024-03-26T05:10:54.635373+0000","last_became_active":"2024-03-26T05:07:59.959705+0000","last_became_peered":"2024-03-26T05:07:59.959705+0000","last_unstale":"2024-03-26T05:10:54.635373+0000","last_undegraded":"2024-03-26T05:10:54.635373+0000","last_fullsized":"2024-03-26T05:10:54.635373+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:45.842392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.842392+0000","last_clean_scrub_stamp":"2024-03-26T05:09:45.842392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.470889+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":433,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.630296+0000","last_change":"2024-03-26T05:09:54.412717+0000","last_active":"2024-03-26T05:10:54.630296+0000","last_peered":"2024-03-26T05:10:54.630296+0000","last_clean":"2024-03-26T05:10:54.630296+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:10:54.630296+0000","last_undegraded":"2024-03-26T05:10:54.630296+0000","last_fullsized":"2024-03-26T05:10:54.630296+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:54.412617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:54.412617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:05.830877+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":457,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629658+0000","last_change":"2024-03-26T05:10:45.477455+0000","last_active":"2024-03-26T05:10:54.629658+0000","last_peered":"2024-03-26T05:10:54.629658+0000","last_clean":"2024-03-26T05:10:54.629658+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:10:54.629658+0000","last_undegraded":"2024-03-26T05:10:54.629658+0000","last_fullsized":"2024-03-26T05:10:54.629658+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:45.477372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:10:45.477372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:48.573961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"254'26","reported_seq":473,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634142+0000","last_change":"2024-03-26T05:07:50.662703+0000","last_active":"2024-03-26T05:10:54.634142+0000","last_peered":"2024-03-26T05:10:54.634142+0000","last_clean":"2024-03-26T05:10:54.634142+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:10:54.634142+0000","last_undegraded":"2024-03-26T05:10:54.634142+0000","last_fullsized":"2024-03-26T05:10:54.634142+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:07:50.662564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:50.662564+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.470785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634596+0000","last_change":"2024-03-26T05:10:32.752769+0000","last_active":"2024-03-26T05:10:54.634596+0000","last_peered":"2024-03-26T05:10:54.634596+0000","last_clean":"2024-03-26T05:10:54.634596+0000","last_became_active":"2024-03-26T05:07:59.955125+0000","last_became_peered":"2024-03-26T05:07:59.955125+0000","last_unstale":"2024-03-26T05:10:54.634596+0000","last_undegraded":"2024-03-26T05:10:54.634596+0000","last_fullsized":"2024-03-26T05:10:54.634596+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:10:32.752678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:32.752678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:49.036523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":585,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.632861+0000","last_change":"2024-03-26T05:09:31.493239+0000","last_active":"2024-03-26T05:10:54.632861+0000","last_peered":"2024-03-26T05:10:54.632861+0000","last_clean":"2024-03-26T05:10:54.632861+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:10:54.632861+0000","last_undegraded":"2024-03-26T05:10:54.632861+0000","last_fullsized":"2024-03-26T05:10:54.632861+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:09:31.493102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:31.493102+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.758195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":441,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634529+0000","last_change":"2024-03-26T05:09:55.509651+0000","last_active":"2024-03-26T05:10:54.634529+0000","last_peered":"2024-03-26T05:10:54.634529+0000","last_clean":"2024-03-26T05:10:54.634529+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:10:54.634529+0000","last_undegraded":"2024-03-26T05:10:54.634529+0000","last_fullsized":"2024-03-26T05:10:54.634529+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:09:55.509344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.509344+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:04.412791+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"254'26","reported_seq":490,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.645875+0000","last_change":"2024-03-26T05:10:41.134236+0000","last_active":"2024-03-26T05:10:55.645875+0000","last_peered":"2024-03-26T05:10:55.645875+0000","last_clean":"2024-03-26T05:10:55.645875+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:10:55.645875+0000","last_undegraded":"2024-03-26T05:10:55.645875+0000","last_fullsized":"2024-03-26T05:10:55.645875+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:10:41.134129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.134129+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.559878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":440,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635314+0000","last_change":"2024-03-26T05:09:46.565079+0000","last_active":"2024-03-26T05:10:54.635314+0000","last_peered":"2024-03-26T05:10:54.635314+0000","last_clean":"2024-03-26T05:10:54.635314+0000","last_became_active":"2024-03-26T05:07:59.995261+0000","last_became_peered":"2024-03-26T05:07:59.995261+0000","last_unstale":"2024-03-26T05:10:54.635314+0000","last_undegraded":"2024-03-26T05:10:54.635314+0000","last_fullsized":"2024-03-26T05:10:54.635314+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:46.564952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:46.564952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:56.415792+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":436,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.646061+0000","last_change":"2024-03-26T05:09:36.861688+0000","last_active":"2024-03-26T05:10:55.646061+0000","last_peered":"2024-03-26T05:10:55.646061+0000","last_clean":"2024-03-26T05:10:55.646061+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:10:55.646061+0000","last_undegraded":"2024-03-26T05:10:55.646061+0000","last_fullsized":"2024-03-26T05:10:55.646061+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:36.861410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:36.861410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:00.073195+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":439,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.636793+0000","last_change":"2024-03-26T05:09:56.528351+0000","last_active":"2024-03-26T05:10:54.636793+0000","last_peered":"2024-03-26T05:10:54.636793+0000","last_clean":"2024-03-26T05:10:54.636793+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:10:54.636793+0000","last_undegraded":"2024-03-26T05:10:54.636793+0000","last_fullsized":"2024-03-26T05:10:54.636793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:56.528075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:56.528075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:14.800367+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"254'25","reported_seq":554,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.632984+0000","last_change":"2024-03-26T05:10:39.649729+0000","last_active":"2024-03-26T05:10:54.632984+0000","last_peered":"2024-03-26T05:10:54.632984+0000","last_clean":"2024-03-26T05:10:54.632984+0000","last_became_active":"2024-03-26T05:07:59.962007+0000","last_became_peered":"2024-03-26T05:07:59.962007+0000","last_unstale":"2024-03-26T05:10:54.632984+0000","last_undegraded":"2024-03-26T05:10:54.632984+0000","last_fullsized":"2024-03-26T05:10:54.632984+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:10:39.649620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:39.649620+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.262212+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":434,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634831+0000","last_change":"2024-03-26T05:10:43.824858+0000","last_active":"2024-03-26T05:10:54.634831+0000","last_peered":"2024-03-26T05:10:54.634831+0000","last_clean":"2024-03-26T05:10:54.634831+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:10:54.634831+0000","last_undegraded":"2024-03-26T05:10:54.634831+0000","last_fullsized":"2024-03-26T05:10:54.634831+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:43.824746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.824746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:48.191458+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":438,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.631378+0000","last_change":"2024-03-26T05:09:35.497868+0000","last_active":"2024-03-26T05:10:54.631378+0000","last_peered":"2024-03-26T05:10:54.631378+0000","last_clean":"2024-03-26T05:10:54.631378+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:10:54.631378+0000","last_undegraded":"2024-03-26T05:10:54.631378+0000","last_fullsized":"2024-03-26T05:10:54.631378+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:35.497586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:35.497586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.562634+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":463,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.644659+0000","last_change":"2024-03-26T05:10:46.108129+0000","last_active":"2024-03-26T05:10:55.644659+0000","last_peered":"2024-03-26T05:10:55.644659+0000","last_clean":"2024-03-26T05:10:55.644659+0000","last_became_active":"2024-03-26T05:07:59.993996+0000","last_became_peered":"2024-03-26T05:07:59.993996+0000","last_unstale":"2024-03-26T05:10:55.644659+0000","last_undegraded":"2024-03-26T05:10:55.644659+0000","last_fullsized":"2024-03-26T05:10:55.644659+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:46.108051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:46.108051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:12.619539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"254'10","reported_seq":449,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.636945+0000","last_change":"2024-03-26T05:10:29.575129+0000","last_active":"2024-03-26T05:10:54.636945+0000","last_peered":"2024-03-26T05:10:54.636945+0000","last_clean":"2024-03-26T05:10:54.636945+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:10:54.636945+0000","last_undegraded":"2024-03-26T05:10:54.636945+0000","last_fullsized":"2024-03-26T05:10:54.636945+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'6","last_scrub_stamp":"2024-03-26T05:10:29.575014+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.575014+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:48.708154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":427,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633810+0000","last_change":"2024-03-26T05:10:41.595090+0000","last_active":"2024-03-26T05:10:54.633810+0000","last_peered":"2024-03-26T05:10:54.633810+0000","last_clean":"2024-03-26T05:10:54.633810+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:10:54.633810+0000","last_undegraded":"2024-03-26T05:10:54.633810+0000","last_fullsized":"2024-03-26T05:10:54.633810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:41.594950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.594950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:45.945454+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":438,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634577+0000","last_change":"2024-03-26T05:09:49.476829+0000","last_active":"2024-03-26T05:10:54.634577+0000","last_peered":"2024-03-26T05:10:54.634577+0000","last_clean":"2024-03-26T05:10:54.634577+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:10:54.634577+0000","last_undegraded":"2024-03-26T05:10:54.634577+0000","last_fullsized":"2024-03-26T05:10:54.634577+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:49.476722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:09:49.476722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.905204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":114,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634015+0000","last_change":"2024-03-26T05:07:59.997817+0000","last_active":"2024-03-26T05:10:54.634015+0000","last_peered":"2024-03-26T05:10:54.634015+0000","last_clean":"2024-03-26T05:10:54.634015+0000","last_became_active":"2024-03-26T05:07:59.997439+0000","last_became_peered":"2024-03-26T05:07:59.997439+0000","last_unstale":"2024-03-26T05:10:54.634015+0000","last_undegraded":"2024-03-26T05:10:54.634015+0000","last_fullsized":"2024-03-26T05:10:54.634015+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:06:56.886632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"254'25","reported_seq":470,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635355+0000","last_change":"2024-03-26T05:07:52.593239+0000","last_active":"2024-03-26T05:10:54.635355+0000","last_peered":"2024-03-26T05:10:54.635355+0000","last_clean":"2024-03-26T05:10:54.635355+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:10:54.635355+0000","last_undegraded":"2024-03-26T05:10:54.635355+0000","last_fullsized":"2024-03-26T05:10:54.635355+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:07:52.592943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:52.592943+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.470750+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":451,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.645137+0000","last_change":"2024-03-26T05:10:29.171886+0000","last_active":"2024-03-26T05:10:55.645137+0000","last_peered":"2024-03-26T05:10:55.645137+0000","last_clean":"2024-03-26T05:10:55.645137+0000","last_became_active":"2024-03-26T05:07:59.994210+0000","last_became_peered":"2024-03-26T05:07:59.994210+0000","last_unstale":"2024-03-26T05:10:55.645137+0000","last_undegraded":"2024-03-26T05:10:55.645137+0000","last_fullsized":"2024-03-26T05:10:55.645137+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:29.171780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.171780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:47.646591+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":423,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633559+0000","last_change":"2024-03-26T05:09:26.674360+0000","last_active":"2024-03-26T05:10:54.633559+0000","last_peered":"2024-03-26T05:10:54.633559+0000","last_clean":"2024-03-26T05:10:54.633559+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:10:54.633559+0000","last_undegraded":"2024-03-26T05:10:54.633559+0000","last_fullsized":"2024-03-26T05:10:54.633559+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:26.674245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:09:26.674245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.470940+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":438,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635571+0000","last_change":"2024-03-26T05:09:53.508140+0000","last_active":"2024-03-26T05:10:54.635571+0000","last_peered":"2024-03-26T05:10:54.635571+0000","last_clean":"2024-03-26T05:10:54.635571+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:10:54.635571+0000","last_undegraded":"2024-03-26T05:10:54.635571+0000","last_fullsized":"2024-03-26T05:10:54.635571+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:53.507753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:09:53.507753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:05.567493+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"254'15","reported_seq":461,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635940+0000","last_change":"2024-03-26T05:07:53.425727+0000","last_active":"2024-03-26T05:10:54.635940+0000","last_peered":"2024-03-26T05:10:54.635940+0000","last_clean":"2024-03-26T05:10:54.635940+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:10:54.635940+0000","last_undegraded":"2024-03-26T05:10:54.635940+0000","last_fullsized":"2024-03-26T05:10:54.635940+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-26T05:07:53.425472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.425472+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:56.415774+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":431,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635093+0000","last_change":"2024-03-26T05:09:28.716469+0000","last_active":"2024-03-26T05:10:54.635093+0000","last_peered":"2024-03-26T05:10:54.635093+0000","last_clean":"2024-03-26T05:10:54.635093+0000","last_became_active":"2024-03-26T05:07:59.999703+0000","last_became_peered":"2024-03-26T05:07:59.999703+0000","last_unstale":"2024-03-26T05:10:54.635093+0000","last_undegraded":"2024-03-26T05:10:54.635093+0000","last_fullsized":"2024-03-26T05:10:54.635093+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:28.716348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:09:28.716348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.470874+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":452,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635849+0000","last_change":"2024-03-26T05:10:38.473529+0000","last_active":"2024-03-26T05:10:54.635849+0000","last_peered":"2024-03-26T05:10:54.635849+0000","last_clean":"2024-03-26T05:10:54.635849+0000","last_became_active":"2024-03-26T05:08:00.001601+0000","last_became_peered":"2024-03-26T05:08:00.001601+0000","last_unstale":"2024-03-26T05:10:54.635849+0000","last_undegraded":"2024-03-26T05:10:54.635849+0000","last_fullsized":"2024-03-26T05:10:54.635849+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:38.473446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:10:38.473446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:43.065707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":433,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.630095+0000","last_change":"2024-03-26T05:07:39.699241+0000","last_active":"2024-03-26T05:10:54.630095+0000","last_peered":"2024-03-26T05:10:54.630095+0000","last_clean":"2024-03-26T05:10:54.630095+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:10:54.630095+0000","last_undegraded":"2024-03-26T05:10:54.630095+0000","last_fullsized":"2024-03-26T05:10:54.630095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:39.699122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:39.699122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"254'24","reported_seq":473,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629487+0000","last_change":"2024-03-26T05:07:43.754242+0000","last_active":"2024-03-26T05:10:54.629487+0000","last_peered":"2024-03-26T05:10:54.629487+0000","last_clean":"2024-03-26T05:10:54.629487+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:10:54.629487+0000","last_undegraded":"2024-03-26T05:10:54.629487+0000","last_fullsized":"2024-03-26T05:10:54.629487+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:07:43.754130+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:07:43.754130+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":439,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.644700+0000","last_change":"2024-03-26T05:10:50.061225+0000","last_active":"2024-03-26T05:10:55.644700+0000","last_peered":"2024-03-26T05:10:55.644700+0000","last_clean":"2024-03-26T05:10:55.644700+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:10:55.644700+0000","last_undegraded":"2024-03-26T05:10:55.644700+0000","last_fullsized":"2024-03-26T05:10:55.644700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:50.061128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_clean_scrub_stamp":"2024-03-26T05:10:50.061128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:08.890951+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"216'10","reported_seq":161,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634194+0000","last_change":"2024-03-26T05:10:40.373554+0000","last_active":"2024-03-26T05:10:54.634194+0000","last_peered":"2024-03-26T05:10:54.634194+0000","last_clean":"2024-03-26T05:10:54.634194+0000","last_became_active":"2024-03-26T05:08:00.005648+0000","last_became_peered":"2024-03-26T05:08:00.005648+0000","last_unstale":"2024-03-26T05:10:54.634194+0000","last_undegraded":"2024-03-26T05:10:54.634194+0000","last_fullsized":"2024-03-26T05:10:54.634194+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:10:40.373470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.373470+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:08.010761+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":446,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634855+0000","last_change":"2024-03-26T05:09:45.438723+0000","last_active":"2024-03-26T05:10:54.634855+0000","last_peered":"2024-03-26T05:10:54.634855+0000","last_clean":"2024-03-26T05:10:54.634855+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:10:54.634855+0000","last_undegraded":"2024-03-26T05:10:54.634855+0000","last_fullsized":"2024-03-26T05:10:54.634855+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:45.438615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:45.438615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.866308+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"254'32","reported_seq":489,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634782+0000","last_change":"2024-03-26T05:09:30.638211+0000","last_active":"2024-03-26T05:10:54.634782+0000","last_peered":"2024-03-26T05:10:54.634782+0000","last_clean":"2024-03-26T05:10:54.634782+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:10:54.634782+0000","last_undegraded":"2024-03-26T05:10:54.634782+0000","last_fullsized":"2024-03-26T05:10:54.634782+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:09:30.635624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:30.635624+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.470924+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":441,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.630033+0000","last_change":"2024-03-26T05:10:40.548775+0000","last_active":"2024-03-26T05:10:54.630033+0000","last_peered":"2024-03-26T05:10:54.630033+0000","last_clean":"2024-03-26T05:10:54.630033+0000","last_became_active":"2024-03-26T05:07:59.942512+0000","last_became_peered":"2024-03-26T05:07:59.942512+0000","last_unstale":"2024-03-26T05:10:54.630033+0000","last_undegraded":"2024-03-26T05:10:54.630033+0000","last_fullsized":"2024-03-26T05:10:54.630033+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:40.548688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.548688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.537910+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"120'9","reported_seq":570,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635154+0000","last_change":"2024-03-26T05:10:30.593032+0000","last_active":"2024-03-26T05:10:54.635154+0000","last_peered":"2024-03-26T05:10:54.635154+0000","last_clean":"2024-03-26T05:10:54.635154+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:10:54.635154+0000","last_undegraded":"2024-03-26T05:10:54.635154+0000","last_fullsized":"2024-03-26T05:10:54.635154+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'9","last_scrub_stamp":"2024-03-26T05:10:30.592948+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.592948+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:45.127936+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":451,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.644618+0000","last_change":"2024-03-26T05:09:59.033943+0000","last_active":"2024-03-26T05:10:55.644618+0000","last_peered":"2024-03-26T05:10:55.644618+0000","last_clean":"2024-03-26T05:10:55.644618+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:10:55.644618+0000","last_undegraded":"2024-03-26T05:10:55.644618+0000","last_fullsized":"2024-03-26T05:10:55.644618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:59.033819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:59.033819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:04.791902+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"254'26","reported_seq":143,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634040+0000","last_change":"2024-03-26T05:10:33.467211+0000","last_active":"2024-03-26T05:10:54.634040+0000","last_peered":"2024-03-26T05:10:54.634040+0000","last_clean":"2024-03-26T05:10:54.634040+0000","last_became_active":"2024-03-26T05:07:59.960346+0000","last_became_peered":"2024-03-26T05:07:59.960346+0000","last_unstale":"2024-03-26T05:10:54.634040+0000","last_undegraded":"2024-03-26T05:10:54.634040+0000","last_fullsized":"2024-03-26T05:10:54.634040+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:10:33.467113+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.467113+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:40.892275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":444,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634689+0000","last_change":"2024-03-26T05:10:41.140060+0000","last_active":"2024-03-26T05:10:54.634689+0000","last_peered":"2024-03-26T05:10:54.634689+0000","last_clean":"2024-03-26T05:10:54.634689+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:10:54.634689+0000","last_undegraded":"2024-03-26T05:10:54.634689+0000","last_fullsized":"2024-03-26T05:10:54.634689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:41.139989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.139989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:04.392633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":423,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635246+0000","last_change":"2024-03-26T05:07:55.590013+0000","last_active":"2024-03-26T05:10:54.635246+0000","last_peered":"2024-03-26T05:10:54.635246+0000","last_clean":"2024-03-26T05:10:54.635246+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:10:54.635246+0000","last_undegraded":"2024-03-26T05:10:54.635246+0000","last_fullsized":"2024-03-26T05:10:54.635246+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:55.589886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:07:55.589886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.470768+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":438,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635579+0000","last_change":"2024-03-26T05:09:28.457669+0000","last_active":"2024-03-26T05:10:54.635579+0000","last_peered":"2024-03-26T05:10:54.635579+0000","last_clean":"2024-03-26T05:10:54.635579+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:10:54.635579+0000","last_undegraded":"2024-03-26T05:10:54.635579+0000","last_fullsized":"2024-03-26T05:10:54.635579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:28.457548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:28.457548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:55.339552+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"254'27","reported_seq":496,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.646073+0000","last_change":"2024-03-26T05:10:43.135431+0000","last_active":"2024-03-26T05:10:55.646073+0000","last_peered":"2024-03-26T05:10:55.646073+0000","last_clean":"2024-03-26T05:10:55.646073+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:10:55.646073+0000","last_undegraded":"2024-03-26T05:10:55.646073+0000","last_fullsized":"2024-03-26T05:10:55.646073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:10:43.135325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.135325+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:57.176336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":434,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635298+0000","last_change":"2024-03-26T05:10:35.847292+0000","last_active":"2024-03-26T05:10:54.635298+0000","last_peered":"2024-03-26T05:10:54.635298+0000","last_clean":"2024-03-26T05:10:54.635298+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:10:54.635298+0000","last_undegraded":"2024-03-26T05:10:54.635298+0000","last_fullsized":"2024-03-26T05:10:54.635298+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:35.847191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:35.847191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:41.957309+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633877+0000","last_change":"2024-03-26T05:09:52.050966+0000","last_active":"2024-03-26T05:10:54.633877+0000","last_peered":"2024-03-26T05:10:54.633877+0000","last_clean":"2024-03-26T05:10:54.633877+0000","last_became_active":"2024-03-26T05:07:59.997434+0000","last_became_peered":"2024-03-26T05:07:59.997434+0000","last_unstale":"2024-03-26T05:10:54.633877+0000","last_undegraded":"2024-03-26T05:10:54.633877+0000","last_fullsized":"2024-03-26T05:10:54.633877+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:52.050832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:52.050832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:18.207778+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":473,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.631643+0000","last_change":"2024-03-26T05:09:32.473925+0000","last_active":"2024-03-26T05:10:54.631643+0000","last_peered":"2024-03-26T05:10:54.631643+0000","last_clean":"2024-03-26T05:10:54.631643+0000","last_became_active":"2024-03-26T05:07:59.992217+0000","last_became_peered":"2024-03-26T05:07:59.992217+0000","last_unstale":"2024-03-26T05:10:54.631643+0000","last_undegraded":"2024-03-26T05:10:54.631643+0000","last_fullsized":"2024-03-26T05:10:54.631643+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:09:32.473785+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_clean_scrub_stamp":"2024-03-26T05:09:32.473785+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.115120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"254'18","reported_seq":465,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635390+0000","last_change":"2024-03-26T05:10:36.560789+0000","last_active":"2024-03-26T05:10:54.635390+0000","last_peered":"2024-03-26T05:10:54.635390+0000","last_clean":"2024-03-26T05:10:54.635390+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:10:54.635390+0000","last_undegraded":"2024-03-26T05:10:54.635390+0000","last_fullsized":"2024-03-26T05:10:54.635390+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'18","last_scrub_stamp":"2024-03-26T05:10:36.560689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:36.560689+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:04.290785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.650931+0000","last_change":"2024-03-26T05:10:55.650520+0000","last_active":"2024-03-26T05:10:55.650931+0000","last_peered":"2024-03-26T05:10:55.650931+0000","last_clean":"2024-03-26T05:10:55.650931+0000","last_became_active":"2024-03-26T05:10:55.650187+0000","last_became_peered":"2024-03-26T05:10:55.650187+0000","last_unstale":"2024-03-26T05:10:55.650931+0000","last_undegraded":"2024-03-26T05:10:55.650931+0000","last_fullsized":"2024-03-26T05:10:55.650931+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:03.814877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":118,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634109+0000","last_change":"2024-03-26T05:08:00.007340+0000","last_active":"2024-03-26T05:10:54.634109+0000","last_peered":"2024-03-26T05:10:54.634109+0000","last_clean":"2024-03-26T05:10:54.634109+0000","last_became_active":"2024-03-26T05:08:00.000166+0000","last_became_peered":"2024-03-26T05:08:00.000166+0000","last_unstale":"2024-03-26T05:10:54.634109+0000","last_undegraded":"2024-03-26T05:10:54.634109+0000","last_fullsized":"2024-03-26T05:10:54.634109+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:07:53.088449+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.088449+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":467,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635414+0000","last_change":"2024-03-26T05:09:48.462986+0000","last_active":"2024-03-26T05:10:54.635414+0000","last_peered":"2024-03-26T05:10:54.635414+0000","last_clean":"2024-03-26T05:10:54.635414+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:10:54.635414+0000","last_undegraded":"2024-03-26T05:10:54.635414+0000","last_fullsized":"2024-03-26T05:10:54.635414+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:09:48.462855+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:05:06.832744+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.462855+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.280826+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"254'48","reported_seq":169,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.632550+0000","last_change":"2024-03-26T05:09:59.156101+0000","last_active":"2024-03-26T05:10:54.632550+0000","last_peered":"2024-03-26T05:10:54.632550+0000","last_clean":"2024-03-26T05:10:54.632550+0000","last_became_active":"2024-03-26T05:07:59.947478+0000","last_became_peered":"2024-03-26T05:07:59.947478+0000","last_unstale":"2024-03-26T05:10:54.632550+0000","last_undegraded":"2024-03-26T05:10:54.632550+0000","last_fullsized":"2024-03-26T05:10:54.632550+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'36","last_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:09:59.155894+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.794154+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":573,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635748+0000","last_change":"2024-03-26T05:10:29.818124+0000","last_active":"2024-03-26T05:10:54.635748+0000","last_peered":"2024-03-26T05:10:54.635748+0000","last_clean":"2024-03-26T05:10:54.635748+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:10:54.635748+0000","last_undegraded":"2024-03-26T05:10:54.635748+0000","last_fullsized":"2024-03-26T05:10:54.635748+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:10:29.818015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.818015+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:33.669328+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.636170+0000","last_change":"2024-03-26T05:10:54.636170+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.636170+0000","last_undegraded":"2024-03-26T05:10:54.636170+0000","last_fullsized":"2024-03-26T05:10:54.636170+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":433,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633723+0000","last_change":"2024-03-26T05:10:33.604297+0000","last_active":"2024-03-26T05:10:54.633723+0000","last_peered":"2024-03-26T05:10:54.633723+0000","last_clean":"2024-03-26T05:10:54.633723+0000","last_became_active":"2024-03-26T05:08:55.556543+0000","last_became_peered":"2024-03-26T05:08:55.556543+0000","last_unstale":"2024-03-26T05:10:54.633723+0000","last_undegraded":"2024-03-26T05:10:54.633723+0000","last_fullsized":"2024-03-26T05:10:54.633723+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.604198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.604198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:36.748955+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":446,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635682+0000","last_change":"2024-03-26T05:10:38.869839+0000","last_active":"2024-03-26T05:10:54.635682+0000","last_peered":"2024-03-26T05:10:54.635682+0000","last_clean":"2024-03-26T05:10:54.635682+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:10:54.635682+0000","last_undegraded":"2024-03-26T05:10:54.635682+0000","last_fullsized":"2024-03-26T05:10:54.635682+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:38.869752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:38.869752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.797842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"254'37","reported_seq":500,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629490+0000","last_change":"2024-03-26T05:09:53.374250+0000","last_active":"2024-03-26T05:10:54.629490+0000","last_peered":"2024-03-26T05:10:54.629490+0000","last_clean":"2024-03-26T05:10:54.629490+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:10:54.629490+0000","last_undegraded":"2024-03-26T05:10:54.629490+0000","last_fullsized":"2024-03-26T05:10:54.629490+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'31","last_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_clean_scrub_stamp":"2024-03-26T05:09:53.374102+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:13.729213+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":440,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633466+0000","last_change":"2024-03-26T05:10:37.078616+0000","last_active":"2024-03-26T05:10:54.633466+0000","last_peered":"2024-03-26T05:10:54.633466+0000","last_clean":"2024-03-26T05:10:54.633466+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:10:54.633466+0000","last_undegraded":"2024-03-26T05:10:54.633466+0000","last_fullsized":"2024-03-26T05:10:54.633466+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:37.078539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.432732+0000","last_clean_scrub_stamp":"2024-03-26T05:10:37.078539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.617367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.631571+0000","last_change":"2024-03-26T05:10:54.631571+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.631571+0000","last_undegraded":"2024-03-26T05:10:54.631571+0000","last_fullsized":"2024-03-26T05:10:54.631571+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":434,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633245+0000","last_change":"2024-03-26T05:10:27.852306+0000","last_active":"2024-03-26T05:10:54.633245+0000","last_peered":"2024-03-26T05:10:54.633245+0000","last_clean":"2024-03-26T05:10:54.633245+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:10:54.633245+0000","last_undegraded":"2024-03-26T05:10:54.633245+0000","last_fullsized":"2024-03-26T05:10:54.633245+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:27.852200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_clean_scrub_stamp":"2024-03-26T05:10:27.852200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:39.771724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":452,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633773+0000","last_change":"2024-03-26T05:08:00.000387+0000","last_active":"2024-03-26T05:10:54.633773+0000","last_peered":"2024-03-26T05:10:54.633773+0000","last_clean":"2024-03-26T05:10:54.633773+0000","last_became_active":"2024-03-26T05:08:00.000009+0000","last_became_peered":"2024-03-26T05:08:00.000009+0000","last_unstale":"2024-03-26T05:10:54.633773+0000","last_undegraded":"2024-03-26T05:10:54.633773+0000","last_fullsized":"2024-03-26T05:10:54.633773+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:46.384777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:46.384777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.046272+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635848+0000","last_change":"2024-03-26T05:09:37.393265+0000","last_active":"2024-03-26T05:10:54.635848+0000","last_peered":"2024-03-26T05:10:54.635848+0000","last_clean":"2024-03-26T05:10:54.635848+0000","last_became_active":"2024-03-26T05:07:59.945688+0000","last_became_peered":"2024-03-26T05:07:59.945688+0000","last_unstale":"2024-03-26T05:10:54.635848+0000","last_undegraded":"2024-03-26T05:10:54.635848+0000","last_fullsized":"2024-03-26T05:10:54.635848+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:37.393159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:37.393159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:58.746449+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"254'41","reported_seq":509,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.631715+0000","last_change":"2024-03-26T05:09:34.481126+0000","last_active":"2024-03-26T05:10:54.631715+0000","last_peered":"2024-03-26T05:10:54.631715+0000","last_clean":"2024-03-26T05:10:54.631715+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:10:54.631715+0000","last_undegraded":"2024-03-26T05:10:54.631715+0000","last_fullsized":"2024-03-26T05:10:54.631715+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:09:34.480977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:34.480977+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.370575+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.631777+0000","last_change":"2024-03-26T05:10:54.631777+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.631777+0000","last_undegraded":"2024-03-26T05:10:54.631777+0000","last_fullsized":"2024-03-26T05:10:54.631777+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":431,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.645152+0000","last_change":"2024-03-26T05:09:37.959362+0000","last_active":"2024-03-26T05:10:55.645152+0000","last_peered":"2024-03-26T05:10:55.645152+0000","last_clean":"2024-03-26T05:10:55.645152+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:10:55.645152+0000","last_undegraded":"2024-03-26T05:10:55.645152+0000","last_fullsized":"2024-03-26T05:10:55.645152+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:37.959237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:37.959237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:01.339031+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":440,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633668+0000","last_change":"2024-03-26T05:09:58.378595+0000","last_active":"2024-03-26T05:10:54.633668+0000","last_peered":"2024-03-26T05:10:54.633668+0000","last_clean":"2024-03-26T05:10:54.633668+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:10:54.633668+0000","last_undegraded":"2024-03-26T05:10:54.633668+0000","last_fullsized":"2024-03-26T05:10:54.633668+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:58.378228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:58.378228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.565597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634750+0000","last_change":"2024-03-26T05:09:40.213194+0000","last_active":"2024-03-26T05:10:54.634750+0000","last_peered":"2024-03-26T05:10:54.634750+0000","last_clean":"2024-03-26T05:10:54.634750+0000","last_became_active":"2024-03-26T05:07:59.959978+0000","last_became_peered":"2024-03-26T05:07:59.959978+0000","last_unstale":"2024-03-26T05:10:54.634750+0000","last_undegraded":"2024-03-26T05:10:54.634750+0000","last_fullsized":"2024-03-26T05:10:54.634750+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:40.213116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:40.213116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"254'21","reported_seq":476,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.649896+0000","last_change":"2024-03-26T05:09:27.773009+0000","last_active":"2024-03-26T05:10:55.649896+0000","last_peered":"2024-03-26T05:10:55.649896+0000","last_clean":"2024-03-26T05:10:55.649896+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:10:55.649896+0000","last_undegraded":"2024-03-26T05:10:55.649896+0000","last_fullsized":"2024-03-26T05:10:55.649896+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:09:27.772633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:27.772633+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:56.062107+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.650840+0000","last_change":"2024-03-26T05:10:55.650442+0000","last_active":"2024-03-26T05:10:55.650840+0000","last_peered":"2024-03-26T05:10:55.650840+0000","last_clean":"2024-03-26T05:10:55.650840+0000","last_became_active":"2024-03-26T05:10:55.650103+0000","last_became_peered":"2024-03-26T05:10:55.650103+0000","last_unstale":"2024-03-26T05:10:55.650840+0000","last_undegraded":"2024-03-26T05:10:55.650840+0000","last_fullsized":"2024-03-26T05:10:55.650840+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:23.696755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":419,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633853+0000","last_change":"2024-03-26T05:07:48.635847+0000","last_active":"2024-03-26T05:10:54.633853+0000","last_peered":"2024-03-26T05:10:54.633853+0000","last_clean":"2024-03-26T05:10:54.633853+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:10:54.633853+0000","last_undegraded":"2024-03-26T05:10:54.633853+0000","last_fullsized":"2024-03-26T05:10:54.633853+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:48.635728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:48.635728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.470817+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":118,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634906+0000","last_change":"2024-03-26T05:08:00.004448+0000","last_active":"2024-03-26T05:10:54.634906+0000","last_peered":"2024-03-26T05:10:54.634906+0000","last_clean":"2024-03-26T05:10:54.634906+0000","last_became_active":"2024-03-26T05:07:59.998511+0000","last_became_peered":"2024-03-26T05:07:59.998511+0000","last_unstale":"2024-03-26T05:10:54.634906+0000","last_undegraded":"2024-03-26T05:10:54.634906+0000","last_fullsized":"2024-03-26T05:10:54.634906+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:07:37.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:07:37.347513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":434,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629337+0000","last_change":"2024-03-26T05:09:31.499628+0000","last_active":"2024-03-26T05:10:54.629337+0000","last_peered":"2024-03-26T05:10:54.629337+0000","last_clean":"2024-03-26T05:10:54.629337+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:10:54.629337+0000","last_undegraded":"2024-03-26T05:10:54.629337+0000","last_fullsized":"2024-03-26T05:10:54.629337+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:31.499513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:31.499513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"254'46","reported_seq":650,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.636736+0000","last_change":"2024-03-26T05:10:48.344286+0000","last_active":"2024-03-26T05:10:54.636736+0000","last_peered":"2024-03-26T05:10:54.636736+0000","last_clean":"2024-03-26T05:10:54.636736+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:10:54.636736+0000","last_undegraded":"2024-03-26T05:10:54.636736+0000","last_fullsized":"2024-03-26T05:10:54.636736+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'46","last_scrub_stamp":"2024-03-26T05:10:48.344180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:48.344180+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:57.157542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":435,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.648649+0000","last_change":"2024-03-26T05:09:41.858043+0000","last_active":"2024-03-26T05:10:55.648649+0000","last_peered":"2024-03-26T05:10:55.648649+0000","last_clean":"2024-03-26T05:10:55.648649+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:10:55.648649+0000","last_undegraded":"2024-03-26T05:10:55.648649+0000","last_fullsized":"2024-03-26T05:10:55.648649+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:41.857888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:41.857888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.702593+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.631429+0000","last_change":"2024-03-26T05:10:54.631429+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.631429+0000","last_undegraded":"2024-03-26T05:10:54.631429+0000","last_fullsized":"2024-03-26T05:10:54.631429+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":435,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635989+0000","last_change":"2024-03-26T05:09:36.428895+0000","last_active":"2024-03-26T05:10:54.635989+0000","last_peered":"2024-03-26T05:10:54.635989+0000","last_clean":"2024-03-26T05:10:54.635989+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:10:54.635989+0000","last_undegraded":"2024-03-26T05:10:54.635989+0000","last_fullsized":"2024-03-26T05:10:54.635989+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:36.428762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:36.428762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.905179+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":441,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.637044+0000","last_change":"2024-03-26T05:10:39.454264+0000","last_active":"2024-03-26T05:10:54.637044+0000","last_peered":"2024-03-26T05:10:54.637044+0000","last_clean":"2024-03-26T05:10:54.637044+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:10:54.637044+0000","last_undegraded":"2024-03-26T05:10:54.637044+0000","last_fullsized":"2024-03-26T05:10:54.637044+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:39.454176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:39.454176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:41.956147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":458,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633968+0000","last_change":"2024-03-26T05:09:55.551716+0000","last_active":"2024-03-26T05:10:54.633968+0000","last_peered":"2024-03-26T05:10:54.633968+0000","last_clean":"2024-03-26T05:10:54.633968+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:10:54.633968+0000","last_undegraded":"2024-03-26T05:10:54.633968+0000","last_fullsized":"2024-03-26T05:10:54.633968+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:55.551595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.551595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.543264+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"254'23","reported_seq":255,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635165+0000","last_change":"2024-03-26T05:08:00.019227+0000","last_active":"2024-03-26T05:10:54.635165+0000","last_peered":"2024-03-26T05:10:54.635165+0000","last_clean":"2024-03-26T05:10:54.635165+0000","last_became_active":"2024-03-26T05:08:00.001398+0000","last_became_peered":"2024-03-26T05:08:00.001398+0000","last_unstale":"2024-03-26T05:10:54.635165+0000","last_undegraded":"2024-03-26T05:10:54.635165+0000","last_fullsized":"2024-03-26T05:10:54.635165+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-26T05:07:54.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:54.529843+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.046242+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":460,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633812+0000","last_change":"2024-03-26T05:10:43.076916+0000","last_active":"2024-03-26T05:10:54.633812+0000","last_peered":"2024-03-26T05:10:54.633812+0000","last_clean":"2024-03-26T05:10:54.633812+0000","last_became_active":"2024-03-26T05:07:59.955456+0000","last_became_peered":"2024-03-26T05:07:59.955456+0000","last_unstale":"2024-03-26T05:10:54.633812+0000","last_undegraded":"2024-03-26T05:10:54.633812+0000","last_fullsized":"2024-03-26T05:10:54.633812+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:43.076802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.076802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:54.501193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.636467+0000","last_change":"2024-03-26T05:10:54.636467+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.636467+0000","last_undegraded":"2024-03-26T05:10:54.636467+0000","last_fullsized":"2024-03-26T05:10:54.636467+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634288+0000","last_change":"2024-03-26T05:09:46.172999+0000","last_active":"2024-03-26T05:10:54.634288+0000","last_peered":"2024-03-26T05:10:54.634288+0000","last_clean":"2024-03-26T05:10:54.634288+0000","last_became_active":"2024-03-26T05:07:59.959853+0000","last_became_peered":"2024-03-26T05:07:59.959853+0000","last_unstale":"2024-03-26T05:10:54.634288+0000","last_undegraded":"2024-03-26T05:10:54.634288+0000","last_fullsized":"2024-03-26T05:10:54.634288+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:46.172868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_clean_scrub_stamp":"2024-03-26T05:09:46.172868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452798+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":702,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633349+0000","last_change":"2024-03-26T05:10:39.866608+0000","last_active":"2024-03-26T05:10:54.633349+0000","last_peered":"2024-03-26T05:10:54.633349+0000","last_clean":"2024-03-26T05:10:54.633349+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:10:54.633349+0000","last_undegraded":"2024-03-26T05:10:54.633349+0000","last_fullsized":"2024-03-26T05:10:54.633349+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:10:39.866516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:10:39.866516+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.907367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":452,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634680+0000","last_change":"2024-03-26T05:09:39.605190+0000","last_active":"2024-03-26T05:10:54.634680+0000","last_peered":"2024-03-26T05:10:54.634680+0000","last_clean":"2024-03-26T05:10:54.634680+0000","last_became_active":"2024-03-26T05:07:59.960061+0000","last_became_peered":"2024-03-26T05:07:59.960061+0000","last_unstale":"2024-03-26T05:10:54.634680+0000","last_undegraded":"2024-03-26T05:10:54.634680+0000","last_fullsized":"2024-03-26T05:10:54.634680+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:39.605053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:39.605053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.821576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"254'36","reported_seq":626,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633412+0000","last_change":"2024-03-26T05:09:41.506253+0000","last_active":"2024-03-26T05:10:54.633412+0000","last_peered":"2024-03-26T05:10:54.633412+0000","last_clean":"2024-03-26T05:10:54.633412+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:10:54.633412+0000","last_undegraded":"2024-03-26T05:10:54.633412+0000","last_fullsized":"2024-03-26T05:10:54.633412+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'24","last_scrub_stamp":"2024-03-26T05:09:41.506123+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:09:41.506123+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.758224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":459,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634755+0000","last_change":"2024-03-26T05:10:36.850667+0000","last_active":"2024-03-26T05:10:54.634755+0000","last_peered":"2024-03-26T05:10:54.634755+0000","last_clean":"2024-03-26T05:10:54.634755+0000","last_became_active":"2024-03-26T05:07:59.956080+0000","last_became_peered":"2024-03-26T05:07:59.956080+0000","last_unstale":"2024-03-26T05:10:54.634755+0000","last_undegraded":"2024-03-26T05:10:54.634755+0000","last_fullsized":"2024-03-26T05:10:54.634755+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:36.850583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:36.850583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:53.029005+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.755480+0000","last_change":"2024-03-26T05:10:55.754958+0000","last_active":"2024-03-26T05:10:55.755480+0000","last_peered":"2024-03-26T05:10:55.755480+0000","last_clean":"2024-03-26T05:10:55.755480+0000","last_became_active":"2024-03-26T05:10:55.754590+0000","last_became_peered":"2024-03-26T05:10:55.754590+0000","last_unstale":"2024-03-26T05:10:55.755480+0000","last_undegraded":"2024-03-26T05:10:55.755480+0000","last_fullsized":"2024-03-26T05:10:55.755480+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:05.893327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":441,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.636481+0000","last_change":"2024-03-26T05:08:00.002985+0000","last_active":"2024-03-26T05:10:54.636481+0000","last_peered":"2024-03-26T05:10:54.636481+0000","last_clean":"2024-03-26T05:10:54.636481+0000","last_became_active":"2024-03-26T05:08:00.001657+0000","last_became_peered":"2024-03-26T05:08:00.001657+0000","last_unstale":"2024-03-26T05:10:54.636481+0000","last_undegraded":"2024-03-26T05:10:54.636481+0000","last_fullsized":"2024-03-26T05:10:54.636481+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:39.338092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:39.338092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:56.415750+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":337,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629559+0000","last_change":"2024-03-26T05:09:48.473285+0000","last_active":"2024-03-26T05:10:54.629559+0000","last_peered":"2024-03-26T05:10:54.629559+0000","last_clean":"2024-03-26T05:10:54.629559+0000","last_became_active":"2024-03-26T05:07:59.992489+0000","last_became_peered":"2024-03-26T05:07:59.992489+0000","last_unstale":"2024-03-26T05:10:54.629559+0000","last_undegraded":"2024-03-26T05:10:54.629559+0000","last_fullsized":"2024-03-26T05:10:54.629559+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.473160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:55.196315+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":1085,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633322+0000","last_change":"2024-03-26T05:09:55.485007+0000","last_active":"2024-03-26T05:10:54.633322+0000","last_peered":"2024-03-26T05:10:54.633322+0000","last_clean":"2024-03-26T05:10:54.633322+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:10:54.633322+0000","last_undegraded":"2024-03-26T05:10:54.633322+0000","last_fullsized":"2024-03-26T05:10:54.633322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:09:55.484790+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.484790+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:14.311528+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":625,"num_read_kb":620,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"254'30","reported_seq":521,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.644804+0000","last_change":"2024-03-26T05:09:49.909494+0000","last_active":"2024-03-26T05:10:55.644804+0000","last_peered":"2024-03-26T05:10:55.644804+0000","last_clean":"2024-03-26T05:10:55.644804+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:10:55.644804+0000","last_undegraded":"2024-03-26T05:10:55.644804+0000","last_fullsized":"2024-03-26T05:10:55.644804+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'18","last_scrub_stamp":"2024-03-26T05:09:49.909347+0000","last_deep_scrub":"187'18","last_deep_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_clean_scrub_stamp":"2024-03-26T05:09:49.909347+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:16.935034+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634266+0000","last_change":"2024-03-26T05:10:44.404350+0000","last_active":"2024-03-26T05:10:54.634266+0000","last_peered":"2024-03-26T05:10:54.634266+0000","last_clean":"2024-03-26T05:10:54.634266+0000","last_became_active":"2024-03-26T05:07:59.999990+0000","last_became_peered":"2024-03-26T05:07:59.999990+0000","last_unstale":"2024-03-26T05:10:54.634266+0000","last_undegraded":"2024-03-26T05:10:54.634266+0000","last_fullsized":"2024-03-26T05:10:54.634266+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:44.404261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:44.404261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:56.403017+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.632660+0000","last_change":"2024-03-26T05:10:54.632660+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.632660+0000","last_undegraded":"2024-03-26T05:10:54.632660+0000","last_fullsized":"2024-03-26T05:10:54.632660+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":429,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634390+0000","last_change":"2024-03-26T05:10:34.408142+0000","last_active":"2024-03-26T05:10:54.634390+0000","last_peered":"2024-03-26T05:10:54.634390+0000","last_clean":"2024-03-26T05:10:54.634390+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:10:54.634390+0000","last_undegraded":"2024-03-26T05:10:54.634390+0000","last_fullsized":"2024-03-26T05:10:54.634390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:34.408045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:10:34.408045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:39.555764+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":453,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.636651+0000","last_change":"2024-03-26T05:09:54.538181+0000","last_active":"2024-03-26T05:10:54.636651+0000","last_peered":"2024-03-26T05:10:54.636651+0000","last_clean":"2024-03-26T05:10:54.636651+0000","last_became_active":"2024-03-26T05:08:00.003099+0000","last_became_peered":"2024-03-26T05:08:00.003099+0000","last_unstale":"2024-03-26T05:10:54.636651+0000","last_undegraded":"2024-03-26T05:10:54.636651+0000","last_fullsized":"2024-03-26T05:10:54.636651+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:54.538056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:09:54.538056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:06.436896+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"254'44","reported_seq":572,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.649069+0000","last_change":"2024-03-26T05:09:53.978359+0000","last_active":"2024-03-26T05:10:55.649069+0000","last_peered":"2024-03-26T05:10:55.649069+0000","last_clean":"2024-03-26T05:10:55.649069+0000","last_became_active":"2024-03-26T05:07:59.994226+0000","last_became_peered":"2024-03-26T05:07:59.994226+0000","last_unstale":"2024-03-26T05:10:55.649069+0000","last_undegraded":"2024-03-26T05:10:55.649069+0000","last_fullsized":"2024-03-26T05:10:55.649069+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'34","last_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:09:53.978203+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:00.272840+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":459,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.631843+0000","last_change":"2024-03-26T05:10:42.594397+0000","last_active":"2024-03-26T05:10:54.631843+0000","last_peered":"2024-03-26T05:10:54.631843+0000","last_clean":"2024-03-26T05:10:54.631843+0000","last_became_active":"2024-03-26T05:07:59.991484+0000","last_became_peered":"2024-03-26T05:07:59.991484+0000","last_unstale":"2024-03-26T05:10:54.631843+0000","last_undegraded":"2024-03-26T05:10:54.631843+0000","last_fullsized":"2024-03-26T05:10:54.631843+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:42.594296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:10:42.594296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:54.067648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.632133+0000","last_change":"2024-03-26T05:10:54.632133+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.632133+0000","last_undegraded":"2024-03-26T05:10:54.632133+0000","last_fullsized":"2024-03-26T05:10:54.632133+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":432,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.645236+0000","last_change":"2024-03-26T05:09:28.793178+0000","last_active":"2024-03-26T05:10:55.645236+0000","last_peered":"2024-03-26T05:10:55.645236+0000","last_clean":"2024-03-26T05:10:55.645236+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:10:55.645236+0000","last_undegraded":"2024-03-26T05:10:55.645236+0000","last_fullsized":"2024-03-26T05:10:55.645236+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:28.792827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:28.792827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:56.062066+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634078+0000","last_change":"2024-03-26T05:09:42.163715+0000","last_active":"2024-03-26T05:10:54.634078+0000","last_peered":"2024-03-26T05:10:54.634078+0000","last_clean":"2024-03-26T05:10:54.634078+0000","last_became_active":"2024-03-26T05:07:59.960768+0000","last_became_peered":"2024-03-26T05:07:59.960768+0000","last_unstale":"2024-03-26T05:10:54.634078+0000","last_undegraded":"2024-03-26T05:10:54.634078+0000","last_fullsized":"2024-03-26T05:10:54.634078+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:42.163609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:42.163609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:04.022763+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"254'26","reported_seq":489,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634202+0000","last_change":"2024-03-26T05:09:47.483160+0000","last_active":"2024-03-26T05:10:54.634202+0000","last_peered":"2024-03-26T05:10:54.634202+0000","last_clean":"2024-03-26T05:10:54.634202+0000","last_became_active":"2024-03-26T05:07:59.960351+0000","last_became_peered":"2024-03-26T05:07:59.960351+0000","last_unstale":"2024-03-26T05:10:54.634202+0000","last_undegraded":"2024-03-26T05:10:54.634202+0000","last_fullsized":"2024-03-26T05:10:54.634202+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:09:47.483012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:47.483012+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.652907+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"216'11","reported_seq":601,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.650057+0000","last_change":"2024-03-26T05:10:34.204907+0000","last_active":"2024-03-26T05:10:55.650057+0000","last_peered":"2024-03-26T05:10:55.650057+0000","last_clean":"2024-03-26T05:10:55.650057+0000","last_became_active":"2024-03-26T05:07:59.994682+0000","last_became_peered":"2024-03-26T05:07:59.994682+0000","last_unstale":"2024-03-26T05:10:55.650057+0000","last_undegraded":"2024-03-26T05:10:55.650057+0000","last_fullsized":"2024-03-26T05:10:55.650057+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:10:34.204798+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:10:34.204798+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.783859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.634480+0000","last_change":"2024-03-26T05:10:54.634480+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.634480+0000","last_undegraded":"2024-03-26T05:10:54.634480+0000","last_fullsized":"2024-03-26T05:10:54.634480+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":435,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.635228+0000","last_change":"2024-03-26T05:10:34.839543+0000","last_active":"2024-03-26T05:10:54.635228+0000","last_peered":"2024-03-26T05:10:54.635228+0000","last_clean":"2024-03-26T05:10:54.635228+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:10:54.635228+0000","last_undegraded":"2024-03-26T05:10:54.635228+0000","last_fullsized":"2024-03-26T05:10:54.635228+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:34.837732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:34.837732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.929082+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":452,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-26T05:10:55.645769+0000","last_change":"2024-03-26T05:09:47.958661+0000","last_active":"2024-03-26T05:10:55.645769+0000","last_peered":"2024-03-26T05:10:55.645769+0000","last_clean":"2024-03-26T05:10:55.645769+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:10:55.645769+0000","last_undegraded":"2024-03-26T05:10:55.645769+0000","last_fullsized":"2024-03-26T05:10:55.645769+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:09:47.958370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:47.958370+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:58.030382+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"254'34","reported_seq":540,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629370+0000","last_change":"2024-03-26T05:09:47.484748+0000","last_active":"2024-03-26T05:10:54.629370+0000","last_peered":"2024-03-26T05:10:54.629370+0000","last_clean":"2024-03-26T05:10:54.629370+0000","last_became_active":"2024-03-26T05:07:59.942510+0000","last_became_peered":"2024-03-26T05:07:59.942510+0000","last_unstale":"2024-03-26T05:10:54.629370+0000","last_undegraded":"2024-03-26T05:10:54.629370+0000","last_fullsized":"2024-03-26T05:10:54.629370+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:09:47.484585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:47.484585+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:13.731149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":559,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.637268+0000","last_change":"2024-03-26T05:07:53.583159+0000","last_active":"2024-03-26T05:10:54.637268+0000","last_peered":"2024-03-26T05:10:54.637268+0000","last_clean":"2024-03-26T05:10:54.637268+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:10:54.637268+0000","last_undegraded":"2024-03-26T05:10:54.637268+0000","last_fullsized":"2024-03-26T05:10:54.637268+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:07:53.583016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.583016+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.470802+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.631731+0000","last_change":"2024-03-26T05:10:54.631731+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.631731+0000","last_undegraded":"2024-03-26T05:10:54.631731+0000","last_fullsized":"2024-03-26T05:10:54.631731+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":430,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629407+0000","last_change":"2024-03-26T05:07:50.818124+0000","last_active":"2024-03-26T05:10:54.629407+0000","last_peered":"2024-03-26T05:10:54.629407+0000","last_clean":"2024-03-26T05:10:54.629407+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:10:54.629407+0000","last_undegraded":"2024-03-26T05:10:54.629407+0000","last_fullsized":"2024-03-26T05:10:54.629407+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:50.818000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:50.818000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":431,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634742+0000","last_change":"2024-03-26T05:09:38.690272+0000","last_active":"2024-03-26T05:10:54.634742+0000","last_peered":"2024-03-26T05:10:54.634742+0000","last_clean":"2024-03-26T05:10:54.634742+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:10:54.634742+0000","last_undegraded":"2024-03-26T05:10:54.634742+0000","last_fullsized":"2024-03-26T05:10:54.634742+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:38.690131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:38.690131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:58.560653+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"254'52","reported_seq":520,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.637163+0000","last_change":"2024-03-26T05:09:34.644416+0000","last_active":"2024-03-26T05:10:54.637163+0000","last_peered":"2024-03-26T05:10:54.637163+0000","last_clean":"2024-03-26T05:10:54.637163+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:10:54.637163+0000","last_undegraded":"2024-03-26T05:10:54.637163+0000","last_fullsized":"2024-03-26T05:10:54.637163+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:09:34.643076+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:09:34.643076+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.470907+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":434,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629213+0000","last_change":"2024-03-26T05:10:33.411222+0000","last_active":"2024-03-26T05:10:54.629213+0000","last_peered":"2024-03-26T05:10:54.629213+0000","last_clean":"2024-03-26T05:10:54.629213+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:10:54.629213+0000","last_undegraded":"2024-03-26T05:10:54.629213+0000","last_fullsized":"2024-03-26T05:10:54.629213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.411139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.411139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:54.372120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.634379+0000","last_change":"2024-03-26T05:10:54.634379+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.634379+0000","last_undegraded":"2024-03-26T05:10:54.634379+0000","last_fullsized":"2024-03-26T05:10:54.634379+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":114,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633844+0000","last_change":"2024-03-26T05:08:00.006006+0000","last_active":"2024-03-26T05:10:54.633844+0000","last_peered":"2024-03-26T05:10:54.633844+0000","last_clean":"2024-03-26T05:10:54.633844+0000","last_became_active":"2024-03-26T05:08:00.005676+0000","last_became_peered":"2024-03-26T05:08:00.005676+0000","last_unstale":"2024-03-26T05:10:54.633844+0000","last_undegraded":"2024-03-26T05:10:54.633844+0000","last_fullsized":"2024-03-26T05:10:54.633844+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_clean_scrub_stamp":"2024-03-26T05:07:52.465619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"216'10","reported_seq":589,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.630185+0000","last_change":"2024-03-26T05:09:36.472026+0000","last_active":"2024-03-26T05:10:54.630185+0000","last_peered":"2024-03-26T05:10:54.630185+0000","last_clean":"2024-03-26T05:10:54.630185+0000","last_became_active":"2024-03-26T05:07:59.997429+0000","last_became_peered":"2024-03-26T05:07:59.997429+0000","last_unstale":"2024-03-26T05:10:54.630185+0000","last_undegraded":"2024-03-26T05:10:54.630185+0000","last_fullsized":"2024-03-26T05:10:54.630185+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:09:36.471890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:36.471890+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":439,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.637068+0000","last_change":"2024-03-26T05:10:40.541468+0000","last_active":"2024-03-26T05:10:54.637068+0000","last_peered":"2024-03-26T05:10:54.637068+0000","last_clean":"2024-03-26T05:10:54.637068+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:10:54.637068+0000","last_undegraded":"2024-03-26T05:10:54.637068+0000","last_fullsized":"2024-03-26T05:10:54.637068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:40.541340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.541340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.162836+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"254'27","reported_seq":487,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.637639+0000","last_change":"2024-03-26T05:10:43.423183+0000","last_active":"2024-03-26T05:10:54.637639+0000","last_peered":"2024-03-26T05:10:54.637639+0000","last_clean":"2024-03-26T05:10:54.637639+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:10:54.637639+0000","last_undegraded":"2024-03-26T05:10:54.637639+0000","last_fullsized":"2024-03-26T05:10:54.637639+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:10:43.423040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.423040+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:50.942552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.632832+0000","last_change":"2024-03-26T05:10:54.632832+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.632832+0000","last_undegraded":"2024-03-26T05:10:54.632832+0000","last_fullsized":"2024-03-26T05:10:54.632832+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":439,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634694+0000","last_change":"2024-03-26T05:10:44.608828+0000","last_active":"2024-03-26T05:10:54.634694+0000","last_peered":"2024-03-26T05:10:54.634694+0000","last_clean":"2024-03-26T05:10:54.634694+0000","last_became_active":"2024-03-26T05:07:59.957995+0000","last_became_peered":"2024-03-26T05:07:59.957995+0000","last_unstale":"2024-03-26T05:10:54.634694+0000","last_undegraded":"2024-03-26T05:10:54.634694+0000","last_fullsized":"2024-03-26T05:10:54.634694+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:44.608717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:44.608717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.528157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634118+0000","last_change":"2024-03-26T05:10:48.413966+0000","last_active":"2024-03-26T05:10:54.634118+0000","last_peered":"2024-03-26T05:10:54.634118+0000","last_clean":"2024-03-26T05:10:54.634118+0000","last_became_active":"2024-03-26T05:07:59.947747+0000","last_became_peered":"2024-03-26T05:07:59.947747+0000","last_unstale":"2024-03-26T05:10:54.634118+0000","last_undegraded":"2024-03-26T05:10:54.634118+0000","last_fullsized":"2024-03-26T05:10:54.634118+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:48.413868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.446060+0000","last_clean_scrub_stamp":"2024-03-26T05:10:48.413868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:13.855843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":441,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.629150+0000","last_change":"2024-03-26T05:07:49.831449+0000","last_active":"2024-03-26T05:10:54.629150+0000","last_peered":"2024-03-26T05:10:54.629150+0000","last_clean":"2024-03-26T05:10:54.629150+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:10:54.629150+0000","last_undegraded":"2024-03-26T05:10:54.629150+0000","last_fullsized":"2024-03-26T05:10:54.629150+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:49.831323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:49.831323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"254'25","reported_seq":488,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.630054+0000","last_change":"2024-03-26T05:10:48.381843+0000","last_active":"2024-03-26T05:10:54.630054+0000","last_peered":"2024-03-26T05:10:54.630054+0000","last_clean":"2024-03-26T05:10:54.630054+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:10:54.630054+0000","last_undegraded":"2024-03-26T05:10:54.630054+0000","last_fullsized":"2024-03-26T05:10:54.630054+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:10:48.381743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:48.381743+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:49.809560+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.636169+0000","last_change":"2024-03-26T05:10:54.636169+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.636169+0000","last_undegraded":"2024-03-26T05:10:54.636169+0000","last_fullsized":"2024-03-26T05:10:54.636169+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":428,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.634972+0000","last_change":"2024-03-26T05:07:38.496117+0000","last_active":"2024-03-26T05:10:54.634972+0000","last_peered":"2024-03-26T05:10:54.634972+0000","last_clean":"2024-03-26T05:10:54.634972+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:10:54.634972+0000","last_undegraded":"2024-03-26T05:10:54.634972+0000","last_fullsized":"2024-03-26T05:10:54.634972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:38.495883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:38.495883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.046205+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":432,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.636889+0000","last_change":"2024-03-26T05:09:56.509351+0000","last_active":"2024-03-26T05:10:54.636889+0000","last_peered":"2024-03-26T05:10:54.636889+0000","last_clean":"2024-03-26T05:10:54.636889+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:10:54.636889+0000","last_undegraded":"2024-03-26T05:10:54.636889+0000","last_fullsized":"2024-03-26T05:10:54.636889+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:09:56.509231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:58.290248+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":460,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.631553+0000","last_change":"2024-03-26T05:09:45.550058+0000","last_active":"2024-03-26T05:10:54.631553+0000","last_peered":"2024-03-26T05:10:54.631553+0000","last_clean":"2024-03-26T05:10:54.631553+0000","last_became_active":"2024-03-26T05:06:06.521150+0000","last_became_peered":"2024-03-26T05:06:06.521150+0000","last_unstale":"2024-03-26T05:10:54.631553+0000","last_undegraded":"2024-03-26T05:10:54.631553+0000","last_fullsized":"2024-03-26T05:10:54.631553+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:09:45.549936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:04.471132+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"254'31","reported_seq":486,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.630504+0000","last_change":"2024-03-26T05:09:58.433286+0000","last_active":"2024-03-26T05:10:54.630504+0000","last_peered":"2024-03-26T05:10:54.630504+0000","last_clean":"2024-03-26T05:10:54.630504+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:10:54.630504+0000","last_undegraded":"2024-03-26T05:10:54.630504+0000","last_fullsized":"2024-03-26T05:10:54.630504+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:09:58.433038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:58.433038+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.060742+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":265,"state":"creating+peering","last_fresh":"2024-03-26T05:10:54.634088+0000","last_change":"2024-03-26T05:10:54.634088+0000","last_active":"2024-03-26T05:10:54.621176+0000","last_peered":"2024-03-26T05:10:54.621176+0000","last_clean":"2024-03-26T05:10:54.621176+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T05:10:54.634088+0000","last_undegraded":"2024-03-26T05:10:54.634088+0000","last_fullsized":"2024-03-26T05:10:54.634088+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":429,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.630305+0000","last_change":"2024-03-26T05:09:38.506191+0000","last_active":"2024-03-26T05:10:54.630305+0000","last_peered":"2024-03-26T05:10:54.630305+0000","last_clean":"2024-03-26T05:10:54.630305+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:10:54.630305+0000","last_undegraded":"2024-03-26T05:10:54.630305+0000","last_fullsized":"2024-03-26T05:10:54.630305+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:38.506070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:38.506070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"216'11","reported_seq":597,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633879+0000","last_change":"2024-03-26T05:10:30.100941+0000","last_active":"2024-03-26T05:10:54.633879+0000","last_peered":"2024-03-26T05:10:54.633879+0000","last_clean":"2024-03-26T05:10:54.633879+0000","last_became_active":"2024-03-26T05:07:59.960519+0000","last_became_peered":"2024-03-26T05:07:59.960519+0000","last_unstale":"2024-03-26T05:10:54.633879+0000","last_undegraded":"2024-03-26T05:10:54.633879+0000","last_fullsized":"2024-03-26T05:10:54.633879+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:10:30.100859+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.100859+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:56.657116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":445,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.633467+0000","last_change":"2024-03-26T05:09:55.525744+0000","last_active":"2024-03-26T05:10:54.633467+0000","last_peered":"2024-03-26T05:10:54.633467+0000","last_clean":"2024-03-26T05:10:54.633467+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:10:54.633467+0000","last_undegraded":"2024-03-26T05:10:54.633467+0000","last_fullsized":"2024-03-26T05:10:54.633467+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:55.525633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.525633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:57.130990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"254'20","reported_seq":467,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-26T05:10:54.630040+0000","last_change":"2024-03-26T05:09:49.460933+0000","last_active":"2024-03-26T05:10:54.630040+0000","last_peered":"2024-03-26T05:10:54.630040+0000","last_clean":"2024-03-26T05:10:54.630040+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:10:54.630040+0000","last_undegraded":"2024-03-26T05:10:54.630040+0000","last_fullsized":"2024-03-26T05:10:54.630040+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:09:49.460784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:49.460784+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:18.489025+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":5},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":625,"num_read_kb":620,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1413,"num_read_kb":1138,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":1870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":934,"ondisk_log_size":934,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005187,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":199404,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93545748,"statfs":{"total":95995035648,"available":95790845952,"internally_reserved":0,"allocated":1671168,"data_stored":1240782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":202504650},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332232,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":293816,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93451336,"statfs":{"total":95995035648,"available":95694168064,"internally_reserved":0,"allocated":1679360,"data_stored":1255770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303004,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93442148,"statfs":{"total":95995035648,"available":95684759552,"internally_reserved":0,"allocated":1126400,"data_stored":666336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738535,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":468240,"kb_used_data":126084,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93276912,"statfs":{"total":95995035648,"available":95515557888,"internally_reserved":0,"allocated":129110016,"data_stored":128659423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411485,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298808,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93446344,"statfs":{"total":95995035648,"available":95689056256,"internally_reserved":0,"allocated":1089536,"data_stored":665546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117143,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":361296,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93383856,"statfs":{"total":95995035648,"available":95625068544,"internally_reserved":0,"allocated":1114112,"data_stored":665882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368835015},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920915,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":271036,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":93474116,"statfs":{"total":95995035648,"available":95717494784,"internally_reserved":0,"allocated":1097728,"data_stored":657882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659277,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415400,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93329752,"statfs":{"total":95995035648,"available":95569666048,"internally_reserved":0,"allocated":1077248,"data_stored":657139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:10:57.665 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:10:57.672 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:57.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:57.774 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:57.799 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:57.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:57.899 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:57.927 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:57.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:57.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:57.986 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:57 smithi080 ceph-mon[40404]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:10:57.986 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:57 smithi080 ceph-mon[29691]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:10:58.029 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:58.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:57 smithi132 ceph-mon[33500]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:10:58.052 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:58.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:10:58.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:58.181 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:58.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:58.283 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:58.310 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:58.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:58.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:58.412 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:58.436 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:58.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:58.445 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:58.479 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:58.538 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:58.561 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:58.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:58.652 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:10:58.664 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:58.719 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:58.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:58.820 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:58.848 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:58.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:58.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:58.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:58 smithi080 ceph-mon[29691]: pgmap v479: 148 pgs: 13 creating+peering, 135 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:10:58.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:58 smithi080 ceph-mon[40404]: pgmap v479: 148 pgs: 13 creating+peering, 135 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:10:58.941 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:10:58.950 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:58.978 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:58.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:59.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:58 smithi132 ceph-mon[33500]: pgmap v479: 148 pgs: 13 creating+peering, 135 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:10:59.079 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:59.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:59.104 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:59.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:59.205 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:10:59.234 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:59.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:10:59.336 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:10:59.359 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:59.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:59.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:10:59.461 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:10:59.484 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:59.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:59.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:10:59.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:10:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:59.610 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:59.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:10:59.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:10:59.738 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:59.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:59.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:10:59.840 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:10:59.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:59 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:10:59.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:59 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:10:59.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:10:59 smithi080 ceph-mon[40404]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T05:10:59.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:59 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:10:59.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:59 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:10:59.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:10:59 smithi080 ceph-mon[29691]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T05:10:59.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:59.868 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:10:59.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:10:59.969 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:10:59.994 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:10:59.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:10:59.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:00.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:59 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:11:00.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:59 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:11:00.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:10:59 smithi132 ceph-mon[33500]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T05:11:00.096 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:00.124 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:00.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:00.214 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:11:00.214 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:11:00.225 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:00.253 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:00.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:00.355 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:00.379 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:00.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:00.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:00.481 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:00.504 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:00.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:00.606 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:00.633 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:00.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:00.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:00.735 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:00.736 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:00 smithi080 ceph-mon[29691]: pgmap v481: 148 pgs: 6 creating+peering, 142 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-26T05:11:00.736 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:00 smithi080 ceph-mon[40404]: pgmap v481: 148 pgs: 6 creating+peering, 142 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-26T05:11:00.760 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:00.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:00.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:00.862 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:00.885 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:00.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:00.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:00.987 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:01.010 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:01.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:01.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:00 smithi132 ceph-mon[33500]: pgmap v481: 148 pgs: 6 creating+peering, 142 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-26T05:11:01.111 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:01.139 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:01.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:01.241 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:01.267 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:01.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:01.369 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:01.395 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:01.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:01.497 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:01.523 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:01.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:01.625 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:01.650 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:01.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:01.752 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:01.777 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:01.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:01.878 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:01.904 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:01.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:02.006 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:02.031 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:02.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:02.132 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:02.159 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:02.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:02.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:02.161 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T05:11:02.261 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:02.285 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:02.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:02.386 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:02.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:02.411 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:02.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:02.445 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:02.511 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:02.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:02.537 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:02.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:02.638 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:02.663 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:02.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:02.764 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:02.787 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:02.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:02.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:02 smithi132 ceph-mon[33500]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T05:11:02.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:02 smithi080 ceph-mon[29691]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T05:11:02.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:02 smithi080 ceph-mon[40404]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T05:11:02.889 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:02.915 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:02.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:03.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:03.039 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:03.040 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:03.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:03.142 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:03.166 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:03.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:03.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:03.268 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:03.292 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:03.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:03.394 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:03.418 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:03.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:03.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:03.519 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:03.544 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:03.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:03.612 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:03 smithi132 ceph-mon[33500]: pgmap v483: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:11:03.646 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:03.669 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:03.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:03.771 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:03.794 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:03.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:03.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:03.813 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:03 smithi080 ceph-mon[40404]: pgmap v483: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:11:03.813 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:03 smithi080 ceph-mon[29691]: pgmap v483: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:11:03.895 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:03.919 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:03.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:03.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:04.021 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:04.044 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:04.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:04.116 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:11:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:11:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:11:04.146 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:04.171 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:04.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:04.272 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:04.276 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:11:04.276 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:11:04.297 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:04.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:04.399 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:04.424 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:04.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:04.525 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:04.551 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:04.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:04.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:04.652 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:04.677 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:04.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:04.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:04.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:04.803 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:04.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:04.904 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:04.930 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:04.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:04.942 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T05:11:04.971 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:11:04.973 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:11:05.032 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:05.054 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:05.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:05.156 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:05.181 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:05.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:05.225 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:05.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:05.307 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:05.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:05.408 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:05.433 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:05.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:05.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:05.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:05.559 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:05.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:05.580 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:05 smithi132 ceph-mon[33500]: pgmap v484: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-03-26T05:11:05.581 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:11:05.581 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:11:05.581 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:05 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2656703426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:11:05.581 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:11:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:11:05.662 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:05.686 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:05.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:05.787 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:05.811 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:05.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:05.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:05 smithi080 ceph-mon[40404]: pgmap v484: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-03-26T05:11:05.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:11:05.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:11:05.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:05 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2656703426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:11:05.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:05 smithi080 ceph-mon[29691]: pgmap v484: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-03-26T05:11:05.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:11:05.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:11:05.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:05 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2656703426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:11:05.912 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:05.933 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:11:05.933 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-26T05:11:05.941 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:05.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:06.042 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:06.066 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:06.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:06.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:06.190 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:06.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:06.292 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:06.318 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:06.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:06.419 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:06.443 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:06.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:06.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:06.545 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:06.568 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:06.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:06.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:06.604 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[40404]: from='client.25241 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:11:06.604 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:06.604 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2331159647' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:11:06.604 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:11:06.604 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2656703426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:11:06.604 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2331159647' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:11:06.604 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[40404]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T05:11:06.604 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[40404]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T05:11:06.604 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[29691]: from='client.25241 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:11:06.605 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:06.605 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2331159647' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:11:06.605 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:11:06.605 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2656703426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:11:06.605 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2331159647' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:11:06.605 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[29691]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T05:11:06.605 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:06 smithi080 ceph-mon[29691]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T05:11:06.670 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:06.683 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:06 smithi132 ceph-mon[33500]: from='client.25241 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:11:06.683 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:06 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:06.683 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:06 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2331159647' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:11:06.684 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:06 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:11:06.684 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:06 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2656703426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:11:06.684 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:06 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2331159647' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:11:06.684 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:06 smithi132 ceph-mon[33500]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T05:11:06.684 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:06 smithi132 ceph-mon[33500]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T05:11:06.695 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:06.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:06.796 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:06.821 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:06.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:06.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:06.923 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:06.930 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:11:06.930 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:11:06.947 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:06.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:07.049 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:07.073 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:07.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:07.175 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:07.199 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:07.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:07.301 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:07.324 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:07.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:07.426 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:07.453 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:07.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:07.554 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:07.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:07.579 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:07.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:07.645 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T05:11:07.679 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:07.703 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:07.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:07.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:07.754 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:07 smithi080 ceph-mon[29691]: pgmap v485: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:11:07.755 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:07 smithi080 ceph-mon[40404]: pgmap v485: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:11:07.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:07 smithi132 ceph-mon[33500]: pgmap v485: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:11:07.804 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:07.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:07.829 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:07.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:07.886 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:07.931 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:07.955 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:07.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:08.039 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-03-26T05:11:03.920119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":2279,"num_read_kb":2093,"num_write":1093,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1774542,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1158,"ondisk_log_size":1158,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2601732,"kb_used_data":125456,"kb_used_omap":12,"kb_used_meta":2476147,"kb_avail":747359484,"statfs":{"total":767960285184,"available":765296111616,"internally_reserved":0,"allocated":128466944,"data_stored":124864319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2535575088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.420062"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":447,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657766+0000","last_change":"2024-03-26T05:09:39.468117+0000","last_active":"2024-03-26T05:10:58.657766+0000","last_peered":"2024-03-26T05:10:58.657766+0000","last_clean":"2024-03-26T05:10:58.657766+0000","last_became_active":"2024-03-26T05:07:59.996436+0000","last_became_peered":"2024-03-26T05:07:59.996436+0000","last_unstale":"2024-03-26T05:10:58.657766+0000","last_undegraded":"2024-03-26T05:10:58.657766+0000","last_fullsized":"2024-03-26T05:10:58.657766+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:39.468037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:39.468037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:01.482846+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":450,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656202+0000","last_change":"2024-03-26T05:09:44.530686+0000","last_active":"2024-03-26T05:10:58.656202+0000","last_peered":"2024-03-26T05:10:58.656202+0000","last_clean":"2024-03-26T05:10:58.656202+0000","last_became_active":"2024-03-26T05:07:59.997089+0000","last_became_peered":"2024-03-26T05:07:59.997089+0000","last_unstale":"2024-03-26T05:10:58.656202+0000","last_undegraded":"2024-03-26T05:10:58.656202+0000","last_fullsized":"2024-03-26T05:10:58.656202+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:44.530563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:44.530563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.884815+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":460,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101296+0000","last_change":"2024-03-26T05:09:24.726025+0000","last_active":"2024-03-26T05:10:59.101296+0000","last_peered":"2024-03-26T05:10:59.101296+0000","last_clean":"2024-03-26T05:10:59.101296+0000","last_became_active":"2024-03-26T05:07:59.994431+0000","last_became_peered":"2024-03-26T05:07:59.994431+0000","last_unstale":"2024-03-26T05:10:59.101296+0000","last_undegraded":"2024-03-26T05:10:59.101296+0000","last_fullsized":"2024-03-26T05:10:59.101296+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:24.725911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:24.725911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:00.073176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"254'21","reported_seq":480,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657702+0000","last_change":"2024-03-26T05:09:55.417464+0000","last_active":"2024-03-26T05:10:58.657702+0000","last_peered":"2024-03-26T05:10:58.657702+0000","last_clean":"2024-03-26T05:10:58.657702+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:10:58.657702+0000","last_undegraded":"2024-03-26T05:10:58.657702+0000","last_fullsized":"2024-03-26T05:10:58.657702+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:09:55.417325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.417325+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:24.296213+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":425,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657974+0000","last_change":"2024-03-26T05:09:59.528706+0000","last_active":"2024-03-26T05:10:58.657974+0000","last_peered":"2024-03-26T05:10:58.657974+0000","last_clean":"2024-03-26T05:10:58.657974+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:10:58.657974+0000","last_undegraded":"2024-03-26T05:10:58.657974+0000","last_fullsized":"2024-03-26T05:10:58.657974+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:59.528596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:59.528596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:25.643478+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":439,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657573+0000","last_change":"2024-03-26T05:10:29.404548+0000","last_active":"2024-03-26T05:10:58.657573+0000","last_peered":"2024-03-26T05:10:58.657573+0000","last_clean":"2024-03-26T05:10:58.657573+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:10:58.657573+0000","last_undegraded":"2024-03-26T05:10:58.657573+0000","last_fullsized":"2024-03-26T05:10:58.657573+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:29.404404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.404404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:51.272613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":446,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656623+0000","last_change":"2024-03-26T05:09:40.635212+0000","last_active":"2024-03-26T05:10:58.656623+0000","last_peered":"2024-03-26T05:10:58.656623+0000","last_clean":"2024-03-26T05:10:58.656623+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:10:58.656623+0000","last_undegraded":"2024-03-26T05:10:58.656623+0000","last_fullsized":"2024-03-26T05:10:58.656623+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:40.635129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:09:40.635129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.760848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"254'51","reported_seq":524,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656891+0000","last_change":"2024-03-26T05:07:46.642266+0000","last_active":"2024-03-26T05:10:58.656891+0000","last_peered":"2024-03-26T05:10:58.656891+0000","last_clean":"2024-03-26T05:10:58.656891+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:10:58.656891+0000","last_undegraded":"2024-03-26T05:10:58.656891+0000","last_fullsized":"2024-03-26T05:10:58.656891+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'39","last_scrub_stamp":"2024-03-26T05:07:46.641260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:46.641260+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.411572+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":443,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.102186+0000","last_change":"2024-03-26T05:10:30.211104+0000","last_active":"2024-03-26T05:10:59.102186+0000","last_peered":"2024-03-26T05:10:59.102186+0000","last_clean":"2024-03-26T05:10:59.102186+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:10:59.102186+0000","last_undegraded":"2024-03-26T05:10:59.102186+0000","last_fullsized":"2024-03-26T05:10:59.102186+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:30.211007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.211007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:37.046198+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":128,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280521+0000","last_change":"2024-03-26T05:09:57.144879+0000","last_active":"2024-03-26T05:10:59.280521+0000","last_peered":"2024-03-26T05:10:59.280521+0000","last_clean":"2024-03-26T05:10:59.280521+0000","last_became_active":"2024-03-26T05:08:00.000128+0000","last_became_peered":"2024-03-26T05:08:00.000128+0000","last_unstale":"2024-03-26T05:10:59.280521+0000","last_undegraded":"2024-03-26T05:10:59.280521+0000","last_fullsized":"2024-03-26T05:10:59.280521+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:57.144806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:57.144806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:10.794551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":128,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280984+0000","last_change":"2024-03-26T05:10:29.515852+0000","last_active":"2024-03-26T05:10:59.280984+0000","last_peered":"2024-03-26T05:10:59.280984+0000","last_clean":"2024-03-26T05:10:59.280984+0000","last_became_active":"2024-03-26T05:07:59.947557+0000","last_became_peered":"2024-03-26T05:07:59.947557+0000","last_unstale":"2024-03-26T05:10:59.280984+0000","last_undegraded":"2024-03-26T05:10:59.280984+0000","last_fullsized":"2024-03-26T05:10:59.280984+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:29.515767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.515767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:52.847327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"254'27","reported_seq":492,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658105+0000","last_change":"2024-03-26T05:09:33.456052+0000","last_active":"2024-03-26T05:10:58.658105+0000","last_peered":"2024-03-26T05:10:58.658105+0000","last_clean":"2024-03-26T05:10:58.658105+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:10:58.658105+0000","last_undegraded":"2024-03-26T05:10:58.658105+0000","last_fullsized":"2024-03-26T05:10:58.658105+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:09:33.455867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:33.455867+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:01.482830+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":128,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280242+0000","last_change":"2024-03-26T05:09:48.186435+0000","last_active":"2024-03-26T05:10:59.280242+0000","last_peered":"2024-03-26T05:10:59.280242+0000","last_clean":"2024-03-26T05:10:59.280242+0000","last_became_active":"2024-03-26T05:07:59.998777+0000","last_became_peered":"2024-03-26T05:07:59.998777+0000","last_unstale":"2024-03-26T05:10:59.280242+0000","last_undegraded":"2024-03-26T05:10:59.280242+0000","last_fullsized":"2024-03-26T05:10:59.280242+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:48.186287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.186287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:00.482583+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":452,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.064458+0000","last_change":"2024-03-26T05:10:41.797043+0000","last_active":"2024-03-26T05:10:59.064458+0000","last_peered":"2024-03-26T05:10:59.064458+0000","last_clean":"2024-03-26T05:10:59.064458+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:10:59.064458+0000","last_undegraded":"2024-03-26T05:10:59.064458+0000","last_fullsized":"2024-03-26T05:10:59.064458+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:41.796930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.796930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:56.003329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":120,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280752+0000","last_change":"2024-03-26T05:08:00.011593+0000","last_active":"2024-03-26T05:10:59.280752+0000","last_peered":"2024-03-26T05:10:59.280752+0000","last_clean":"2024-03-26T05:10:59.280752+0000","last_became_active":"2024-03-26T05:08:00.011262+0000","last_became_peered":"2024-03-26T05:08:00.011262+0000","last_unstale":"2024-03-26T05:10:59.280752+0000","last_undegraded":"2024-03-26T05:10:59.280752+0000","last_fullsized":"2024-03-26T05:10:59.280752+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:53.514177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.514177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"254'13","reported_seq":468,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657889+0000","last_change":"2024-03-26T05:10:32.425969+0000","last_active":"2024-03-26T05:10:58.657889+0000","last_peered":"2024-03-26T05:10:58.657889+0000","last_clean":"2024-03-26T05:10:58.657889+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:10:58.657889+0000","last_undegraded":"2024-03-26T05:10:58.657889+0000","last_fullsized":"2024-03-26T05:10:58.657889+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'13","last_scrub_stamp":"2024-03-26T05:10:32.425855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:32.425855+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:48.496500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":456,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658700+0000","last_change":"2024-03-26T05:09:56.453730+0000","last_active":"2024-03-26T05:10:58.658700+0000","last_peered":"2024-03-26T05:10:58.658700+0000","last_clean":"2024-03-26T05:10:58.658700+0000","last_became_active":"2024-03-26T05:07:59.942829+0000","last_became_peered":"2024-03-26T05:07:59.942829+0000","last_unstale":"2024-03-26T05:10:58.658700+0000","last_undegraded":"2024-03-26T05:10:58.658700+0000","last_fullsized":"2024-03-26T05:10:58.658700+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:56.453608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:09:56.453608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:14.181663+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":438,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657568+0000","last_change":"2024-03-26T05:09:57.476795+0000","last_active":"2024-03-26T05:10:58.657568+0000","last_peered":"2024-03-26T05:10:58.657568+0000","last_clean":"2024-03-26T05:10:58.657568+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:10:58.657568+0000","last_undegraded":"2024-03-26T05:10:58.657568+0000","last_fullsized":"2024-03-26T05:10:58.657568+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:57.476689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:57.476689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:15.765524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":455,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.102272+0000","last_change":"2024-03-26T05:10:38.179665+0000","last_active":"2024-03-26T05:10:59.102272+0000","last_peered":"2024-03-26T05:10:59.102272+0000","last_clean":"2024-03-26T05:10:59.102272+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:10:59.102272+0000","last_undegraded":"2024-03-26T05:10:59.102272+0000","last_fullsized":"2024-03-26T05:10:59.102272+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:38.179521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:10:38.179521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:45.972937+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"254'27","reported_seq":493,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101394+0000","last_change":"2024-03-26T05:09:48.440867+0000","last_active":"2024-03-26T05:10:59.101394+0000","last_peered":"2024-03-26T05:10:59.101394+0000","last_clean":"2024-03-26T05:10:59.101394+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:10:59.101394+0000","last_undegraded":"2024-03-26T05:10:59.101394+0000","last_fullsized":"2024-03-26T05:10:59.101394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:09:48.440764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.440764+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:06.366696+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":446,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657575+0000","last_change":"2024-03-26T05:09:45.842503+0000","last_active":"2024-03-26T05:10:58.657575+0000","last_peered":"2024-03-26T05:10:58.657575+0000","last_clean":"2024-03-26T05:10:58.657575+0000","last_became_active":"2024-03-26T05:07:59.959705+0000","last_became_peered":"2024-03-26T05:07:59.959705+0000","last_unstale":"2024-03-26T05:10:58.657575+0000","last_undegraded":"2024-03-26T05:10:58.657575+0000","last_fullsized":"2024-03-26T05:10:58.657575+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:45.842392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.842392+0000","last_clean_scrub_stamp":"2024-03-26T05:09:45.842392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.411603+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":439,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658801+0000","last_change":"2024-03-26T05:09:54.412717+0000","last_active":"2024-03-26T05:10:58.658801+0000","last_peered":"2024-03-26T05:10:58.658801+0000","last_clean":"2024-03-26T05:10:58.658801+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:10:58.658801+0000","last_undegraded":"2024-03-26T05:10:58.658801+0000","last_fullsized":"2024-03-26T05:10:58.658801+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:54.412617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:54.412617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:05.830877+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":463,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658157+0000","last_change":"2024-03-26T05:10:45.477455+0000","last_active":"2024-03-26T05:10:58.658157+0000","last_peered":"2024-03-26T05:10:58.658157+0000","last_clean":"2024-03-26T05:10:58.658157+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:10:58.658157+0000","last_undegraded":"2024-03-26T05:10:58.658157+0000","last_fullsized":"2024-03-26T05:10:58.658157+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:45.477372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:10:45.477372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:48.573961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"254'26","reported_seq":479,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656383+0000","last_change":"2024-03-26T05:07:50.662703+0000","last_active":"2024-03-26T05:10:58.656383+0000","last_peered":"2024-03-26T05:10:58.656383+0000","last_clean":"2024-03-26T05:10:58.656383+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:10:58.656383+0000","last_undegraded":"2024-03-26T05:10:58.656383+0000","last_fullsized":"2024-03-26T05:10:58.656383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:07:50.662564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:50.662564+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.411519+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":461,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.065238+0000","last_change":"2024-03-26T05:10:32.752769+0000","last_active":"2024-03-26T05:10:59.065238+0000","last_peered":"2024-03-26T05:10:59.065238+0000","last_clean":"2024-03-26T05:10:59.065238+0000","last_became_active":"2024-03-26T05:07:59.955125+0000","last_became_peered":"2024-03-26T05:07:59.955125+0000","last_unstale":"2024-03-26T05:10:59.065238+0000","last_undegraded":"2024-03-26T05:10:59.065238+0000","last_fullsized":"2024-03-26T05:10:59.065238+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:10:32.752678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:32.752678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:49.036523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":592,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.674963+0000","last_change":"2024-03-26T05:09:31.493239+0000","last_active":"2024-03-26T05:10:59.674963+0000","last_peered":"2024-03-26T05:10:59.674963+0000","last_clean":"2024-03-26T05:10:59.674963+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:10:59.674963+0000","last_undegraded":"2024-03-26T05:10:59.674963+0000","last_fullsized":"2024-03-26T05:10:59.674963+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:09:31.493102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:31.493102+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:02.820260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":447,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656328+0000","last_change":"2024-03-26T05:09:55.509651+0000","last_active":"2024-03-26T05:10:58.656328+0000","last_peered":"2024-03-26T05:10:58.656328+0000","last_clean":"2024-03-26T05:10:58.656328+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:10:58.656328+0000","last_undegraded":"2024-03-26T05:10:58.656328+0000","last_fullsized":"2024-03-26T05:10:58.656328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:09:55.509344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.509344+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:04.412791+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"254'26","reported_seq":494,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.102546+0000","last_change":"2024-03-26T05:10:41.134236+0000","last_active":"2024-03-26T05:10:59.102546+0000","last_peered":"2024-03-26T05:10:59.102546+0000","last_clean":"2024-03-26T05:10:59.102546+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:10:59.102546+0000","last_undegraded":"2024-03-26T05:10:59.102546+0000","last_fullsized":"2024-03-26T05:10:59.102546+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:10:41.134129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.134129+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.559878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":446,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657708+0000","last_change":"2024-03-26T05:09:46.565079+0000","last_active":"2024-03-26T05:10:58.657708+0000","last_peered":"2024-03-26T05:10:58.657708+0000","last_clean":"2024-03-26T05:10:58.657708+0000","last_became_active":"2024-03-26T05:07:59.995261+0000","last_became_peered":"2024-03-26T05:07:59.995261+0000","last_unstale":"2024-03-26T05:10:58.657708+0000","last_undegraded":"2024-03-26T05:10:58.657708+0000","last_fullsized":"2024-03-26T05:10:58.657708+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:46.564952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:46.564952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":440,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101862+0000","last_change":"2024-03-26T05:09:36.861688+0000","last_active":"2024-03-26T05:10:59.101862+0000","last_peered":"2024-03-26T05:10:59.101862+0000","last_clean":"2024-03-26T05:10:59.101862+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:10:59.101862+0000","last_undegraded":"2024-03-26T05:10:59.101862+0000","last_fullsized":"2024-03-26T05:10:59.101862+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:36.861410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:36.861410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:00.073195+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":445,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656143+0000","last_change":"2024-03-26T05:09:56.528351+0000","last_active":"2024-03-26T05:10:58.656143+0000","last_peered":"2024-03-26T05:10:58.656143+0000","last_clean":"2024-03-26T05:10:58.656143+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:10:58.656143+0000","last_undegraded":"2024-03-26T05:10:58.656143+0000","last_fullsized":"2024-03-26T05:10:58.656143+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:56.528075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:56.528075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:14.800367+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"254'25","reported_seq":560,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.675884+0000","last_change":"2024-03-26T05:10:39.649729+0000","last_active":"2024-03-26T05:10:59.675884+0000","last_peered":"2024-03-26T05:10:59.675884+0000","last_clean":"2024-03-26T05:10:59.675884+0000","last_became_active":"2024-03-26T05:07:59.962007+0000","last_became_peered":"2024-03-26T05:07:59.962007+0000","last_unstale":"2024-03-26T05:10:59.675884+0000","last_undegraded":"2024-03-26T05:10:59.675884+0000","last_fullsized":"2024-03-26T05:10:59.675884+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:10:39.649620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:39.649620+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.262212+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":440,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.064900+0000","last_change":"2024-03-26T05:10:43.824858+0000","last_active":"2024-03-26T05:10:59.064900+0000","last_peered":"2024-03-26T05:10:59.064900+0000","last_clean":"2024-03-26T05:10:59.064900+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:10:59.064900+0000","last_undegraded":"2024-03-26T05:10:59.064900+0000","last_fullsized":"2024-03-26T05:10:59.064900+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:43.824746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.824746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:48.191458+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":444,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.674911+0000","last_change":"2024-03-26T05:09:35.497868+0000","last_active":"2024-03-26T05:10:59.674911+0000","last_peered":"2024-03-26T05:10:59.674911+0000","last_clean":"2024-03-26T05:10:59.674911+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:10:59.674911+0000","last_undegraded":"2024-03-26T05:10:59.674911+0000","last_fullsized":"2024-03-26T05:10:59.674911+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:35.497586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:35.497586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.562634+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":467,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101491+0000","last_change":"2024-03-26T05:10:46.108129+0000","last_active":"2024-03-26T05:10:59.101491+0000","last_peered":"2024-03-26T05:10:59.101491+0000","last_clean":"2024-03-26T05:10:59.101491+0000","last_became_active":"2024-03-26T05:07:59.993996+0000","last_became_peered":"2024-03-26T05:07:59.993996+0000","last_unstale":"2024-03-26T05:10:59.101491+0000","last_undegraded":"2024-03-26T05:10:59.101491+0000","last_fullsized":"2024-03-26T05:10:59.101491+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:46.108051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:46.108051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:12.619539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"254'10","reported_seq":455,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656228+0000","last_change":"2024-03-26T05:10:29.575129+0000","last_active":"2024-03-26T05:10:58.656228+0000","last_peered":"2024-03-26T05:10:58.656228+0000","last_clean":"2024-03-26T05:10:58.656228+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:10:58.656228+0000","last_undegraded":"2024-03-26T05:10:58.656228+0000","last_fullsized":"2024-03-26T05:10:58.656228+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'6","last_scrub_stamp":"2024-03-26T05:10:29.575014+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.575014+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:48.708154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":433,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658063+0000","last_change":"2024-03-26T05:10:41.595090+0000","last_active":"2024-03-26T05:10:58.658063+0000","last_peered":"2024-03-26T05:10:58.658063+0000","last_clean":"2024-03-26T05:10:58.658063+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:10:58.658063+0000","last_undegraded":"2024-03-26T05:10:58.658063+0000","last_fullsized":"2024-03-26T05:10:58.658063+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:41.594950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.594950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:45.945454+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":444,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.064408+0000","last_change":"2024-03-26T05:09:49.476829+0000","last_active":"2024-03-26T05:10:59.064408+0000","last_peered":"2024-03-26T05:10:59.064408+0000","last_clean":"2024-03-26T05:10:59.064408+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:10:59.064408+0000","last_undegraded":"2024-03-26T05:10:59.064408+0000","last_fullsized":"2024-03-26T05:10:59.064408+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:49.476722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:09:49.476722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.014157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":120,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280994+0000","last_change":"2024-03-26T05:07:59.997817+0000","last_active":"2024-03-26T05:10:59.280994+0000","last_peered":"2024-03-26T05:10:59.280994+0000","last_clean":"2024-03-26T05:10:59.280994+0000","last_became_active":"2024-03-26T05:07:59.997439+0000","last_became_peered":"2024-03-26T05:07:59.997439+0000","last_unstale":"2024-03-26T05:10:59.280994+0000","last_undegraded":"2024-03-26T05:10:59.280994+0000","last_fullsized":"2024-03-26T05:10:59.280994+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:06:56.886632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"254'25","reported_seq":476,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656784+0000","last_change":"2024-03-26T05:07:52.593239+0000","last_active":"2024-03-26T05:10:58.656784+0000","last_peered":"2024-03-26T05:10:58.656784+0000","last_clean":"2024-03-26T05:10:58.656784+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:10:58.656784+0000","last_undegraded":"2024-03-26T05:10:58.656784+0000","last_fullsized":"2024-03-26T05:10:58.656784+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:07:52.592943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:52.592943+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.411483+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":455,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101700+0000","last_change":"2024-03-26T05:10:29.171886+0000","last_active":"2024-03-26T05:10:59.101700+0000","last_peered":"2024-03-26T05:10:59.101700+0000","last_clean":"2024-03-26T05:10:59.101700+0000","last_became_active":"2024-03-26T05:07:59.994210+0000","last_became_peered":"2024-03-26T05:07:59.994210+0000","last_unstale":"2024-03-26T05:10:59.101700+0000","last_undegraded":"2024-03-26T05:10:59.101700+0000","last_fullsized":"2024-03-26T05:10:59.101700+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:29.171780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.171780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:47.646591+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":429,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657904+0000","last_change":"2024-03-26T05:09:26.674360+0000","last_active":"2024-03-26T05:10:58.657904+0000","last_peered":"2024-03-26T05:10:58.657904+0000","last_clean":"2024-03-26T05:10:58.657904+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:10:58.657904+0000","last_undegraded":"2024-03-26T05:10:58.657904+0000","last_fullsized":"2024-03-26T05:10:58.657904+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:26.674245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:09:26.674245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.411652+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":444,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.065427+0000","last_change":"2024-03-26T05:09:53.508140+0000","last_active":"2024-03-26T05:10:59.065427+0000","last_peered":"2024-03-26T05:10:59.065427+0000","last_clean":"2024-03-26T05:10:59.065427+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:10:59.065427+0000","last_undegraded":"2024-03-26T05:10:59.065427+0000","last_fullsized":"2024-03-26T05:10:59.065427+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:53.507753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:09:53.507753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:05.567493+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"254'15","reported_seq":467,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656925+0000","last_change":"2024-03-26T05:07:53.425727+0000","last_active":"2024-03-26T05:10:58.656925+0000","last_peered":"2024-03-26T05:10:58.656925+0000","last_clean":"2024-03-26T05:10:58.656925+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:10:58.656925+0000","last_undegraded":"2024-03-26T05:10:58.656925+0000","last_fullsized":"2024-03-26T05:10:58.656925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-26T05:07:53.425472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.425472+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":437,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656631+0000","last_change":"2024-03-26T05:09:28.716469+0000","last_active":"2024-03-26T05:10:58.656631+0000","last_peered":"2024-03-26T05:10:58.656631+0000","last_clean":"2024-03-26T05:10:58.656631+0000","last_became_active":"2024-03-26T05:07:59.999703+0000","last_became_peered":"2024-03-26T05:07:59.999703+0000","last_unstale":"2024-03-26T05:10:58.656631+0000","last_undegraded":"2024-03-26T05:10:58.656631+0000","last_fullsized":"2024-03-26T05:10:58.656631+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:28.716348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:09:28.716348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.411587+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":458,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656822+0000","last_change":"2024-03-26T05:10:38.473529+0000","last_active":"2024-03-26T05:10:58.656822+0000","last_peered":"2024-03-26T05:10:58.656822+0000","last_clean":"2024-03-26T05:10:58.656822+0000","last_became_active":"2024-03-26T05:08:00.001601+0000","last_became_peered":"2024-03-26T05:08:00.001601+0000","last_unstale":"2024-03-26T05:10:58.656822+0000","last_undegraded":"2024-03-26T05:10:58.656822+0000","last_fullsized":"2024-03-26T05:10:58.656822+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:38.473446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:10:38.473446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:43.065707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":439,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658350+0000","last_change":"2024-03-26T05:07:39.699241+0000","last_active":"2024-03-26T05:10:58.658350+0000","last_peered":"2024-03-26T05:10:58.658350+0000","last_clean":"2024-03-26T05:10:58.658350+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:10:58.658350+0000","last_undegraded":"2024-03-26T05:10:58.658350+0000","last_fullsized":"2024-03-26T05:10:58.658350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:39.699122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:39.699122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:01.482758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"254'24","reported_seq":479,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658068+0000","last_change":"2024-03-26T05:07:43.754242+0000","last_active":"2024-03-26T05:10:58.658068+0000","last_peered":"2024-03-26T05:10:58.658068+0000","last_clean":"2024-03-26T05:10:58.658068+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:10:58.658068+0000","last_undegraded":"2024-03-26T05:10:58.658068+0000","last_fullsized":"2024-03-26T05:10:58.658068+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:07:43.754130+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:07:43.754130+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:01.482781+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":443,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101978+0000","last_change":"2024-03-26T05:10:50.061225+0000","last_active":"2024-03-26T05:10:59.101978+0000","last_peered":"2024-03-26T05:10:59.101978+0000","last_clean":"2024-03-26T05:10:59.101978+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:10:59.101978+0000","last_undegraded":"2024-03-26T05:10:59.101978+0000","last_fullsized":"2024-03-26T05:10:59.101978+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:50.061128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_clean_scrub_stamp":"2024-03-26T05:10:50.061128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:08.890951+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"216'10","reported_seq":168,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280951+0000","last_change":"2024-03-26T05:10:40.373554+0000","last_active":"2024-03-26T05:10:59.280951+0000","last_peered":"2024-03-26T05:10:59.280951+0000","last_clean":"2024-03-26T05:10:59.280951+0000","last_became_active":"2024-03-26T05:08:00.005648+0000","last_became_peered":"2024-03-26T05:08:00.005648+0000","last_unstale":"2024-03-26T05:10:59.280951+0000","last_undegraded":"2024-03-26T05:10:59.280951+0000","last_fullsized":"2024-03-26T05:10:59.280951+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:10:40.373470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.373470+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:08.010761+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":452,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.065148+0000","last_change":"2024-03-26T05:09:45.438723+0000","last_active":"2024-03-26T05:10:59.065148+0000","last_peered":"2024-03-26T05:10:59.065148+0000","last_clean":"2024-03-26T05:10:59.065148+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:10:59.065148+0000","last_undegraded":"2024-03-26T05:10:59.065148+0000","last_fullsized":"2024-03-26T05:10:59.065148+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:45.438615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:45.438615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.866308+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"254'32","reported_seq":495,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658108+0000","last_change":"2024-03-26T05:09:30.638211+0000","last_active":"2024-03-26T05:10:58.658108+0000","last_peered":"2024-03-26T05:10:58.658108+0000","last_clean":"2024-03-26T05:10:58.658108+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:10:58.658108+0000","last_undegraded":"2024-03-26T05:10:58.658108+0000","last_fullsized":"2024-03-26T05:10:58.658108+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:09:30.635624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:30.635624+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.411637+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":447,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658803+0000","last_change":"2024-03-26T05:10:40.548775+0000","last_active":"2024-03-26T05:10:58.658803+0000","last_peered":"2024-03-26T05:10:58.658803+0000","last_clean":"2024-03-26T05:10:58.658803+0000","last_became_active":"2024-03-26T05:07:59.942512+0000","last_became_peered":"2024-03-26T05:07:59.942512+0000","last_unstale":"2024-03-26T05:10:58.658803+0000","last_undegraded":"2024-03-26T05:10:58.658803+0000","last_fullsized":"2024-03-26T05:10:58.658803+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:40.548688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.548688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.537910+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"120'9","reported_seq":577,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658286+0000","last_change":"2024-03-26T05:10:30.593032+0000","last_active":"2024-03-26T05:10:58.658286+0000","last_peered":"2024-03-26T05:10:58.658286+0000","last_clean":"2024-03-26T05:10:58.658286+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:10:58.658286+0000","last_undegraded":"2024-03-26T05:10:58.658286+0000","last_fullsized":"2024-03-26T05:10:58.658286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'9","last_scrub_stamp":"2024-03-26T05:10:30.592948+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.592948+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:45.127936+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":455,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101809+0000","last_change":"2024-03-26T05:09:59.033943+0000","last_active":"2024-03-26T05:10:59.101809+0000","last_peered":"2024-03-26T05:10:59.101809+0000","last_clean":"2024-03-26T05:10:59.101809+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:10:59.101809+0000","last_undegraded":"2024-03-26T05:10:59.101809+0000","last_fullsized":"2024-03-26T05:10:59.101809+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:59.033819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:59.033819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:04.791902+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"254'26","reported_seq":149,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280816+0000","last_change":"2024-03-26T05:10:33.467211+0000","last_active":"2024-03-26T05:10:59.280816+0000","last_peered":"2024-03-26T05:10:59.280816+0000","last_clean":"2024-03-26T05:10:59.280816+0000","last_became_active":"2024-03-26T05:07:59.960346+0000","last_became_peered":"2024-03-26T05:07:59.960346+0000","last_unstale":"2024-03-26T05:10:59.280816+0000","last_undegraded":"2024-03-26T05:10:59.280816+0000","last_fullsized":"2024-03-26T05:10:59.280816+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:10:33.467113+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.467113+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:40.892275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":450,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101624+0000","last_change":"2024-03-26T05:10:41.140060+0000","last_active":"2024-03-26T05:10:59.101624+0000","last_peered":"2024-03-26T05:10:59.101624+0000","last_clean":"2024-03-26T05:10:59.101624+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:10:59.101624+0000","last_undegraded":"2024-03-26T05:10:59.101624+0000","last_fullsized":"2024-03-26T05:10:59.101624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:41.139989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.139989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:04.392633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":429,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657492+0000","last_change":"2024-03-26T05:07:55.590013+0000","last_active":"2024-03-26T05:10:58.657492+0000","last_peered":"2024-03-26T05:10:58.657492+0000","last_clean":"2024-03-26T05:10:58.657492+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:10:58.657492+0000","last_undegraded":"2024-03-26T05:10:58.657492+0000","last_fullsized":"2024-03-26T05:10:58.657492+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:55.589886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:07:55.589886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.411501+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":444,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.064871+0000","last_change":"2024-03-26T05:09:28.457669+0000","last_active":"2024-03-26T05:10:59.064871+0000","last_peered":"2024-03-26T05:10:59.064871+0000","last_clean":"2024-03-26T05:10:59.064871+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:10:59.064871+0000","last_undegraded":"2024-03-26T05:10:59.064871+0000","last_fullsized":"2024-03-26T05:10:59.064871+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:28.457548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:28.457548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.014190+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"254'27","reported_seq":500,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.102645+0000","last_change":"2024-03-26T05:10:43.135431+0000","last_active":"2024-03-26T05:10:59.102645+0000","last_peered":"2024-03-26T05:10:59.102645+0000","last_clean":"2024-03-26T05:10:59.102645+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:10:59.102645+0000","last_undegraded":"2024-03-26T05:10:59.102645+0000","last_fullsized":"2024-03-26T05:10:59.102645+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:10:43.135325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.135325+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:57.176336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":440,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.065274+0000","last_change":"2024-03-26T05:10:35.847292+0000","last_active":"2024-03-26T05:10:59.065274+0000","last_peered":"2024-03-26T05:10:59.065274+0000","last_clean":"2024-03-26T05:10:59.065274+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:10:59.065274+0000","last_undegraded":"2024-03-26T05:10:59.065274+0000","last_fullsized":"2024-03-26T05:10:59.065274+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:35.847191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:35.847191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:41.957309+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":128,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280998+0000","last_change":"2024-03-26T05:09:52.050966+0000","last_active":"2024-03-26T05:10:59.280998+0000","last_peered":"2024-03-26T05:10:59.280998+0000","last_clean":"2024-03-26T05:10:59.280998+0000","last_became_active":"2024-03-26T05:07:59.997434+0000","last_became_peered":"2024-03-26T05:07:59.997434+0000","last_unstale":"2024-03-26T05:10:59.280998+0000","last_undegraded":"2024-03-26T05:10:59.280998+0000","last_fullsized":"2024-03-26T05:10:59.280998+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:52.050832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:52.050832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:18.207778+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":479,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.675141+0000","last_change":"2024-03-26T05:09:32.473925+0000","last_active":"2024-03-26T05:10:59.675141+0000","last_peered":"2024-03-26T05:10:59.675141+0000","last_clean":"2024-03-26T05:10:59.675141+0000","last_became_active":"2024-03-26T05:07:59.992217+0000","last_became_peered":"2024-03-26T05:07:59.992217+0000","last_unstale":"2024-03-26T05:10:59.675141+0000","last_undegraded":"2024-03-26T05:10:59.675141+0000","last_fullsized":"2024-03-26T05:10:59.675141+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:09:32.473785+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_clean_scrub_stamp":"2024-03-26T05:09:32.473785+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"254'18","reported_seq":471,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657661+0000","last_change":"2024-03-26T05:10:36.560789+0000","last_active":"2024-03-26T05:10:58.657661+0000","last_peered":"2024-03-26T05:10:58.657661+0000","last_clean":"2024-03-26T05:10:58.657661+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:10:58.657661+0000","last_undegraded":"2024-03-26T05:10:58.657661+0000","last_fullsized":"2024-03-26T05:10:58.657661+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'18","last_scrub_stamp":"2024-03-26T05:10:36.560689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:36.560689+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:04.290785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101249+0000","last_change":"2024-03-26T05:10:55.650520+0000","last_active":"2024-03-26T05:10:59.101249+0000","last_peered":"2024-03-26T05:10:59.101249+0000","last_clean":"2024-03-26T05:10:59.101249+0000","last_became_active":"2024-03-26T05:10:55.650187+0000","last_became_peered":"2024-03-26T05:10:55.650187+0000","last_unstale":"2024-03-26T05:10:59.101249+0000","last_undegraded":"2024-03-26T05:10:59.101249+0000","last_fullsized":"2024-03-26T05:10:59.101249+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:55.192012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":124,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280572+0000","last_change":"2024-03-26T05:08:00.007340+0000","last_active":"2024-03-26T05:10:59.280572+0000","last_peered":"2024-03-26T05:10:59.280572+0000","last_clean":"2024-03-26T05:10:59.280572+0000","last_became_active":"2024-03-26T05:08:00.000166+0000","last_became_peered":"2024-03-26T05:08:00.000166+0000","last_unstale":"2024-03-26T05:10:59.280572+0000","last_undegraded":"2024-03-26T05:10:59.280572+0000","last_fullsized":"2024-03-26T05:10:59.280572+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:07:53.088449+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.088449+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":473,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.065661+0000","last_change":"2024-03-26T05:09:48.462986+0000","last_active":"2024-03-26T05:10:59.065661+0000","last_peered":"2024-03-26T05:10:59.065661+0000","last_clean":"2024-03-26T05:10:59.065661+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:10:59.065661+0000","last_undegraded":"2024-03-26T05:10:59.065661+0000","last_fullsized":"2024-03-26T05:10:59.065661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:09:48.462855+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:05:06.832744+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.462855+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.280826+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"254'48","reported_seq":175,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280211+0000","last_change":"2024-03-26T05:09:59.156101+0000","last_active":"2024-03-26T05:10:59.280211+0000","last_peered":"2024-03-26T05:10:59.280211+0000","last_clean":"2024-03-26T05:10:59.280211+0000","last_became_active":"2024-03-26T05:07:59.947478+0000","last_became_peered":"2024-03-26T05:07:59.947478+0000","last_unstale":"2024-03-26T05:10:59.280211+0000","last_undegraded":"2024-03-26T05:10:59.280211+0000","last_fullsized":"2024-03-26T05:10:59.280211+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'36","last_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:09:59.155894+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.794154+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":580,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.064471+0000","last_change":"2024-03-26T05:10:29.818124+0000","last_active":"2024-03-26T05:10:59.064471+0000","last_peered":"2024-03-26T05:10:59.064471+0000","last_clean":"2024-03-26T05:10:59.064471+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:10:59.064471+0000","last_undegraded":"2024-03-26T05:10:59.064471+0000","last_fullsized":"2024-03-26T05:10:59.064471+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:10:29.818015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.818015+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:33.669328+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656950+0000","last_change":"2024-03-26T05:10:55.653437+0000","last_active":"2024-03-26T05:10:58.656950+0000","last_peered":"2024-03-26T05:10:58.656950+0000","last_clean":"2024-03-26T05:10:58.656950+0000","last_became_active":"2024-03-26T05:10:55.652957+0000","last_became_peered":"2024-03-26T05:10:55.652957+0000","last_unstale":"2024-03-26T05:10:58.656950+0000","last_undegraded":"2024-03-26T05:10:58.656950+0000","last_fullsized":"2024-03-26T05:10:58.656950+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:57.103157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":439,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657993+0000","last_change":"2024-03-26T05:10:33.604297+0000","last_active":"2024-03-26T05:10:58.657993+0000","last_peered":"2024-03-26T05:10:58.657993+0000","last_clean":"2024-03-26T05:10:58.657993+0000","last_became_active":"2024-03-26T05:08:55.556543+0000","last_became_peered":"2024-03-26T05:08:55.556543+0000","last_unstale":"2024-03-26T05:10:58.657993+0000","last_undegraded":"2024-03-26T05:10:58.657993+0000","last_fullsized":"2024-03-26T05:10:58.657993+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.604198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.604198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:36.748955+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":452,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.065589+0000","last_change":"2024-03-26T05:10:38.869839+0000","last_active":"2024-03-26T05:10:59.065589+0000","last_peered":"2024-03-26T05:10:59.065589+0000","last_clean":"2024-03-26T05:10:59.065589+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:10:59.065589+0000","last_undegraded":"2024-03-26T05:10:59.065589+0000","last_fullsized":"2024-03-26T05:10:59.065589+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:38.869752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:38.869752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.797842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"254'37","reported_seq":506,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658362+0000","last_change":"2024-03-26T05:09:53.374250+0000","last_active":"2024-03-26T05:10:58.658362+0000","last_peered":"2024-03-26T05:10:58.658362+0000","last_clean":"2024-03-26T05:10:58.658362+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:10:58.658362+0000","last_undegraded":"2024-03-26T05:10:58.658362+0000","last_fullsized":"2024-03-26T05:10:58.658362+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'31","last_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_clean_scrub_stamp":"2024-03-26T05:09:53.374102+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:13.729213+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":446,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.100884+0000","last_change":"2024-03-26T05:10:37.078616+0000","last_active":"2024-03-26T05:10:59.100884+0000","last_peered":"2024-03-26T05:10:59.100884+0000","last_clean":"2024-03-26T05:10:59.100884+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:10:59.100884+0000","last_undegraded":"2024-03-26T05:10:59.100884+0000","last_fullsized":"2024-03-26T05:10:59.100884+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:37.078539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.432732+0000","last_clean_scrub_stamp":"2024-03-26T05:10:37.078539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.617367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280901+0000","last_change":"2024-03-26T05:10:55.648932+0000","last_active":"2024-03-26T05:10:59.280901+0000","last_peered":"2024-03-26T05:10:59.280901+0000","last_clean":"2024-03-26T05:10:59.280901+0000","last_became_active":"2024-03-26T05:10:55.648385+0000","last_became_peered":"2024-03-26T05:10:55.648385+0000","last_unstale":"2024-03-26T05:10:59.280901+0000","last_undegraded":"2024-03-26T05:10:59.280901+0000","last_fullsized":"2024-03-26T05:10:59.280901+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:57.657240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":440,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.064589+0000","last_change":"2024-03-26T05:10:27.852306+0000","last_active":"2024-03-26T05:10:59.064589+0000","last_peered":"2024-03-26T05:10:59.064589+0000","last_clean":"2024-03-26T05:10:59.064589+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:10:59.064589+0000","last_undegraded":"2024-03-26T05:10:59.064589+0000","last_fullsized":"2024-03-26T05:10:59.064589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:27.852200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_clean_scrub_stamp":"2024-03-26T05:10:27.852200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:39.771724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":458,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101492+0000","last_change":"2024-03-26T05:08:00.000387+0000","last_active":"2024-03-26T05:10:59.101492+0000","last_peered":"2024-03-26T05:10:59.101492+0000","last_clean":"2024-03-26T05:10:59.101492+0000","last_became_active":"2024-03-26T05:08:00.000009+0000","last_became_peered":"2024-03-26T05:08:00.000009+0000","last_unstale":"2024-03-26T05:10:59.101492+0000","last_undegraded":"2024-03-26T05:10:59.101492+0000","last_fullsized":"2024-03-26T05:10:59.101492+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:46.384777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:46.384777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":456,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.065627+0000","last_change":"2024-03-26T05:09:37.393265+0000","last_active":"2024-03-26T05:10:59.065627+0000","last_peered":"2024-03-26T05:10:59.065627+0000","last_clean":"2024-03-26T05:10:59.065627+0000","last_became_active":"2024-03-26T05:07:59.945688+0000","last_became_peered":"2024-03-26T05:07:59.945688+0000","last_unstale":"2024-03-26T05:10:59.065627+0000","last_undegraded":"2024-03-26T05:10:59.065627+0000","last_fullsized":"2024-03-26T05:10:59.065627+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:37.393159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:37.393159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"254'41","reported_seq":515,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.675502+0000","last_change":"2024-03-26T05:09:34.481126+0000","last_active":"2024-03-26T05:10:59.675502+0000","last_peered":"2024-03-26T05:10:59.675502+0000","last_clean":"2024-03-26T05:10:59.675502+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:10:59.675502+0000","last_undegraded":"2024-03-26T05:10:59.675502+0000","last_fullsized":"2024-03-26T05:10:59.675502+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:09:34.480977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:34.480977+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.370575+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280778+0000","last_change":"2024-03-26T05:10:55.650295+0000","last_active":"2024-03-26T05:10:59.280778+0000","last_peered":"2024-03-26T05:10:59.280778+0000","last_clean":"2024-03-26T05:10:59.280778+0000","last_became_active":"2024-03-26T05:10:55.649911+0000","last_became_peered":"2024-03-26T05:10:55.649911+0000","last_unstale":"2024-03-26T05:10:59.280778+0000","last_undegraded":"2024-03-26T05:10:59.280778+0000","last_fullsized":"2024-03-26T05:10:59.280778+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.750335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":435,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101193+0000","last_change":"2024-03-26T05:09:37.959362+0000","last_active":"2024-03-26T05:10:59.101193+0000","last_peered":"2024-03-26T05:10:59.101193+0000","last_clean":"2024-03-26T05:10:59.101193+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:10:59.101193+0000","last_undegraded":"2024-03-26T05:10:59.101193+0000","last_fullsized":"2024-03-26T05:10:59.101193+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:37.959237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:37.959237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:01.339031+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":446,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101017+0000","last_change":"2024-03-26T05:09:58.378595+0000","last_active":"2024-03-26T05:10:59.101017+0000","last_peered":"2024-03-26T05:10:59.101017+0000","last_clean":"2024-03-26T05:10:59.101017+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:10:59.101017+0000","last_undegraded":"2024-03-26T05:10:59.101017+0000","last_fullsized":"2024-03-26T05:10:59.101017+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:58.378228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:58.378228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.565597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":128,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280683+0000","last_change":"2024-03-26T05:09:40.213194+0000","last_active":"2024-03-26T05:10:59.280683+0000","last_peered":"2024-03-26T05:10:59.280683+0000","last_clean":"2024-03-26T05:10:59.280683+0000","last_became_active":"2024-03-26T05:07:59.959978+0000","last_became_peered":"2024-03-26T05:07:59.959978+0000","last_unstale":"2024-03-26T05:10:59.280683+0000","last_undegraded":"2024-03-26T05:10:59.280683+0000","last_fullsized":"2024-03-26T05:10:59.280683+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:40.213116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:40.213116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"254'21","reported_seq":480,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.102411+0000","last_change":"2024-03-26T05:09:27.773009+0000","last_active":"2024-03-26T05:10:59.102411+0000","last_peered":"2024-03-26T05:10:59.102411+0000","last_clean":"2024-03-26T05:10:59.102411+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:10:59.102411+0000","last_undegraded":"2024-03-26T05:10:59.102411+0000","last_fullsized":"2024-03-26T05:10:59.102411+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:09:27.772633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:27.772633+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.126453+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.100868+0000","last_change":"2024-03-26T05:10:55.650442+0000","last_active":"2024-03-26T05:10:59.100868+0000","last_peered":"2024-03-26T05:10:59.100868+0000","last_clean":"2024-03-26T05:10:59.100868+0000","last_became_active":"2024-03-26T05:10:55.650103+0000","last_became_peered":"2024-03-26T05:10:55.650103+0000","last_unstale":"2024-03-26T05:10:59.100868+0000","last_undegraded":"2024-03-26T05:10:59.100868+0000","last_fullsized":"2024-03-26T05:10:59.100868+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:58.474837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":425,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656892+0000","last_change":"2024-03-26T05:07:48.635847+0000","last_active":"2024-03-26T05:10:58.656892+0000","last_peered":"2024-03-26T05:10:58.656892+0000","last_clean":"2024-03-26T05:10:58.656892+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:10:58.656892+0000","last_undegraded":"2024-03-26T05:10:58.656892+0000","last_fullsized":"2024-03-26T05:10:58.656892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:48.635728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:48.635728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.411553+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":124,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280578+0000","last_change":"2024-03-26T05:08:00.004448+0000","last_active":"2024-03-26T05:10:59.280578+0000","last_peered":"2024-03-26T05:10:59.280578+0000","last_clean":"2024-03-26T05:10:59.280578+0000","last_became_active":"2024-03-26T05:07:59.998511+0000","last_became_peered":"2024-03-26T05:07:59.998511+0000","last_unstale":"2024-03-26T05:10:59.280578+0000","last_undegraded":"2024-03-26T05:10:59.280578+0000","last_fullsized":"2024-03-26T05:10:59.280578+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:07:37.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:07:37.347513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":440,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657942+0000","last_change":"2024-03-26T05:09:31.499628+0000","last_active":"2024-03-26T05:10:58.657942+0000","last_peered":"2024-03-26T05:10:58.657942+0000","last_clean":"2024-03-26T05:10:58.657942+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:10:58.657942+0000","last_undegraded":"2024-03-26T05:10:58.657942+0000","last_fullsized":"2024-03-26T05:10:58.657942+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:31.499513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:31.499513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:01.482901+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"254'46","reported_seq":657,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657384+0000","last_change":"2024-03-26T05:10:48.344286+0000","last_active":"2024-03-26T05:10:58.657384+0000","last_peered":"2024-03-26T05:10:58.657384+0000","last_clean":"2024-03-26T05:10:58.657384+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:10:58.657384+0000","last_undegraded":"2024-03-26T05:10:58.657384+0000","last_fullsized":"2024-03-26T05:10:58.657384+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'46","last_scrub_stamp":"2024-03-26T05:10:48.344180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:48.344180+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:57.157542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":440,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101046+0000","last_change":"2024-03-26T05:09:41.858043+0000","last_active":"2024-03-26T05:10:59.101046+0000","last_peered":"2024-03-26T05:10:59.101046+0000","last_clean":"2024-03-26T05:10:59.101046+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:10:59.101046+0000","last_undegraded":"2024-03-26T05:10:59.101046+0000","last_fullsized":"2024-03-26T05:10:59.101046+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:41.857888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:41.857888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.702593+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280242+0000","last_change":"2024-03-26T05:10:55.651127+0000","last_active":"2024-03-26T05:10:59.280242+0000","last_peered":"2024-03-26T05:10:59.280242+0000","last_clean":"2024-03-26T05:10:59.280242+0000","last_became_active":"2024-03-26T05:10:55.650612+0000","last_became_peered":"2024-03-26T05:10:55.650612+0000","last_unstale":"2024-03-26T05:10:59.280242+0000","last_undegraded":"2024-03-26T05:10:59.280242+0000","last_fullsized":"2024-03-26T05:10:59.280242+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:24.207175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":441,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.065765+0000","last_change":"2024-03-26T05:09:36.428895+0000","last_active":"2024-03-26T05:10:59.065765+0000","last_peered":"2024-03-26T05:10:59.065765+0000","last_clean":"2024-03-26T05:10:59.065765+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:10:59.065765+0000","last_undegraded":"2024-03-26T05:10:59.065765+0000","last_fullsized":"2024-03-26T05:10:59.065765+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:36.428762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:36.428762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.014133+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":447,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656234+0000","last_change":"2024-03-26T05:10:39.454264+0000","last_active":"2024-03-26T05:10:58.656234+0000","last_peered":"2024-03-26T05:10:58.656234+0000","last_clean":"2024-03-26T05:10:58.656234+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:10:58.656234+0000","last_undegraded":"2024-03-26T05:10:58.656234+0000","last_fullsized":"2024-03-26T05:10:58.656234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:39.454176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:39.454176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:41.956147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":464,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.100812+0000","last_change":"2024-03-26T05:09:55.551716+0000","last_active":"2024-03-26T05:10:59.100812+0000","last_peered":"2024-03-26T05:10:59.100812+0000","last_clean":"2024-03-26T05:10:59.100812+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:10:59.100812+0000","last_undegraded":"2024-03-26T05:10:59.100812+0000","last_fullsized":"2024-03-26T05:10:59.100812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:55.551595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.551595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.543264+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"254'23","reported_seq":261,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101416+0000","last_change":"2024-03-26T05:08:00.019227+0000","last_active":"2024-03-26T05:10:59.101416+0000","last_peered":"2024-03-26T05:10:59.101416+0000","last_clean":"2024-03-26T05:10:59.101416+0000","last_became_active":"2024-03-26T05:08:00.001398+0000","last_became_peered":"2024-03-26T05:08:00.001398+0000","last_unstale":"2024-03-26T05:10:59.101416+0000","last_undegraded":"2024-03-26T05:10:59.101416+0000","last_fullsized":"2024-03-26T05:10:59.101416+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-26T05:07:54.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:54.529843+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":466,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.102152+0000","last_change":"2024-03-26T05:10:43.076916+0000","last_active":"2024-03-26T05:10:59.102152+0000","last_peered":"2024-03-26T05:10:59.102152+0000","last_clean":"2024-03-26T05:10:59.102152+0000","last_became_active":"2024-03-26T05:07:59.955456+0000","last_became_peered":"2024-03-26T05:07:59.955456+0000","last_unstale":"2024-03-26T05:10:59.102152+0000","last_undegraded":"2024-03-26T05:10:59.102152+0000","last_fullsized":"2024-03-26T05:10:59.102152+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:43.076802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.076802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:54.501193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.655914+0000","last_change":"2024-03-26T05:10:55.651844+0000","last_active":"2024-03-26T05:10:58.655914+0000","last_peered":"2024-03-26T05:10:58.655914+0000","last_clean":"2024-03-26T05:10:58.655914+0000","last_became_active":"2024-03-26T05:10:55.651432+0000","last_became_peered":"2024-03-26T05:10:55.651432+0000","last_unstale":"2024-03-26T05:10:58.655914+0000","last_undegraded":"2024-03-26T05:10:58.655914+0000","last_fullsized":"2024-03-26T05:10:58.655914+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:16.458743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":128,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.281256+0000","last_change":"2024-03-26T05:09:46.172999+0000","last_active":"2024-03-26T05:10:59.281256+0000","last_peered":"2024-03-26T05:10:59.281256+0000","last_clean":"2024-03-26T05:10:59.281256+0000","last_became_active":"2024-03-26T05:07:59.959853+0000","last_became_peered":"2024-03-26T05:07:59.959853+0000","last_unstale":"2024-03-26T05:10:59.281256+0000","last_undegraded":"2024-03-26T05:10:59.281256+0000","last_fullsized":"2024-03-26T05:10:59.281256+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:46.172868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_clean_scrub_stamp":"2024-03-26T05:09:46.172868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452798+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":708,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.064762+0000","last_change":"2024-03-26T05:10:39.866608+0000","last_active":"2024-03-26T05:10:59.064762+0000","last_peered":"2024-03-26T05:10:59.064762+0000","last_clean":"2024-03-26T05:10:59.064762+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:10:59.064762+0000","last_undegraded":"2024-03-26T05:10:59.064762+0000","last_fullsized":"2024-03-26T05:10:59.064762+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:10:39.866516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:10:39.866516+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.907367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":458,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657178+0000","last_change":"2024-03-26T05:09:39.605190+0000","last_active":"2024-03-26T05:10:58.657178+0000","last_peered":"2024-03-26T05:10:58.657178+0000","last_clean":"2024-03-26T05:10:58.657178+0000","last_became_active":"2024-03-26T05:07:59.960061+0000","last_became_peered":"2024-03-26T05:07:59.960061+0000","last_unstale":"2024-03-26T05:10:58.657178+0000","last_undegraded":"2024-03-26T05:10:58.657178+0000","last_fullsized":"2024-03-26T05:10:58.657178+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:39.605053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:39.605053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.821576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"254'36","reported_seq":633,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.675716+0000","last_change":"2024-03-26T05:09:41.506253+0000","last_active":"2024-03-26T05:10:59.675716+0000","last_peered":"2024-03-26T05:10:59.675716+0000","last_clean":"2024-03-26T05:10:59.675716+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:10:59.675716+0000","last_undegraded":"2024-03-26T05:10:59.675716+0000","last_fullsized":"2024-03-26T05:10:59.675716+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'24","last_scrub_stamp":"2024-03-26T05:09:41.506123+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:09:41.506123+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:02.820289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":465,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.065281+0000","last_change":"2024-03-26T05:10:36.850667+0000","last_active":"2024-03-26T05:10:59.065281+0000","last_peered":"2024-03-26T05:10:59.065281+0000","last_clean":"2024-03-26T05:10:59.065281+0000","last_became_active":"2024-03-26T05:07:59.956080+0000","last_became_peered":"2024-03-26T05:07:59.956080+0000","last_unstale":"2024-03-26T05:10:59.065281+0000","last_undegraded":"2024-03-26T05:10:59.065281+0000","last_fullsized":"2024-03-26T05:10:59.065281+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:36.850583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:36.850583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:53.029005+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101867+0000","last_change":"2024-03-26T05:10:55.754958+0000","last_active":"2024-03-26T05:10:59.101867+0000","last_peered":"2024-03-26T05:10:59.101867+0000","last_clean":"2024-03-26T05:10:59.101867+0000","last_became_active":"2024-03-26T05:10:55.754590+0000","last_became_peered":"2024-03-26T05:10:55.754590+0000","last_unstale":"2024-03-26T05:10:59.101867+0000","last_undegraded":"2024-03-26T05:10:59.101867+0000","last_fullsized":"2024-03-26T05:10:59.101867+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:06.042120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":447,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657056+0000","last_change":"2024-03-26T05:08:00.002985+0000","last_active":"2024-03-26T05:10:58.657056+0000","last_peered":"2024-03-26T05:10:58.657056+0000","last_clean":"2024-03-26T05:10:58.657056+0000","last_became_active":"2024-03-26T05:08:00.001657+0000","last_became_peered":"2024-03-26T05:08:00.001657+0000","last_unstale":"2024-03-26T05:10:58.657056+0000","last_undegraded":"2024-03-26T05:10:58.657056+0000","last_fullsized":"2024-03-26T05:10:58.657056+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:39.338092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:39.338092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":343,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658255+0000","last_change":"2024-03-26T05:09:48.473285+0000","last_active":"2024-03-26T05:10:58.658255+0000","last_peered":"2024-03-26T05:10:58.658255+0000","last_clean":"2024-03-26T05:10:58.658255+0000","last_became_active":"2024-03-26T05:07:59.992489+0000","last_became_peered":"2024-03-26T05:07:59.992489+0000","last_unstale":"2024-03-26T05:10:58.658255+0000","last_undegraded":"2024-03-26T05:10:58.658255+0000","last_fullsized":"2024-03-26T05:10:58.658255+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.473160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:01.482950+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":1096,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.064630+0000","last_change":"2024-03-26T05:09:55.485007+0000","last_active":"2024-03-26T05:10:59.064630+0000","last_peered":"2024-03-26T05:10:59.064630+0000","last_clean":"2024-03-26T05:10:59.064630+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:10:59.064630+0000","last_undegraded":"2024-03-26T05:10:59.064630+0000","last_fullsized":"2024-03-26T05:10:59.064630+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:09:55.484790+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.484790+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:14.311528+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":630,"num_read_kb":625,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"254'30","reported_seq":525,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.102081+0000","last_change":"2024-03-26T05:09:49.909494+0000","last_active":"2024-03-26T05:10:59.102081+0000","last_peered":"2024-03-26T05:10:59.102081+0000","last_clean":"2024-03-26T05:10:59.102081+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:10:59.102081+0000","last_undegraded":"2024-03-26T05:10:59.102081+0000","last_fullsized":"2024-03-26T05:10:59.102081+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'18","last_scrub_stamp":"2024-03-26T05:09:49.909347+0000","last_deep_scrub":"187'18","last_deep_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_clean_scrub_stamp":"2024-03-26T05:09:49.909347+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:16.935034+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":128,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.281194+0000","last_change":"2024-03-26T05:10:44.404350+0000","last_active":"2024-03-26T05:10:59.281194+0000","last_peered":"2024-03-26T05:10:59.281194+0000","last_clean":"2024-03-26T05:10:59.281194+0000","last_became_active":"2024-03-26T05:07:59.999990+0000","last_became_peered":"2024-03-26T05:07:59.999990+0000","last_unstale":"2024-03-26T05:10:59.281194+0000","last_undegraded":"2024-03-26T05:10:59.281194+0000","last_fullsized":"2024-03-26T05:10:59.281194+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:44.404261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:44.404261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:56.403017+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.065297+0000","last_change":"2024-03-26T05:10:55.652321+0000","last_active":"2024-03-26T05:10:59.065297+0000","last_peered":"2024-03-26T05:10:59.065297+0000","last_clean":"2024-03-26T05:10:59.065297+0000","last_became_active":"2024-03-26T05:10:55.651931+0000","last_became_peered":"2024-03-26T05:10:55.651931+0000","last_unstale":"2024-03-26T05:10:59.065297+0000","last_undegraded":"2024-03-26T05:10:59.065297+0000","last_fullsized":"2024-03-26T05:10:59.065297+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:23.581179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":435,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656109+0000","last_change":"2024-03-26T05:10:34.408142+0000","last_active":"2024-03-26T05:10:58.656109+0000","last_peered":"2024-03-26T05:10:58.656109+0000","last_clean":"2024-03-26T05:10:58.656109+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:10:58.656109+0000","last_undegraded":"2024-03-26T05:10:58.656109+0000","last_fullsized":"2024-03-26T05:10:58.656109+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:34.408045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:10:34.408045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:39.555764+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":459,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657274+0000","last_change":"2024-03-26T05:09:54.538181+0000","last_active":"2024-03-26T05:10:58.657274+0000","last_peered":"2024-03-26T05:10:58.657274+0000","last_clean":"2024-03-26T05:10:58.657274+0000","last_became_active":"2024-03-26T05:08:00.003099+0000","last_became_peered":"2024-03-26T05:08:00.003099+0000","last_unstale":"2024-03-26T05:10:58.657274+0000","last_undegraded":"2024-03-26T05:10:58.657274+0000","last_fullsized":"2024-03-26T05:10:58.657274+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:54.538056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:09:54.538056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:06.436896+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"254'44","reported_seq":576,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.102178+0000","last_change":"2024-03-26T05:09:53.978359+0000","last_active":"2024-03-26T05:10:59.102178+0000","last_peered":"2024-03-26T05:10:59.102178+0000","last_clean":"2024-03-26T05:10:59.102178+0000","last_became_active":"2024-03-26T05:07:59.994226+0000","last_became_peered":"2024-03-26T05:07:59.994226+0000","last_unstale":"2024-03-26T05:10:59.102178+0000","last_undegraded":"2024-03-26T05:10:59.102178+0000","last_fullsized":"2024-03-26T05:10:59.102178+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'34","last_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:09:53.978203+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:00.272840+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":465,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.675535+0000","last_change":"2024-03-26T05:10:42.594397+0000","last_active":"2024-03-26T05:10:59.675535+0000","last_peered":"2024-03-26T05:10:59.675535+0000","last_clean":"2024-03-26T05:10:59.675535+0000","last_became_active":"2024-03-26T05:07:59.991484+0000","last_became_peered":"2024-03-26T05:07:59.991484+0000","last_unstale":"2024-03-26T05:10:59.675535+0000","last_undegraded":"2024-03-26T05:10:59.675535+0000","last_fullsized":"2024-03-26T05:10:59.675535+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:42.594296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:10:42.594296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:54.067648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280895+0000","last_change":"2024-03-26T05:10:55.648891+0000","last_active":"2024-03-26T05:10:59.280895+0000","last_peered":"2024-03-26T05:10:59.280895+0000","last_clean":"2024-03-26T05:10:59.280895+0000","last_became_active":"2024-03-26T05:10:55.648533+0000","last_became_peered":"2024-03-26T05:10:55.648533+0000","last_unstale":"2024-03-26T05:10:59.280895+0000","last_undegraded":"2024-03-26T05:10:59.280895+0000","last_fullsized":"2024-03-26T05:10:59.280895+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:06.188725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":436,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101791+0000","last_change":"2024-03-26T05:09:28.793178+0000","last_active":"2024-03-26T05:10:59.101791+0000","last_peered":"2024-03-26T05:10:59.101791+0000","last_clean":"2024-03-26T05:10:59.101791+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:10:59.101791+0000","last_undegraded":"2024-03-26T05:10:59.101791+0000","last_fullsized":"2024-03-26T05:10:59.101791+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:28.792827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:28.792827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.126431+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":128,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280868+0000","last_change":"2024-03-26T05:09:42.163715+0000","last_active":"2024-03-26T05:10:59.280868+0000","last_peered":"2024-03-26T05:10:59.280868+0000","last_clean":"2024-03-26T05:10:59.280868+0000","last_became_active":"2024-03-26T05:07:59.960768+0000","last_became_peered":"2024-03-26T05:07:59.960768+0000","last_unstale":"2024-03-26T05:10:59.280868+0000","last_undegraded":"2024-03-26T05:10:59.280868+0000","last_fullsized":"2024-03-26T05:10:59.280868+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:42.163609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:42.163609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:04.022763+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"254'26","reported_seq":495,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101272+0000","last_change":"2024-03-26T05:09:47.483160+0000","last_active":"2024-03-26T05:10:59.101272+0000","last_peered":"2024-03-26T05:10:59.101272+0000","last_clean":"2024-03-26T05:10:59.101272+0000","last_became_active":"2024-03-26T05:07:59.960351+0000","last_became_peered":"2024-03-26T05:07:59.960351+0000","last_unstale":"2024-03-26T05:10:59.101272+0000","last_undegraded":"2024-03-26T05:10:59.101272+0000","last_fullsized":"2024-03-26T05:10:59.101272+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:09:47.483012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:47.483012+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.107710+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"216'11","reported_seq":606,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.102507+0000","last_change":"2024-03-26T05:10:34.204907+0000","last_active":"2024-03-26T05:10:59.102507+0000","last_peered":"2024-03-26T05:10:59.102507+0000","last_clean":"2024-03-26T05:10:59.102507+0000","last_became_active":"2024-03-26T05:07:59.994682+0000","last_became_peered":"2024-03-26T05:07:59.994682+0000","last_unstale":"2024-03-26T05:10:59.102507+0000","last_undegraded":"2024-03-26T05:10:59.102507+0000","last_fullsized":"2024-03-26T05:10:59.102507+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:10:34.204798+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:10:34.204798+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.783859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657897+0000","last_change":"2024-03-26T05:10:55.648350+0000","last_active":"2024-03-26T05:10:58.657897+0000","last_peered":"2024-03-26T05:10:58.657897+0000","last_clean":"2024-03-26T05:10:58.657897+0000","last_became_active":"2024-03-26T05:10:55.647949+0000","last_became_peered":"2024-03-26T05:10:55.647949+0000","last_unstale":"2024-03-26T05:10:58.657897+0000","last_undegraded":"2024-03-26T05:10:58.657897+0000","last_fullsized":"2024-03-26T05:10:58.657897+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:54.976918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":441,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.065375+0000","last_change":"2024-03-26T05:10:34.839543+0000","last_active":"2024-03-26T05:10:59.065375+0000","last_peered":"2024-03-26T05:10:59.065375+0000","last_clean":"2024-03-26T05:10:59.065375+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:10:59.065375+0000","last_undegraded":"2024-03-26T05:10:59.065375+0000","last_fullsized":"2024-03-26T05:10:59.065375+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:34.837732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:34.837732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.929082+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":456,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.102445+0000","last_change":"2024-03-26T05:09:47.958661+0000","last_active":"2024-03-26T05:10:59.102445+0000","last_peered":"2024-03-26T05:10:59.102445+0000","last_clean":"2024-03-26T05:10:59.102445+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:10:59.102445+0000","last_undegraded":"2024-03-26T05:10:59.102445+0000","last_fullsized":"2024-03-26T05:10:59.102445+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:09:47.958370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:47.958370+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.126473+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"254'34","reported_seq":546,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658216+0000","last_change":"2024-03-26T05:09:47.484748+0000","last_active":"2024-03-26T05:10:58.658216+0000","last_peered":"2024-03-26T05:10:58.658216+0000","last_clean":"2024-03-26T05:10:58.658216+0000","last_became_active":"2024-03-26T05:07:59.942510+0000","last_became_peered":"2024-03-26T05:07:59.942510+0000","last_unstale":"2024-03-26T05:10:58.658216+0000","last_undegraded":"2024-03-26T05:10:58.658216+0000","last_fullsized":"2024-03-26T05:10:58.658216+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:09:47.484585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:47.484585+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:13.731149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":566,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656404+0000","last_change":"2024-03-26T05:07:53.583159+0000","last_active":"2024-03-26T05:10:58.656404+0000","last_peered":"2024-03-26T05:10:58.656404+0000","last_clean":"2024-03-26T05:10:58.656404+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:10:58.656404+0000","last_undegraded":"2024-03-26T05:10:58.656404+0000","last_fullsized":"2024-03-26T05:10:58.656404+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:07:53.583016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.583016+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.411537+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.065191+0000","last_change":"2024-03-26T05:10:55.652074+0000","last_active":"2024-03-26T05:10:59.065191+0000","last_peered":"2024-03-26T05:10:59.065191+0000","last_clean":"2024-03-26T05:10:59.065191+0000","last_became_active":"2024-03-26T05:10:55.651719+0000","last_became_peered":"2024-03-26T05:10:55.651719+0000","last_unstale":"2024-03-26T05:10:59.065191+0000","last_undegraded":"2024-03-26T05:10:59.065191+0000","last_fullsized":"2024-03-26T05:10:59.065191+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:18.255604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":436,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657972+0000","last_change":"2024-03-26T05:07:50.818124+0000","last_active":"2024-03-26T05:10:58.657972+0000","last_peered":"2024-03-26T05:10:58.657972+0000","last_clean":"2024-03-26T05:10:58.657972+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:10:58.657972+0000","last_undegraded":"2024-03-26T05:10:58.657972+0000","last_fullsized":"2024-03-26T05:10:58.657972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:50.818000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:50.818000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:01.482797+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":437,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657282+0000","last_change":"2024-03-26T05:09:38.690272+0000","last_active":"2024-03-26T05:10:58.657282+0000","last_peered":"2024-03-26T05:10:58.657282+0000","last_clean":"2024-03-26T05:10:58.657282+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:10:58.657282+0000","last_undegraded":"2024-03-26T05:10:58.657282+0000","last_fullsized":"2024-03-26T05:10:58.657282+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:38.690131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:38.690131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"254'52","reported_seq":526,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656375+0000","last_change":"2024-03-26T05:09:34.644416+0000","last_active":"2024-03-26T05:10:58.656375+0000","last_peered":"2024-03-26T05:10:58.656375+0000","last_clean":"2024-03-26T05:10:58.656375+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:10:58.656375+0000","last_undegraded":"2024-03-26T05:10:58.656375+0000","last_fullsized":"2024-03-26T05:10:58.656375+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:09:34.643076+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:09:34.643076+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.411620+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":440,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657754+0000","last_change":"2024-03-26T05:10:33.411222+0000","last_active":"2024-03-26T05:10:58.657754+0000","last_peered":"2024-03-26T05:10:58.657754+0000","last_clean":"2024-03-26T05:10:58.657754+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:10:58.657754+0000","last_undegraded":"2024-03-26T05:10:58.657754+0000","last_fullsized":"2024-03-26T05:10:58.657754+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.411139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.411139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:54.372120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657036+0000","last_change":"2024-03-26T05:10:55.652478+0000","last_active":"2024-03-26T05:10:58.657036+0000","last_peered":"2024-03-26T05:10:58.657036+0000","last_clean":"2024-03-26T05:10:58.657036+0000","last_became_active":"2024-03-26T05:10:55.652104+0000","last_became_peered":"2024-03-26T05:10:55.652104+0000","last_unstale":"2024-03-26T05:10:58.657036+0000","last_undegraded":"2024-03-26T05:10:58.657036+0000","last_fullsized":"2024-03-26T05:10:58.657036+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:10.886402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":120,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.280755+0000","last_change":"2024-03-26T05:08:00.006006+0000","last_active":"2024-03-26T05:10:59.280755+0000","last_peered":"2024-03-26T05:10:59.280755+0000","last_clean":"2024-03-26T05:10:59.280755+0000","last_became_active":"2024-03-26T05:08:00.005676+0000","last_became_peered":"2024-03-26T05:08:00.005676+0000","last_unstale":"2024-03-26T05:10:59.280755+0000","last_undegraded":"2024-03-26T05:10:59.280755+0000","last_fullsized":"2024-03-26T05:10:59.280755+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_clean_scrub_stamp":"2024-03-26T05:07:52.465619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:10:59.452733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"216'10","reported_seq":596,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658765+0000","last_change":"2024-03-26T05:09:36.472026+0000","last_active":"2024-03-26T05:10:58.658765+0000","last_peered":"2024-03-26T05:10:58.658765+0000","last_clean":"2024-03-26T05:10:58.658765+0000","last_became_active":"2024-03-26T05:07:59.997429+0000","last_became_peered":"2024-03-26T05:07:59.997429+0000","last_unstale":"2024-03-26T05:10:58.658765+0000","last_undegraded":"2024-03-26T05:10:58.658765+0000","last_fullsized":"2024-03-26T05:10:58.658765+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:09:36.471890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:36.471890+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:01.482885+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":445,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656314+0000","last_change":"2024-03-26T05:10:40.541468+0000","last_active":"2024-03-26T05:10:58.656314+0000","last_peered":"2024-03-26T05:10:58.656314+0000","last_clean":"2024-03-26T05:10:58.656314+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:10:58.656314+0000","last_undegraded":"2024-03-26T05:10:58.656314+0000","last_fullsized":"2024-03-26T05:10:58.656314+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:40.541340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.541340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.162836+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"254'27","reported_seq":493,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657620+0000","last_change":"2024-03-26T05:10:43.423183+0000","last_active":"2024-03-26T05:10:58.657620+0000","last_peered":"2024-03-26T05:10:58.657620+0000","last_clean":"2024-03-26T05:10:58.657620+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:10:58.657620+0000","last_undegraded":"2024-03-26T05:10:58.657620+0000","last_fullsized":"2024-03-26T05:10:58.657620+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:10:43.423040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.423040+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:50.942552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.064867+0000","last_change":"2024-03-26T05:10:55.651677+0000","last_active":"2024-03-26T05:10:59.064867+0000","last_peered":"2024-03-26T05:10:59.064867+0000","last_clean":"2024-03-26T05:10:59.064867+0000","last_became_active":"2024-03-26T05:10:55.651348+0000","last_became_peered":"2024-03-26T05:10:55.651348+0000","last_unstale":"2024-03-26T05:10:59.064867+0000","last_undegraded":"2024-03-26T05:10:59.064867+0000","last_fullsized":"2024-03-26T05:10:59.064867+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:58.278849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":445,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658040+0000","last_change":"2024-03-26T05:10:44.608828+0000","last_active":"2024-03-26T05:10:58.658040+0000","last_peered":"2024-03-26T05:10:58.658040+0000","last_clean":"2024-03-26T05:10:58.658040+0000","last_became_active":"2024-03-26T05:07:59.957995+0000","last_became_peered":"2024-03-26T05:07:59.957995+0000","last_unstale":"2024-03-26T05:10:58.658040+0000","last_undegraded":"2024-03-26T05:10:58.658040+0000","last_fullsized":"2024-03-26T05:10:58.658040+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:44.608717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:44.608717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.528157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":128,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.281190+0000","last_change":"2024-03-26T05:10:48.413966+0000","last_active":"2024-03-26T05:10:59.281190+0000","last_peered":"2024-03-26T05:10:59.281190+0000","last_clean":"2024-03-26T05:10:59.281190+0000","last_became_active":"2024-03-26T05:07:59.947747+0000","last_became_peered":"2024-03-26T05:07:59.947747+0000","last_unstale":"2024-03-26T05:10:59.281190+0000","last_undegraded":"2024-03-26T05:10:59.281190+0000","last_fullsized":"2024-03-26T05:10:59.281190+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:48.413868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.446060+0000","last_clean_scrub_stamp":"2024-03-26T05:10:48.413868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:13.855843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":447,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657765+0000","last_change":"2024-03-26T05:07:49.831449+0000","last_active":"2024-03-26T05:10:58.657765+0000","last_peered":"2024-03-26T05:10:58.657765+0000","last_clean":"2024-03-26T05:10:58.657765+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:10:58.657765+0000","last_undegraded":"2024-03-26T05:10:58.657765+0000","last_fullsized":"2024-03-26T05:10:58.657765+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:49.831323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:49.831323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:01.482813+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"254'25","reported_seq":494,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658589+0000","last_change":"2024-03-26T05:10:48.381843+0000","last_active":"2024-03-26T05:10:58.658589+0000","last_peered":"2024-03-26T05:10:58.658589+0000","last_clean":"2024-03-26T05:10:58.658589+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:10:58.658589+0000","last_undegraded":"2024-03-26T05:10:58.658589+0000","last_fullsized":"2024-03-26T05:10:58.658589+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:10:48.381743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:48.381743+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:49.809560+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.655914+0000","last_change":"2024-03-26T05:10:55.754903+0000","last_active":"2024-03-26T05:10:58.655914+0000","last_peered":"2024-03-26T05:10:58.655914+0000","last_clean":"2024-03-26T05:10:58.655914+0000","last_became_active":"2024-03-26T05:10:55.754440+0000","last_became_peered":"2024-03-26T05:10:55.754440+0000","last_unstale":"2024-03-26T05:10:58.655914+0000","last_undegraded":"2024-03-26T05:10:58.655914+0000","last_fullsized":"2024-03-26T05:10:58.655914+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:56.709837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":434,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101210+0000","last_change":"2024-03-26T05:07:38.496117+0000","last_active":"2024-03-26T05:10:59.101210+0000","last_peered":"2024-03-26T05:10:59.101210+0000","last_clean":"2024-03-26T05:10:59.101210+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:10:59.101210+0000","last_undegraded":"2024-03-26T05:10:59.101210+0000","last_fullsized":"2024-03-26T05:10:59.101210+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:38.495883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:38.495883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":438,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657548+0000","last_change":"2024-03-26T05:09:56.509351+0000","last_active":"2024-03-26T05:10:58.657548+0000","last_peered":"2024-03-26T05:10:58.657548+0000","last_clean":"2024-03-26T05:10:58.657548+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:10:58.657548+0000","last_undegraded":"2024-03-26T05:10:58.657548+0000","last_fullsized":"2024-03-26T05:10:58.657548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:09:56.509231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":466,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.675359+0000","last_change":"2024-03-26T05:09:45.550058+0000","last_active":"2024-03-26T05:10:59.675359+0000","last_peered":"2024-03-26T05:10:59.675359+0000","last_clean":"2024-03-26T05:10:59.675359+0000","last_became_active":"2024-03-26T05:06:06.521150+0000","last_became_peered":"2024-03-26T05:06:06.521150+0000","last_unstale":"2024-03-26T05:10:59.675359+0000","last_undegraded":"2024-03-26T05:10:59.675359+0000","last_fullsized":"2024-03-26T05:10:59.675359+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:09:45.549936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:04.471132+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"254'31","reported_seq":492,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.659201+0000","last_change":"2024-03-26T05:09:58.433286+0000","last_active":"2024-03-26T05:10:58.659201+0000","last_peered":"2024-03-26T05:10:58.659201+0000","last_clean":"2024-03-26T05:10:58.659201+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:10:58.659201+0000","last_undegraded":"2024-03-26T05:10:58.659201+0000","last_fullsized":"2024-03-26T05:10:58.659201+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:09:58.433038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:58.433038+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:03.060742+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":15,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.656525+0000","last_change":"2024-03-26T05:10:55.755077+0000","last_active":"2024-03-26T05:10:58.656525+0000","last_peered":"2024-03-26T05:10:58.656525+0000","last_clean":"2024-03-26T05:10:58.656525+0000","last_became_active":"2024-03-26T05:10:55.754445+0000","last_became_peered":"2024-03-26T05:10:55.754445+0000","last_unstale":"2024-03-26T05:10:58.656525+0000","last_undegraded":"2024-03-26T05:10:58.656525+0000","last_fullsized":"2024-03-26T05:10:58.656525+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:56.935906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":435,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658812+0000","last_change":"2024-03-26T05:09:38.506191+0000","last_active":"2024-03-26T05:10:58.658812+0000","last_peered":"2024-03-26T05:10:58.658812+0000","last_clean":"2024-03-26T05:10:58.658812+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:10:58.658812+0000","last_undegraded":"2024-03-26T05:10:58.658812+0000","last_fullsized":"2024-03-26T05:10:58.658812+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:38.506070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:38.506070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:01.482865+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"216'11","reported_seq":604,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:59.101595+0000","last_change":"2024-03-26T05:10:30.100941+0000","last_active":"2024-03-26T05:10:59.101595+0000","last_peered":"2024-03-26T05:10:59.101595+0000","last_clean":"2024-03-26T05:10:59.101595+0000","last_became_active":"2024-03-26T05:07:59.960519+0000","last_became_peered":"2024-03-26T05:07:59.960519+0000","last_unstale":"2024-03-26T05:10:59.101595+0000","last_undegraded":"2024-03-26T05:10:59.101595+0000","last_fullsized":"2024-03-26T05:10:59.101595+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:10:30.100859+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.100859+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:56.657116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":451,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.657804+0000","last_change":"2024-03-26T05:09:55.525744+0000","last_active":"2024-03-26T05:10:58.657804+0000","last_peered":"2024-03-26T05:10:58.657804+0000","last_clean":"2024-03-26T05:10:58.657804+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:10:58.657804+0000","last_undegraded":"2024-03-26T05:10:58.657804+0000","last_fullsized":"2024-03-26T05:10:58.657804+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:55.525633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.525633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:02.421989+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"254'20","reported_seq":473,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T05:10:58.658674+0000","last_change":"2024-03-26T05:09:49.460933+0000","last_active":"2024-03-26T05:10:58.658674+0000","last_peered":"2024-03-26T05:10:58.658674+0000","last_clean":"2024-03-26T05:10:58.658674+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:10:58.658674+0000","last_undegraded":"2024-03-26T05:10:58.658674+0000","last_fullsized":"2024-03-26T05:10:58.658674+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:09:49.460784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:49.460784+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:18.489025+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":630,"num_read_kb":625,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1413,"num_read_kb":1138,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":1870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":934,"ondisk_log_size":934,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005188,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":199452,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93545700,"statfs":{"total":95995035648,"available":95790796800,"internally_reserved":0,"allocated":1720320,"data_stored":1274087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":202504650},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332233,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":293848,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93451304,"statfs":{"total":95995035648,"available":95694135296,"internally_reserved":0,"allocated":1716224,"data_stored":1280838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303040,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93442112,"statfs":{"total":95995035648,"available":95684722688,"internally_reserved":0,"allocated":1163264,"data_stored":691404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738536,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":458680,"kb_used_data":116520,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93286472,"statfs":{"total":95995035648,"available":95525347328,"internally_reserved":0,"allocated":119316480,"data_stored":118854091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411486,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298844,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93446308,"statfs":{"total":95995035648,"available":95689019392,"internally_reserved":0,"allocated":1126400,"data_stored":690614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117144,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":361332,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93383820,"statfs":{"total":95995035648,"available":95625031680,"internally_reserved":0,"allocated":1150976,"data_stored":690950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368835015},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920916,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":271084,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":93474068,"statfs":{"total":95995035648,"available":95717445632,"internally_reserved":0,"allocated":1146880,"data_stored":691539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659278,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415452,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93329700,"statfs":{"total":95995035648,"available":95569612800,"internally_reserved":0,"allocated":1126400,"data_stored":690796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:11:08.045 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T05:11:08.045 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:11:08.056 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:08.079 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:08.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:08.180 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:08.204 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:08.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:08.305 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:11:08.306 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:08.324 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:11:08.330 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:08.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:08.343 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:11:08.363 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:11:08.383 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:11:08.403 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:11:08.422 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:11:08.431 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:08.441 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:11:08.443 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:11:08.443 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:11:08.443 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:11:08.443 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:11:08.443 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:11:08.443 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:11:08.443 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:11:08.443 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:11:08.456 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:08.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:08.558 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:08.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:08.582 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:08.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:08.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:08 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1742976953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:08.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:08 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1742976953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:08.680 DEBUG:teuthology.orchestra.run.smithi080:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-26T05:11:08.680 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_2 2024-03-26T05:11:08.681 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:11:08.683 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T05:11:08.683 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T05:11:08.683 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:08.707 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:08.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:08.733 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:hints = 1 2024-03-26T05:11:08.734 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T05:11:08.734 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Object prefix: benchmark_data_smithi080_136801 2024-03-26T05:11:08.734 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:08 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1742976953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:08.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:11:08.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 0 0 0 0 0 0 - 0 2024-03-26T05:11:08.809 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:08.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:08.853 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:08.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:08.907 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:11:08.954 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:09.007 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:09.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:09.109 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:09.147 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:09.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:09.249 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:09.302 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:09.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:09.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:09.317 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:09.404 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:09.441 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:09.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:09.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:09.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:09.581 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:09.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:09.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:09 smithi132 ceph-mon[33500]: pgmap v488: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-03-26T05:11:09.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:09 smithi132 ceph-mon[33500]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:11:09.667 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:09.683 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:09.724 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:09.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:09.734 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 1 16 3159 3143 196.387 196.438 0.00515587 0.0050751 2024-03-26T05:11:09.784 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:09 smithi080 ceph-mon[29691]: pgmap v488: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-03-26T05:11:09.784 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:09 smithi080 ceph-mon[29691]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:11:09.784 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:09 smithi080 ceph-mon[40404]: pgmap v488: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-03-26T05:11:09.785 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:09 smithi080 ceph-mon[40404]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18340 7.1 7:8b7e6025:::benchmark_data_smithi080_80848_object18339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.064655+0000", 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 504.74921028699998, 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.22265895099999999, 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.809 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18340 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064655+0000", 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064655+0000", 2024-03-26T05:11:09.810 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064656+0000", 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.424e-06 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064739+0000", 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2670999999999995e-05 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064754+0000", 2024-03-26T05:11:09.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4780000000000001e-05 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064763+0000", 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8740000000000001e-06 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064819+0000", 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6740999999999998e-05 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.064924+0000", 2024-03-26T05:11:09.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104545 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.065047+0000", 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012353 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.065997+0000", 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00094926299999999999 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.813 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.066103+0000", 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000106575 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287201+0000", 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.22109801900000001 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287266+0000", 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4567999999999996e-05 2024-03-26T05:11:09.814 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287314+0000", 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7960999999999997e-05 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18358 7.1 7:8417ed37:::benchmark_data_smithi080_80848_object18357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.084602+0000", 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 504.729263527, 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20301822899999999, 2024-03-26T05:11:09.815 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18358 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.084602+0000", 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.084602+0000", 2024-03-26T05:11:09.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.084603+0000", 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2330000000000001e-06 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.099940+0000", 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.015337491 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.099960+0000", 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.9471999999999999e-05 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.099967+0000", 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2880000000000004e-06 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.100023+0000", 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6268999999999999e-05 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.100143+0000", 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011924700000000001 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.100270+0000", 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012700999999999999 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.101482+0000", 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001212185 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.101961+0000", 2024-03-26T05:11:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00047930800000000001 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287548+0000", 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.185586844 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287582+0000", 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.3637000000000003e-05 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287620+0000", 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8244999999999997e-05 2024-03-26T05:11:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24701.0:18365 7.1 7:86ff0e4b:::benchmark_data_smithi080_80848_object18364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:02:45.102039+0000", 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 504.71182581400001, 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18571283699999999, 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24701", 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 18365 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.821 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102039+0000", 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102039+0000", 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.822 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102040+0000", 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.139e-06 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102104+0000", 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3891000000000002e-05 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102114+0000", 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9869999999999994e-06 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102122+0000", 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4699999999999996e-06 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102179+0000", 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7643999999999999e-05 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102319+0000", 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013995500000000001 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,6", 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.102422+0000", 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000102944 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.103247+0000", 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082511399999999995 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.103513+0000", 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000265944 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287662+0000", 2024-03-26T05:11:09.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18414857300000001 2024-03-26T05:11:09.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287715+0000", 2024-03-26T05:11:09.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2778999999999997e-05 2024-03-26T05:11:09.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:02:45.287752+0000", 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7397000000000001e-05 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24356.0:1904 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94789128800384 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:05:35.868777+0000", 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 333.945088537, 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.55502071600000003, 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24356", 2024-03-26T05:11:09.827 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 1904 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868777+0000", 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868777+0000", 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868778+0000", 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0410000000000001e-06 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868783+0000", 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0180000000000001e-06 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868793+0000", 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0161e-05 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868800+0000", 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2559999999999996e-06 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868862+0000", 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1681999999999997e-05 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op must wait for map", 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.868868+0000", 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0129999999999999e-06 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.884702+0000", 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.015834014 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:35.884713+0000", 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.076e-05 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:36.409253+0000", 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.524540177 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for degraded object", 2024-03-26T05:11:09.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:36.409736+0000", 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00048286599999999998 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:36.421137+0000", 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011401029 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:36.421240+0000", 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103022 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:05:36.423797+0000", 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002557677 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161558 8.2 8:4e8f99be:::benchmark_data_smithi080_109213_object47041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.515801+0000", 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.298063962, 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.97921734999999999, 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.833 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161558 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.515801+0000", 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.515801+0000", 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.835 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.835 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.515802+0000", 2024-03-26T05:11:09.835 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:09.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1659999999999999e-06 2024-03-26T05:11:09.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.836 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.515807+0000", 2024-03-26T05:11:09.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1370000000000001e-06 2024-03-26T05:11:09.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.515821+0000", 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3627e-05 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.515828+0000", 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7320000000000001e-06 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.515891+0000", 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3089000000000005e-05 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.515902+0000", 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0672999999999999e-05 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.489611+0000", 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.97370939400000001 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.489701+0000", 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0452999999999996e-05 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.489779+0000", 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7175999999999994e-05 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.494901+0000", 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.005122578 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.494947+0000", 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5633000000000002e-05 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.494957+0000", 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0319000000000001e-05 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.494991+0000", 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.3380000000000002e-05 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.495019+0000", 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.7993000000000001e-05 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.840 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161559 8.2 8:48d62e8c:::benchmark_data_smithi080_109213_object47042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.516279+0000", 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.29758571000001, 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.97890422700000002, 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161559 2024-03-26T05:11:09.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.516279+0000", 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.516279+0000", 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.516281+0000", 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.141e-06 2024-03-26T05:11:09.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.516288+0000", 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3780000000000003e-06 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.516301+0000", 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2962000000000001e-05 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.516309+0000", 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8990000000000001e-06 2024-03-26T05:11:09.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.516375+0000", 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6376000000000003e-05 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.516386+0000", 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0838e-05 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.491967+0000", 2024-03-26T05:11:09.844 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.97558080400000002 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.492047+0000", 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0520999999999997e-05 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.492137+0000", 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.9190000000000005e-05 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.845 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.495072+0000", 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0029358050000000001 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.495100+0000", 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.7628999999999999e-05 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.495131+0000", 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0722000000000002e-05 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.846 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.495156+0000", 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.5290999999999999e-05 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.495184+0000", 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.7671e-05 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161579 8.2 8:45f0800c:::benchmark_data_smithi080_109213_object47062:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.547727+0000", 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.266138016, 2024-03-26T05:11:09.847 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.95585242699999995, 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161579 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.547727+0000", 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.848 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.547727+0000", 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.547729+0000", 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.7090000000000001e-06 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.547735+0000", 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5069999999999997e-06 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.849 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.547752+0000", 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6181999999999999e-05 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.547760+0000", 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6000000000000007e-06 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.547834+0000", 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4258999999999995e-05 2024-03-26T05:11:09.850 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.547846+0000", 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1544e-05 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.493703+0000", 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.94585670099999997 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.493771+0000", 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8423000000000005e-05 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.493836+0000", 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4467999999999993e-05 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.495192+0000", 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0013561580000000001 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503394+0000", 2024-03-26T05:11:09.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0082020670000000004 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503450+0000", 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6100999999999997e-05 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503513+0000", 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3472000000000005e-05 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503580+0000", 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6235999999999994e-05 2024-03-26T05:11:09.853 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161597 8.2 8:4db2e94d:::benchmark_data_smithi080_109213_object47080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.568692+0000", 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.24517317199999, 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.93523727800000001, 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.854 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161597 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.568692+0000", 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.568692+0000", 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.568694+0000", 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.8899999999999999e-06 2024-03-26T05:11:09.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.568704+0000", 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0225e-05 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.568726+0000", 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.1713999999999999e-05 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.568735+0000", 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6269999999999995e-06 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.568867+0000", 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013119800000000001 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.568875+0000", 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7649999999999999e-06 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.495216+0000", 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92634063300000002 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.495276+0000", 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0164000000000002e-05 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.495338+0000", 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2226999999999999e-05 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503594+0000", 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0082559880000000006 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.858 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503662+0000", 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7883e-05 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503822+0000", 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016008400000000001 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503881+0000", 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8653999999999998e-05 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503929+0000", 2024-03-26T05:11:09.859 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8226000000000002e-05 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161603 8.2 8:4f00e5fb:::benchmark_data_smithi080_109213_object47086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.576329+0000", 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.237536542, 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92775187100000001, 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161603 2024-03-26T05:11:09.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.576329+0000", 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.576329+0000", 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.576330+0000", 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3799999999999999e-06 2024-03-26T05:11:09.861 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.576336+0000", 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3960000000000003e-06 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.576351+0000", 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4652e-05 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.576359+0000", 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8439999999999998e-06 2024-03-26T05:11:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.576412+0000", 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3585999999999998e-05 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.576420+0000", 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4059999999999999e-06 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.497041+0000", 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.920620726 2024-03-26T05:11:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.497159+0000", 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011882100000000001 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.497294+0000", 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000134685 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503602+0000", 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0063081739999999997 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503714+0000", 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000111296 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503988+0000", 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00027433699999999999 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.504045+0000", 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6972000000000003e-05 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.504080+0000", 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5596000000000003e-05 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161605 8.2 8:4ee7f3c4:::benchmark_data_smithi080_109213_object47088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.580872+0000", 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.23299299599999, 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92337200399999997, 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161605 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.580872+0000", 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.580872+0000", 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.580874+0000", 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4649999999999999e-06 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.580882+0000", 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.1850000000000004e-06 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.580896+0000", 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4012999999999999e-05 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.580904+0000", 2024-03-26T05:11:09.868 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0430000000000008e-06 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.580965+0000", 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1235999999999994e-05 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.580972+0000", 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.455e-06 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.499904+0000", 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.918932525 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.869 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.500031+0000", 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000127083 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.500159+0000", 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012809400000000001 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503729+0000", 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0035698639999999998 2024-03-26T05:11:09.870 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.503771+0000", 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1461000000000002e-05 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.504125+0000", 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035450299999999999 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.504205+0000", 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9905000000000002e-05 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.871 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.504244+0000", 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9171999999999997e-05 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161613 8.2 8:4f455ce3:::benchmark_data_smithi080_109213_object47096:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.593626+0000", 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.220239306, 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92641095299999998, 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.872 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.873 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161613 2024-03-26T05:11:09.873 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.873 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.873 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.873 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.873 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593626+0000", 2024-03-26T05:11:09.874 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.874 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.874 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.874 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.874 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593626+0000", 2024-03-26T05:11:09.874 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.874 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593627+0000", 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3790000000000001e-06 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593633+0000", 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.088e-06 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593648+0000", 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4790000000000001e-05 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.875 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593656+0000", 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5889999999999997e-06 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593725+0000", 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9197999999999997e-05 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593732+0000", 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5259999999999996e-06 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.876 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.504283+0000", 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.91055014899999998 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.504425+0000", 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014258599999999999 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.504552+0000", 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012664600000000001 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.877 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.514122+0000", 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0095697990000000004 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.516711+0000", 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00258939 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.519936+0000", 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0032252650000000002 2024-03-26T05:11:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.519988+0000", 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1601999999999998e-05 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.520037+0000", 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8946000000000002e-05 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.879 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161614 8.2 8:4b2a961c:::benchmark_data_smithi080_109213_object47097:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.593675+0000", 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.22019001, 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.93633813499999996, 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161614 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.880 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593675+0000", 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593675+0000", 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593676+0000", 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6400000000000003e-07 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.881 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593678+0000", 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.728e-06 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593685+0000", 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5180000000000001e-06 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593689+0000", 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6880000000000001e-06 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.882 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593758+0000", 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9423000000000003e-05 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593766+0000", 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.49e-06 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.507245+0000", 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.91347835399999999 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.883 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.507365+0000", 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120674 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.507484+0000", 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011851 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.519847+0000", 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01236282 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.884 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.525099+0000", 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0052525150000000001 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.529773+0000", 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0046737269999999999 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.529843+0000", 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9799e-05 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.885 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.886 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.886 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530013+0000", 2024-03-26T05:11:09.886 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017042500000000001 2024-03-26T05:11:09.886 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.886 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.886 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.886 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.886 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.886 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161615 8.2 8:4e3b70b7:::benchmark_data_smithi080_109213_object47098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.886 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.593745+0000", 2024-03-26T05:11:09.886 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.220120313, 2024-03-26T05:11:09.886 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.93655654200000005, 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161615 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593745+0000", 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593745+0000", 2024-03-26T05:11:09.887 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593745+0000", 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3600000000000004e-07 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593752+0000", 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3169999999999998e-06 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593759+0000", 2024-03-26T05:11:09.888 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3209999999999999e-06 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593765+0000", 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.088e-06 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593793+0000", 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.8189000000000001e-05 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.593797+0000", 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.467e-06 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.889 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.510120+0000", 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.91632354400000005 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.510243+0000", 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012265900000000001 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.510372+0000", 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012937199999999999 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.890 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.519894+0000", 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0095220779999999998 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.525104+0000", 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0052093959999999998 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530219+0000", 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0051155300000000001 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.891 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530266+0000", 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6962000000000002e-05 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530301+0000", 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5083000000000003e-05 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161629 8.2 8:4c26b1eb:::benchmark_data_smithi080_109213_object47112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.608333+0000", 2024-03-26T05:11:09.892 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.205531893, 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92208931500000002, 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161629 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.608333+0000", 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.893 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.608333+0000", 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.608335+0000", 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5239999999999999e-06 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.608341+0000", 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3969999999999999e-06 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.894 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.608358+0000", 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.7132000000000001e-05 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.608367+0000", 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.1630000000000002e-06 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.608465+0000", 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7690999999999997e-05 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.895 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.608477+0000", 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2071e-05 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.512177+0000", 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.90370013500000002 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.512280+0000", 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000102774 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.896 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.512377+0000", 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6922999999999993e-05 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.525108+0000", 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.012731170999999999 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.525150+0000", 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2163000000000003e-05 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.897 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530337+0000", 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0051864750000000003 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530390+0000", 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3378999999999998e-05 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530423+0000", 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2317000000000002e-05 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.898 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161641 8.2 8:4f43c5cd:::benchmark_data_smithi080_109213_object47124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.625420+0000", 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.18844513800001, 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.90513500899999999, 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161641 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.625420+0000", 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.625420+0000", 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.900 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.625421+0000", 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.406e-06 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.625431+0000", 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5000000000000005e-06 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.625445+0000", 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3784000000000001e-05 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.901 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.625454+0000", 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0010000000000007e-06 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.625519+0000", 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5402999999999997e-05 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.625531+0000", 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.152e-05 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.902 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.514170+0000", 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.88863941000000002 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.514284+0000", 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000114418 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.514379+0000", 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4413000000000006e-05 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.903 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.525188+0000", 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010808760000000001 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.525195+0000", 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8599999999999993e-06 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530467+0000", 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0052716259999999997 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.904 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530512+0000", 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4666999999999998e-05 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530555+0000", 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3241000000000002e-05 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161645 8.2 8:4d864c9b:::benchmark_data_smithi080_109213_object47128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.630483+0000", 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.18338241799999, 2024-03-26T05:11:09.905 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.90023133200000005, 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161645 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.630483+0000", 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.906 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.630483+0000", 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.630484+0000", 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.229e-06 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.630489+0000", 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.48e-06 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.907 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.630502+0000", 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2880999999999999e-05 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.630509+0000", 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6320000000000002e-06 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.630569+0000", 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9626999999999997e-05 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.630580+0000", 2024-03-26T05:11:09.908 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1829e-05 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.516759+0000", 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.88617869199999999 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.516878+0000", 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000118886 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.909 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.517012+0000", 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013366699999999999 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.525200+0000", 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081884899999999997 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.525239+0000", 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9032999999999997e-05 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.910 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530626+0000", 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.005386399 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530676+0000", 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0862000000000002e-05 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530714+0000", 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7625e-05 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.912 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161664 8.2 8:4521494f:::benchmark_data_smithi080_109213_object47147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.912 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.656975+0000", 2024-03-26T05:11:09.912 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.156890307, 2024-03-26T05:11:09.912 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.87898795299999999, 2024-03-26T05:11:09.912 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.912 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.912 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.912 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.913 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161664 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.656975+0000", 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.656975+0000", 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.914 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.656977+0000", 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.114e-06 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.656987+0000", 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8859999999999999e-06 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.657017+0000", 2024-03-26T05:11:09.915 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.9785999999999999e-05 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.657033+0000", 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6612999999999998e-05 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.657151+0000", 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000117637 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.657175+0000", 2024-03-26T05:11:09.916 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.3862999999999998e-05 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.520065+0000", 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.86289045799999997 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.520201+0000", 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013545000000000001 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.520322+0000", 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012174499999999999 2024-03-26T05:11:09.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.525204+0000", 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0048813329999999999 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.529710+0000", 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.004506342 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.535885+0000", 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0061745389999999997 2024-03-26T05:11:09.918 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.535933+0000", 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8859000000000001e-05 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.535963+0000", 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.9328000000000001e-05 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.919 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161665 8.2 8:4f5c10fc:::benchmark_data_smithi080_109213_object47148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.657076+0000", 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.156789609, 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.87907144100000001, 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161665 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.657076+0000", 2024-03-26T05:11:09.920 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.657076+0000", 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.657077+0000", 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.44e-06 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.657084+0000", 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.567e-06 2024-03-26T05:11:09.921 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.657103+0000", 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.9639e-05 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.657119+0000", 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5773e-05 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.657289+0000", 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000170235 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.922 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.657298+0000", 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0289999999999994e-06 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.522874+0000", 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.86557616299999995 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.523016+0000", 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014158700000000001 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.923 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.523146+0000", 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000130097 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.529727+0000", 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006581015 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530066+0000", 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00033900400000000001 2024-03-26T05:11:09.924 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.536034+0000", 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0059683870000000003 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.536107+0000", 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2427000000000003e-05 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.536147+0000", 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0077999999999998e-05 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.925 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161667 8.2 8:4e704fdf:::benchmark_data_smithi080_109213_object47150:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.660526+0000", 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.153338815, 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.88030389399999998, 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161667 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.926 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.660526+0000", 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.660526+0000", 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.660527+0000", 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8100000000000001e-07 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.927 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.660535+0000", 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0290000000000005e-06 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.660551+0000", 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5525000000000001e-05 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.660560+0000", 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.1689999999999998e-06 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.660624+0000", 2024-03-26T05:11:09.928 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3643e-05 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.660629+0000", 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3000000000000001e-06 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.525268+0000", 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.864638566 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.525342+0000", 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4737e-05 2024-03-26T05:11:09.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.525455+0000", 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000112797 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530028+0000", 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045731510000000001 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530179+0000", 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000150884 2024-03-26T05:11:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.540716+0000", 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010536413999999999 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.540772+0000", 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6017999999999998e-05 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.540830+0000", 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8680000000000001e-05 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.931 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:161670 8.2 8:4d204339:::benchmark_data_smithi080_109213_object47153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:16.670451+0000", 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.143414314, 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.87054631000000005, 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 161670 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:09.932 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.670451+0000", 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.670451+0000", 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.670452+0000", 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6099999999999999e-07 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.933 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.670456+0000", 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3109999999999999e-06 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.670468+0000", 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1912999999999999e-05 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.670474+0000", 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9139999999999996e-06 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.670527+0000", 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3526999999999998e-05 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:16.670534+0000", 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5250000000000002e-06 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.527125+0000", 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.85659110500000002 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.935 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.527241+0000", 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115809 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.527367+0000", 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012592300000000001 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530080+0000", 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0027134390000000002 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.936 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.530785+0000", 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070496999999999997 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.540882+0000", 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010096635 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.540947+0000", 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4943999999999993e-05 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:17.540997+0000", 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0433999999999999e-05 2024-03-26T05:11:09.937 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.938 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.938 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.938 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:09.938 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:09.938 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:11:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:09.939 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:09.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:10.039 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:10.077 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:10.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:10.178 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:10.215 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:10.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:10.316 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:10.352 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:10.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:10.454 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:10.489 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:10.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:10.591 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:10.631 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:10.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:10.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:10.732 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:10.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 2 16 6539 6523 203.807 211.25 0.00548908 0.00489723 2024-03-26T05:11:10.771 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:10.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:10.873 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:10.909 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:10.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:10.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:10.981 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T05:11:11.011 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:11.061 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:11.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:11.163 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:11.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:11.187 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:11.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:11.280 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:11:11.288 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:11.323 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:11.365 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:11.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:11.468 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:11.501 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:11.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:11.603 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:11.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:11 smithi080 ceph-mon[29691]: pgmap v489: 148 pgs: 148 active+clean; 101 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 13 MiB/s wr, 202 op/s 2024-03-26T05:11:11.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:11 smithi080 ceph-mon[40404]: pgmap v489: 148 pgs: 148 active+clean; 101 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 13 MiB/s wr, 202 op/s 2024-03-26T05:11:11.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:11.640 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:11.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:11.734 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 3 15 9276 9261 192.909 171.125 0.00507439 0.00517375 2024-03-26T05:11:11.741 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:11.761 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:11 smithi132 ceph-mon[33500]: pgmap v489: 148 pgs: 148 active+clean; 101 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 13 MiB/s wr, 202 op/s 2024-03-26T05:11:11.789 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:11.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:11.891 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:11.927 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:11.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:12.028 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:12.062 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:12.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:12.163 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:12.205 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:12.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:12.306 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:12.329 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:12.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:12.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:12.431 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:12.469 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:12.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:12.572 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:12.609 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:12.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:12.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:12.623 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:12.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:12.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 4 16 11671 11655 182.081 149.625 0.0053751 0.00548249 2024-03-26T05:11:12.746 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:12.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:12.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:12.847 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:12.883 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:12.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:12.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:12.985 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:13.020 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:13.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:13.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:13.126 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:11:13.128 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:13.131 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:11:13.172 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:13.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:13.274 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:13.308 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:13.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:13.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:13.410 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:13.445 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:13.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:13.547 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:13.590 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:13.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:13.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:13.614 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:13 smithi132 ceph-mon[33500]: pgmap v490: 148 pgs: 148 active+clean; 316 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 39 MiB/s wr, 632 op/s 2024-03-26T05:11:13.692 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:13.729 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:13.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:13.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 5 15 14331 14316 178.925 166.312 0.00657317 0.00558301 2024-03-26T05:11:13.825 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:11:13.825 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:11:13.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.825 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.25268.0:14568 9.a 9:5990b3e7:::benchmark_data_smithi080_136801_object14567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-26T05:11:13.825 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:11:13.827925+0000", 2024-03-26T05:11:13.825 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0047644929999999999, 2024-03-26T05:11:13.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.004800726, 2024-03-26T05:11:13.825 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:13.825 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:11:13.825 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:13.825 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.25268", 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 14568 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.827925+0000", 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.827925+0000", 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:13.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.827926+0000", 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0299999999999997e-07 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.827971+0000", 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4839999999999998e-05 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.827982+0000", 2024-03-26T05:11:13.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1316e-05 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.827990+0000", 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3049999999999993e-06 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.828047+0000", 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7096999999999999e-05 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.828156+0000", 2024-03-26T05:11:13.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108569 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,7", 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.828279+0000", 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000123523 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.829982+0000", 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001702739 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.829995+0000", 2024-03-26T05:11:13.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2774e-05 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.25268.0:14576 9.a 9:5b8d0cd4:::benchmark_data_smithi080_136801_object14575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:11:13.830264+0000", 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.002424837, 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0025662269999999999, 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.25268", 2024-03-26T05:11:13.830 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 14576 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.830264+0000", 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.830264+0000", 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.830265+0000", 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7300000000000005e-07 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.830329+0000", 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3975999999999999e-05 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.830339+0000", 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.012e-05 2024-03-26T05:11:13.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.830347+0000", 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2089999999999999e-06 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.830399+0000", 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2831999999999998e-05 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.830513+0000", 2024-03-26T05:11:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011333299999999999 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,7", 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.830655+0000", 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000141822 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.25268.0:14573 9.b e271/265 9:dae29063:::benchmark_data_smithi080_136801_object14572:head v 271'906, mlcod=271'904)", 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:11:13.830994+0000", 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.001695053, 2024-03-26T05:11:13.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0019191430000000001, 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.830994+0000", 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.830994+0000", 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:13.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.830995+0000", 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.61e-07 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.831036+0000", 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0784000000000001e-05 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.831048+0000", 2024-03-26T05:11:13.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2493e-05 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.831055+0000", 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6619999999999999e-06 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.831313+0000", 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000258473 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:13.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.831334+0000", 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.052e-05 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.25268.0:14575 9.9 e271/265 9:9bb5de74:::benchmark_data_smithi080_136801_object14574:head v 271'956, mlcod=271'954)", 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:11:13.831370+0000", 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0013190599999999999, 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001614047, 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:13.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.831370+0000", 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.831370+0000", 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.831371+0000", 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3e-07 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.831764+0000", 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00039260899999999998 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.831774+0000", 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0429e-05 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.831781+0000", 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1709999999999996e-06 2024-03-26T05:11:13.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.832186+0000", 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040463899999999999 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:11:13.832205+0000", 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.8810000000000001e-05 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-03-26T05:11:13.841 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 4 2024-03-26T05:11:13.842 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:11:13.842 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:13.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:13 smithi080 ceph-mon[40404]: pgmap v490: 148 pgs: 148 active+clean; 316 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 39 MiB/s wr, 632 op/s 2024-03-26T05:11:13.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:13 smithi080 ceph-mon[29691]: pgmap v490: 148 pgs: 148 active+clean; 316 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 39 MiB/s wr, 632 op/s 2024-03-26T05:11:13.880 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:13.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:13.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:13.981 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:14.013 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:14.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:14.115 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:14.151 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:14.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:14.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:14.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:14.288 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:14.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:14.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:11:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:11:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:11:14.389 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:14.419 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:14.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:14.521 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:14.561 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:14.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:14.663 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:14.704 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:14.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:14.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:14.716 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T05:11:14.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 6 16 17119 17103 178.133 174.188 0.00834294 0.00560732 2024-03-26T05:11:14.806 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:14.847 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:14.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:14.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:14.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:14 smithi080 ceph-mon[29691]: from='client.25274 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:11:14.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:14 smithi080 ceph-mon[40404]: from='client.25274 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:11:14.950 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:14.987 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:14.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:14.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:15.004 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:14 smithi132 ceph-mon[33500]: from='client.25274 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:11:15.089 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:15.128 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:15.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:15.175 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:15.229 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:15.256 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-03-26T05:11:11.922212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331618654,"num_objects":5281,"num_object_clones":0,"num_object_copies":15843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5281,"num_whiteouts":0,"num_read":2289,"num_read_kb":2103,"num_write":6144,"num_write_kb":325725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1774542,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6209,"ondisk_log_size":6209,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3299068,"kb_used_data":788156,"kb_used_omap":12,"kb_used_meta":2510835,"kb_avail":746662148,"statfs":{"total":767960285184,"available":764582039552,"internally_reserved":0,"allocated":807071744,"data_stored":803370322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2571095600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":331022336,"num_objects":5051,"num_object_clones":0,"num_object_copies":15153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5051,"num_write_kb":323264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002092"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":453,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499030+0000","last_change":"2024-03-26T05:09:39.468117+0000","last_active":"2024-03-26T05:11:06.499030+0000","last_peered":"2024-03-26T05:11:06.499030+0000","last_clean":"2024-03-26T05:11:06.499030+0000","last_became_active":"2024-03-26T05:07:59.996436+0000","last_became_peered":"2024-03-26T05:07:59.996436+0000","last_unstale":"2024-03-26T05:11:06.499030+0000","last_undegraded":"2024-03-26T05:11:06.499030+0000","last_fullsized":"2024-03-26T05:11:06.499030+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:39.468037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:39.468037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.559083+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":455,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676266+0000","last_change":"2024-03-26T05:09:44.530686+0000","last_active":"2024-03-26T05:11:06.676266+0000","last_peered":"2024-03-26T05:11:06.676266+0000","last_clean":"2024-03-26T05:11:06.676266+0000","last_became_active":"2024-03-26T05:07:59.997089+0000","last_became_peered":"2024-03-26T05:07:59.997089+0000","last_unstale":"2024-03-26T05:11:06.676266+0000","last_undegraded":"2024-03-26T05:11:06.676266+0000","last_fullsized":"2024-03-26T05:11:06.676266+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:44.530563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:44.530563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.884815+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":466,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676482+0000","last_change":"2024-03-26T05:09:24.726025+0000","last_active":"2024-03-26T05:11:06.676482+0000","last_peered":"2024-03-26T05:11:06.676482+0000","last_clean":"2024-03-26T05:11:06.676482+0000","last_became_active":"2024-03-26T05:07:59.994431+0000","last_became_peered":"2024-03-26T05:07:59.994431+0000","last_unstale":"2024-03-26T05:11:06.676482+0000","last_undegraded":"2024-03-26T05:11:06.676482+0000","last_fullsized":"2024-03-26T05:11:06.676482+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:24.725911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:24.725911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:10.038602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"254'21","reported_seq":486,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499022+0000","last_change":"2024-03-26T05:09:55.417464+0000","last_active":"2024-03-26T05:11:06.499022+0000","last_peered":"2024-03-26T05:11:06.499022+0000","last_clean":"2024-03-26T05:11:06.499022+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:11:06.499022+0000","last_undegraded":"2024-03-26T05:11:06.499022+0000","last_fullsized":"2024-03-26T05:11:06.499022+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:09:55.417325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.417325+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:24.296213+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":431,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526701+0000","last_change":"2024-03-26T05:09:59.528706+0000","last_active":"2024-03-26T05:11:06.526701+0000","last_peered":"2024-03-26T05:11:06.526701+0000","last_clean":"2024-03-26T05:11:06.526701+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:11:06.526701+0000","last_undegraded":"2024-03-26T05:11:06.526701+0000","last_fullsized":"2024-03-26T05:11:06.526701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:59.528596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:59.528596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:25.643478+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":445,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.498893+0000","last_change":"2024-03-26T05:10:29.404548+0000","last_active":"2024-03-26T05:11:06.498893+0000","last_peered":"2024-03-26T05:11:06.498893+0000","last_clean":"2024-03-26T05:11:06.498893+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:11:06.498893+0000","last_undegraded":"2024-03-26T05:11:06.498893+0000","last_fullsized":"2024-03-26T05:11:06.498893+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:29.404404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.404404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:51.272613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":452,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526513+0000","last_change":"2024-03-26T05:09:40.635212+0000","last_active":"2024-03-26T05:11:06.526513+0000","last_peered":"2024-03-26T05:11:06.526513+0000","last_clean":"2024-03-26T05:11:06.526513+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:11:06.526513+0000","last_undegraded":"2024-03-26T05:11:06.526513+0000","last_fullsized":"2024-03-26T05:11:06.526513+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:40.635129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:09:40.635129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"254'51","reported_seq":530,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526390+0000","last_change":"2024-03-26T05:07:46.642266+0000","last_active":"2024-03-26T05:11:06.526390+0000","last_peered":"2024-03-26T05:11:06.526390+0000","last_clean":"2024-03-26T05:11:06.526390+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:11:06.526390+0000","last_undegraded":"2024-03-26T05:11:06.526390+0000","last_fullsized":"2024-03-26T05:11:06.526390+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'39","last_scrub_stamp":"2024-03-26T05:07:46.641260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:46.641260+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.460082+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":449,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676241+0000","last_change":"2024-03-26T05:10:30.211104+0000","last_active":"2024-03-26T05:11:06.676241+0000","last_peered":"2024-03-26T05:11:06.676241+0000","last_clean":"2024-03-26T05:11:06.676241+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:11:06.676241+0000","last_undegraded":"2024-03-26T05:11:06.676241+0000","last_fullsized":"2024-03-26T05:11:06.676241+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:30.211007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.211007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:37.046198+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":133,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676485+0000","last_change":"2024-03-26T05:09:57.144879+0000","last_active":"2024-03-26T05:11:06.676485+0000","last_peered":"2024-03-26T05:11:06.676485+0000","last_clean":"2024-03-26T05:11:06.676485+0000","last_became_active":"2024-03-26T05:08:00.000128+0000","last_became_peered":"2024-03-26T05:08:00.000128+0000","last_unstale":"2024-03-26T05:11:06.676485+0000","last_undegraded":"2024-03-26T05:11:06.676485+0000","last_fullsized":"2024-03-26T05:11:06.676485+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:57.144806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:57.144806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:10.794551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":133,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677477+0000","last_change":"2024-03-26T05:10:29.515852+0000","last_active":"2024-03-26T05:11:06.677477+0000","last_peered":"2024-03-26T05:11:06.677477+0000","last_clean":"2024-03-26T05:11:06.677477+0000","last_became_active":"2024-03-26T05:07:59.947557+0000","last_became_peered":"2024-03-26T05:07:59.947557+0000","last_unstale":"2024-03-26T05:11:06.677477+0000","last_undegraded":"2024-03-26T05:11:06.677477+0000","last_fullsized":"2024-03-26T05:11:06.677477+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:29.515767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.515767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:52.847327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"254'27","reported_seq":498,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499159+0000","last_change":"2024-03-26T05:09:33.456052+0000","last_active":"2024-03-26T05:11:06.499159+0000","last_peered":"2024-03-26T05:11:06.499159+0000","last_clean":"2024-03-26T05:11:06.499159+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:11:06.499159+0000","last_undegraded":"2024-03-26T05:11:06.499159+0000","last_fullsized":"2024-03-26T05:11:06.499159+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'21","last_scrub_stamp":"2024-03-26T05:09:33.455867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:33.455867+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.559052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":133,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676029+0000","last_change":"2024-03-26T05:09:48.186435+0000","last_active":"2024-03-26T05:11:06.676029+0000","last_peered":"2024-03-26T05:11:06.676029+0000","last_clean":"2024-03-26T05:11:06.676029+0000","last_became_active":"2024-03-26T05:07:59.998777+0000","last_became_peered":"2024-03-26T05:07:59.998777+0000","last_unstale":"2024-03-26T05:11:06.676029+0000","last_undegraded":"2024-03-26T05:11:06.676029+0000","last_fullsized":"2024-03-26T05:11:06.676029+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:48.186287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.186287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.534209+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":457,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677570+0000","last_change":"2024-03-26T05:10:41.797043+0000","last_active":"2024-03-26T05:11:06.677570+0000","last_peered":"2024-03-26T05:11:06.677570+0000","last_clean":"2024-03-26T05:11:06.677570+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:11:06.677570+0000","last_undegraded":"2024-03-26T05:11:06.677570+0000","last_fullsized":"2024-03-26T05:11:06.677570+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:41.796930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.796930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:56.003329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":125,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677289+0000","last_change":"2024-03-26T05:08:00.011593+0000","last_active":"2024-03-26T05:11:06.677289+0000","last_peered":"2024-03-26T05:11:06.677289+0000","last_clean":"2024-03-26T05:11:06.677289+0000","last_became_active":"2024-03-26T05:08:00.011262+0000","last_became_peered":"2024-03-26T05:08:00.011262+0000","last_unstale":"2024-03-26T05:11:06.677289+0000","last_undegraded":"2024-03-26T05:11:06.677289+0000","last_fullsized":"2024-03-26T05:11:06.677289+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:53.514177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.514177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.551188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"254'13","reported_seq":474,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499185+0000","last_change":"2024-03-26T05:10:32.425969+0000","last_active":"2024-03-26T05:11:06.499185+0000","last_peered":"2024-03-26T05:11:06.499185+0000","last_clean":"2024-03-26T05:11:06.499185+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:11:06.499185+0000","last_undegraded":"2024-03-26T05:11:06.499185+0000","last_fullsized":"2024-03-26T05:11:06.499185+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'13","last_scrub_stamp":"2024-03-26T05:10:32.425855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:32.425855+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:48.496500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":462,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499608+0000","last_change":"2024-03-26T05:09:56.453730+0000","last_active":"2024-03-26T05:11:06.499608+0000","last_peered":"2024-03-26T05:11:06.499608+0000","last_clean":"2024-03-26T05:11:06.499608+0000","last_became_active":"2024-03-26T05:07:59.942829+0000","last_became_peered":"2024-03-26T05:07:59.942829+0000","last_unstale":"2024-03-26T05:11:06.499608+0000","last_undegraded":"2024-03-26T05:11:06.499608+0000","last_fullsized":"2024-03-26T05:11:06.499608+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:56.453608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:09:56.453608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:14.181663+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":443,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677155+0000","last_change":"2024-03-26T05:09:57.476795+0000","last_active":"2024-03-26T05:11:06.677155+0000","last_peered":"2024-03-26T05:11:06.677155+0000","last_clean":"2024-03-26T05:11:06.677155+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:11:06.677155+0000","last_undegraded":"2024-03-26T05:11:06.677155+0000","last_fullsized":"2024-03-26T05:11:06.677155+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:57.476689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:57.476689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:15.765524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676214+0000","last_change":"2024-03-26T05:10:38.179665+0000","last_active":"2024-03-26T05:11:06.676214+0000","last_peered":"2024-03-26T05:11:06.676214+0000","last_clean":"2024-03-26T05:11:06.676214+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:11:06.676214+0000","last_undegraded":"2024-03-26T05:11:06.676214+0000","last_fullsized":"2024-03-26T05:11:06.676214+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:38.179521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:10:38.179521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:45.972937+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"254'27","reported_seq":498,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.500822+0000","last_change":"2024-03-26T05:09:48.440867+0000","last_active":"2024-03-26T05:11:06.500822+0000","last_peered":"2024-03-26T05:11:06.500822+0000","last_clean":"2024-03-26T05:11:06.500822+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:11:06.500822+0000","last_undegraded":"2024-03-26T05:11:06.500822+0000","last_fullsized":"2024-03-26T05:11:06.500822+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:09:48.440764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.440764+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":452,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526980+0000","last_change":"2024-03-26T05:09:45.842503+0000","last_active":"2024-03-26T05:11:06.526980+0000","last_peered":"2024-03-26T05:11:06.526980+0000","last_clean":"2024-03-26T05:11:06.526980+0000","last_became_active":"2024-03-26T05:07:59.959705+0000","last_became_peered":"2024-03-26T05:07:59.959705+0000","last_unstale":"2024-03-26T05:11:06.526980+0000","last_undegraded":"2024-03-26T05:11:06.526980+0000","last_fullsized":"2024-03-26T05:11:06.526980+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:45.842392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.842392+0000","last_clean_scrub_stamp":"2024-03-26T05:09:45.842392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.460115+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":445,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499592+0000","last_change":"2024-03-26T05:09:54.412717+0000","last_active":"2024-03-26T05:11:06.499592+0000","last_peered":"2024-03-26T05:11:06.499592+0000","last_clean":"2024-03-26T05:11:06.499592+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:11:06.499592+0000","last_undegraded":"2024-03-26T05:11:06.499592+0000","last_fullsized":"2024-03-26T05:11:06.499592+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:54.412617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:54.412617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":469,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499553+0000","last_change":"2024-03-26T05:10:45.477455+0000","last_active":"2024-03-26T05:11:06.499553+0000","last_peered":"2024-03-26T05:11:06.499553+0000","last_clean":"2024-03-26T05:11:06.499553+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:11:06.499553+0000","last_undegraded":"2024-03-26T05:11:06.499553+0000","last_fullsized":"2024-03-26T05:11:06.499553+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:45.477372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:10:45.477372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:48.573961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"254'26","reported_seq":485,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526578+0000","last_change":"2024-03-26T05:07:50.662703+0000","last_active":"2024-03-26T05:11:06.526578+0000","last_peered":"2024-03-26T05:11:06.526578+0000","last_clean":"2024-03-26T05:11:06.526578+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:11:06.526578+0000","last_undegraded":"2024-03-26T05:11:06.526578+0000","last_fullsized":"2024-03-26T05:11:06.526578+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:07:50.662564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:50.662564+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.460033+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":466,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677957+0000","last_change":"2024-03-26T05:10:32.752769+0000","last_active":"2024-03-26T05:11:06.677957+0000","last_peered":"2024-03-26T05:11:06.677957+0000","last_clean":"2024-03-26T05:11:06.677957+0000","last_became_active":"2024-03-26T05:07:59.955125+0000","last_became_peered":"2024-03-26T05:07:59.955125+0000","last_unstale":"2024-03-26T05:11:06.677957+0000","last_undegraded":"2024-03-26T05:11:06.677957+0000","last_fullsized":"2024-03-26T05:11:06.677957+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:10:32.752678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:32.752678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:49.036523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":600,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676766+0000","last_change":"2024-03-26T05:09:31.493239+0000","last_active":"2024-03-26T05:11:06.676766+0000","last_peered":"2024-03-26T05:11:06.676766+0000","last_clean":"2024-03-26T05:11:06.676766+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:11:06.676766+0000","last_undegraded":"2024-03-26T05:11:06.676766+0000","last_fullsized":"2024-03-26T05:11:06.676766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:09:31.493102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:31.493102+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:12.760290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":452,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676399+0000","last_change":"2024-03-26T05:09:55.509651+0000","last_active":"2024-03-26T05:11:06.676399+0000","last_peered":"2024-03-26T05:11:06.676399+0000","last_clean":"2024-03-26T05:11:06.676399+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:11:06.676399+0000","last_undegraded":"2024-03-26T05:11:06.676399+0000","last_fullsized":"2024-03-26T05:11:06.676399+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:09:55.509344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.509344+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"254'26","reported_seq":500,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676474+0000","last_change":"2024-03-26T05:10:41.134236+0000","last_active":"2024-03-26T05:11:06.676474+0000","last_peered":"2024-03-26T05:11:06.676474+0000","last_clean":"2024-03-26T05:11:06.676474+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:11:06.676474+0000","last_undegraded":"2024-03-26T05:11:06.676474+0000","last_fullsized":"2024-03-26T05:11:06.676474+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:10:41.134129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.134129+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.559878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":451,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677237+0000","last_change":"2024-03-26T05:09:46.565079+0000","last_active":"2024-03-26T05:11:06.677237+0000","last_peered":"2024-03-26T05:11:06.677237+0000","last_clean":"2024-03-26T05:11:06.677237+0000","last_became_active":"2024-03-26T05:07:59.995261+0000","last_became_peered":"2024-03-26T05:07:59.995261+0000","last_unstale":"2024-03-26T05:11:06.677237+0000","last_undegraded":"2024-03-26T05:11:06.677237+0000","last_fullsized":"2024-03-26T05:11:06.677237+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:46.564952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:46.564952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:07.322416+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676730+0000","last_change":"2024-03-26T05:09:36.861688+0000","last_active":"2024-03-26T05:11:06.676730+0000","last_peered":"2024-03-26T05:11:06.676730+0000","last_clean":"2024-03-26T05:11:06.676730+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:11:06.676730+0000","last_undegraded":"2024-03-26T05:11:06.676730+0000","last_fullsized":"2024-03-26T05:11:06.676730+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:36.861410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:36.861410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:10.038618+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":451,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.525842+0000","last_change":"2024-03-26T05:09:56.528351+0000","last_active":"2024-03-26T05:11:06.525842+0000","last_peered":"2024-03-26T05:11:06.525842+0000","last_clean":"2024-03-26T05:11:06.525842+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:11:06.525842+0000","last_undegraded":"2024-03-26T05:11:06.525842+0000","last_fullsized":"2024-03-26T05:11:06.525842+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:56.528075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:56.528075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:14.800367+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"254'25","reported_seq":566,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677026+0000","last_change":"2024-03-26T05:10:39.649729+0000","last_active":"2024-03-26T05:11:06.677026+0000","last_peered":"2024-03-26T05:11:06.677026+0000","last_clean":"2024-03-26T05:11:06.677026+0000","last_became_active":"2024-03-26T05:07:59.962007+0000","last_became_peered":"2024-03-26T05:07:59.962007+0000","last_unstale":"2024-03-26T05:11:06.677026+0000","last_undegraded":"2024-03-26T05:11:06.677026+0000","last_fullsized":"2024-03-26T05:11:06.677026+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:10:39.649620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:39.649620+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.262212+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":445,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677427+0000","last_change":"2024-03-26T05:10:43.824858+0000","last_active":"2024-03-26T05:11:06.677427+0000","last_peered":"2024-03-26T05:11:06.677427+0000","last_clean":"2024-03-26T05:11:06.677427+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:11:06.677427+0000","last_undegraded":"2024-03-26T05:11:06.677427+0000","last_fullsized":"2024-03-26T05:11:06.677427+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:43.824746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.824746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:48.191458+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":450,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.675770+0000","last_change":"2024-03-26T05:09:35.497868+0000","last_active":"2024-03-26T05:11:06.675770+0000","last_peered":"2024-03-26T05:11:06.675770+0000","last_clean":"2024-03-26T05:11:06.675770+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:11:06.675770+0000","last_undegraded":"2024-03-26T05:11:06.675770+0000","last_fullsized":"2024-03-26T05:11:06.675770+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:35.497586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:35.497586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":473,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676793+0000","last_change":"2024-03-26T05:10:46.108129+0000","last_active":"2024-03-26T05:11:06.676793+0000","last_peered":"2024-03-26T05:11:06.676793+0000","last_clean":"2024-03-26T05:11:06.676793+0000","last_became_active":"2024-03-26T05:07:59.993996+0000","last_became_peered":"2024-03-26T05:07:59.993996+0000","last_unstale":"2024-03-26T05:11:06.676793+0000","last_undegraded":"2024-03-26T05:11:06.676793+0000","last_fullsized":"2024-03-26T05:11:06.676793+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:46.108051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:46.108051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:12.619539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"254'10","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.525910+0000","last_change":"2024-03-26T05:10:29.575129+0000","last_active":"2024-03-26T05:11:06.525910+0000","last_peered":"2024-03-26T05:11:06.525910+0000","last_clean":"2024-03-26T05:11:06.525910+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:11:06.525910+0000","last_undegraded":"2024-03-26T05:11:06.525910+0000","last_fullsized":"2024-03-26T05:11:06.525910+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'6","last_scrub_stamp":"2024-03-26T05:10:29.575014+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.575014+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:48.708154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":439,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526855+0000","last_change":"2024-03-26T05:10:41.595090+0000","last_active":"2024-03-26T05:11:06.526855+0000","last_peered":"2024-03-26T05:11:06.526855+0000","last_clean":"2024-03-26T05:11:06.526855+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:11:06.526855+0000","last_undegraded":"2024-03-26T05:11:06.526855+0000","last_fullsized":"2024-03-26T05:11:06.526855+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:41.594950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.594950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:45.945454+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":449,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677355+0000","last_change":"2024-03-26T05:09:49.476829+0000","last_active":"2024-03-26T05:11:06.677355+0000","last_peered":"2024-03-26T05:11:06.677355+0000","last_clean":"2024-03-26T05:11:06.677355+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:11:06.677355+0000","last_undegraded":"2024-03-26T05:11:06.677355+0000","last_fullsized":"2024-03-26T05:11:06.677355+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:49.476722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:09:49.476722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":125,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677528+0000","last_change":"2024-03-26T05:07:59.997817+0000","last_active":"2024-03-26T05:11:06.677528+0000","last_peered":"2024-03-26T05:11:06.677528+0000","last_clean":"2024-03-26T05:11:06.677528+0000","last_became_active":"2024-03-26T05:07:59.997439+0000","last_became_peered":"2024-03-26T05:07:59.997439+0000","last_unstale":"2024-03-26T05:11:06.677528+0000","last_undegraded":"2024-03-26T05:11:06.677528+0000","last_fullsized":"2024-03-26T05:11:06.677528+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:06:56.886632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.551115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"254'25","reported_seq":482,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526330+0000","last_change":"2024-03-26T05:07:52.593239+0000","last_active":"2024-03-26T05:11:06.526330+0000","last_peered":"2024-03-26T05:11:06.526330+0000","last_clean":"2024-03-26T05:11:06.526330+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:11:06.526330+0000","last_undegraded":"2024-03-26T05:11:06.526330+0000","last_fullsized":"2024-03-26T05:11:06.526330+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:07:52.592943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:52.592943+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.459997+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.675474+0000","last_change":"2024-03-26T05:10:29.171886+0000","last_active":"2024-03-26T05:11:06.675474+0000","last_peered":"2024-03-26T05:11:06.675474+0000","last_clean":"2024-03-26T05:11:06.675474+0000","last_became_active":"2024-03-26T05:07:59.994210+0000","last_became_peered":"2024-03-26T05:07:59.994210+0000","last_unstale":"2024-03-26T05:11:06.675474+0000","last_undegraded":"2024-03-26T05:11:06.675474+0000","last_fullsized":"2024-03-26T05:11:06.675474+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:29.171780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.171780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:47.646591+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":435,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526607+0000","last_change":"2024-03-26T05:09:26.674360+0000","last_active":"2024-03-26T05:11:06.526607+0000","last_peered":"2024-03-26T05:11:06.526607+0000","last_clean":"2024-03-26T05:11:06.526607+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:11:06.526607+0000","last_undegraded":"2024-03-26T05:11:06.526607+0000","last_fullsized":"2024-03-26T05:11:06.526607+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:26.674245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:09:26.674245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.460180+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":449,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.678935+0000","last_change":"2024-03-26T05:09:53.508140+0000","last_active":"2024-03-26T05:11:06.678935+0000","last_peered":"2024-03-26T05:11:06.678935+0000","last_clean":"2024-03-26T05:11:06.678935+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:11:06.678935+0000","last_undegraded":"2024-03-26T05:11:06.678935+0000","last_fullsized":"2024-03-26T05:11:06.678935+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:53.507753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:09:53.507753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"254'15","reported_seq":472,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677567+0000","last_change":"2024-03-26T05:07:53.425727+0000","last_active":"2024-03-26T05:11:06.677567+0000","last_peered":"2024-03-26T05:11:06.677567+0000","last_clean":"2024-03-26T05:11:06.677567+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:11:06.677567+0000","last_undegraded":"2024-03-26T05:11:06.677567+0000","last_fullsized":"2024-03-26T05:11:06.677567+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-26T05:07:53.425472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.425472+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:07.322400+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":443,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.525910+0000","last_change":"2024-03-26T05:09:28.716469+0000","last_active":"2024-03-26T05:11:06.525910+0000","last_peered":"2024-03-26T05:11:06.525910+0000","last_clean":"2024-03-26T05:11:06.525910+0000","last_became_active":"2024-03-26T05:07:59.999703+0000","last_became_peered":"2024-03-26T05:07:59.999703+0000","last_unstale":"2024-03-26T05:11:06.525910+0000","last_undegraded":"2024-03-26T05:11:06.525910+0000","last_fullsized":"2024-03-26T05:11:06.525910+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:28.716348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:09:28.716348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.460099+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":463,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677503+0000","last_change":"2024-03-26T05:10:38.473529+0000","last_active":"2024-03-26T05:11:06.677503+0000","last_peered":"2024-03-26T05:11:06.677503+0000","last_clean":"2024-03-26T05:11:06.677503+0000","last_became_active":"2024-03-26T05:08:00.001601+0000","last_became_peered":"2024-03-26T05:08:00.001601+0000","last_unstale":"2024-03-26T05:11:06.677503+0000","last_undegraded":"2024-03-26T05:11:06.677503+0000","last_fullsized":"2024-03-26T05:11:06.677503+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:38.473446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:10:38.473446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:43.065707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":445,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499225+0000","last_change":"2024-03-26T05:07:39.699241+0000","last_active":"2024-03-26T05:11:06.499225+0000","last_peered":"2024-03-26T05:11:06.499225+0000","last_clean":"2024-03-26T05:11:06.499225+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:11:06.499225+0000","last_undegraded":"2024-03-26T05:11:06.499225+0000","last_fullsized":"2024-03-26T05:11:06.499225+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:39.699122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:39.699122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.558896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"254'24","reported_seq":485,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499305+0000","last_change":"2024-03-26T05:07:43.754242+0000","last_active":"2024-03-26T05:11:06.499305+0000","last_peered":"2024-03-26T05:11:06.499305+0000","last_clean":"2024-03-26T05:11:06.499305+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:11:06.499305+0000","last_undegraded":"2024-03-26T05:11:06.499305+0000","last_fullsized":"2024-03-26T05:11:06.499305+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'16","last_scrub_stamp":"2024-03-26T05:07:43.754130+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:07:43.754130+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.558956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":449,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676266+0000","last_change":"2024-03-26T05:10:50.061225+0000","last_active":"2024-03-26T05:11:06.676266+0000","last_peered":"2024-03-26T05:11:06.676266+0000","last_clean":"2024-03-26T05:11:06.676266+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:11:06.676266+0000","last_undegraded":"2024-03-26T05:11:06.676266+0000","last_fullsized":"2024-03-26T05:11:06.676266+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:50.061128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_clean_scrub_stamp":"2024-03-26T05:10:50.061128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:08.890951+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"216'10","reported_seq":175,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677390+0000","last_change":"2024-03-26T05:10:40.373554+0000","last_active":"2024-03-26T05:11:06.677390+0000","last_peered":"2024-03-26T05:11:06.677390+0000","last_clean":"2024-03-26T05:11:06.677390+0000","last_became_active":"2024-03-26T05:08:00.005648+0000","last_became_peered":"2024-03-26T05:08:00.005648+0000","last_unstale":"2024-03-26T05:11:06.677390+0000","last_undegraded":"2024-03-26T05:11:06.677390+0000","last_fullsized":"2024-03-26T05:11:06.677390+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:10:40.373470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.373470+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:08.010761+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":457,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677302+0000","last_change":"2024-03-26T05:09:45.438723+0000","last_active":"2024-03-26T05:11:06.677302+0000","last_peered":"2024-03-26T05:11:06.677302+0000","last_clean":"2024-03-26T05:11:06.677302+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:11:06.677302+0000","last_undegraded":"2024-03-26T05:11:06.677302+0000","last_fullsized":"2024-03-26T05:11:06.677302+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:45.438615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:45.438615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.866308+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"254'32","reported_seq":501,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526408+0000","last_change":"2024-03-26T05:09:30.638211+0000","last_active":"2024-03-26T05:11:06.526408+0000","last_peered":"2024-03-26T05:11:06.526408+0000","last_clean":"2024-03-26T05:11:06.526408+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:11:06.526408+0000","last_undegraded":"2024-03-26T05:11:06.526408+0000","last_fullsized":"2024-03-26T05:11:06.526408+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:09:30.635624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:30.635624+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.460166+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":453,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499881+0000","last_change":"2024-03-26T05:10:40.548775+0000","last_active":"2024-03-26T05:11:06.499881+0000","last_peered":"2024-03-26T05:11:06.499881+0000","last_clean":"2024-03-26T05:11:06.499881+0000","last_became_active":"2024-03-26T05:07:59.942512+0000","last_became_peered":"2024-03-26T05:07:59.942512+0000","last_unstale":"2024-03-26T05:11:06.499881+0000","last_undegraded":"2024-03-26T05:11:06.499881+0000","last_fullsized":"2024-03-26T05:11:06.499881+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:40.548688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.548688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.537910+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"120'9","reported_seq":585,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526742+0000","last_change":"2024-03-26T05:10:30.593032+0000","last_active":"2024-03-26T05:11:06.526742+0000","last_peered":"2024-03-26T05:11:06.526742+0000","last_clean":"2024-03-26T05:11:06.526742+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:11:06.526742+0000","last_undegraded":"2024-03-26T05:11:06.526742+0000","last_fullsized":"2024-03-26T05:11:06.526742+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'9","last_scrub_stamp":"2024-03-26T05:10:30.592948+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.592948+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:45.127936+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.675906+0000","last_change":"2024-03-26T05:09:59.033943+0000","last_active":"2024-03-26T05:11:06.675906+0000","last_peered":"2024-03-26T05:11:06.675906+0000","last_clean":"2024-03-26T05:11:06.675906+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:11:06.675906+0000","last_undegraded":"2024-03-26T05:11:06.675906+0000","last_fullsized":"2024-03-26T05:11:06.675906+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:59.033819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:59.033819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:10.038677+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"254'26","reported_seq":154,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677168+0000","last_change":"2024-03-26T05:10:33.467211+0000","last_active":"2024-03-26T05:11:06.677168+0000","last_peered":"2024-03-26T05:11:06.677168+0000","last_clean":"2024-03-26T05:11:06.677168+0000","last_became_active":"2024-03-26T05:07:59.960346+0000","last_became_peered":"2024-03-26T05:07:59.960346+0000","last_unstale":"2024-03-26T05:11:06.677168+0000","last_undegraded":"2024-03-26T05:11:06.677168+0000","last_fullsized":"2024-03-26T05:11:06.677168+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:10:33.467113+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.467113+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:40.892275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":455,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.500991+0000","last_change":"2024-03-26T05:10:41.140060+0000","last_active":"2024-03-26T05:11:06.500991+0000","last_peered":"2024-03-26T05:11:06.500991+0000","last_clean":"2024-03-26T05:11:06.500991+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:11:06.500991+0000","last_undegraded":"2024-03-26T05:11:06.500991+0000","last_fullsized":"2024-03-26T05:11:06.500991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:41.139989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.139989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:04.392633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":435,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526949+0000","last_change":"2024-03-26T05:07:55.590013+0000","last_active":"2024-03-26T05:11:06.526949+0000","last_peered":"2024-03-26T05:11:06.526949+0000","last_clean":"2024-03-26T05:11:06.526949+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:11:06.526949+0000","last_undegraded":"2024-03-26T05:11:06.526949+0000","last_fullsized":"2024-03-26T05:11:06.526949+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:55.589886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:07:55.589886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.460015+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":449,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676454+0000","last_change":"2024-03-26T05:09:28.457669+0000","last_active":"2024-03-26T05:11:06.676454+0000","last_peered":"2024-03-26T05:11:06.676454+0000","last_clean":"2024-03-26T05:11:06.676454+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:11:06.676454+0000","last_undegraded":"2024-03-26T05:11:06.676454+0000","last_fullsized":"2024-03-26T05:11:06.676454+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:28.457548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:28.457548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"254'27","reported_seq":506,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676582+0000","last_change":"2024-03-26T05:10:43.135431+0000","last_active":"2024-03-26T05:11:06.676582+0000","last_peered":"2024-03-26T05:11:06.676582+0000","last_clean":"2024-03-26T05:11:06.676582+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:11:06.676582+0000","last_undegraded":"2024-03-26T05:11:06.676582+0000","last_fullsized":"2024-03-26T05:11:06.676582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:10:43.135325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.135325+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:57.176336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":445,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.678890+0000","last_change":"2024-03-26T05:10:35.847292+0000","last_active":"2024-03-26T05:11:06.678890+0000","last_peered":"2024-03-26T05:11:06.678890+0000","last_clean":"2024-03-26T05:11:06.678890+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:11:06.678890+0000","last_undegraded":"2024-03-26T05:11:06.678890+0000","last_fullsized":"2024-03-26T05:11:06.678890+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:35.847191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:35.847191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:41.957309+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":133,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677393+0000","last_change":"2024-03-26T05:09:52.050966+0000","last_active":"2024-03-26T05:11:06.677393+0000","last_peered":"2024-03-26T05:11:06.677393+0000","last_clean":"2024-03-26T05:11:06.677393+0000","last_became_active":"2024-03-26T05:07:59.997434+0000","last_became_peered":"2024-03-26T05:07:59.997434+0000","last_unstale":"2024-03-26T05:11:06.677393+0000","last_undegraded":"2024-03-26T05:11:06.677393+0000","last_fullsized":"2024-03-26T05:11:06.677393+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:52.050832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:52.050832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:18.207778+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":485,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.675661+0000","last_change":"2024-03-26T05:09:32.473925+0000","last_active":"2024-03-26T05:11:06.675661+0000","last_peered":"2024-03-26T05:11:06.675661+0000","last_clean":"2024-03-26T05:11:06.675661+0000","last_became_active":"2024-03-26T05:07:59.992217+0000","last_became_peered":"2024-03-26T05:07:59.992217+0000","last_unstale":"2024-03-26T05:11:06.675661+0000","last_undegraded":"2024-03-26T05:11:06.675661+0000","last_fullsized":"2024-03-26T05:11:06.675661+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:09:32.473785+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_clean_scrub_stamp":"2024-03-26T05:09:32.473785+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"254'18","reported_seq":477,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526994+0000","last_change":"2024-03-26T05:10:36.560789+0000","last_active":"2024-03-26T05:11:06.526994+0000","last_peered":"2024-03-26T05:11:06.526994+0000","last_clean":"2024-03-26T05:11:06.526994+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:11:06.526994+0000","last_undegraded":"2024-03-26T05:11:06.526994+0000","last_fullsized":"2024-03-26T05:11:06.526994+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'18","last_scrub_stamp":"2024-03-26T05:10:36.560689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:36.560689+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:04.290785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"271'408","reported_seq":430,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:10.826249+0000","last_change":"2024-03-26T05:10:55.650520+0000","last_active":"2024-03-26T05:11:10.826249+0000","last_peered":"2024-03-26T05:11:10.826249+0000","last_clean":"2024-03-26T05:11:10.826249+0000","last_became_active":"2024-03-26T05:10:55.650187+0000","last_became_peered":"2024-03-26T05:10:55.650187+0000","last_unstale":"2024-03-26T05:11:10.826249+0000","last_undegraded":"2024-03-26T05:11:10.826249+0000","last_fullsized":"2024-03-26T05:11:10.826249+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:14.112864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26738688,"num_objects":408,"num_object_clones":0,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":408,"num_write_kb":26112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":129,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677937+0000","last_change":"2024-03-26T05:08:00.007340+0000","last_active":"2024-03-26T05:11:06.677937+0000","last_peered":"2024-03-26T05:11:06.677937+0000","last_clean":"2024-03-26T05:11:06.677937+0000","last_became_active":"2024-03-26T05:08:00.000166+0000","last_became_peered":"2024-03-26T05:08:00.000166+0000","last_unstale":"2024-03-26T05:11:06.677937+0000","last_undegraded":"2024-03-26T05:11:06.677937+0000","last_fullsized":"2024-03-26T05:11:06.677937+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:07:53.088449+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.088449+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.551172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":478,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.678148+0000","last_change":"2024-03-26T05:09:48.462986+0000","last_active":"2024-03-26T05:11:06.678148+0000","last_peered":"2024-03-26T05:11:06.678148+0000","last_clean":"2024-03-26T05:11:06.678148+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:11:06.678148+0000","last_undegraded":"2024-03-26T05:11:06.678148+0000","last_fullsized":"2024-03-26T05:11:06.678148+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:09:48.462855+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:05:06.832744+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.462855+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"254'48","reported_seq":180,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676627+0000","last_change":"2024-03-26T05:09:59.156101+0000","last_active":"2024-03-26T05:11:06.676627+0000","last_peered":"2024-03-26T05:11:06.676627+0000","last_clean":"2024-03-26T05:11:06.676627+0000","last_became_active":"2024-03-26T05:07:59.947478+0000","last_became_peered":"2024-03-26T05:07:59.947478+0000","last_unstale":"2024-03-26T05:11:06.676627+0000","last_undegraded":"2024-03-26T05:11:06.676627+0000","last_fullsized":"2024-03-26T05:11:06.676627+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'36","last_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:09:59.155894+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.551237+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":587,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676874+0000","last_change":"2024-03-26T05:10:29.818124+0000","last_active":"2024-03-26T05:11:06.676874+0000","last_peered":"2024-03-26T05:11:06.676874+0000","last_clean":"2024-03-26T05:11:06.676874+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:11:06.676874+0000","last_undegraded":"2024-03-26T05:11:06.676874+0000","last_fullsized":"2024-03-26T05:11:06.676874+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:10:29.818015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.818015+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:33.669328+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"271'339","reported_seq":360,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:10.363228+0000","last_change":"2024-03-26T05:10:55.653437+0000","last_active":"2024-03-26T05:11:10.363228+0000","last_peered":"2024-03-26T05:11:10.363228+0000","last_clean":"2024-03-26T05:11:10.363228+0000","last_became_active":"2024-03-26T05:10:55.652957+0000","last_became_peered":"2024-03-26T05:10:55.652957+0000","last_unstale":"2024-03-26T05:11:10.363228+0000","last_undegraded":"2024-03-26T05:11:10.363228+0000","last_fullsized":"2024-03-26T05:11:10.363228+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:22.932408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22216704,"num_objects":339,"num_object_clones":0,"num_object_copies":1017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":339,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":445,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526744+0000","last_change":"2024-03-26T05:10:33.604297+0000","last_active":"2024-03-26T05:11:06.526744+0000","last_peered":"2024-03-26T05:11:06.526744+0000","last_clean":"2024-03-26T05:11:06.526744+0000","last_became_active":"2024-03-26T05:08:55.556543+0000","last_became_peered":"2024-03-26T05:08:55.556543+0000","last_unstale":"2024-03-26T05:11:06.526744+0000","last_undegraded":"2024-03-26T05:11:06.526744+0000","last_fullsized":"2024-03-26T05:11:06.526744+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.604198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.604198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:36.748955+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":457,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.679182+0000","last_change":"2024-03-26T05:10:38.869839+0000","last_active":"2024-03-26T05:11:06.679182+0000","last_peered":"2024-03-26T05:11:06.679182+0000","last_clean":"2024-03-26T05:11:06.679182+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:11:06.679182+0000","last_undegraded":"2024-03-26T05:11:06.679182+0000","last_fullsized":"2024-03-26T05:11:06.679182+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:38.869752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:38.869752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.797842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"254'37","reported_seq":512,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499377+0000","last_change":"2024-03-26T05:09:53.374250+0000","last_active":"2024-03-26T05:11:06.499377+0000","last_peered":"2024-03-26T05:11:06.499377+0000","last_clean":"2024-03-26T05:11:06.499377+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:11:06.499377+0000","last_undegraded":"2024-03-26T05:11:06.499377+0000","last_fullsized":"2024-03-26T05:11:06.499377+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'31","last_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_clean_scrub_stamp":"2024-03-26T05:09:53.374102+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:13.729213+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":451,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.500504+0000","last_change":"2024-03-26T05:10:37.078616+0000","last_active":"2024-03-26T05:11:06.500504+0000","last_peered":"2024-03-26T05:11:06.500504+0000","last_clean":"2024-03-26T05:11:06.500504+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:11:06.500504+0000","last_undegraded":"2024-03-26T05:11:06.500504+0000","last_fullsized":"2024-03-26T05:11:06.500504+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:37.078539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.432732+0000","last_clean_scrub_stamp":"2024-03-26T05:10:37.078539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.617367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"271'279","reported_seq":299,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:09.907735+0000","last_change":"2024-03-26T05:10:55.648932+0000","last_active":"2024-03-26T05:11:09.907735+0000","last_peered":"2024-03-26T05:11:09.907735+0000","last_clean":"2024-03-26T05:11:09.907735+0000","last_became_active":"2024-03-26T05:10:55.648385+0000","last_became_peered":"2024-03-26T05:10:55.648385+0000","last_unstale":"2024-03-26T05:11:09.907735+0000","last_undegraded":"2024-03-26T05:11:09.907735+0000","last_fullsized":"2024-03-26T05:11:09.907735+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:19.782801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18284544,"num_objects":279,"num_object_clones":0,"num_object_copies":837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":279,"num_write_kb":17856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":445,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676904+0000","last_change":"2024-03-26T05:10:27.852306+0000","last_active":"2024-03-26T05:11:06.676904+0000","last_peered":"2024-03-26T05:11:06.676904+0000","last_clean":"2024-03-26T05:11:06.676904+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:11:06.676904+0000","last_undegraded":"2024-03-26T05:11:06.676904+0000","last_fullsized":"2024-03-26T05:11:06.676904+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:27.852200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_clean_scrub_stamp":"2024-03-26T05:10:27.852200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:39.771724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":463,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.500850+0000","last_change":"2024-03-26T05:08:00.000387+0000","last_active":"2024-03-26T05:11:06.500850+0000","last_peered":"2024-03-26T05:11:06.500850+0000","last_clean":"2024-03-26T05:11:06.500850+0000","last_became_active":"2024-03-26T05:08:00.000009+0000","last_became_peered":"2024-03-26T05:08:00.000009+0000","last_unstale":"2024-03-26T05:11:06.500850+0000","last_undegraded":"2024-03-26T05:11:06.500850+0000","last_fullsized":"2024-03-26T05:11:06.500850+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:46.384777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:46.384777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.102022+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.679155+0000","last_change":"2024-03-26T05:09:37.393265+0000","last_active":"2024-03-26T05:11:06.679155+0000","last_peered":"2024-03-26T05:11:06.679155+0000","last_clean":"2024-03-26T05:11:06.679155+0000","last_became_active":"2024-03-26T05:07:59.945688+0000","last_became_peered":"2024-03-26T05:07:59.945688+0000","last_unstale":"2024-03-26T05:11:06.679155+0000","last_undegraded":"2024-03-26T05:11:06.679155+0000","last_fullsized":"2024-03-26T05:11:06.679155+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:37.393159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:37.393159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:07.024891+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"254'41","reported_seq":521,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676943+0000","last_change":"2024-03-26T05:09:34.481126+0000","last_active":"2024-03-26T05:11:06.676943+0000","last_peered":"2024-03-26T05:11:06.676943+0000","last_clean":"2024-03-26T05:11:06.676943+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:11:06.676943+0000","last_undegraded":"2024-03-26T05:11:06.676943+0000","last_fullsized":"2024-03-26T05:11:06.676943+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'25","last_scrub_stamp":"2024-03-26T05:09:34.480977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:34.480977+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"271'215","reported_seq":236,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:09.909517+0000","last_change":"2024-03-26T05:10:55.650295+0000","last_active":"2024-03-26T05:11:09.909517+0000","last_peered":"2024-03-26T05:11:09.909517+0000","last_clean":"2024-03-26T05:11:09.909517+0000","last_became_active":"2024-03-26T05:10:55.649911+0000","last_became_peered":"2024-03-26T05:10:55.649911+0000","last_unstale":"2024-03-26T05:11:09.909517+0000","last_undegraded":"2024-03-26T05:11:09.909517+0000","last_fullsized":"2024-03-26T05:11:09.909517+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:05.955335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14090240,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":215,"num_write_kb":13760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":441,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.675971+0000","last_change":"2024-03-26T05:09:37.959362+0000","last_active":"2024-03-26T05:11:06.675971+0000","last_peered":"2024-03-26T05:11:06.675971+0000","last_clean":"2024-03-26T05:11:06.675971+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:11:06.675971+0000","last_undegraded":"2024-03-26T05:11:06.675971+0000","last_fullsized":"2024-03-26T05:11:06.675971+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:37.959237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:37.959237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:10.038661+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":451,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.500962+0000","last_change":"2024-03-26T05:09:58.378595+0000","last_active":"2024-03-26T05:11:06.500962+0000","last_peered":"2024-03-26T05:11:06.500962+0000","last_clean":"2024-03-26T05:11:06.500962+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:11:06.500962+0000","last_undegraded":"2024-03-26T05:11:06.500962+0000","last_fullsized":"2024-03-26T05:11:06.500962+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:58.378228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:58.378228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.565597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":133,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676491+0000","last_change":"2024-03-26T05:09:40.213194+0000","last_active":"2024-03-26T05:11:06.676491+0000","last_peered":"2024-03-26T05:11:06.676491+0000","last_clean":"2024-03-26T05:11:06.676491+0000","last_became_active":"2024-03-26T05:07:59.959978+0000","last_became_peered":"2024-03-26T05:07:59.959978+0000","last_unstale":"2024-03-26T05:11:06.676491+0000","last_undegraded":"2024-03-26T05:11:06.676491+0000","last_fullsized":"2024-03-26T05:11:06.676491+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:40.213116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:40.213116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.551219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"254'21","reported_seq":486,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676358+0000","last_change":"2024-03-26T05:09:27.773009+0000","last_active":"2024-03-26T05:11:06.676358+0000","last_peered":"2024-03-26T05:11:06.676358+0000","last_clean":"2024-03-26T05:11:06.676358+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:11:06.676358+0000","last_undegraded":"2024-03-26T05:11:06.676358+0000","last_fullsized":"2024-03-26T05:11:06.676358+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:09:27.772633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:27.772633+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:10.038554+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"271'450","reported_seq":472,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:10.828685+0000","last_change":"2024-03-26T05:10:55.650442+0000","last_active":"2024-03-26T05:11:10.828685+0000","last_peered":"2024-03-26T05:11:10.828685+0000","last_clean":"2024-03-26T05:11:10.828685+0000","last_became_active":"2024-03-26T05:10:55.650103+0000","last_became_peered":"2024-03-26T05:10:55.650103+0000","last_unstale":"2024-03-26T05:11:10.828685+0000","last_undegraded":"2024-03-26T05:11:10.828685+0000","last_fullsized":"2024-03-26T05:11:10.828685+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":450,"log_dups_size":0,"ondisk_log_size":450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:16.916628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29491200,"num_objects":450,"num_object_clones":0,"num_object_copies":1350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":450,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":431,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526700+0000","last_change":"2024-03-26T05:07:48.635847+0000","last_active":"2024-03-26T05:11:06.526700+0000","last_peered":"2024-03-26T05:11:06.526700+0000","last_clean":"2024-03-26T05:11:06.526700+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:11:06.526700+0000","last_undegraded":"2024-03-26T05:11:06.526700+0000","last_fullsized":"2024-03-26T05:11:06.526700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:48.635728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:48.635728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.460064+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":129,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677960+0000","last_change":"2024-03-26T05:08:00.004448+0000","last_active":"2024-03-26T05:11:06.677960+0000","last_peered":"2024-03-26T05:11:06.677960+0000","last_clean":"2024-03-26T05:11:06.677960+0000","last_became_active":"2024-03-26T05:07:59.998511+0000","last_became_peered":"2024-03-26T05:07:59.998511+0000","last_unstale":"2024-03-26T05:11:06.677960+0000","last_undegraded":"2024-03-26T05:11:06.677960+0000","last_fullsized":"2024-03-26T05:11:06.677960+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:07:37.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:07:37.347513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.551135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499204+0000","last_change":"2024-03-26T05:09:31.499628+0000","last_active":"2024-03-26T05:11:06.499204+0000","last_peered":"2024-03-26T05:11:06.499204+0000","last_clean":"2024-03-26T05:11:06.499204+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:11:06.499204+0000","last_undegraded":"2024-03-26T05:11:06.499204+0000","last_fullsized":"2024-03-26T05:11:06.499204+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:31.499513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:31.499513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.559176+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"254'46","reported_seq":664,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676482+0000","last_change":"2024-03-26T05:10:48.344286+0000","last_active":"2024-03-26T05:11:06.676482+0000","last_peered":"2024-03-26T05:11:06.676482+0000","last_clean":"2024-03-26T05:11:06.676482+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:11:06.676482+0000","last_undegraded":"2024-03-26T05:11:06.676482+0000","last_fullsized":"2024-03-26T05:11:06.676482+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'46","last_scrub_stamp":"2024-03-26T05:10:48.344180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:48.344180+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:57.157542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.675854+0000","last_change":"2024-03-26T05:09:41.858043+0000","last_active":"2024-03-26T05:11:06.675854+0000","last_peered":"2024-03-26T05:11:06.675854+0000","last_clean":"2024-03-26T05:11:06.675854+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:11:06.675854+0000","last_undegraded":"2024-03-26T05:11:06.675854+0000","last_fullsized":"2024-03-26T05:11:06.675854+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:41.857888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:41.857888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.702593+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"271'210","reported_seq":231,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:09.883527+0000","last_change":"2024-03-26T05:10:55.651127+0000","last_active":"2024-03-26T05:11:09.883527+0000","last_peered":"2024-03-26T05:11:09.883527+0000","last_clean":"2024-03-26T05:11:09.883527+0000","last_became_active":"2024-03-26T05:10:55.650612+0000","last_became_peered":"2024-03-26T05:10:55.650612+0000","last_unstale":"2024-03-26T05:11:09.883527+0000","last_undegraded":"2024-03-26T05:11:09.883527+0000","last_fullsized":"2024-03-26T05:11:09.883527+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:21.211594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13762560,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":210,"num_write_kb":13440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.679249+0000","last_change":"2024-03-26T05:09:36.428895+0000","last_active":"2024-03-26T05:11:06.679249+0000","last_peered":"2024-03-26T05:11:06.679249+0000","last_clean":"2024-03-26T05:11:06.679249+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:11:06.679249+0000","last_undegraded":"2024-03-26T05:11:06.679249+0000","last_fullsized":"2024-03-26T05:11:06.679249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:36.428762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:36.428762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":452,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676714+0000","last_change":"2024-03-26T05:10:39.454264+0000","last_active":"2024-03-26T05:11:06.676714+0000","last_peered":"2024-03-26T05:11:06.676714+0000","last_clean":"2024-03-26T05:11:06.676714+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:11:06.676714+0000","last_undegraded":"2024-03-26T05:11:06.676714+0000","last_fullsized":"2024-03-26T05:11:06.676714+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:39.454176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:39.454176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:41.956147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":469,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.500381+0000","last_change":"2024-03-26T05:09:55.551716+0000","last_active":"2024-03-26T05:11:06.500381+0000","last_peered":"2024-03-26T05:11:06.500381+0000","last_clean":"2024-03-26T05:11:06.500381+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:11:06.500381+0000","last_undegraded":"2024-03-26T05:11:06.500381+0000","last_fullsized":"2024-03-26T05:11:06.500381+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:55.551595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.551595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.543264+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"254'23","reported_seq":266,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.501215+0000","last_change":"2024-03-26T05:08:00.019227+0000","last_active":"2024-03-26T05:11:06.501215+0000","last_peered":"2024-03-26T05:11:06.501215+0000","last_clean":"2024-03-26T05:11:06.501215+0000","last_became_active":"2024-03-26T05:08:00.001398+0000","last_became_peered":"2024-03-26T05:08:00.001398+0000","last_unstale":"2024-03-26T05:11:06.501215+0000","last_undegraded":"2024-03-26T05:11:06.501215+0000","last_fullsized":"2024-03-26T05:11:06.501215+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-26T05:07:54.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:07:54.529843+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.101992+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":471,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.501001+0000","last_change":"2024-03-26T05:10:43.076916+0000","last_active":"2024-03-26T05:11:06.501001+0000","last_peered":"2024-03-26T05:11:06.501001+0000","last_clean":"2024-03-26T05:11:06.501001+0000","last_became_active":"2024-03-26T05:07:59.955456+0000","last_became_peered":"2024-03-26T05:07:59.955456+0000","last_unstale":"2024-03-26T05:11:06.501001+0000","last_undegraded":"2024-03-26T05:11:06.501001+0000","last_fullsized":"2024-03-26T05:11:06.501001+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:43.076802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.076802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:54.501193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"271'365","reported_seq":387,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:10.615374+0000","last_change":"2024-03-26T05:10:55.651844+0000","last_active":"2024-03-26T05:11:10.615374+0000","last_peered":"2024-03-26T05:11:10.615374+0000","last_clean":"2024-03-26T05:11:10.615374+0000","last_became_active":"2024-03-26T05:10:55.651432+0000","last_became_peered":"2024-03-26T05:10:55.651432+0000","last_unstale":"2024-03-26T05:11:10.615374+0000","last_undegraded":"2024-03-26T05:11:10.615374+0000","last_fullsized":"2024-03-26T05:11:10.615374+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:17.254815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23920640,"num_objects":365,"num_object_clones":0,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":365,"num_write_kb":23360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":133,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.678367+0000","last_change":"2024-03-26T05:09:46.172999+0000","last_active":"2024-03-26T05:11:06.678367+0000","last_peered":"2024-03-26T05:11:06.678367+0000","last_clean":"2024-03-26T05:11:06.678367+0000","last_became_active":"2024-03-26T05:07:59.959853+0000","last_became_peered":"2024-03-26T05:07:59.959853+0000","last_unstale":"2024-03-26T05:11:06.678367+0000","last_undegraded":"2024-03-26T05:11:06.678367+0000","last_fullsized":"2024-03-26T05:11:06.678367+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:46.172868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_clean_scrub_stamp":"2024-03-26T05:09:46.172868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.551204+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":713,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677169+0000","last_change":"2024-03-26T05:10:39.866608+0000","last_active":"2024-03-26T05:11:06.677169+0000","last_peered":"2024-03-26T05:11:06.677169+0000","last_clean":"2024-03-26T05:11:06.677169+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:11:06.677169+0000","last_undegraded":"2024-03-26T05:11:06.677169+0000","last_fullsized":"2024-03-26T05:11:06.677169+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:10:39.866516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:10:39.866516+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.907367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":464,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.525991+0000","last_change":"2024-03-26T05:09:39.605190+0000","last_active":"2024-03-26T05:11:06.525991+0000","last_peered":"2024-03-26T05:11:06.525991+0000","last_clean":"2024-03-26T05:11:06.525991+0000","last_became_active":"2024-03-26T05:07:59.960061+0000","last_became_peered":"2024-03-26T05:07:59.960061+0000","last_unstale":"2024-03-26T05:11:06.525991+0000","last_undegraded":"2024-03-26T05:11:06.525991+0000","last_fullsized":"2024-03-26T05:11:06.525991+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:39.605053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:39.605053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"254'36","reported_seq":641,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676843+0000","last_change":"2024-03-26T05:09:41.506253+0000","last_active":"2024-03-26T05:11:06.676843+0000","last_peered":"2024-03-26T05:11:06.676843+0000","last_clean":"2024-03-26T05:11:06.676843+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:11:06.676843+0000","last_undegraded":"2024-03-26T05:11:06.676843+0000","last_fullsized":"2024-03-26T05:11:06.676843+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'24","last_scrub_stamp":"2024-03-26T05:09:41.506123+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:09:41.506123+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:12.760311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":470,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.678080+0000","last_change":"2024-03-26T05:10:36.850667+0000","last_active":"2024-03-26T05:11:06.678080+0000","last_peered":"2024-03-26T05:11:06.678080+0000","last_clean":"2024-03-26T05:11:06.678080+0000","last_became_active":"2024-03-26T05:07:59.956080+0000","last_became_peered":"2024-03-26T05:07:59.956080+0000","last_unstale":"2024-03-26T05:11:06.678080+0000","last_undegraded":"2024-03-26T05:11:06.678080+0000","last_fullsized":"2024-03-26T05:11:06.678080+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:36.850583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:36.850583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:53.029005+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"271'431","reported_seq":453,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:10.817073+0000","last_change":"2024-03-26T05:10:55.754958+0000","last_active":"2024-03-26T05:11:10.817073+0000","last_peered":"2024-03-26T05:11:10.817073+0000","last_clean":"2024-03-26T05:11:10.817073+0000","last_became_active":"2024-03-26T05:10:55.754590+0000","last_became_peered":"2024-03-26T05:10:55.754590+0000","last_unstale":"2024-03-26T05:11:10.817073+0000","last_undegraded":"2024-03-26T05:11:10.817073+0000","last_fullsized":"2024-03-26T05:11:10.817073+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:16.154652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28246016,"num_objects":431,"num_object_clones":0,"num_object_copies":1293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":431,"num_write_kb":27584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":452,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676038+0000","last_change":"2024-03-26T05:08:00.002985+0000","last_active":"2024-03-26T05:11:06.676038+0000","last_peered":"2024-03-26T05:11:06.676038+0000","last_clean":"2024-03-26T05:11:06.676038+0000","last_became_active":"2024-03-26T05:08:00.001657+0000","last_became_peered":"2024-03-26T05:08:00.001657+0000","last_unstale":"2024-03-26T05:11:06.676038+0000","last_undegraded":"2024-03-26T05:11:06.676038+0000","last_fullsized":"2024-03-26T05:11:06.676038+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:39.338092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:39.338092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:07.322379+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":349,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499304+0000","last_change":"2024-03-26T05:09:48.473285+0000","last_active":"2024-03-26T05:11:06.499304+0000","last_peered":"2024-03-26T05:11:06.499304+0000","last_clean":"2024-03-26T05:11:06.499304+0000","last_became_active":"2024-03-26T05:07:59.992489+0000","last_became_peered":"2024-03-26T05:07:59.992489+0000","last_unstale":"2024-03-26T05:11:06.499304+0000","last_undegraded":"2024-03-26T05:11:06.499304+0000","last_fullsized":"2024-03-26T05:11:06.499304+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.473160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.559205+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":1111,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:07.966270+0000","last_change":"2024-03-26T05:09:55.485007+0000","last_active":"2024-03-26T05:11:07.966270+0000","last_peered":"2024-03-26T05:11:07.966270+0000","last_clean":"2024-03-26T05:11:07.966270+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:11:07.966270+0000","last_undegraded":"2024-03-26T05:11:07.966270+0000","last_fullsized":"2024-03-26T05:11:07.966270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:09:55.484790+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.484790+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:14.311528+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":640,"num_read_kb":635,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"254'30","reported_seq":531,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676357+0000","last_change":"2024-03-26T05:09:49.909494+0000","last_active":"2024-03-26T05:11:06.676357+0000","last_peered":"2024-03-26T05:11:06.676357+0000","last_clean":"2024-03-26T05:11:06.676357+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:11:06.676357+0000","last_undegraded":"2024-03-26T05:11:06.676357+0000","last_fullsized":"2024-03-26T05:11:06.676357+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'18","last_scrub_stamp":"2024-03-26T05:09:49.909347+0000","last_deep_scrub":"187'18","last_deep_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_clean_scrub_stamp":"2024-03-26T05:09:49.909347+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:16.935034+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":133,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.678249+0000","last_change":"2024-03-26T05:10:44.404350+0000","last_active":"2024-03-26T05:11:06.678249+0000","last_peered":"2024-03-26T05:11:06.678249+0000","last_clean":"2024-03-26T05:11:06.678249+0000","last_became_active":"2024-03-26T05:07:59.999990+0000","last_became_peered":"2024-03-26T05:07:59.999990+0000","last_unstale":"2024-03-26T05:11:06.678249+0000","last_undegraded":"2024-03-26T05:11:06.678249+0000","last_fullsized":"2024-03-26T05:11:06.678249+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:44.404261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:44.404261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:56.403017+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"271'227","reported_seq":248,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:09.880326+0000","last_change":"2024-03-26T05:10:55.652321+0000","last_active":"2024-03-26T05:11:09.880326+0000","last_peered":"2024-03-26T05:11:09.880326+0000","last_clean":"2024-03-26T05:11:09.880326+0000","last_became_active":"2024-03-26T05:10:55.651931+0000","last_became_peered":"2024-03-26T05:10:55.651931+0000","last_unstale":"2024-03-26T05:11:09.880326+0000","last_undegraded":"2024-03-26T05:11:09.880326+0000","last_fullsized":"2024-03-26T05:11:09.880326+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:06.446893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14876672,"num_objects":227,"num_object_clones":0,"num_object_copies":681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":14528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":440,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676138+0000","last_change":"2024-03-26T05:10:34.408142+0000","last_active":"2024-03-26T05:11:06.676138+0000","last_peered":"2024-03-26T05:11:06.676138+0000","last_clean":"2024-03-26T05:11:06.676138+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:11:06.676138+0000","last_undegraded":"2024-03-26T05:11:06.676138+0000","last_fullsized":"2024-03-26T05:11:06.676138+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:34.408045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:10:34.408045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:39.555764+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":464,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676313+0000","last_change":"2024-03-26T05:09:54.538181+0000","last_active":"2024-03-26T05:11:06.676313+0000","last_peered":"2024-03-26T05:11:06.676313+0000","last_clean":"2024-03-26T05:11:06.676313+0000","last_became_active":"2024-03-26T05:08:00.003099+0000","last_became_peered":"2024-03-26T05:08:00.003099+0000","last_unstale":"2024-03-26T05:11:06.676313+0000","last_undegraded":"2024-03-26T05:11:06.676313+0000","last_fullsized":"2024-03-26T05:11:06.676313+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:54.538056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:35.745100+0000","last_clean_scrub_stamp":"2024-03-26T05:09:54.538056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"254'44","reported_seq":582,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676109+0000","last_change":"2024-03-26T05:09:53.978359+0000","last_active":"2024-03-26T05:11:06.676109+0000","last_peered":"2024-03-26T05:11:06.676109+0000","last_clean":"2024-03-26T05:11:06.676109+0000","last_became_active":"2024-03-26T05:07:59.994226+0000","last_became_peered":"2024-03-26T05:07:59.994226+0000","last_unstale":"2024-03-26T05:11:06.676109+0000","last_undegraded":"2024-03-26T05:11:06.676109+0000","last_fullsized":"2024-03-26T05:11:06.676109+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'34","last_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:09:53.978203+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:10.038639+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":471,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677032+0000","last_change":"2024-03-26T05:10:42.594397+0000","last_active":"2024-03-26T05:11:06.677032+0000","last_peered":"2024-03-26T05:11:06.677032+0000","last_clean":"2024-03-26T05:11:06.677032+0000","last_became_active":"2024-03-26T05:07:59.991484+0000","last_became_peered":"2024-03-26T05:07:59.991484+0000","last_unstale":"2024-03-26T05:11:06.677032+0000","last_undegraded":"2024-03-26T05:11:06.677032+0000","last_fullsized":"2024-03-26T05:11:06.677032+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:42.594296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:10:42.594296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:54.067648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"271'233","reported_seq":254,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:09.907782+0000","last_change":"2024-03-26T05:10:55.648891+0000","last_active":"2024-03-26T05:11:09.907782+0000","last_peered":"2024-03-26T05:11:09.907782+0000","last_clean":"2024-03-26T05:11:09.907782+0000","last_became_active":"2024-03-26T05:10:55.648533+0000","last_became_peered":"2024-03-26T05:10:55.648533+0000","last_unstale":"2024-03-26T05:11:09.907782+0000","last_undegraded":"2024-03-26T05:11:09.907782+0000","last_fullsized":"2024-03-26T05:11:09.907782+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.423291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15269888,"num_objects":233,"num_object_clones":0,"num_object_copies":699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":233,"num_write_kb":14912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":442,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.675528+0000","last_change":"2024-03-26T05:09:28.793178+0000","last_active":"2024-03-26T05:11:06.675528+0000","last_peered":"2024-03-26T05:11:06.675528+0000","last_clean":"2024-03-26T05:11:06.675528+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:11:06.675528+0000","last_undegraded":"2024-03-26T05:11:06.675528+0000","last_fullsized":"2024-03-26T05:11:06.675528+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:28.792827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:28.792827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:10.038521+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":133,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677294+0000","last_change":"2024-03-26T05:09:42.163715+0000","last_active":"2024-03-26T05:11:06.677294+0000","last_peered":"2024-03-26T05:11:06.677294+0000","last_clean":"2024-03-26T05:11:06.677294+0000","last_became_active":"2024-03-26T05:07:59.960768+0000","last_became_peered":"2024-03-26T05:07:59.960768+0000","last_unstale":"2024-03-26T05:11:06.677294+0000","last_undegraded":"2024-03-26T05:11:06.677294+0000","last_fullsized":"2024-03-26T05:11:06.677294+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:42.163609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:42.163609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.551255+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"254'26","reported_seq":500,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.501327+0000","last_change":"2024-03-26T05:09:47.483160+0000","last_active":"2024-03-26T05:11:06.501327+0000","last_peered":"2024-03-26T05:11:06.501327+0000","last_clean":"2024-03-26T05:11:06.501327+0000","last_became_active":"2024-03-26T05:07:59.960351+0000","last_became_peered":"2024-03-26T05:07:59.960351+0000","last_unstale":"2024-03-26T05:11:06.501327+0000","last_undegraded":"2024-03-26T05:11:06.501327+0000","last_fullsized":"2024-03-26T05:11:06.501327+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-26T05:09:47.483012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:47.483012+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:10.083498+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"216'11","reported_seq":614,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676519+0000","last_change":"2024-03-26T05:10:34.204907+0000","last_active":"2024-03-26T05:11:06.676519+0000","last_peered":"2024-03-26T05:11:06.676519+0000","last_clean":"2024-03-26T05:11:06.676519+0000","last_became_active":"2024-03-26T05:07:59.994682+0000","last_became_peered":"2024-03-26T05:07:59.994682+0000","last_unstale":"2024-03-26T05:11:06.676519+0000","last_undegraded":"2024-03-26T05:11:06.676519+0000","last_fullsized":"2024-03-26T05:11:06.676519+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:10:34.204798+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:10:34.204798+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.783859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"271'339","reported_seq":360,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:10.632284+0000","last_change":"2024-03-26T05:10:55.648350+0000","last_active":"2024-03-26T05:11:10.632284+0000","last_peered":"2024-03-26T05:11:10.632284+0000","last_clean":"2024-03-26T05:11:10.632284+0000","last_became_active":"2024-03-26T05:10:55.647949+0000","last_became_peered":"2024-03-26T05:10:55.647949+0000","last_unstale":"2024-03-26T05:11:10.632284+0000","last_undegraded":"2024-03-26T05:11:10.632284+0000","last_fullsized":"2024-03-26T05:11:10.632284+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:10.616095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22216704,"num_objects":339,"num_object_clones":0,"num_object_copies":1017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":339,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677695+0000","last_change":"2024-03-26T05:10:34.839543+0000","last_active":"2024-03-26T05:11:06.677695+0000","last_peered":"2024-03-26T05:11:06.677695+0000","last_clean":"2024-03-26T05:11:06.677695+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:11:06.677695+0000","last_undegraded":"2024-03-26T05:11:06.677695+0000","last_fullsized":"2024-03-26T05:11:06.677695+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:34.837732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:34.837732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.929082+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":462,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676671+0000","last_change":"2024-03-26T05:09:47.958661+0000","last_active":"2024-03-26T05:11:06.676671+0000","last_peered":"2024-03-26T05:11:06.676671+0000","last_clean":"2024-03-26T05:11:06.676671+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:11:06.676671+0000","last_undegraded":"2024-03-26T05:11:06.676671+0000","last_fullsized":"2024-03-26T05:11:06.676671+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:09:47.958370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:47.958370+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:10.038576+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"254'34","reported_seq":552,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499247+0000","last_change":"2024-03-26T05:09:47.484748+0000","last_active":"2024-03-26T05:11:06.499247+0000","last_peered":"2024-03-26T05:11:06.499247+0000","last_clean":"2024-03-26T05:11:06.499247+0000","last_became_active":"2024-03-26T05:07:59.942510+0000","last_became_peered":"2024-03-26T05:07:59.942510+0000","last_unstale":"2024-03-26T05:11:06.499247+0000","last_undegraded":"2024-03-26T05:11:06.499247+0000","last_fullsized":"2024-03-26T05:11:06.499247+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:09:47.484585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:47.484585+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:13.731149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":574,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526434+0000","last_change":"2024-03-26T05:07:53.583159+0000","last_active":"2024-03-26T05:11:06.526434+0000","last_peered":"2024-03-26T05:11:06.526434+0000","last_clean":"2024-03-26T05:11:06.526434+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:11:06.526434+0000","last_undegraded":"2024-03-26T05:11:06.526434+0000","last_fullsized":"2024-03-26T05:11:06.526434+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:07:53.583016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:07:53.583016+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:13.475031+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"271'220","reported_seq":241,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:09.869698+0000","last_change":"2024-03-26T05:10:55.652074+0000","last_active":"2024-03-26T05:11:09.869698+0000","last_peered":"2024-03-26T05:11:09.869698+0000","last_clean":"2024-03-26T05:11:09.869698+0000","last_became_active":"2024-03-26T05:10:55.651719+0000","last_became_peered":"2024-03-26T05:10:55.651719+0000","last_unstale":"2024-03-26T05:11:09.869698+0000","last_undegraded":"2024-03-26T05:11:09.869698+0000","last_fullsized":"2024-03-26T05:11:09.869698+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:56.482218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14417920,"num_objects":220,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":220,"num_write_kb":14080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":442,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499074+0000","last_change":"2024-03-26T05:07:50.818124+0000","last_active":"2024-03-26T05:11:06.499074+0000","last_peered":"2024-03-26T05:11:06.499074+0000","last_clean":"2024-03-26T05:11:06.499074+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:11:06.499074+0000","last_undegraded":"2024-03-26T05:11:06.499074+0000","last_fullsized":"2024-03-26T05:11:06.499074+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:50.818000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:50.818000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.558987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":443,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526866+0000","last_change":"2024-03-26T05:09:38.690272+0000","last_active":"2024-03-26T05:11:06.526866+0000","last_peered":"2024-03-26T05:11:06.526866+0000","last_clean":"2024-03-26T05:11:06.526866+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:11:06.526866+0000","last_undegraded":"2024-03-26T05:11:06.526866+0000","last_fullsized":"2024-03-26T05:11:06.526866+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:38.690131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:38.690131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"254'52","reported_seq":532,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526289+0000","last_change":"2024-03-26T05:09:34.644416+0000","last_active":"2024-03-26T05:11:06.526289+0000","last_peered":"2024-03-26T05:11:06.526289+0000","last_clean":"2024-03-26T05:11:06.526289+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:11:06.526289+0000","last_undegraded":"2024-03-26T05:11:06.526289+0000","last_fullsized":"2024-03-26T05:11:06.526289+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:09:34.643076+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:09:34.643076+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.460133+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499046+0000","last_change":"2024-03-26T05:10:33.411222+0000","last_active":"2024-03-26T05:11:06.499046+0000","last_peered":"2024-03-26T05:11:06.499046+0000","last_clean":"2024-03-26T05:11:06.499046+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:11:06.499046+0000","last_undegraded":"2024-03-26T05:11:06.499046+0000","last_fullsized":"2024-03-26T05:11:06.499046+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.411139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.411139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:54.372120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"271'371","reported_seq":392,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:10.617738+0000","last_change":"2024-03-26T05:10:55.652478+0000","last_active":"2024-03-26T05:11:10.617738+0000","last_peered":"2024-03-26T05:11:10.617738+0000","last_clean":"2024-03-26T05:11:10.617738+0000","last_became_active":"2024-03-26T05:10:55.652104+0000","last_became_peered":"2024-03-26T05:10:55.652104+0000","last_unstale":"2024-03-26T05:11:10.617738+0000","last_undegraded":"2024-03-26T05:11:10.617738+0000","last_fullsized":"2024-03-26T05:11:10.617738+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:04.343417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24313856,"num_objects":371,"num_object_clones":0,"num_object_copies":1113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":371,"num_write_kb":23744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":125,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677045+0000","last_change":"2024-03-26T05:08:00.006006+0000","last_active":"2024-03-26T05:11:06.677045+0000","last_peered":"2024-03-26T05:11:06.677045+0000","last_clean":"2024-03-26T05:11:06.677045+0000","last_became_active":"2024-03-26T05:08:00.005676+0000","last_became_peered":"2024-03-26T05:08:00.005676+0000","last_unstale":"2024-03-26T05:11:06.677045+0000","last_undegraded":"2024-03-26T05:11:06.677045+0000","last_fullsized":"2024-03-26T05:11:06.677045+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_clean_scrub_stamp":"2024-03-26T05:07:52.465619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:09.551154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"216'10","reported_seq":604,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499714+0000","last_change":"2024-03-26T05:09:36.472026+0000","last_active":"2024-03-26T05:11:06.499714+0000","last_peered":"2024-03-26T05:11:06.499714+0000","last_clean":"2024-03-26T05:11:06.499714+0000","last_became_active":"2024-03-26T05:07:59.997429+0000","last_became_peered":"2024-03-26T05:07:59.997429+0000","last_unstale":"2024-03-26T05:11:06.499714+0000","last_undegraded":"2024-03-26T05:11:06.499714+0000","last_fullsized":"2024-03-26T05:11:06.499714+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:09:36.471890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:36.471890+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.559147+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":451,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.525931+0000","last_change":"2024-03-26T05:10:40.541468+0000","last_active":"2024-03-26T05:11:06.525931+0000","last_peered":"2024-03-26T05:11:06.525931+0000","last_clean":"2024-03-26T05:11:06.525931+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:11:06.525931+0000","last_undegraded":"2024-03-26T05:11:06.525931+0000","last_fullsized":"2024-03-26T05:11:06.525931+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:40.541340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.541340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.162836+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"254'27","reported_seq":498,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.677056+0000","last_change":"2024-03-26T05:10:43.423183+0000","last_active":"2024-03-26T05:11:06.677056+0000","last_peered":"2024-03-26T05:11:06.677056+0000","last_clean":"2024-03-26T05:11:06.677056+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:11:06.677056+0000","last_undegraded":"2024-03-26T05:11:06.677056+0000","last_fullsized":"2024-03-26T05:11:06.677056+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:10:43.423040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.423040+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:50.942552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"271'227","reported_seq":247,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:09.882527+0000","last_change":"2024-03-26T05:10:55.651677+0000","last_active":"2024-03-26T05:11:09.882527+0000","last_peered":"2024-03-26T05:11:09.882527+0000","last_clean":"2024-03-26T05:11:09.882527+0000","last_became_active":"2024-03-26T05:10:55.651348+0000","last_became_peered":"2024-03-26T05:10:55.651348+0000","last_unstale":"2024-03-26T05:11:09.882527+0000","last_undegraded":"2024-03-26T05:11:09.882527+0000","last_fullsized":"2024-03-26T05:11:09.882527+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:57.432492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14876672,"num_objects":227,"num_object_clones":0,"num_object_copies":681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":14528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":451,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526373+0000","last_change":"2024-03-26T05:10:44.608828+0000","last_active":"2024-03-26T05:11:06.526373+0000","last_peered":"2024-03-26T05:11:06.526373+0000","last_clean":"2024-03-26T05:11:06.526373+0000","last_became_active":"2024-03-26T05:07:59.957995+0000","last_became_peered":"2024-03-26T05:07:59.957995+0000","last_unstale":"2024-03-26T05:11:06.526373+0000","last_undegraded":"2024-03-26T05:11:06.526373+0000","last_fullsized":"2024-03-26T05:11:06.526373+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:44.608717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:44.608717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.528157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":133,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.678133+0000","last_change":"2024-03-26T05:10:48.413966+0000","last_active":"2024-03-26T05:11:06.678133+0000","last_peered":"2024-03-26T05:11:06.678133+0000","last_clean":"2024-03-26T05:11:06.678133+0000","last_became_active":"2024-03-26T05:07:59.947747+0000","last_became_peered":"2024-03-26T05:07:59.947747+0000","last_unstale":"2024-03-26T05:11:06.678133+0000","last_undegraded":"2024-03-26T05:11:06.678133+0000","last_fullsized":"2024-03-26T05:11:06.678133+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:48.413868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.446060+0000","last_clean_scrub_stamp":"2024-03-26T05:10:48.413868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:13.855843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":453,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499039+0000","last_change":"2024-03-26T05:07:49.831449+0000","last_active":"2024-03-26T05:11:06.499039+0000","last_peered":"2024-03-26T05:11:06.499039+0000","last_clean":"2024-03-26T05:11:06.499039+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:11:06.499039+0000","last_undegraded":"2024-03-26T05:11:06.499039+0000","last_fullsized":"2024-03-26T05:11:06.499039+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:49.831323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:07:49.831323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.559016+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"254'25","reported_seq":500,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499543+0000","last_change":"2024-03-26T05:10:48.381843+0000","last_active":"2024-03-26T05:11:06.499543+0000","last_peered":"2024-03-26T05:11:06.499543+0000","last_clean":"2024-03-26T05:11:06.499543+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:11:06.499543+0000","last_undegraded":"2024-03-26T05:11:06.499543+0000","last_fullsized":"2024-03-26T05:11:06.499543+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:10:48.381743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:48.381743+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:49.809560+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"271'347","reported_seq":368,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:10.369187+0000","last_change":"2024-03-26T05:10:55.754903+0000","last_active":"2024-03-26T05:11:10.369187+0000","last_peered":"2024-03-26T05:11:10.369187+0000","last_clean":"2024-03-26T05:11:10.369187+0000","last_became_active":"2024-03-26T05:10:55.754440+0000","last_became_peered":"2024-03-26T05:10:55.754440+0000","last_unstale":"2024-03-26T05:11:10.369187+0000","last_undegraded":"2024-03-26T05:11:10.369187+0000","last_fullsized":"2024-03-26T05:11:10.369187+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:13.183303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22740992,"num_objects":347,"num_object_clones":0,"num_object_copies":1041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":347,"num_write_kb":22208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":439,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.500998+0000","last_change":"2024-03-26T05:07:38.496117+0000","last_active":"2024-03-26T05:11:06.500998+0000","last_peered":"2024-03-26T05:11:06.500998+0000","last_clean":"2024-03-26T05:11:06.500998+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:11:06.500998+0000","last_undegraded":"2024-03-26T05:11:06.500998+0000","last_fullsized":"2024-03-26T05:11:06.500998+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:07:38.495883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:07:38.495883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:11.101954+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":443,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676566+0000","last_change":"2024-03-26T05:09:56.509351+0000","last_active":"2024-03-26T05:11:06.676566+0000","last_peered":"2024-03-26T05:11:06.676566+0000","last_clean":"2024-03-26T05:11:06.676566+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:11:06.676566+0000","last_undegraded":"2024-03-26T05:11:06.676566+0000","last_fullsized":"2024-03-26T05:11:06.676566+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:09:56.509231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:07.322432+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":472,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.676705+0000","last_change":"2024-03-26T05:09:45.550058+0000","last_active":"2024-03-26T05:11:06.676705+0000","last_peered":"2024-03-26T05:11:06.676705+0000","last_clean":"2024-03-26T05:11:06.676705+0000","last_became_active":"2024-03-26T05:06:06.521150+0000","last_became_peered":"2024-03-26T05:06:06.521150+0000","last_unstale":"2024-03-26T05:11:06.676705+0000","last_undegraded":"2024-03-26T05:11:06.676705+0000","last_fullsized":"2024-03-26T05:11:06.676705+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:09:45.549936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"254'31","reported_seq":498,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499751+0000","last_change":"2024-03-26T05:09:58.433286+0000","last_active":"2024-03-26T05:11:06.499751+0000","last_peered":"2024-03-26T05:11:06.499751+0000","last_clean":"2024-03-26T05:11:06.499751+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:11:06.499751+0000","last_undegraded":"2024-03-26T05:11:06.499751+0000","last_fullsized":"2024-03-26T05:11:06.499751+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:09:58.433038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:58.433038+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.559253+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"271'390","reported_seq":412,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:10.622762+0000","last_change":"2024-03-26T05:10:55.755077+0000","last_active":"2024-03-26T05:11:10.622762+0000","last_peered":"2024-03-26T05:11:10.622762+0000","last_clean":"2024-03-26T05:11:10.622762+0000","last_became_active":"2024-03-26T05:10:55.754445+0000","last_became_peered":"2024-03-26T05:10:55.754445+0000","last_unstale":"2024-03-26T05:11:10.622762+0000","last_undegraded":"2024-03-26T05:11:10.622762+0000","last_fullsized":"2024-03-26T05:11:10.622762+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:13.274589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25559040,"num_objects":390,"num_object_clones":0,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":390,"num_write_kb":24960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":441,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499743+0000","last_change":"2024-03-26T05:09:38.506191+0000","last_active":"2024-03-26T05:11:06.499743+0000","last_peered":"2024-03-26T05:11:06.499743+0000","last_clean":"2024-03-26T05:11:06.499743+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:11:06.499743+0000","last_undegraded":"2024-03-26T05:11:06.499743+0000","last_fullsized":"2024-03-26T05:11:06.499743+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:38.506070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:38.506070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.559113+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"216'11","reported_seq":611,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.500613+0000","last_change":"2024-03-26T05:10:30.100941+0000","last_active":"2024-03-26T05:11:06.500613+0000","last_peered":"2024-03-26T05:11:06.500613+0000","last_clean":"2024-03-26T05:11:06.500613+0000","last_became_active":"2024-03-26T05:07:59.960519+0000","last_became_peered":"2024-03-26T05:07:59.960519+0000","last_unstale":"2024-03-26T05:11:06.500613+0000","last_undegraded":"2024-03-26T05:11:06.500613+0000","last_fullsized":"2024-03-26T05:11:06.500613+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:10:30.100859+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.100859+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:56.657116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":457,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.526522+0000","last_change":"2024-03-26T05:09:55.525744+0000","last_active":"2024-03-26T05:11:06.526522+0000","last_peered":"2024-03-26T05:11:06.526522+0000","last_clean":"2024-03-26T05:11:06.526522+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:11:06.526522+0000","last_undegraded":"2024-03-26T05:11:06.526522+0000","last_fullsized":"2024-03-26T05:11:06.526522+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:55.525633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.525633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:08.460195+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"254'20","reported_seq":479,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-26T05:11:06.499325+0000","last_change":"2024-03-26T05:09:49.460933+0000","last_active":"2024-03-26T05:11:06.499325+0000","last_peered":"2024-03-26T05:11:06.499325+0000","last_clean":"2024-03-26T05:11:06.499325+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:11:06.499325+0000","last_undegraded":"2024-03-26T05:11:06.499325+0000","last_fullsized":"2024-03-26T05:11:06.499325+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:09:49.460784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:49.460784+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:18.489025+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":331022336,"num_objects":5051,"num_object_clones":0,"num_object_copies":15153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":323264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":935264256,"data_stored":935264256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5051,"ondisk_log_size":5051,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":640,"num_read_kb":635,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1413,"num_read_kb":1138,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":1870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":934,"ondisk_log_size":934,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005190,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":234056,"kb_used_data":36288,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93511096,"statfs":{"total":95995035648,"available":95755362304,"internally_reserved":0,"allocated":37158912,"data_stored":36696811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":202504650},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332235,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":332536,"kb_used_data":40368,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93412616,"statfs":{"total":95995035648,"available":95654518784,"internally_reserved":0,"allocated":41336832,"data_stored":40889599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575013,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":519132,"kb_used_data":190868,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93226020,"statfs":{"total":95995035648,"available":95463444480,"internally_reserved":0,"allocated":195448832,"data_stored":194965125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738538,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":515220,"kb_used_data":168908,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93229932,"statfs":{"total":95995035648,"available":95467450368,"internally_reserved":0,"allocated":172961792,"data_stored":172487556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411488,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":351352,"kb_used_data":49456,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93393800,"statfs":{"total":95995035648,"available":95635251200,"internally_reserved":0,"allocated":50642944,"data_stored":50195311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117146,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":533648,"kb_used_data":181768,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93211504,"statfs":{"total":95995035648,"available":95448580096,"internally_reserved":0,"allocated":186130432,"data_stored":185658559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":360315335},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920918,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300236,"kb_used_data":30276,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":93444916,"statfs":{"total":95995035648,"available":95687593984,"internally_reserved":0,"allocated":31002624,"data_stored":30535436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659280,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":512888,"kb_used_data":90224,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":93232264,"statfs":{"total":95995035648,"available":95469838336,"internally_reserved":0,"allocated":92389376,"data_stored":91941925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":432798149},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110755840,"data_stored":110755840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":194379776,"data_stored":194379776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62652416,"data_stored":62652416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":202244096,"data_stored":202244096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":43646976,"data_stored":43646976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":138215424,"data_stored":138215424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49610752,"data_stored":49610752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":133758976,"data_stored":133758976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:11:15.261 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool set unique_pool_2 pgp_num 16 2024-03-26T05:11:15.264 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:15.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:15.366 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:15.401 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:15.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:15.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:15.541 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:15.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:15.559 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:11:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:11:15.560 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:15 smithi080 ceph-mon[29691]: pgmap v491: 148 pgs: 148 active+clean; 316 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 631 op/s 2024-03-26T05:11:15.643 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:15.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:15.684 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:15.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:15.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 7 15 19683 19668 175.585 160.312 0.00480677 0.00568921 2024-03-26T05:11:15.784 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:15.842 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:15.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:15.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:15.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:15 smithi080 ceph-mon[40404]: pgmap v491: 148 pgs: 148 active+clean; 316 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 631 op/s 2024-03-26T05:11:15.945 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:15.981 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:15.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:15.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:16.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:15 smithi132 ceph-mon[33500]: pgmap v491: 148 pgs: 148 active+clean; 316 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 631 op/s 2024-03-26T05:11:16.083 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:16.130 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:16.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:16.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:16.232 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:16.266 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:16.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:16.367 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:16.404 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:16.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:16.506 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:16.545 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:16.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:16.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:16.647 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:16.649 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:16 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:16.693 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:16.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:16.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:16.722 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:11:16.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 8 16 21616 21600 168.729 120.75 0.00207224 0.00588188 2024-03-26T05:11:16.771 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:11:16.795 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:16.801 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:11:16.802 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:16 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:16.802 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:16 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:16.826 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:16.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:16.833 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:11:16.866 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:11:16.909 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:11:16.928 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:16.935 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:11:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:16.952 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:16.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:16.962 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:11:16.970 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:11:16.970 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:11:16.970 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:11:16.970 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:11:16.970 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:11:16.970 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:11:16.970 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:11:16.970 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:11:17.053 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:17.084 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:17.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:17.186 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:17.221 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:17.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:17.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:17.323 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:17.361 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:17.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:17.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:17.463 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:17.489 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:17.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:17.591 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:17.639 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:17.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:17.677 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:11:17.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 9 16 23420 23404 162.509 112.75 0.00261556 0.00613107 2024-03-26T05:11:17.739 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:17.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:17.763 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:17.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:17.863 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:17.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:17 smithi080 ceph-mon[40404]: pgmap v492: 148 pgs: 148 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T05:11:17.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:17 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1672359099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:17.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:17 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2064578901' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:11:17.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:17 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:11:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:17 smithi080 ceph-mon[29691]: pgmap v492: 148 pgs: 148 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T05:11:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:17 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1672359099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:17 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2064578901' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:11:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:17 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:11:17.887 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:17.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:17.899 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:17 smithi132 ceph-mon[33500]: pgmap v492: 148 pgs: 148 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T05:11:17.899 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:17 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1672359099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:17.899 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:17 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2064578901' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:11:17.899 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:17 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:11:17.989 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:18.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:18.033 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:18.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:18.134 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:18.158 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:18.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:18.260 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:18.286 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:18.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:18.388 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:18.426 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:18.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:18.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:18.528 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:18.553 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:18.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:18.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:18.655 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:18.689 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:18.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:18.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 10 16 24828 24812 155.057 88 0.0108172 0.00644297 2024-03-26T05:11:18.791 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:18.820 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:18.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:18.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:18 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:11:18.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:18 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2064578901' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:11:18.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:18 smithi080 ceph-mon[40404]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T05:11:18.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:18 smithi080 ceph-mon[40404]: pgmap v494: 148 pgs: 148 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T05:11:18.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:18 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:11:18.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:18 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2064578901' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:11:18.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:18 smithi080 ceph-mon[29691]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T05:11:18.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:18 smithi080 ceph-mon[29691]: pgmap v494: 148 pgs: 148 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T05:11:18.921 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:18.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:18.959 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:18.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:18.962 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:11:18.962 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:11:18.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:18 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:11:18.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:18 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2064578901' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:11:18.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:18 smithi132 ceph-mon[33500]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T05:11:18.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:18 smithi132 ceph-mon[33500]: pgmap v494: 148 pgs: 148 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T05:11:19.061 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:19.095 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:19.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:19.197 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:19.222 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:19.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:19.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:19.324 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:19.361 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:19.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:19.463 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:19.487 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:19.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:19.589 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:19.625 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:19.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:19.726 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:19.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 11 15 26583 26568 150.938 109.75 0.00637011 0.00661943 2024-03-26T05:11:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:19.758 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:19.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:19.859 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:19.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:19 smithi080 ceph-mon[40404]: 4.6 scrub starts 2024-03-26T05:11:19.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:19 smithi080 ceph-mon[40404]: 4.6 scrub ok 2024-03-26T05:11:19.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:19 smithi080 ceph-mon[40404]: 4.11 scrub starts 2024-03-26T05:11:19.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:19 smithi080 ceph-mon[40404]: 4.11 scrub ok 2024-03-26T05:11:19.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:19 smithi080 ceph-mon[40404]: 6.4 scrub starts 2024-03-26T05:11:19.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:19 smithi080 ceph-mon[40404]: 6.4 scrub ok 2024-03-26T05:11:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:19 smithi080 ceph-mon[29691]: 4.6 scrub starts 2024-03-26T05:11:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:19 smithi080 ceph-mon[29691]: 4.6 scrub ok 2024-03-26T05:11:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:19 smithi080 ceph-mon[29691]: 4.11 scrub starts 2024-03-26T05:11:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:19 smithi080 ceph-mon[29691]: 4.11 scrub ok 2024-03-26T05:11:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:19 smithi080 ceph-mon[29691]: 6.4 scrub starts 2024-03-26T05:11:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:19 smithi080 ceph-mon[29691]: 6.4 scrub ok 2024-03-26T05:11:19.883 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:19.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:19.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:19.972 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:19.985 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:20.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:20.034 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:20.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:19 smithi132 ceph-mon[33500]: 4.6 scrub starts 2024-03-26T05:11:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:19 smithi132 ceph-mon[33500]: 4.6 scrub ok 2024-03-26T05:11:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:19 smithi132 ceph-mon[33500]: 4.11 scrub starts 2024-03-26T05:11:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:19 smithi132 ceph-mon[33500]: 4.11 scrub ok 2024-03-26T05:11:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:19 smithi132 ceph-mon[33500]: 6.4 scrub starts 2024-03-26T05:11:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:19 smithi132 ceph-mon[33500]: 6.4 scrub ok 2024-03-26T05:11:20.135 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:20.171 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:20.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:20.273 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:20.303 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:20.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:20.404 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:20.441 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:20.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:20.543 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:20.579 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:20.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:20.680 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:20.721 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:20.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:20.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 12 16 28261 28245 147.092 104.812 0.0064162 0.0067924 2024-03-26T05:11:20.823 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:20.852 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T05:11:20.860 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:20.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:20.962 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:21.008 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:21.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:21.109 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:21.147 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:21.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:21.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[40404]: 5.17 scrub starts 2024-03-26T05:11:21.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[40404]: 5.17 scrub ok 2024-03-26T05:11:21.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[40404]: 3.16 scrub starts 2024-03-26T05:11:21.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[40404]: 4.12 scrub starts 2024-03-26T05:11:21.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[40404]: 3.16 scrub ok 2024-03-26T05:11:21.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[40404]: 4.12 scrub ok 2024-03-26T05:11:21.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[40404]: 5.1f scrub starts 2024-03-26T05:11:21.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[40404]: 5.1f scrub ok 2024-03-26T05:11:21.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[40404]: pgmap v495: 148 pgs: 148 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-26T05:11:21.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:11:21.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:11:21.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[29691]: 5.17 scrub starts 2024-03-26T05:11:21.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[29691]: 5.17 scrub ok 2024-03-26T05:11:21.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[29691]: 3.16 scrub starts 2024-03-26T05:11:21.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[29691]: 4.12 scrub starts 2024-03-26T05:11:21.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[29691]: 3.16 scrub ok 2024-03-26T05:11:21.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[29691]: 4.12 scrub ok 2024-03-26T05:11:21.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[29691]: 5.1f scrub starts 2024-03-26T05:11:21.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[29691]: 5.1f scrub ok 2024-03-26T05:11:21.231 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[29691]: pgmap v495: 148 pgs: 148 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-26T05:11:21.231 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:11:21.231 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:11:21.248 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:21.265 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:21.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:20 smithi132 ceph-mon[33500]: 5.17 scrub starts 2024-03-26T05:11:21.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:20 smithi132 ceph-mon[33500]: 5.17 scrub ok 2024-03-26T05:11:21.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:20 smithi132 ceph-mon[33500]: 3.16 scrub starts 2024-03-26T05:11:21.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:20 smithi132 ceph-mon[33500]: 4.12 scrub starts 2024-03-26T05:11:21.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:20 smithi132 ceph-mon[33500]: 3.16 scrub ok 2024-03-26T05:11:21.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:20 smithi132 ceph-mon[33500]: 4.12 scrub ok 2024-03-26T05:11:21.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:20 smithi132 ceph-mon[33500]: 5.1f scrub starts 2024-03-26T05:11:21.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:20 smithi132 ceph-mon[33500]: 5.1f scrub ok 2024-03-26T05:11:21.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:20 smithi132 ceph-mon[33500]: pgmap v495: 148 pgs: 148 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-26T05:11:21.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:11:21.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:11:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:21.305 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:21.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:21.406 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:21.446 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:21.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:21.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:21.548 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:21.585 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:21.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:21.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:21.687 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:21.732 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:21.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:21.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:21.735 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 13 15 30312 30297 145.641 128.25 0.0076473 0.00685998 2024-03-26T05:11:21.835 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:21.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:21.874 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:21.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:21.976 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:22.013 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:22.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: 6.5 scrub starts 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: 6.f scrub starts 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: 6.5 scrub ok 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: 6.f scrub ok 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: 4.1d scrub starts 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: 4.1d scrub ok 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: 2.2 scrub starts 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: 2.2 scrub ok 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: 4.5 scrub starts 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: 4.5 scrub ok 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:11:22.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:11:22.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: Adjusting osd_memory_target on smithi132 to 3248M 2024-03-26T05:11:22.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:11:22.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:11:22.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:11:22.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:11:22.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: 3.14 scrub starts 2024-03-26T05:11:22.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T05:11:22.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:21 smithi132 ceph-mon[33500]: 3.14 scrub ok 2024-03-26T05:11:22.115 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:22.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: 6.5 scrub starts 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: 6.f scrub starts 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: 6.5 scrub ok 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: 6.f scrub ok 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: 4.1d scrub starts 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: 4.1d scrub ok 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: 2.2 scrub starts 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: 2.2 scrub ok 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: 4.5 scrub starts 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: 4.5 scrub ok 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:11:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:11:22.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: Adjusting osd_memory_target on smithi132 to 3248M 2024-03-26T05:11:22.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:11:22.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:11:22.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:11:22.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:11:22.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: 3.14 scrub starts 2024-03-26T05:11:22.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T05:11:22.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[40404]: 3.14 scrub ok 2024-03-26T05:11:22.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: 6.5 scrub starts 2024-03-26T05:11:22.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: 6.f scrub starts 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: 6.5 scrub ok 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: 6.f scrub ok 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: 4.1d scrub starts 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: 4.1d scrub ok 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: 2.2 scrub starts 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: 2.2 scrub ok 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: 4.5 scrub starts 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: 4.5 scrub ok 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: Adjusting osd_memory_target on smithi132 to 3248M 2024-03-26T05:11:22.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:11:22.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:11:22.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:11:22.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:11:22.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: 3.14 scrub starts 2024-03-26T05:11:22.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T05:11:22.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:21 smithi080 ceph-mon[29691]: 3.14 scrub ok 2024-03-26T05:11:22.152 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:22.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:22.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:22.254 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:22.291 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:22.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:22.392 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:22.426 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:22.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:22.528 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:22.565 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:22.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:22.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:22.667 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:22.736 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 14 16 32512 32496 145.055 137.438 0.00598409 0.00688878 2024-03-26T05:11:22.745 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:22.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:22.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:22.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:22.883 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:22.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:22.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:22.985 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:23.018 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:23.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:23.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:23.118 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:22 smithi132 ceph-mon[33500]: 6.0 scrub starts 2024-03-26T05:11:23.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:22 smithi132 ceph-mon[33500]: 6.0 scrub ok 2024-03-26T05:11:23.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:22 smithi132 ceph-mon[33500]: 6.1d scrub starts 2024-03-26T05:11:23.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:22 smithi132 ceph-mon[33500]: 6.1d scrub ok 2024-03-26T05:11:23.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:22 smithi132 ceph-mon[33500]: 3.10 scrub starts 2024-03-26T05:11:23.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:22 smithi132 ceph-mon[33500]: 3.10 scrub ok 2024-03-26T05:11:23.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:22 smithi132 ceph-mon[33500]: pgmap v497: 148 pgs: 148 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 175 MiB/s wr, 2.80k op/s 2024-03-26T05:11:23.120 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:23.121 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:11:23.170 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:23.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:23.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:23.272 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:23.276 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[40404]: 6.0 scrub starts 2024-03-26T05:11:23.277 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[40404]: 6.0 scrub ok 2024-03-26T05:11:23.277 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[40404]: 6.1d scrub starts 2024-03-26T05:11:23.277 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[40404]: 6.1d scrub ok 2024-03-26T05:11:23.277 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[40404]: 3.10 scrub starts 2024-03-26T05:11:23.277 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[40404]: 3.10 scrub ok 2024-03-26T05:11:23.277 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[40404]: pgmap v497: 148 pgs: 148 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 175 MiB/s wr, 2.80k op/s 2024-03-26T05:11:23.277 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[29691]: 6.0 scrub starts 2024-03-26T05:11:23.277 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[29691]: 6.0 scrub ok 2024-03-26T05:11:23.277 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[29691]: 6.1d scrub starts 2024-03-26T05:11:23.277 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[29691]: 6.1d scrub ok 2024-03-26T05:11:23.278 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[29691]: 3.10 scrub starts 2024-03-26T05:11:23.278 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[29691]: 3.10 scrub ok 2024-03-26T05:11:23.278 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:22 smithi080 ceph-mon[29691]: pgmap v497: 148 pgs: 148 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 175 MiB/s wr, 2.80k op/s 2024-03-26T05:11:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:23.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:23.318 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:11:23.318 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:11:23.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:23.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:23.465 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:23.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:23.566 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:23.604 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:23.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:23.706 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:23.736 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 15 16 34227 34211 142.53 107.188 0.00620355 0.00701005 2024-03-26T05:11:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:23.750 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:23.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:23.813 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:23.851 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:23.902 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:23.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:24.003 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:24.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:24.044 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:24.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:24.133 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:23 smithi132 ceph-mon[33500]: 4.b scrub starts 2024-03-26T05:11:24.133 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:23 smithi132 ceph-mon[33500]: 4.b scrub ok 2024-03-26T05:11:24.133 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:23 smithi132 ceph-mon[33500]: 6.d scrub starts 2024-03-26T05:11:24.133 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:23 smithi132 ceph-mon[33500]: 6.d scrub ok 2024-03-26T05:11:24.145 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:24.173 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T05:11:24.185 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:24.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:24.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:24.262 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:23 smithi080 ceph-mon[40404]: 4.b scrub starts 2024-03-26T05:11:24.262 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:23 smithi080 ceph-mon[40404]: 4.b scrub ok 2024-03-26T05:11:24.262 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:23 smithi080 ceph-mon[40404]: 6.d scrub starts 2024-03-26T05:11:24.262 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:23 smithi080 ceph-mon[40404]: 6.d scrub ok 2024-03-26T05:11:24.263 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:23 smithi080 ceph-mon[29691]: 4.b scrub starts 2024-03-26T05:11:24.263 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:23 smithi080 ceph-mon[29691]: 4.b scrub ok 2024-03-26T05:11:24.263 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:23 smithi080 ceph-mon[29691]: 6.d scrub starts 2024-03-26T05:11:24.263 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:23 smithi080 ceph-mon[29691]: 6.d scrub ok 2024-03-26T05:11:24.263 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:11:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:11:24.286 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:24.325 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:24.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:24.426 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:24.463 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:24.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:24.565 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:24.602 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:24.607 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:24.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:24.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:24.657 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:11:24.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:24.736 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 16 16 35866 35850 140.023 102.438 0.00935086 0.00713488 2024-03-26T05:11:24.744 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:24.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:24.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:24.879 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:24.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:24.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:24.979 INFO:teuthology.orchestra.run.smithi080.stderr:set pool 9 pgp_num to 16 2024-03-26T05:11:24.981 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:25.008 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:25.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:25.029 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:24 smithi132 ceph-mon[33500]: 3.6 scrub starts 2024-03-26T05:11:25.029 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:24 smithi132 ceph-mon[33500]: 3.6 scrub ok 2024-03-26T05:11:25.029 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:24 smithi132 ceph-mon[33500]: pgmap v498: 148 pgs: 148 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-26T05:11:25.029 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:24 smithi132 ceph-mon[33500]: 5.10 scrub starts 2024-03-26T05:11:25.029 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:24 smithi132 ceph-mon[33500]: 5.10 scrub ok 2024-03-26T05:11:25.029 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2904762322' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T05:11:25.029 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:24 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T05:11:25.029 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:24 smithi132 ceph-mon[33500]: 4.13 scrub starts 2024-03-26T05:11:25.029 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:24 smithi132 ceph-mon[33500]: 4.13 scrub ok 2024-03-26T05:11:25.110 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:25.148 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:25.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:25.249 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:25.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:25.285 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:25.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:25.298 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[40404]: 3.6 scrub starts 2024-03-26T05:11:25.298 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[40404]: 3.6 scrub ok 2024-03-26T05:11:25.298 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[40404]: pgmap v498: 148 pgs: 148 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-26T05:11:25.298 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[40404]: 5.10 scrub starts 2024-03-26T05:11:25.298 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[40404]: 5.10 scrub ok 2024-03-26T05:11:25.298 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2904762322' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T05:11:25.299 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T05:11:25.299 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[40404]: 4.13 scrub starts 2024-03-26T05:11:25.299 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[40404]: 4.13 scrub ok 2024-03-26T05:11:25.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[29691]: 3.6 scrub starts 2024-03-26T05:11:25.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[29691]: 3.6 scrub ok 2024-03-26T05:11:25.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[29691]: pgmap v498: 148 pgs: 148 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-26T05:11:25.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[29691]: 5.10 scrub starts 2024-03-26T05:11:25.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[29691]: 5.10 scrub ok 2024-03-26T05:11:25.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2904762322' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T05:11:25.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T05:11:25.300 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[29691]: 4.13 scrub starts 2024-03-26T05:11:25.300 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:24 smithi080 ceph-mon[29691]: 4.13 scrub ok 2024-03-26T05:11:25.385 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:25.423 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:25.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:25.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:25.525 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:25.574 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:25.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:25.675 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:25.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:25.715 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:25.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:25.736 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 17 15 37661 37646 138.389 112.25 0.00731476 0.00722173 2024-03-26T05:11:25.772 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:11:25.816 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:25.855 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:25.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:25.957 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:25.994 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:25.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: 3.1f scrub starts 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: 3.1f scrub ok 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: 4.7 scrub starts 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: 4.7 scrub ok 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: 6.a scrub starts 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: 6.a scrub ok 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: 4.a scrub starts 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: 4.a scrub ok 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T05:11:26.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: 3.11 scrub starts 2024-03-26T05:11:26.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:25 smithi132 ceph-mon[33500]: 3.11 scrub ok 2024-03-26T05:11:26.096 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:26.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: 3.1f scrub starts 2024-03-26T05:11:26.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: 3.1f scrub ok 2024-03-26T05:11:26.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: 4.7 scrub starts 2024-03-26T05:11:26.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: 4.7 scrub ok 2024-03-26T05:11:26.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: 6.a scrub starts 2024-03-26T05:11:26.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: 6.a scrub ok 2024-03-26T05:11:26.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: 4.a scrub starts 2024-03-26T05:11:26.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: 4.a scrub ok 2024-03-26T05:11:26.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:11:26.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:11:26.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T05:11:26.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T05:11:26.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:26.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: 3.11 scrub starts 2024-03-26T05:11:26.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[40404]: 3.11 scrub ok 2024-03-26T05:11:26.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: 3.1f scrub starts 2024-03-26T05:11:26.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: 3.1f scrub ok 2024-03-26T05:11:26.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: 4.7 scrub starts 2024-03-26T05:11:26.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: 4.7 scrub ok 2024-03-26T05:11:26.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: 6.a scrub starts 2024-03-26T05:11:26.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: 6.a scrub ok 2024-03-26T05:11:26.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: 4.a scrub starts 2024-03-26T05:11:26.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: 4.a scrub ok 2024-03-26T05:11:26.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:11:26.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:11:26.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T05:11:26.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T05:11:26.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:26.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: 3.11 scrub starts 2024-03-26T05:11:26.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:25 smithi080 ceph-mon[29691]: 3.11 scrub ok 2024-03-26T05:11:26.135 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:26.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:26.238 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:26.281 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:26.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:26.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:26.383 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:26.423 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:26.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:26.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:26.526 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:26.527 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:11:26.527 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:11:26.563 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:26.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:26.665 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:26.686 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:26.696 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:26.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:26.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:26.736 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 18 16 39405 39389 136.752 108.938 0.00992841 0.00730653 2024-03-26T05:11:26.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:26.831 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:26.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:26.933 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:26.980 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:26.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:26.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:27.082 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:27.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[29691]: 5.c scrub starts 2024-03-26T05:11:27.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[29691]: 5.c scrub ok 2024-03-26T05:11:27.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[29691]: 5.11 scrub starts 2024-03-26T05:11:27.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[29691]: 5.11 scrub ok 2024-03-26T05:11:27.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[29691]: pgmap v500: 148 pgs: 148 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-26T05:11:27.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[29691]: 6.2 scrub starts 2024-03-26T05:11:27.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[29691]: 6.2 scrub ok 2024-03-26T05:11:27.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[29691]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T05:11:27.113 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[40404]: 5.c scrub starts 2024-03-26T05:11:27.113 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[40404]: 5.c scrub ok 2024-03-26T05:11:27.113 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[40404]: 5.11 scrub starts 2024-03-26T05:11:27.113 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[40404]: 5.11 scrub ok 2024-03-26T05:11:27.113 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[40404]: pgmap v500: 148 pgs: 148 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-26T05:11:27.113 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[40404]: 6.2 scrub starts 2024-03-26T05:11:27.113 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[40404]: 6.2 scrub ok 2024-03-26T05:11:27.113 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:27 smithi080 ceph-mon[40404]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T05:11:27.121 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:27.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:27.207 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:26 smithi132 ceph-mon[33500]: 5.c scrub starts 2024-03-26T05:11:27.207 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:27 smithi132 ceph-mon[33500]: 5.c scrub ok 2024-03-26T05:11:27.207 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:27 smithi132 ceph-mon[33500]: 5.11 scrub starts 2024-03-26T05:11:27.207 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:27 smithi132 ceph-mon[33500]: 5.11 scrub ok 2024-03-26T05:11:27.207 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:27 smithi132 ceph-mon[33500]: pgmap v500: 148 pgs: 148 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-26T05:11:27.207 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:27 smithi132 ceph-mon[33500]: 6.2 scrub starts 2024-03-26T05:11:27.207 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:27 smithi132 ceph-mon[33500]: 6.2 scrub ok 2024-03-26T05:11:27.208 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:27 smithi132 ceph-mon[33500]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T05:11:27.223 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:27.258 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T05:11:27.271 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:27.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:27.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:27.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:27.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:27.423 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:27.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:27.524 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:27.565 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:27.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:27.667 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:27.682 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:27.703 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:27.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:27.736 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 19 16 41178 41162 135.387 110.812 0.00897042 0.00738091 2024-03-26T05:11:27.805 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:27.842 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:27.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:27.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:27.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:27.983 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:27.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:27.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:28.014 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:11:28.085 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:28.121 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:28.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:28.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:28.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:28 smithi132 ceph-mon[33500]: 5.b scrub starts 2024-03-26T05:11:28.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:28 smithi132 ceph-mon[33500]: 5.b scrub ok 2024-03-26T05:11:28.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:28 smithi132 ceph-mon[33500]: 5.0 scrub starts 2024-03-26T05:11:28.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:28 smithi132 ceph-mon[33500]: 5.0 scrub ok 2024-03-26T05:11:28.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:28 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3979078097' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:11:28.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:28 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:11:28.223 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:28.260 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:28.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:28.361 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:28.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:28 smithi080 ceph-mon[29691]: 5.b scrub starts 2024-03-26T05:11:28.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:28 smithi080 ceph-mon[29691]: 5.b scrub ok 2024-03-26T05:11:28.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:28 smithi080 ceph-mon[29691]: 5.0 scrub starts 2024-03-26T05:11:28.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:28 smithi080 ceph-mon[29691]: 5.0 scrub ok 2024-03-26T05:11:28.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:28 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3979078097' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:11:28.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:28 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:11:28.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:28 smithi080 ceph-mon[40404]: 5.b scrub starts 2024-03-26T05:11:28.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:28 smithi080 ceph-mon[40404]: 5.b scrub ok 2024-03-26T05:11:28.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:28 smithi080 ceph-mon[40404]: 5.0 scrub starts 2024-03-26T05:11:28.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:28 smithi080 ceph-mon[40404]: 5.0 scrub ok 2024-03-26T05:11:28.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:28 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3979078097' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:11:28.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:28 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:11:28.401 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:28.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:28.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:28.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:28.539 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:28.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:28.640 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:28.675 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:28.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:28.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:28.736 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:11:28.746538+0000 min lat: 0.00115197 max lat: 0.140466 avg lat: 0.00742959 2024-03-26T05:11:28.736 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:11:28.736 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 20 16 43063 43047 134.507 117.812 0.00508228 0.00742959 2024-03-26T05:11:28.778 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:28.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:28.828 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:28.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:28.929 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:28.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:28.974 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:28.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:29.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[29691]: 5.1e scrub starts 2024-03-26T05:11:29.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[29691]: 5.1e scrub ok 2024-03-26T05:11:29.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[29691]: 3.19 scrub starts 2024-03-26T05:11:29.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[29691]: 3.19 scrub ok 2024-03-26T05:11:29.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[29691]: pgmap v502: 148 pgs: 148 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 797 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T05:11:29.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[29691]: 3.d scrub starts 2024-03-26T05:11:29.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[29691]: 3.d scrub ok 2024-03-26T05:11:29.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:11:29.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[29691]: Cluster is now healthy 2024-03-26T05:11:29.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:11:29.068 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[29691]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T05:11:29.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[40404]: 5.1e scrub starts 2024-03-26T05:11:29.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[40404]: 5.1e scrub ok 2024-03-26T05:11:29.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[40404]: 3.19 scrub starts 2024-03-26T05:11:29.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[40404]: 3.19 scrub ok 2024-03-26T05:11:29.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[40404]: pgmap v502: 148 pgs: 148 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 797 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T05:11:29.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[40404]: 3.d scrub starts 2024-03-26T05:11:29.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[40404]: 3.d scrub ok 2024-03-26T05:11:29.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:11:29.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[40404]: Cluster is now healthy 2024-03-26T05:11:29.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:11:29.069 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:29 smithi080 ceph-mon[40404]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T05:11:29.075 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:29.112 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:29.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:29.168 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:29 smithi132 ceph-mon[33500]: 5.1e scrub starts 2024-03-26T05:11:29.168 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:29 smithi132 ceph-mon[33500]: 5.1e scrub ok 2024-03-26T05:11:29.168 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:29 smithi132 ceph-mon[33500]: 3.19 scrub starts 2024-03-26T05:11:29.168 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:29 smithi132 ceph-mon[33500]: 3.19 scrub ok 2024-03-26T05:11:29.168 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:29 smithi132 ceph-mon[33500]: pgmap v502: 148 pgs: 148 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 797 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T05:11:29.168 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:29 smithi132 ceph-mon[33500]: 3.d scrub starts 2024-03-26T05:11:29.168 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:29 smithi132 ceph-mon[33500]: 3.d scrub ok 2024-03-26T05:11:29.168 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:29 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:11:29.168 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:29 smithi132 ceph-mon[33500]: Cluster is now healthy 2024-03-26T05:11:29.169 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:29 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:11:29.169 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:29 smithi132 ceph-mon[33500]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T05:11:29.214 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:29.253 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:29.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:29.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:29.355 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:29.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:29.397 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:29.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:29.445 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:11:29.483 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:11:29.498 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:29.521 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:11:29.536 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:29.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:29.569 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:11:29.611 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:11:29.639 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:29.652 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:11:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:29.681 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:29.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:29.694 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:11:29.694 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:11:29.694 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:11:29.695 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2024-03-26T05:11:29.695 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2024-03-26T05:11:29.695 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:11:29.736 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:11:29.736 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 21 16 44743 44727 133.102 105 0.0126953 0.00750752 2024-03-26T05:11:29.744 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:11:29.744 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:11:29.744 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:11:29.744 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:11:29.745 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:11:29.745 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:11:29.745 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:11:29.745 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:11:29.782 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:29.825 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:29.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:29.927 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:29.967 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:29.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:29.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:30.069 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:30.105 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:30.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:30.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:30.207 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:30.250 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:30.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:30.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:30 smithi132 ceph-mon[33500]: 4.2 scrub starts 2024-03-26T05:11:30.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:30 smithi132 ceph-mon[33500]: 4.2 scrub ok 2024-03-26T05:11:30.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:30 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:11:30.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:30 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:11:30.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:30 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1328161882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:30.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:30 smithi132 ceph-mon[33500]: 3.9 scrub starts 2024-03-26T05:11:30.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:30 smithi132 ceph-mon[33500]: 3.9 scrub ok 2024-03-26T05:11:30.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:30 smithi132 ceph-mon[33500]: 3.f deep-scrub starts 2024-03-26T05:11:30.351 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:30.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[40404]: 4.2 scrub starts 2024-03-26T05:11:30.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[40404]: 4.2 scrub ok 2024-03-26T05:11:30.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:11:30.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:11:30.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1328161882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:30.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[40404]: 3.9 scrub starts 2024-03-26T05:11:30.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[40404]: 3.9 scrub ok 2024-03-26T05:11:30.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[40404]: 3.f deep-scrub starts 2024-03-26T05:11:30.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[29691]: 4.2 scrub starts 2024-03-26T05:11:30.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[29691]: 4.2 scrub ok 2024-03-26T05:11:30.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:11:30.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:11:30.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1328161882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:30.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[29691]: 3.9 scrub starts 2024-03-26T05:11:30.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[29691]: 3.9 scrub ok 2024-03-26T05:11:30.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:30 smithi080 ceph-mon[29691]: 3.f deep-scrub starts 2024-03-26T05:11:30.386 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:30.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:30.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:30.527 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:30.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:30.628 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:30.664 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:30.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:30.736 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 22 16 46552 46536 132.191 113.062 0.00620187 0.00755931 2024-03-26T05:11:30.752 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T05:11:30.767 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:30.806 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:30.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:30.831 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:11:30.908 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:30.957 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:30.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:31.059 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:31.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:31.102 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:31.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:31.137 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:31.203 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:31.226 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:11:31.226 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:11:31.226 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-03-26T05:11:31.227 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T05:11:31.227 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T05:11:31.227 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-03-26T05:11:31.235 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:31.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:31.272 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:31 smithi132 ceph-mon[33500]: 4.3 scrub starts 2024-03-26T05:11:31.272 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:31 smithi132 ceph-mon[33500]: 4.3 scrub ok 2024-03-26T05:11:31.272 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:31 smithi132 ceph-mon[33500]: 4.1c scrub starts 2024-03-26T05:11:31.272 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:31 smithi132 ceph-mon[33500]: 4.1c scrub ok 2024-03-26T05:11:31.272 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:31 smithi132 ceph-mon[33500]: 3.4 deep-scrub starts 2024-03-26T05:11:31.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:31 smithi132 ceph-mon[33500]: 3.4 deep-scrub ok 2024-03-26T05:11:31.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:31 smithi132 ceph-mon[33500]: pgmap v504: 148 pgs: 148 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-26T05:11:31.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:31 smithi132 ceph-mon[33500]: 3.f deep-scrub ok 2024-03-26T05:11:31.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:31 smithi132 ceph-mon[33500]: 4.1a scrub starts 2024-03-26T05:11:31.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:31 smithi132 ceph-mon[33500]: 4.1a scrub ok 2024-03-26T05:11:31.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:31 smithi132 ceph-mon[33500]: 3.1c scrub starts 2024-03-26T05:11:31.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:31 smithi132 ceph-mon[33500]: 3.1c scrub ok 2024-03-26T05:11:31.278 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:31.338 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:31.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[29691]: 4.3 scrub starts 2024-03-26T05:11:31.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[29691]: 4.3 scrub ok 2024-03-26T05:11:31.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[29691]: 4.1c scrub starts 2024-03-26T05:11:31.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[29691]: 4.1c scrub ok 2024-03-26T05:11:31.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[29691]: 3.4 deep-scrub starts 2024-03-26T05:11:31.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[29691]: 3.4 deep-scrub ok 2024-03-26T05:11:31.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[29691]: pgmap v504: 148 pgs: 148 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-26T05:11:31.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[29691]: 3.f deep-scrub ok 2024-03-26T05:11:31.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[29691]: 4.1a scrub starts 2024-03-26T05:11:31.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[29691]: 4.1a scrub ok 2024-03-26T05:11:31.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[29691]: 3.1c scrub starts 2024-03-26T05:11:31.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[29691]: 3.1c scrub ok 2024-03-26T05:11:31.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[40404]: 4.3 scrub starts 2024-03-26T05:11:31.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[40404]: 4.3 scrub ok 2024-03-26T05:11:31.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[40404]: 4.1c scrub starts 2024-03-26T05:11:31.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[40404]: 4.1c scrub ok 2024-03-26T05:11:31.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[40404]: 3.4 deep-scrub starts 2024-03-26T05:11:31.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[40404]: 3.4 deep-scrub ok 2024-03-26T05:11:31.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[40404]: pgmap v504: 148 pgs: 148 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-26T05:11:31.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[40404]: 3.f deep-scrub ok 2024-03-26T05:11:31.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[40404]: 4.1a scrub starts 2024-03-26T05:11:31.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[40404]: 4.1a scrub ok 2024-03-26T05:11:31.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[40404]: 3.1c scrub starts 2024-03-26T05:11:31.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:31 smithi080 ceph-mon[40404]: 3.1c scrub ok 2024-03-26T05:11:31.379 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:31.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:31.481 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:31.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:31.539 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:31.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:31.640 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:31.679 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:31.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:31.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 23 16 48158 48142 130.807 100.375 0.00727199 0.00763957 2024-03-26T05:11:31.780 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:31.816 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:31.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:31.917 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:31.957 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:31.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:32.058 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:32.105 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:32.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:32.206 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:32.244 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:32.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:32.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:32 smithi132 ceph-mon[33500]: 6.9 scrub starts 2024-03-26T05:11:32.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:32 smithi132 ceph-mon[33500]: 6.9 scrub ok 2024-03-26T05:11:32.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:32 smithi132 ceph-mon[33500]: 5.19 scrub starts 2024-03-26T05:11:32.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:32 smithi132 ceph-mon[33500]: 5.19 scrub ok 2024-03-26T05:11:32.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:32 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:11:32.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:32 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:11:32.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:32 smithi132 ceph-mon[33500]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T05:11:32.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:32 smithi132 ceph-mon[33500]: 6.7 scrub starts 2024-03-26T05:11:32.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:32 smithi132 ceph-mon[33500]: 6.7 scrub ok 2024-03-26T05:11:32.346 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:32.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[29691]: 6.9 scrub starts 2024-03-26T05:11:32.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[29691]: 6.9 scrub ok 2024-03-26T05:11:32.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[29691]: 5.19 scrub starts 2024-03-26T05:11:32.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[29691]: 5.19 scrub ok 2024-03-26T05:11:32.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:11:32.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:11:32.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[29691]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T05:11:32.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[29691]: 6.7 scrub starts 2024-03-26T05:11:32.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[29691]: 6.7 scrub ok 2024-03-26T05:11:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[40404]: 6.9 scrub starts 2024-03-26T05:11:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[40404]: 6.9 scrub ok 2024-03-26T05:11:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[40404]: 5.19 scrub starts 2024-03-26T05:11:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[40404]: 5.19 scrub ok 2024-03-26T05:11:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:11:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:11:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[40404]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T05:11:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[40404]: 6.7 scrub starts 2024-03-26T05:11:32.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:32 smithi080 ceph-mon[40404]: 6.7 scrub ok 2024-03-26T05:11:32.382 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:32.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:32.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:32.484 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:32.486 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:11:32.502 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:32.533 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:32.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:32.634 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:32.666 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:32.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:32.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 24 16 49848 49832 129.757 105.625 0.0157696 0.00770181 2024-03-26T05:11:32.768 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:32.806 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:32.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:32.907 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:32.948 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:32.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:33.050 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:33.078 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:33.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:33.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:33 smithi080 ceph-mon[29691]: 6.6 scrub starts 2024-03-26T05:11:33.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:33 smithi080 ceph-mon[29691]: 6.6 scrub ok 2024-03-26T05:11:33.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:33 smithi080 ceph-mon[29691]: pgmap v506: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-26T05:11:33.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:33 smithi080 ceph-mon[40404]: 6.6 scrub starts 2024-03-26T05:11:33.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:33 smithi080 ceph-mon[40404]: 6.6 scrub ok 2024-03-26T05:11:33.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:33 smithi080 ceph-mon[40404]: pgmap v506: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-26T05:11:33.179 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:33.216 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:33.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:33.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:33 smithi132 ceph-mon[33500]: 6.6 scrub starts 2024-03-26T05:11:33.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:33 smithi132 ceph-mon[33500]: 6.6 scrub ok 2024-03-26T05:11:33.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:33 smithi132 ceph-mon[33500]: pgmap v506: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-26T05:11:33.317 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:33.367 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:33.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:33.377 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:33.468 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:33.514 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:33.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:33.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:33.616 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:33.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:33.659 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:33.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:33.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 25 16 51316 51300 128.237 91.75 0.00764688 0.0077927 2024-03-26T05:11:33.760 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:33.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:33.802 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:33.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:33.903 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:33.943 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:33.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:34.044 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:34.059 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:34 smithi080 ceph-mon[29691]: 3.15 scrub starts 2024-03-26T05:11:34.059 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:34 smithi080 ceph-mon[29691]: 3.15 scrub ok 2024-03-26T05:11:34.059 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:34 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1917602050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:11:34.059 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:34 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:11:34.060 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:11:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:11:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:11:34.060 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:34 smithi080 ceph-mon[40404]: 3.15 scrub starts 2024-03-26T05:11:34.060 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:34 smithi080 ceph-mon[40404]: 3.15 scrub ok 2024-03-26T05:11:34.060 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:34 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1917602050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:11:34.060 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:34 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:11:34.062 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:11:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:34.114 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:34.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:34.215 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:34.251 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:34.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:34.325 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:34 smithi132 ceph-mon[33500]: 3.15 scrub starts 2024-03-26T05:11:34.326 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:34 smithi132 ceph-mon[33500]: 3.15 scrub ok 2024-03-26T05:11:34.326 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:34 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1917602050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:11:34.326 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:34 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:11:34.353 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:34.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:34.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:34.493 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:34.531 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:34.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:34.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:34.633 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:34.667 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:34.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:34.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 26 16 52836 52820 126.958 95 0.00258121 0.00786214 2024-03-26T05:11:34.769 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:34.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:34.811 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:34.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:34.912 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:34.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:34.953 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:34.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:35.054 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:35.094 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:35.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:35.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:35.196 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:35.238 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:35.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:35.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:35.340 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:35.354 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:35 smithi132 ceph-mon[33500]: 4.19 scrub starts 2024-03-26T05:11:35.354 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:35 smithi132 ceph-mon[33500]: 4.19 scrub ok 2024-03-26T05:11:35.354 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:35 smithi132 ceph-mon[33500]: pgmap v507: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 690 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-26T05:11:35.354 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:35 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:11:35.354 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:35 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:11:35.354 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:35 smithi132 ceph-mon[33500]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T05:11:35.354 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:35 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:11:35.355 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:35 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:11:35.355 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:11:35.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[40404]: 4.19 scrub starts 2024-03-26T05:11:35.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[40404]: 4.19 scrub ok 2024-03-26T05:11:35.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[40404]: pgmap v507: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 690 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-26T05:11:35.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:11:35.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:11:35.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[40404]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T05:11:35.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:11:35.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:11:35.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:11:35.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[29691]: 4.19 scrub starts 2024-03-26T05:11:35.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[29691]: 4.19 scrub ok 2024-03-26T05:11:35.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[29691]: pgmap v507: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 690 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-26T05:11:35.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:11:35.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:11:35.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[29691]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T05:11:35.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:11:35.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:11:35.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:11:35.382 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:35.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:35.484 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:35.525 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:35.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:35.597 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:11:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:11:35.627 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:35.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:35.657 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:35.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:35.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 27 16 54234 54218 125.492 87.375 0.00402648 0.00796352 2024-03-26T05:11:35.758 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:35.798 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:35.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:35.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:35.900 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:35.923 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:35.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:35.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:36.024 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:36.058 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:36.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:36.106 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:36 smithi080 ceph-mon[40404]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T05:11:36.106 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:36.106 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:36 smithi080 ceph-mon[29691]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T05:11:36.106 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:36.160 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:36.185 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:36.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:36.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:36.289 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:36.315 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:36.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:36.417 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:36.447 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:36.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:36.510 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:36 smithi132 ceph-mon[33500]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T05:11:36.510 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:36.549 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:36.581 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:36.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:36.683 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:36.721 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:36.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:36.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:36.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 28 16 55277 55261 123.338 65.1875 0.0175797 0.00810186 2024-03-26T05:11:36.805 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:11:36.822 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:36.825 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:11:36.847 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:11:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:36.847 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:36.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:36.888 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:11:36.916 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:11:36.937 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:11:36.948 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:36.965 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:11:36.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:36.988 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:36.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:37.008 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:11:37.011 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:11:37.012 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:11:37.012 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:11:37.012 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:11:37.012 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:11:37.012 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:11:37.012 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:11:37.012 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:11:37.079 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:11:37.079 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198054 8.9 8:99f5ccfa:::benchmark_data_smithi080_109213_object83537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:53.895008+0000", 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 101.68330165499999, 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20899105000000001, 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198054 2024-03-26T05:11:37.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895008+0000", 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895008+0000", 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895010+0000", 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.72e-06 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895014+0000", 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.69e-06 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895028+0000", 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3808000000000001e-05 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895036+0000", 2024-03-26T05:11:37.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6459999999999993e-06 2024-03-26T05:11:37.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.981756+0000", 2024-03-26T05:11:37.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.086720452000000003 2024-03-26T05:11:37.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.981876+0000", 2024-03-26T05:11:37.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119364 2024-03-26T05:11:37.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.982000+0000", 2024-03-26T05:11:37.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012400099999999999 2024-03-26T05:11:37.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992267+0000", 2024-03-26T05:11:37.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010267642 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992404+0000", 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000136713 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.103815+0000", 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.111411407 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.103923+0000", 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000107909 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.103999+0000", 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5698000000000006e-05 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198055 8.9 8:9376f681:::benchmark_data_smithi080_109213_object83538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.086 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:53.895393+0000", 2024-03-26T05:11:37.087 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 101.682916412, 2024-03-26T05:11:37.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.208813532, 2024-03-26T05:11:37.087 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.087 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.087 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.087 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.087 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.087 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198055 2024-03-26T05:11:37.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.087 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895393+0000", 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895393+0000", 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895394+0000", 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6400000000000002e-07 2024-03-26T05:11:37.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895397+0000", 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5499999999999999e-06 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895410+0000", 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2989e-05 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895417+0000", 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4330000000000001e-06 2024-03-26T05:11:37.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.984852+0000", 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.089435597000000006 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.984975+0000", 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000122328 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.985100+0000", 2024-03-26T05:11:37.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000125513 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992272+0000", 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0071720559999999996 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002218+0000", 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.009945934 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104058+0000", 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.101839334 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104151+0000", 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3468999999999996e-05 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104207+0000", 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5665000000000003e-05 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198065 8.9 8:9d5183f8:::benchmark_data_smithi080_109213_object83548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:53.906108+0000", 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 101.672201547, 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.19828168400000001, 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198065 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.093 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.906108+0000", 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.906108+0000", 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.906109+0000", 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2099999999999995e-07 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.906112+0000", 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0510000000000001e-06 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.906122+0000", 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7829999999999993e-06 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.906129+0000", 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2080000000000003e-06 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.988438+0000", 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.082309388999999997 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.988554+0000", 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011518099999999999 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.988686+0000", 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132564 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.998891+0000", 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010205182 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002261+0000", 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0033697000000000002 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104254+0000", 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.101992572 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104333+0000", 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9555999999999995e-05 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104390+0000", 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6576999999999997e-05 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.098 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198081 8.9 8:950f9248:::benchmark_data_smithi080_109213_object83564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:53.917093+0000", 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 101.661216161, 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18745711200000001, 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198081 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.099 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.917093+0000", 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.917093+0000", 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.917094+0000", 2024-03-26T05:11:37.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.61e-07 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.917098+0000", 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7610000000000001e-06 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.917111+0000", 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2676e-05 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.917117+0000", 2024-03-26T05:11:37.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9639999999999997e-06 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992434+0000", 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.075316607999999993 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992546+0000", 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000112419 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992657+0000", 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000110328 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002229+0000", 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0095720879999999994 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002294+0000", 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4789000000000006e-05 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104438+0000", 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.102143832 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104498+0000", 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0931999999999999e-05 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104551+0000", 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2054e-05 2024-03-26T05:11:37.104 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198086 8.9 8:9e1cc76a:::benchmark_data_smithi080_109213_object83569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:53.922111+0000", 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 101.656198648, 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18262619699999999, 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198086 2024-03-26T05:11:37.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.922111+0000", 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.922111+0000", 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.922112+0000", 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.184e-06 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.922117+0000", 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2549999999999997e-06 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.922133+0000", 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5877999999999999e-05 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.922142+0000", 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0559999999999994e-06 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.995725+0000", 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.073583027999999995 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.995859+0000", 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000134108 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.995989+0000", 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000129417 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002303+0000", 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0063138350000000003 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002694+0000", 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00039094999999999999 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104638+0000", 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10194389099999999 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104685+0000", 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7749e-05 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104737+0000", 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1845999999999998e-05 2024-03-26T05:11:37.110 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198508 8.9 8:986ea8e3:::benchmark_data_smithi080_109213_object83991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.353239+0000", 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 101.22507078, 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20569538500000001, 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.111 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198508 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353239+0000", 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353239+0000", 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353240+0000", 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4100000000000001e-06 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353245+0000", 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2000000000000002e-06 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353258+0000", 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2106000000000001e-05 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353272+0000", 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4893e-05 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.406729+0000", 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.053456469 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.406825+0000", 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6075000000000004e-05 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.406903+0000", 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8450999999999998e-05 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.409021+0000", 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002117819 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.410391+0000", 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001370008 2024-03-26T05:11:37.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.558821+0000", 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.148429591 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.558877+0000", 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6178000000000003e-05 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.558934+0000", 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7185000000000002e-05 2024-03-26T05:11:37.116 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198509 8.9 8:914b5413:::benchmark_data_smithi080_109213_object83992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.353322+0000", 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 101.224987687, 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.205746507, 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198509 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.117 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353322+0000", 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353322+0000", 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353322+0000", 2024-03-26T05:11:37.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9500000000000003e-07 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353325+0000", 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.909e-06 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353333+0000", 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3699999999999997e-06 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353340+0000", 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1969999999999999e-06 2024-03-26T05:11:37.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.410429+0000", 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.057089411999999999 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.410567+0000", 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000137405 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.410721+0000", 2024-03-26T05:11:37.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015416099999999999 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416309+0000", 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0055884469999999999 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416351+0000", 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2101999999999998e-05 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.558984+0000", 2024-03-26T05:11:37.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14263231300000001 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.559015+0000", 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.1498000000000002e-05 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.559068+0000", 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3198000000000001e-05 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.122 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198524 8.9 8:97161286:::benchmark_data_smithi080_109213_object84007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.366616+0000", 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 101.21169326499999, 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20856360500000001, 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198524 2024-03-26T05:11:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366616+0000", 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366616+0000", 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366618+0000", 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4479999999999999e-06 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366623+0000", 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3310000000000003e-06 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366639+0000", 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.59e-05 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.125 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366648+0000", 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5639999999999997e-06 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.412563+0000", 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.045915688000000003 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.412676+0000", 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000113008 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.412809+0000", 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013275799999999999 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416343+0000", 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0035344500000000002 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416354+0000", 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0794e-05 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575077+0000", 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.15872312899999999 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575136+0000", 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8783000000000003e-05 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575180+0000", 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3751999999999997e-05 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198530 8.9 8:9a064d51:::benchmark_data_smithi080_109213_object84013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.370993+0000", 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 101.207316221, 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20430893899999999, 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198530 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.370993+0000", 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.370993+0000", 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.370995+0000", 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3349999999999999e-06 2024-03-26T05:11:37.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.371001+0000", 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2369999999999997e-06 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.371012+0000", 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0692e-05 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.371018+0000", 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9390000000000002e-06 2024-03-26T05:11:37.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.414470+0000", 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043452706000000001 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.414555+0000", 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4563999999999999e-05 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.414637+0000", 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2443999999999999e-05 2024-03-26T05:11:37.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416408+0000", 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0017704999999999999 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416541+0000", 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000133337 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575224+0000", 2024-03-26T05:11:37.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.15868285500000001 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575262+0000", 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7988999999999998e-05 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575302+0000", 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0340999999999999e-05 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.134 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198704 8.9 8:9c023eba:::benchmark_data_smithi080_109213_object84187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.610221+0000", 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 100.968089046, 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16595127300000001, 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198704 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.610221+0000", 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.610221+0000", 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.610221+0000", 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0800000000000004e-07 2024-03-26T05:11:37.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.610226+0000", 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4700000000000004e-06 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.610236+0000", 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0118999999999999e-05 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.610244+0000", 2024-03-26T05:11:37.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6699999999999994e-06 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.673106+0000", 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.062862228000000006 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.673194+0000", 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7737000000000004e-05 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.673269+0000", 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4981999999999995e-05 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.676172+0000", 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0029031899999999999 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.678056+0000", 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0018840459999999999 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.776084+0000", 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.098027874000000001 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.776127+0000", 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3312999999999997e-05 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.776172+0000", 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4836000000000002e-05 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.140 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:199631 8.9 8:951c68a4:::benchmark_data_smithi080_109213_object85114:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:55.624693+0000", 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 99.953616609999997, 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20792925800000001, 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 199631 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624693+0000", 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624693+0000", 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624695+0000", 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5770000000000001e-06 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624699+0000", 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6999999999999999e-06 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624710+0000", 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0747000000000001e-05 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624717+0000", 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1489999999999997e-06 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.655682+0000", 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.030964476000000001 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.655762+0000", 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0160999999999994e-05 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.655849+0000", 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7125000000000004e-05 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.660996+0000", 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0051472660000000002 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.663215+0000", 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0022190790000000001 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832537+0000", 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.169322062 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832580+0000", 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3037999999999997e-05 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832622+0000", 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1878000000000001e-05 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:199636 8.9 8:959205e3:::benchmark_data_smithi080_109213_object85119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:55.645400+0000", 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 99.932909596000002, 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18738101200000001, 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 199636 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.645400+0000", 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.645400+0000", 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.645401+0000", 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5149999999999999e-06 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.645408+0000", 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9890000000000003e-06 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.645420+0000", 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1250000000000001e-05 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.645428+0000", 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9149999999999996e-06 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.657862+0000", 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.012434007 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.657948+0000", 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6234000000000001e-05 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.658071+0000", 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000123225 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.661254+0000", 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031827270000000002 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.663395+0000", 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021413059999999999 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832675+0000", 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.169279755 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832725+0000", 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0367000000000001e-05 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832781+0000", 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5721999999999999e-05 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:199649 8.9 8:945d934d:::benchmark_data_smithi080_109213_object85132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:55.668536+0000", 2024-03-26T05:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 99.909773759000004, 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16440793000000001, 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 199649 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.668536+0000", 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.668536+0000", 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.668537+0000", 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.508e-06 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.668542+0000", 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.707e-06 2024-03-26T05:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.668555+0000", 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3426e-05 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.668563+0000", 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7840000000000004e-06 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.668621+0000", 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8241e-05 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.668749+0000", 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012782399999999999 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.668879+0000", 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012996199999999999 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.675286+0000", 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0064065399999999996 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.675395+0000", 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108897 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832835+0000", 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.157440305 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832884+0000", 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8857999999999999e-05 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832944+0000", 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9877999999999998e-05 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:199883 8.9 8:9f523895:::benchmark_data_smithi080_109213_object85366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:55.912096+0000", 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 99.666214065999995, 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16546648, 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 199883 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.912096+0000", 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.912096+0000", 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.912097+0000", 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1200000000000001e-06 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.912100+0000", 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6069999999999999e-06 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.912110+0000", 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8360000000000004e-06 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.912117+0000", 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.674e-06 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.972022+0000", 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.059905601000000003 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.972116+0000", 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3177999999999995e-05 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.972194+0000", 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8666999999999997e-05 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.974200+0000", 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020058089999999999 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.975070+0000", 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000870241 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.077444+0000", 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.102374151 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.077505+0000", 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0454000000000001e-05 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.077562+0000", 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7142000000000003e-05 2024-03-26T05:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:199926 8.9 8:9bc5c9ee:::benchmark_data_smithi080_109213_object85409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:55.992851+0000", 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 99.585458690999999, 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.201735579, 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 199926 2024-03-26T05:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992851+0000", 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992851+0000", 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992852+0000", 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5999999999999991e-07 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992855+0000", 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2789999999999999e-06 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992865+0000", 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3999999999999998e-06 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992870+0000", 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4389999999999999e-06 2024-03-26T05:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992911+0000", 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1449000000000002e-05 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.993006+0000", 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4477999999999999e-05 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.993085+0000", 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9538999999999999e-05 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.996438+0000", 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00335213 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.996976+0000", 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053849499999999995 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.194301+0000", 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.197324468 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.194517+0000", 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021599999999999999 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.194586+0000", 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9942000000000002e-05 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:200274 8.9 8:906226e0:::benchmark_data_smithi080_109213_object85757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:56.335381+0000", 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 99.242928086999996, 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.17162375199999999, 2024-03-26T05:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 200274 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.335381+0000", 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.335381+0000", 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.335383+0000", 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4160000000000001e-06 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.335387+0000", 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1479999999999999e-06 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.335397+0000", 2024-03-26T05:11:37.171 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6760000000000002e-06 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.335404+0000", 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0570000000000004e-06 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.380146+0000", 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044741784 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.380281+0000", 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000135666 2024-03-26T05:11:37.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.380483+0000", 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00020144600000000001 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.390206+0000", 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0097238089999999999 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.390355+0000", 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014864300000000001 2024-03-26T05:11:37.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.506924+0000", 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.116569326 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.506961+0000", 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6928000000000003e-05 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.507005+0000", 2024-03-26T05:11:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3853000000000002e-05 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:201000 8.9 8:97ede170:::benchmark_data_smithi080_109213_object86483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:57.050509+0000", 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 98.527800221999996, 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.188192953, 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 201000 2024-03-26T05:11:37.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.050509+0000", 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.050509+0000", 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.050511+0000", 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.703e-06 2024-03-26T05:11:37.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.050516+0000", 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9790000000000001e-06 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.050532+0000", 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.615e-05 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.050540+0000", 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9389999999999997e-06 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.119649+0000", 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.069108881999999996 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.119782+0000", 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000133156 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.119904+0000", 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000121557 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.178 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.132767+0000", 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.012863495000000001 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.136898+0000", 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0041304360000000004 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.238609+0000", 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.101711413 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.238659+0000", 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9845999999999997e-05 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.238702+0000", 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3396999999999998e-05 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:202429 8.9 8:943a69cc:::benchmark_data_smithi080_109213_object87912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:58.502083+0000", 2024-03-26T05:11:37.180 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 97.076226353999999, 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.193700766, 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 202429 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.502083+0000", 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.181 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.502083+0000", 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.502084+0000", 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0049999999999999e-06 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.502091+0000", 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5300000000000002e-06 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.182 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.502104+0000", 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2792999999999999e-05 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.502111+0000", 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5129999999999999e-06 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.517204+0000", 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.015093367 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.183 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.184 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.517335+0000", 2024-03-26T05:11:37.184 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013098799999999999 2024-03-26T05:11:37.184 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.184 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.184 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.184 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.517478+0000", 2024-03-26T05:11:37.184 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014280599999999999 2024-03-26T05:11:37.184 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.184 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.184 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:37.185 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.185 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.520055+0000", 2024-03-26T05:11:37.185 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0025765559999999998 2024-03-26T05:11:37.185 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.520070+0000", 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4970999999999999e-05 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.695655+0000", 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.17558531699999999 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.186 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.695714+0000", 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8881000000000001e-05 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.695784+0000", 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0038999999999998e-05 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:202824 8.9 8:90537ff0:::benchmark_data_smithi080_109213_object88307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.187 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:58.982640+0000", 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 96.595669727000001, 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18913682500000001, 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 202824 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.982640+0000", 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.188 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.982640+0000", 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.982641+0000", 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.42e-06 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.982646+0000", 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0250000000000002e-06 2024-03-26T05:11:37.189 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.982658+0000", 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2099000000000001e-05 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.982666+0000", 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8080000000000005e-06 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.028333+0000", 2024-03-26T05:11:37.190 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.045667253999999997 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.028418+0000", 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4122999999999994e-05 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.028501+0000", 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3112e-05 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.030935+0000", 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002434322 2024-03-26T05:11:37.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.031142+0000", 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00020722500000000001 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.171647+0000", 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14050507800000001 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.171705+0000", 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7630999999999998e-05 2024-03-26T05:11:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.171777+0000", 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1728000000000002e-05 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:203255 8.9 8:9507306b:::benchmark_data_smithi080_109213_object88738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:59.393074+0000", 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 96.185235785000003, 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.21242334900000001, 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:11:37.193 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 203255 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393074+0000", 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:11:37.194 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393074+0000", 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393075+0000", 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0079999999999999e-06 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393079+0000", 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8909999999999997e-06 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393092+0000", 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2927e-05 2024-03-26T05:11:37.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393097+0000", 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7459999999999997e-06 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.475807+0000", 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.082709258999999993 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.475886+0000", 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9282000000000004e-05 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.196 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.476011+0000", 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000125425 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.482040+0000", 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0060289230000000003 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.486423+0000", 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043832439999999997 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:11:37.197 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.605408+0000", 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11898417 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.605444+0000", 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6127999999999997e-05 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.605497+0000", 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3346000000000001e-05 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.198 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:11:37.199 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:11:37.199 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:11:37.242 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:37.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:37.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:37.344 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:37.382 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:37.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:37.484 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:37.528 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:37.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:37.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:37.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:37.673 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:37.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:37.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 29 16 56506 56490 121.733 76.8125 0.00792392 0.00820959 2024-03-26T05:11:37.765 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:37 smithi132 ceph-mon[33500]: pgmap v510: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 157 MiB/s wr, 2.51k op/s 2024-03-26T05:11:37.765 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:37 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2179750738' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:37.770 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:11:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T05:11:37.770 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:11:37.773 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:37.811 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:37.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:37.867 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:11:37.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:37 smithi080 ceph-mon[29691]: pgmap v510: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 157 MiB/s wr, 2.51k op/s 2024-03-26T05:11:37.871 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:37 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2179750738' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:37.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:37 smithi080 ceph-mon[40404]: pgmap v510: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 157 MiB/s wr, 2.51k op/s 2024-03-26T05:11:37.871 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:37 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2179750738' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:37.913 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:37.957 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:37.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:37.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:38.059 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:38.093 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:38.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:38.194 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:38.223 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:38.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:38.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T05:11:38.306 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:38.326 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:38.378 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:38.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:38.479 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:38.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:38.521 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:38.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:38.622 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:38.660 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:38.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:38.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 30 16 57862 57846 120.5 84.75 0.00957034 0.00829365 2024-03-26T05:11:38.763 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:38.807 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:38.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:38.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:38.909 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:38.945 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:38.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:38.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:39.047 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:39.085 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:39.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:39.186 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:39.224 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:39.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:39.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:39.322 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:39.325 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:39.360 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:39.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:39.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:39.462 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:39.500 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:39.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:39.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:39.602 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:39.639 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:39.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:39.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 31 16 59334 59318 119.58 92 0.0263958 0.00835628 2024-03-26T05:11:39.739 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:39 smithi132 ceph-mon[33500]: pgmap v511: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 742 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-26T05:11:39.739 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:39 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:11:39.739 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:39 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:11:39.739 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:39 smithi132 ceph-mon[33500]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T05:11:39.741 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:39.780 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:39.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:39.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:39 smithi080 ceph-mon[29691]: pgmap v511: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 742 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-26T05:11:39.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:39 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:11:39.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:39 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:11:39.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:39 smithi080 ceph-mon[29691]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T05:11:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:39 smithi080 ceph-mon[40404]: pgmap v511: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 742 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-26T05:11:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:39 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:11:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:39 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:11:39.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:39 smithi080 ceph-mon[40404]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T05:11:39.881 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:39.920 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:11:39.925 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:39.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:40.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:40.059 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:40.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:40.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:40.161 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:40.193 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:40.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:40.295 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:40.321 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:40.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:40.422 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:40.459 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:40.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:40.560 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:40.595 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:40.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:40.696 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:40.734 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:40.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:40.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 32 16 60740 60724 118.589 87.875 0.0139601 0.00842754 2024-03-26T05:11:40.836 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:40.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:40.880 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:40.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:40.981 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:41.024 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:41.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:41.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:41.126 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:41.149 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:41.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:41.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:41.251 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:41.278 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:11:41.278 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:11:41.290 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:41.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:41.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:41.394 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:41.417 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:41.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:41.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:41.519 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:41.554 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:41.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:41.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:41.656 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:41.695 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:41.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:41.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 33 16 61835 61819 117.07 68.4375 0.00693749 0.00853735 2024-03-26T05:11:41.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:41 smithi132 ceph-mon[33500]: pgmap v513: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-26T05:11:41.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:41 smithi132 ceph-mon[33500]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T05:11:41.796 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:41.833 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:41.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:41.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:41.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:41 smithi080 ceph-mon[29691]: pgmap v513: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-26T05:11:41.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:41 smithi080 ceph-mon[29691]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T05:11:41.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:41 smithi080 ceph-mon[40404]: pgmap v513: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-26T05:11:41.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:41 smithi080 ceph-mon[40404]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T05:11:41.934 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:41.957 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:41.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:42.059 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:42.093 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:42.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:42.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:42.196 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:42.230 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:42.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:42.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:42.332 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:42.359 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:42.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:42.460 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:42.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:42.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:42.600 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:42.624 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:42.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:42.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:42.726 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:42.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 34 16 62865 62849 115.52 64.375 0.00418644 0.00865129 2024-03-26T05:11:42.752 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:42.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:42.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:42.853 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:42.877 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:42.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:42.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:42.978 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:43.015 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:43.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:43.116 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:43.142 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:43.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:43.244 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:43.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:43.268 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:43.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:43.369 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:43.406 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:43.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:43.508 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:43.534 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:43.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:43.635 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:43.678 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:43.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:43.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 35 16 63735 63719 113.773 54.375 0.00802086 0.00878433 2024-03-26T05:11:43.780 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:43.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:43 smithi132 ceph-mon[33500]: pgmap v515: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 746 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-26T05:11:43.808 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:43.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:43.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:43 smithi080 ceph-mon[40404]: pgmap v515: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 746 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-26T05:11:43.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:43 smithi080 ceph-mon[29691]: pgmap v515: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 746 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-26T05:11:43.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:43.945 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:43.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:43.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:44.047 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:44.078 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:44.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:44.180 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:44.218 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:44.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:44.281 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:11:44.282 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:11:44.282 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-26T05:11:44.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T05:11:44.306 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:11:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:11:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:11:44.320 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:44.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:44.344 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:44.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:44.445 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:44.476 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:44.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:44.550 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T05:11:44.578 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:44.622 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:44.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:44.724 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:44.737 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 36 16 64898 64882 112.631 72.6875 0.00926004 0.00887345 2024-03-26T05:11:44.762 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:44.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:44.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:44.863 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:44.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:44.896 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:44.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:44.922 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:44.996 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:45.029 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:45.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:45.131 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:45.167 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:45.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:45.268 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:45.295 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:45.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:45.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:45.397 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:45.435 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:45.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:45.536 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:45.564 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:45.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:45.665 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:45.693 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:45.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:45.738 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 37 16 65823 65807 111.149 57.8125 0.00748958 0.00899166 2024-03-26T05:11:45.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:45 smithi132 ceph-mon[33500]: pgmap v516: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s 2024-03-26T05:11:45.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:45 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2946424098' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:11:45.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:11:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:11:45.794 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:45.828 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:45.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:45.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:45.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:45 smithi080 ceph-mon[40404]: pgmap v516: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s 2024-03-26T05:11:45.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:45 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2946424098' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:11:45.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:45 smithi080 ceph-mon[29691]: pgmap v516: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s 2024-03-26T05:11:45.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:45 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2946424098' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:11:45.930 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:45.953 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:45.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:46.055 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:46.092 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:46.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:46.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:46.194 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:46.200 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:11:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:46.240 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:46.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:46.341 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:46.364 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:46.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:46.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:46.466 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:46.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:46.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:46.601 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:46.637 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:46.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:46.738 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 38 16 66857 66841 109.925 64.625 0.00382275 0.00906699 2024-03-26T05:11:46.739 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:46.763 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:46.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:46.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:46 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:46.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:46 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:11:46.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:46 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2946424098' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:11:46.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:46 smithi132 ceph-mon[33500]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T05:11:46.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:46 smithi132 ceph-mon[33500]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T05:11:46.865 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:46.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:46 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:46.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:46 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:11:46.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:46 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2946424098' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:11:46.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:46 smithi080 ceph-mon[29691]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T05:11:46.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:46 smithi080 ceph-mon[29691]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T05:11:46.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:46 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:46.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:46 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:11:46.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:46 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2946424098' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:11:46.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:46 smithi080 ceph-mon[40404]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T05:11:46.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:46 smithi080 ceph-mon[40404]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T05:11:46.901 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:46.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:46.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:46.983 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:11:46.983 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:11:47.004 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:47.044 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:47.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:47.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:47.146 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:47.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:47.186 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:47.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:47.287 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:47.322 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:47.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:47.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:47.424 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:47.448 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:47.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:47.550 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:47.585 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:47.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:47.611 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:47 smithi132 ceph-mon[33500]: pgmap v518: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-26T05:11:47.660 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T05:11:47.686 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:47.738 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 39 16 68018 68002 108.967 72.5625 0.00495574 0.00917177 2024-03-26T05:11:47.742 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:47.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:47.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:47.845 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:47.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:47 smithi080 ceph-mon[29691]: pgmap v518: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-26T05:11:47.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:47 smithi080 ceph-mon[40404]: pgmap v518: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-26T05:11:47.891 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:47.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:47.993 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:48.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:48.035 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:48.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:48.064 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:11:48.136 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:48.181 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:48.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:48.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:48.325 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:48.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:48.427 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:48.466 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:48.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:48.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:48.568 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:48.612 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:48.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:48.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:48.660 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:48.714 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:48.738 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:11:48.748348+0000 min lat: 0.00115197 max lat: 0.194546 avg lat: 0.00923835 2024-03-26T05:11:48.738 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:11:48.738 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 40 16 69258 69242 108.18 77.5 0.0221799 0.00923835 2024-03-26T05:11:48.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:48.761 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:48.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:48.861 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:48.903 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:48.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:49.004 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:49.043 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:49.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:49.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:49.145 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:49.183 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:49.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:49.286 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:49.329 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:49.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:49.430 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:49.434 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:49.475 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:49.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:49.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:49.577 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:49.615 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:49.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:49.717 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:49.738 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 41 16 70639 70623 107.646 86.3125 0.00461134 0.00928437 2024-03-26T05:11:49.748 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:49.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:49.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:49.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:49 smithi132 ceph-mon[33500]: pgmap v520: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 796 B/s rd, 52 MiB/s wr, 831 op/s 2024-03-26T05:11:49.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:49 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:11:49.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:49 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:11:49.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:49 smithi132 ceph-mon[33500]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T05:11:49.850 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:49.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:49 smithi080 ceph-mon[40404]: pgmap v520: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 796 B/s rd, 52 MiB/s wr, 831 op/s 2024-03-26T05:11:49.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:49 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:11:49.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:49 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:11:49.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:49 smithi080 ceph-mon[40404]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T05:11:49.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:49 smithi080 ceph-mon[29691]: pgmap v520: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 796 B/s rd, 52 MiB/s wr, 831 op/s 2024-03-26T05:11:49.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:49 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:11:49.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:49 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:11:49.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:49 smithi080 ceph-mon[29691]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T05:11:49.887 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:49.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:49.989 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:50.015 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:11:50.015 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:11:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:50.029 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:50.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:50.130 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:50.166 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:50.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:50.268 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:50.311 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:50.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:50.412 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:50.453 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:50.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:50.555 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:50.594 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:50.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:50.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:50.634 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:11:50.645 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:11:50.645 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:11:50.694 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:11:50.696 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:50.735 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:50.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:50.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:50.738 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 42 16 72000 71984 107.108 85.0625 0.00800554 0.00933163 2024-03-26T05:11:50.750 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T05:11:50.837 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:50.869 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:50.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:50.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:50.971 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:51.003 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:51.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:51.104 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:51.140 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:51.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:51.242 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:51.280 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:51.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:51.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:51.382 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:51.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:51.429 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:51.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:51.530 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:51.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:51.568 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:51.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:51.669 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:51.702 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:51.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:51.738 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 43 16 73268 73252 106.46 79.25 0.0155715 0.00938808 2024-03-26T05:11:51.747 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:51 smithi080 ceph-mon[29691]: pgmap v522: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-26T05:11:51.747 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:51 smithi080 ceph-mon[29691]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T05:11:51.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:51 smithi080 ceph-mon[40404]: pgmap v522: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-26T05:11:51.747 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:51 smithi080 ceph-mon[40404]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T05:11:51.804 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:51.867 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:51.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:51.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:51.969 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:51.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:51.993 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:51.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:51 smithi132 ceph-mon[33500]: pgmap v522: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-26T05:11:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:51 smithi132 ceph-mon[33500]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T05:11:52.094 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:52.136 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:52.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:52.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:52.237 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:52.271 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:52.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:52.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:52.403 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:52.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:52.435 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:52.445 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:52.454 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:11:52.454 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:11:52.454 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 285, 2024-03-26T05:11:52.454 INFO:teuthology.orchestra.run.smithi080.stdout: "fsid": "dd09e9cc-eb2c-11ee-95d0-87774f69a715", 2024-03-26T05:11:52.454 INFO:teuthology.orchestra.run.smithi080.stdout: "created": "2024-03-26T04:54:51.106954+0000", 2024-03-26T05:11:52.454 INFO:teuthology.orchestra.run.smithi080.stdout: "modified": "2024-03-26T05:11:51.512086+0000", 2024-03-26T05:11:52.454 INFO:teuthology.orchestra.run.smithi080.stdout: "last_up_change": "2024-03-26T05:00:08.801055+0000", 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: "last_in_change": "2024-03-26T05:07:58.901005+0000", 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_num": 5806080, 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_set": [ 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: "nodeep-scrub", 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: "noscrub", 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: "pglog_hardlimit", 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: "purged_snapdirs", 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: "recovery_deletes", 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: "sortbitwise" 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_version": 18, 2024-03-26T05:11:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_snapshot": "", 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_max": 9, 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "max_osd": 8, 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "require_min_compat_client": "luminous", 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "min_compat_client": "luminous", 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "require_osd_release": "squid", 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "allow_crimson": false, 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "pools": [ 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 1, 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": ".mgr", 2024-03-26T05:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T04:58:06.876819+0000", 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 1, 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 1, 2024-03-26T05:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 1, 2024-03-26T05:11:52.458 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 1, 2024-03-26T05:11:52.458 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 1, 2024-03-26T05:11:52.458 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:11:52.458 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:11:52.458 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:11:52.458 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "22", 2024-03-26T05:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "options": { 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_max": 32, 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_min": 1 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": {} 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 7.8899998664855957, 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 7.8899998664855957, 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.37999999523162842, 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 3, 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 3, 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 2, 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "datapool", 2024-03-26T05:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:20.705417+0000", 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 8193, 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 3, 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 3, 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 3, 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 3, 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 3, 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "71", 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 2, 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 71, 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd": {} 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.6500000953674316, 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.6500000953674316, 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 3, 2024-03-26T05:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": ".rgw.root", 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:21.150368+0000", 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "69", 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:11:52.474 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.5, 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.5, 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.5, 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.5, 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 4, 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.log", 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:23.638065+0000", 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "71", 2024-03-26T05:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.25, 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.25, 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2.25, 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2.25, 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 5, 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.control", 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:25.181000+0000", 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "73", 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.25, 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.25, 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.25, 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.25, 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.488 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.488 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 6, 2024-03-26T05:11:52.488 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.meta", 2024-03-26T05:11:52.488 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:27.249998+0000", 2024-03-26T05:11:52.488 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:11:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "75", 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:11:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:11:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:11:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "options": { 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_bias": 4 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:11:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.75, 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.75, 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.75, 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.75, 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 9, 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "unique_pool_2", 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:10:54.457700+0000", 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:11:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 16, 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 16, 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 16, 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 16, 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 16, 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "274", 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:11:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "rados": {} 2024-03-26T05:11:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2, 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2, 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 1, 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2, 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2, 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:11:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 0, 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366", 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 8, 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 265, 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:11:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6802", 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6803", 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:11:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6804", 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6805", 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6808", 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6809", 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6806", 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6807", 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6803/1840686622", 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6805/1840686622", 2024-03-26T05:11:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6809/1840686622", 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6807/1840686622", 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 1, 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23", 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:11:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 13, 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 265, 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6810", 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6811", 2024-03-26T05:11:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6812", 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6813", 2024-03-26T05:11:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6816", 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6817", 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:11:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6814", 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6815", 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6811/1726833996", 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6813/1726833996", 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6817/1726833996", 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6815/1726833996", 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 2, 2024-03-26T05:11:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "536bec9b-9636-4e67-b452-ed1a55c22ab6", 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 18, 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 239, 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6818", 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6819", 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6820", 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6821", 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6824", 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6825", 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6822", 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6823", 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6819/128713757", 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6821/128713757", 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6825/128713757", 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6823/128713757", 2024-03-26T05:11:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 3, 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "640cfdb6-b9c5-446d-8312-504ae12c7818", 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:11:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 27, 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 265, 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6826", 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6827", 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6828", 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6829", 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6832", 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6833", 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6830", 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6831", 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6827/418771441", 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6829/418771441", 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6833/418771441", 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6831/418771441", 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 4, 2024-03-26T05:11:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639", 2024-03-26T05:11:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:11:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:11:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:11:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:11:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:11:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:11:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 33, 2024-03-26T05:11:52.523 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 231, 2024-03-26T05:11:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:11:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:11:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:11:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6800", 2024-03-26T05:11:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:11:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6801", 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6802", 2024-03-26T05:11:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6803", 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6806", 2024-03-26T05:11:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6807", 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6804", 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6805", 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6801/1441071423", 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6803/1441071423", 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6807/1441071423", 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6805/1441071423", 2024-03-26T05:11:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 5, 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc", 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:11:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 41, 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 265, 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6808", 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6809", 2024-03-26T05:11:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6810", 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6811", 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:11:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6814", 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6815", 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:11:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6812", 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6813", 2024-03-26T05:11:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6809/794504625", 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6811/794504625", 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6815/794504625", 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6813/794504625", 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 6, 2024-03-26T05:11:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5", 2024-03-26T05:11:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:11:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:11:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:11:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:11:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:11:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:11:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 51, 2024-03-26T05:11:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 230, 2024-03-26T05:11:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6816", 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6817", 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:11:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6818", 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6819", 2024-03-26T05:11:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6822", 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6823", 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:11:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6820", 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6821", 2024-03-26T05:11:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6817/2209078972", 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6819/2209078972", 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6823/2209078972", 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6821/2209078972", 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 7, 2024-03-26T05:11:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d", 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 61, 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 265, 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6824", 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6825", 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6826", 2024-03-26T05:11:52.544 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:11:52.544 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.544 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.544 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.544 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6827", 2024-03-26T05:11:52.544 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:11:52.544 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.544 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.544 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.544 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:11:52.544 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.544 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6830", 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6831", 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:11:52.545 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6828", 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6829", 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 3655388282 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6825/3655388282", 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6827/3655388282", 2024-03-26T05:11:52.546 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6831/3655388282", 2024-03-26T05:11:52.547 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6829/3655388282", 2024-03-26T05:11:52.547 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:11:52.547 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:11:52.547 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:11:52.547 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.547 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.547 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:11:52.547 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_xinfo": [ 2024-03-26T05:11:52.547 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.547 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 0, 2024-03-26T05:11:52.547 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:11:52.547 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:10:58.088605+0000", 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 1, 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:11:52.548 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:11:24.886782+0000", 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 2, 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:10:53.584821+0000", 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.549 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 3, 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:11:38.476371+0000", 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 4, 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:11:52.550 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:11:30.870031+0000", 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 5, 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:11:48.432617+0000", 2024-03-26T05:11:52.551 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 6, 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:11:34.382839+0000", 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.552 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 7, 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:10:46.851884+0000", 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap": [ 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.1", 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:11:52.553 INFO:teuthology.orchestra.run.smithi080.stdout: 4, 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: 5 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap_items": [ 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.a", 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 4, 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 0 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.554 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.b", 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 4, 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 1 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.1c", 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.555 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 4, 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.2", 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.556 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.7", 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.1f", 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.557 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "6.b", 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 7, 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 4 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 3 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:52.558 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap_primaries": [], 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_temp": [], 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_temp": [], 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "blocklist": { 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3992539385": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/2770889532": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/2466284312": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/490056571": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1986135300": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3796611170": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1993138634": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1832498181": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/2466284312": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:11:52.559 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1650870847": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/1418683843": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3147301987": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/2770889532": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/571905941": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1451012560": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1235125152": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/1418683843": "2024-03-27T05:01:03.689739+0000" 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "range_blocklist": {}, 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profiles": { 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "default": { 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "crush-failure-domain": "osd", 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "k": "2", 2024-03-26T05:11:52.560 INFO:teuthology.orchestra.run.smithi080.stdout: "m": "1", 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: "plugin": "jerasure", 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: "technique": "reed_sol_van" 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps_queue": [], 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: "new_removed_snaps": [], 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: "new_purged_snaps": [], 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_node_flags": {}, 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: "device_class_flags": {}, 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode": { 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode_enabled": false, 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_bucket_count": 0, 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: "degraded_stretch_mode": 0, 2024-03-26T05:11:52.561 INFO:teuthology.orchestra.run.smithi080.stdout: "recovering_stretch_mode": 0, 2024-03-26T05:11:52.562 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode_bucket": 0 2024-03-26T05:11:52.562 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:52.562 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:11:52.563 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:11:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:52.564 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:52.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:52.604 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:11:52.642 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:11:52.665 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:52.683 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:52 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1776146533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:11:52.683 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:52 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3080358685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:52.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:52 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1776146533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:11:52.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:52 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3080358685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:52.684 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:11:52.700 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:52.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:52.724 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:11:52.738 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 44 16 74294 74278 105.498 64.125 0.00820634 0.00947396 2024-03-26T05:11:52.764 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:11:52.802 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:52.807 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:11:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:52.839 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:52.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:52.853 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:11:52.858 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:11:52.858 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:11:52.858 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:11:52.858 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:11:52.858 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:11:52.858 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:11:52.858 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:11:52.858 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:11:52.940 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:52.983 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:52.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:53.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:52 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1776146533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:11:53.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:52 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3080358685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:11:53.083 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:53.107 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:53.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:53.208 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:53.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:53.246 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:53.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:53.347 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:53.443 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:53.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:53.544 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:53.584 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:53.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:53.685 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:53.713 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:53.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:53.738 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 45 16 75403 75387 104.694 69.3125 0.0631177 0.00954243 2024-03-26T05:11:53.815 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:53.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:53.852 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:53.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:53.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:53 smithi080 ceph-mon[29691]: pgmap v524: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 997 op/s 2024-03-26T05:11:53.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:53 smithi080 ceph-mon[40404]: pgmap v524: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 997 op/s 2024-03-26T05:11:53.953 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:53.997 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:53.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:53.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:54.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:53 smithi132 ceph-mon[33500]: pgmap v524: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 997 op/s 2024-03-26T05:11:54.098 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:54.123 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:54.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:54.225 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:54.257 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:54.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:54.307 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:11:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:11:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:11:54.359 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:54.399 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:54.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:54.501 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:54.532 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:54.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:54.634 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:54.675 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:54.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:54.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:54.738 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 46 15 76454 76439 103.847 65.75 0.00615869 0.0096244 2024-03-26T05:11:54.777 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:54.829 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:54.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:54.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:54.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:54 smithi080 ceph-mon[29691]: pgmap v525: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 689 B/s rd, 50 MiB/s wr, 806 op/s 2024-03-26T05:11:54.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:54 smithi080 ceph-mon[40404]: pgmap v525: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 689 B/s rd, 50 MiB/s wr, 806 op/s 2024-03-26T05:11:54.931 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:54.963 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:54.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:54.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:55.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:54 smithi132 ceph-mon[33500]: pgmap v525: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 689 B/s rd, 50 MiB/s wr, 806 op/s 2024-03-26T05:11:55.065 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:55.097 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:55.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:55.203 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:55.244 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:55.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:55.346 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:55.389 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:55.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:55.490 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:55.522 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:55.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:55.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:55.624 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:55.655 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:55.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:55.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:55.739 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 47 16 77643 77627 103.217 74.25 0.0136642 0.00968353 2024-03-26T05:11:55.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:55.789 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:55.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:55.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:55.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:55 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:55.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:11:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:11:55.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:55 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:55.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:55 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:11:55.892 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:55.923 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:55.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:56.025 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:56.063 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:56.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:56.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:56.199 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:56.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:56.301 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:56.348 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:56.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:56.380 INFO:tasks.thrashosds.thrasher:j is {'epoch': 285, 'fsid': 'dd09e9cc-eb2c-11ee-95d0-87774f69a715', 'created': '2024-03-26T04:54:51.106954+0000', 'modified': '2024-03-26T05:11:51.512086+0000', 'last_up_change': '2024-03-26T05:00:08.801055+0000', 'last_in_change': '2024-03-26T05:07:58.901005+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T04:58:06.876819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T05:00:20.705417+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T05:00:21.150368+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T05:00:23.638065+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T05:00:25.181000+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T05:00:27.249998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-26T05:10:54.457700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '274', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '35a52544-0b0b-47d6-8d6a-571c48ceb366', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 265, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6802', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6803', 'nonce': 1840686622}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6804', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6805', 'nonce': 1840686622}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6808', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6809', 'nonce': 1840686622}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6806', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6807', 'nonce': 1840686622}]}, 'public_addr': '172.21.15.80:6803/1840686622', 'cluster_addr': '172.21.15.80:6805/1840686622', 'heartbeat_back_addr': '172.21.15.80:6809/1840686622', 'heartbeat_front_addr': '172.21.15.80:6807/1840686622', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efa5ea33-4f11-49b7-a3f3-f60d5329db23', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 265, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6810', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6811', 'nonce': 1726833996}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6812', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6813', 'nonce': 1726833996}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6816', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6817', 'nonce': 1726833996}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6814', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6815', 'nonce': 1726833996}]}, 'public_addr': '172.21.15.80:6811/1726833996', 'cluster_addr': '172.21.15.80:6813/1726833996', 'heartbeat_back_addr': '172.21.15.80:6817/1726833996', 'heartbeat_front_addr': '172.21.15.80:6815/1726833996', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '536bec9b-9636-4e67-b452-ed1a55c22ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 239, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6818', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6819', 'nonce': 128713757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6820', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6821', 'nonce': 128713757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6824', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6825', 'nonce': 128713757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6822', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6823', 'nonce': 128713757}]}, 'public_addr': '172.21.15.80:6819/128713757', 'cluster_addr': '172.21.15.80:6821/128713757', 'heartbeat_back_addr': '172.21.15.80:6825/128713757', 'heartbeat_front_addr': '172.21.15.80:6823/128713757', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '640cfdb6-b9c5-446d-8312-504ae12c7818', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 265, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6826', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6827', 'nonce': 418771441}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6828', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6829', 'nonce': 418771441}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6832', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6833', 'nonce': 418771441}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6830', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6831', 'nonce': 418771441}]}, 'public_addr': '172.21.15.80:6827/418771441', 'cluster_addr': '172.21.15.80:6829/418771441', 'heartbeat_back_addr': '172.21.15.80:6833/418771441', 'heartbeat_front_addr': '172.21.15.80:6831/418771441', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0cc7822a-ed73-4c79-a30d-1e151ddd6639', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 231, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6800', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6801', 'nonce': 1441071423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6802', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6803', 'nonce': 1441071423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6806', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6807', 'nonce': 1441071423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6804', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6805', 'nonce': 1441071423}]}, 'public_addr': '172.21.15.132:6801/1441071423', 'cluster_addr': '172.21.15.132:6803/1441071423', 'heartbeat_back_addr': '172.21.15.132:6807/1441071423', 'heartbeat_front_addr': '172.21.15.132:6805/1441071423', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 265, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6808', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6809', 'nonce': 794504625}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6810', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6811', 'nonce': 794504625}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6814', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6815', 'nonce': 794504625}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6812', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6813', 'nonce': 794504625}]}, 'public_addr': '172.21.15.132:6809/794504625', 'cluster_addr': '172.21.15.132:6811/794504625', 'heartbeat_back_addr': '172.21.15.132:6815/794504625', 'heartbeat_front_addr': '172.21.15.132:6813/794504625', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2970bbab-11b2-4d1b-b57e-037bdac089b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 230, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6816', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6817', 'nonce': 2209078972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6818', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6819', 'nonce': 2209078972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6822', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6823', 'nonce': 2209078972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6820', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6821', 'nonce': 2209078972}]}, 'public_addr': '172.21.15.132:6817/2209078972', 'cluster_addr': '172.21.15.132:6819/2209078972', 'heartbeat_back_addr': '172.21.15.132:6823/2209078972', 'heartbeat_front_addr': '172.21.15.132:6821/2209078972', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '12f4b466-28f9-469b-8fd0-65b91c6dc22d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 265, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6824', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6825', 'nonce': 3655388282}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6826', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6827', 'nonce': 3655388282}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6830', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6831', 'nonce': 3655388282}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6828', 'nonce': 3655388282}, {'type': 'v1', 'addr': '172.21.15.132:6829', 'nonce': 3655388282}]}, 'public_addr': '172.21.15.132:6825/3655388282', 'cluster_addr': '172.21.15.132:6827/3655388282', 'heartbeat_back_addr': '172.21.15.132:6831/3655388282', 'heartbeat_front_addr': '172.21.15.132:6829/3655388282', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:10:58.088605+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:11:24.886782+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:10:53.584821+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:11:38.476371+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:11:30.870031+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:11:48.432617+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:11:34.382839+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:10:46.851884+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.1', 'osds': [4, 0, 5]}], 'pg_upmap_items': [{'pgid': '3.a', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '3.b', 'mappings': [{'from': 4, 'to': 1}]}, {'pgid': '3.1c', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '6.b', 'mappings': [{'from': 7, 'to': 4}, {'from': 1, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.80:0/3992539385': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/2770889532': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:6801/2466284312': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/490056571': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1986135300': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/3796611170': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1993138634': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1832498181': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/2466284312': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1650870847': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6801/1418683843': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:0/3147301987': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6801/2770889532': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/571905941': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1451012560': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1235125152': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/1418683843': '2024-03-27T05:01:03.689739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T05:11:56.381 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:11:56.451 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:56.481 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:56.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:56.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:56.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:56.616 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:56.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:56.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:56.718 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:11:56.742 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 48 16 78856 78840 102.645 75.8125 0.0115859 0.00973698 2024-03-26T05:11:56.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:56.765 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:56.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:11:56.866 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:56.902 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:56.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:56.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:57.004 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:57.043 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:57.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:57.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:57.145 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:57.183 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:57.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:57.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:57.285 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:57.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:57.331 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:57.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:57.432 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:57.463 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:57.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:57.565 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:57.604 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:57.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:57.706 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:57.739 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 49 16 80143 80127 102.192 80.4375 0.00998034 0.00978098 2024-03-26T05:11:57.740 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:57.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:57.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:57 smithi132 ceph-mon[33500]: pgmap v526: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T05:11:57.827 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:57 smithi080 ceph-mon[40404]: pgmap v526: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T05:11:57.827 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:57 smithi080 ceph-mon[29691]: pgmap v526: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T05:11:57.842 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:57.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:57.875 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:57.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:57.976 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:58.013 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:58.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:58.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:58.115 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:58.148 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:58.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:58.250 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:58.280 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:58.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:58.382 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:11:58.413 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:58.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:58.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:11:58.427 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:11:58.515 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:58.559 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:58.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:58.663 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:58.705 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:58.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:58.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:58.739 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 50 16 81273 81257 101.56 70.625 0.00731737 0.00984113 2024-03-26T05:11:58.781 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:11:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:11:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:81316 9.b 9:d48f36ed:::benchmark_data_smithi080_136801_object81315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-03-26T05:11:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:11:58.786219+0000", 2024-03-26T05:11:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0034417029999999999, 2024-03-26T05:11:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0034763060000000002, 2024-03-26T05:11:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:11:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:11:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:11:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:11:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:11:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 81316 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786219+0000", 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786219+0000", 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786221+0000", 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.863e-06 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786249+0000", 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.7545000000000001e-05 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786271+0000", 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2050000000000001e-05 2024-03-26T05:11:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786282+0000", 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0998e-05 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786457+0000", 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017589600000000001 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786621+0000", 2024-03-26T05:11:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016384700000000001 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786784+0000", 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016306 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:81317 9.b 9.e84136db (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:11:58.786329+0000", 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0033322360000000001, 2024-03-26T05:11:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0034456030000000002, 2024-03-26T05:11:58.787 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:11:58.787 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "queued for pg", 2024-03-26T05:11:58.787 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:11:58.789 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:11:58.789 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:11:58.789 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 81317 2024-03-26T05:11:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:11:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:11:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786329+0000", 2024-03-26T05:11:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:11:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:11:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786329+0000", 2024-03-26T05:11:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:11:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786329+0000", 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6300000000000005e-07 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786349+0000", 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9598000000000001e-05 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786357+0000", 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9610000000000005e-06 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.786363+0000", 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9850000000000003e-06 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:81321 9.9 9:9c6d6c7b:::benchmark_data_smithi080_136801_object81320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:11:58.788989+0000", 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.00067223800000000002, 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00085327299999999999, 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:11:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 81321 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.788989+0000", 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.788989+0000", 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:11:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.788990+0000", 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.367e-06 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.789047+0000", 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6949e-05 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:11:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.789063+0000", 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6147999999999999e-05 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.789070+0000", 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2690000000000002e-06 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.789124+0000", 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3933e-05 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:11:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.789280+0000", 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015536900000000001 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,7", 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.789438+0000", 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015871700000000001 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:81322 9.9 9.93bbae79 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:11:58.789089+0000", 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.00057196999999999999, 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00082308900000000005, 2024-03-26T05:11:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "queued for pg", 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 81322 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.789089+0000", 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.789089+0000", 2024-03-26T05:11:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.789089+0000", 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1799999999999995e-07 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.789131+0000", 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1350999999999998e-05 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:11:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.789139+0000", 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1459999999999996e-06 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:11:58.789146+0000", 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.725e-06 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 4 2024-03-26T05:11:58.799 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:11:58.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:58.840 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:58.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:58.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:58.942 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:11:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:58.984 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:58.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:11:59.085 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:11:59.121 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:59.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:11:59.223 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:59.239 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:11:59.258 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:59.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:59.360 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:11:59.403 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:59.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:59.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:11:59.504 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:59.539 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:59.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:59.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:11:59.547 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:11:59.549 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:59 smithi080 ceph-mon[29691]: pgmap v527: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T05:11:59.549 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:59 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3694190828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:11:59.549 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:11:59 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:11:59.550 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:59 smithi080 ceph-mon[40404]: pgmap v527: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T05:11:59.550 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:59 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3694190828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:11:59.550 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:11:59 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:11:59.641 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:11:59.678 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:59.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:59.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:11:59.740 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 51 15 82499 82484 101.072 76.6875 0.00899429 0.00988943 2024-03-26T05:11:59.780 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:11:59.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:59 smithi132 ceph-mon[33500]: pgmap v527: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T05:11:59.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:59 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3694190828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:11:59.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:11:59 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:11:59.822 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:11:59.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:59.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:11:59.924 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:11:59.957 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:11:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:11:59.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:00.059 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:00.092 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:00.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:00.193 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:00.231 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:00.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:00.334 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:00.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:00.372 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:00.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:00.473 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:00.504 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:00.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:00.532 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T05:12:00.606 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:00.639 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:00.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:00.741 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:00.745 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 52 16 83750 83734 100.623 78.125 0.0116649 0.00993252 2024-03-26T05:12:00.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:00 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:12:00.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:00 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:12:00.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:00 smithi132 ceph-mon[33500]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T05:12:00.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:00 smithi132 ceph-mon[33500]: 3.c scrub starts 2024-03-26T05:12:00.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:00 smithi132 ceph-mon[33500]: 3.c scrub ok 2024-03-26T05:12:00.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:00 smithi132 ceph-mon[33500]: 3.12 scrub starts 2024-03-26T05:12:00.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:00 smithi132 ceph-mon[33500]: 3.12 scrub ok 2024-03-26T05:12:00.814 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:00.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:00.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:00.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:12:00.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:12:00.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[29691]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T05:12:00.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[29691]: 3.c scrub starts 2024-03-26T05:12:00.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[29691]: 3.c scrub ok 2024-03-26T05:12:00.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[29691]: 3.12 scrub starts 2024-03-26T05:12:00.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[29691]: 3.12 scrub ok 2024-03-26T05:12:00.857 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:12:00.857 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:12:00.857 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[40404]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T05:12:00.857 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[40404]: 3.c scrub starts 2024-03-26T05:12:00.857 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[40404]: 3.c scrub ok 2024-03-26T05:12:00.857 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[40404]: 3.12 scrub starts 2024-03-26T05:12:00.857 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:00 smithi080 ceph-mon[40404]: 3.12 scrub ok 2024-03-26T05:12:00.919 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:00.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:00.963 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:00.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:01.065 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:01.104 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:01.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:01.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:01.205 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:01.255 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:01.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:01.357 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:01.397 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:01.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:01.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:01.499 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:01.533 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:01.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:01.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:01.636 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:01.667 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:01.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:01.683 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:01.744 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 53 15 84967 84952 100.16 76.125 0.0100481 0.0099796 2024-03-26T05:12:01.768 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:01.777 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:01.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:01 smithi132 ceph-mon[33500]: pgmap v529: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 608 B/s rd, 42 MiB/s wr, 677 op/s 2024-03-26T05:12:01.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:01 smithi132 ceph-mon[33500]: 5.18 scrub starts 2024-03-26T05:12:01.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:01 smithi132 ceph-mon[33500]: 5.18 scrub ok 2024-03-26T05:12:01.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:01 smithi132 ceph-mon[33500]: 6.19 scrub starts 2024-03-26T05:12:01.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:01 smithi132 ceph-mon[33500]: 6.19 scrub ok 2024-03-26T05:12:01.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:01 smithi132 ceph-mon[33500]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T05:12:01.821 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:01.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:01.835 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:01 smithi080 ceph-mon[40404]: pgmap v529: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 608 B/s rd, 42 MiB/s wr, 677 op/s 2024-03-26T05:12:01.835 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:01 smithi080 ceph-mon[40404]: 5.18 scrub starts 2024-03-26T05:12:01.835 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:01 smithi080 ceph-mon[40404]: 5.18 scrub ok 2024-03-26T05:12:01.835 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:01 smithi080 ceph-mon[40404]: 6.19 scrub starts 2024-03-26T05:12:01.835 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:01 smithi080 ceph-mon[40404]: 6.19 scrub ok 2024-03-26T05:12:01.835 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:01 smithi080 ceph-mon[40404]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T05:12:01.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:01 smithi080 ceph-mon[29691]: pgmap v529: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 608 B/s rd, 42 MiB/s wr, 677 op/s 2024-03-26T05:12:01.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:01 smithi080 ceph-mon[29691]: 5.18 scrub starts 2024-03-26T05:12:01.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:01 smithi080 ceph-mon[29691]: 5.18 scrub ok 2024-03-26T05:12:01.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:01 smithi080 ceph-mon[29691]: 6.19 scrub starts 2024-03-26T05:12:01.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:01 smithi080 ceph-mon[29691]: 6.19 scrub ok 2024-03-26T05:12:01.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:01 smithi080 ceph-mon[29691]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T05:12:01.924 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:01.961 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:01.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:01.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:02.065 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:02.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:02.102 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:02.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:02.203 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:02.244 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:02.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:02.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:02.346 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:02.384 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:02.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:02.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:02.486 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:02.520 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:02.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:02.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:02.622 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:02.663 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[40404]: 4.8 scrub starts 2024-03-26T05:12:02.663 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[40404]: 4.8 scrub ok 2024-03-26T05:12:02.663 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:12:02.663 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:12:02.664 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:12:02.664 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:12:02.664 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[40404]: 3.13 scrub starts 2024-03-26T05:12:02.664 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[40404]: 3.13 scrub ok 2024-03-26T05:12:02.664 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[29691]: 4.8 scrub starts 2024-03-26T05:12:02.664 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[29691]: 4.8 scrub ok 2024-03-26T05:12:02.664 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:12:02.664 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:12:02.664 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:12:02.664 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:12:02.665 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[29691]: 3.13 scrub starts 2024-03-26T05:12:02.665 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:02 smithi080 ceph-mon[29691]: 3.13 scrub ok 2024-03-26T05:12:02.667 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:02.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:02.744 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 54 16 86147 86131 99.6701 73.6875 0.0125603 0.0100276 2024-03-26T05:12:02.770 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:02.809 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:02.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:02.911 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:02.950 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:02.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:02.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:03.001 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:12:03.014 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:12:03.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:02 smithi132 ceph-mon[33500]: 4.8 scrub starts 2024-03-26T05:12:03.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:02 smithi132 ceph-mon[33500]: 4.8 scrub ok 2024-03-26T05:12:03.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:02 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:12:03.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:02 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:12:03.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:02 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:12:03.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:02 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:12:03.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:02 smithi132 ceph-mon[33500]: 3.13 scrub starts 2024-03-26T05:12:03.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:02 smithi132 ceph-mon[33500]: 3.13 scrub ok 2024-03-26T05:12:03.052 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:03.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:03.087 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:03.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:03.188 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:03.200 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:12:03.234 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:03.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:03.337 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:03.360 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:03.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:03.462 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:03.530 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:03.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:03.631 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:03.673 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:03.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:03.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:03.744 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 55 16 87292 87276 99.159 71.5625 0.0106384 0.01008 2024-03-26T05:12:03.775 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:03.816 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:03.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:03.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:03.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[29691]: pgmap v531: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-26T05:12:03.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[29691]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T05:12:03.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[29691]: 2.0 scrub starts 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[29691]: 2.0 scrub ok 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[29691]: 5.5 scrub starts 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[29691]: 5.5 scrub ok 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[40404]: pgmap v531: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[40404]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[40404]: 2.0 scrub starts 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[40404]: 2.0 scrub ok 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[40404]: 5.5 scrub starts 2024-03-26T05:12:03.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:03 smithi080 ceph-mon[40404]: 5.5 scrub ok 2024-03-26T05:12:03.918 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:03.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:03 smithi132 ceph-mon[33500]: pgmap v531: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-26T05:12:03.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:03 smithi132 ceph-mon[33500]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T05:12:03.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:03 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:12:03.944 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:03 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:12:03.944 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:03 smithi132 ceph-mon[33500]: 2.0 scrub starts 2024-03-26T05:12:03.944 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:03 smithi132 ceph-mon[33500]: 2.0 scrub ok 2024-03-26T05:12:03.944 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:03 smithi132 ceph-mon[33500]: 5.5 scrub starts 2024-03-26T05:12:03.944 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:03 smithi132 ceph-mon[33500]: 5.5 scrub ok 2024-03-26T05:12:03.949 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:03.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:03.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:04.051 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:04.106 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:04.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:04.207 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:04.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:04.250 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:04.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:04.351 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:04.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:12:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:12:04.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:04.390 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:04.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:04.491 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:04.532 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:04.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:04.620 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:04 smithi132 ceph-mon[33500]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:12:04.620 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:04 smithi132 ceph-mon[33500]: 6.11 scrub starts 2024-03-26T05:12:04.620 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:04 smithi132 ceph-mon[33500]: 6.11 scrub ok 2024-03-26T05:12:04.620 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:04 smithi132 ceph-mon[33500]: 3.5 scrub starts 2024-03-26T05:12:04.620 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:04 smithi132 ceph-mon[33500]: 3.5 scrub ok 2024-03-26T05:12:04.620 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:04 smithi132 ceph-mon[33500]: 5.9 scrub starts 2024-03-26T05:12:04.620 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:04 smithi132 ceph-mon[33500]: 5.9 scrub ok 2024-03-26T05:12:04.620 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:04 smithi132 ceph-mon[33500]: 6.18 scrub starts 2024-03-26T05:12:04.621 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:04 smithi132 ceph-mon[33500]: 6.18 scrub ok 2024-03-26T05:12:04.621 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:12:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=cleanup t=2024-03-26T05:12:04.356865715Z level=info msg="Completed cleanup jobs" duration=1.385876ms 2024-03-26T05:12:04.632 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:04.682 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[29691]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:12:04.683 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[29691]: 6.11 scrub starts 2024-03-26T05:12:04.683 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[29691]: 6.11 scrub ok 2024-03-26T05:12:04.683 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[29691]: 3.5 scrub starts 2024-03-26T05:12:04.683 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[29691]: 3.5 scrub ok 2024-03-26T05:12:04.683 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[29691]: 5.9 scrub starts 2024-03-26T05:12:04.683 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[29691]: 5.9 scrub ok 2024-03-26T05:12:04.683 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[29691]: 6.18 scrub starts 2024-03-26T05:12:04.683 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[29691]: 6.18 scrub ok 2024-03-26T05:12:04.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[40404]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:12:04.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[40404]: 6.11 scrub starts 2024-03-26T05:12:04.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[40404]: 6.11 scrub ok 2024-03-26T05:12:04.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[40404]: 3.5 scrub starts 2024-03-26T05:12:04.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[40404]: 3.5 scrub ok 2024-03-26T05:12:04.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[40404]: 5.9 scrub starts 2024-03-26T05:12:04.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[40404]: 5.9 scrub ok 2024-03-26T05:12:04.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[40404]: 6.18 scrub starts 2024-03-26T05:12:04.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:04 smithi080 ceph-mon[40404]: 6.18 scrub ok 2024-03-26T05:12:04.685 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:04.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:04.744 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 56 16 88431 88415 98.6594 71.1875 0.00721351 0.0101265 2024-03-26T05:12:04.788 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:04.827 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:04.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:04.929 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:04.968 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:04.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:05.070 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:05.103 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:05.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:05.204 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:05.242 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:05.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:05.343 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:05.370 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":532,"stamp":"2024-03-26T05:12:02.565491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5457975646,"num_objects":83503,"num_object_clones":0,"num_object_copies":250509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83503,"num_whiteouts":0,"num_read":2339,"num_read_kb":2153,"num_write":84366,"num_write_kb":5331933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1774542,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81531,"ondisk_log_size":81531,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18726620,"kb_used_data":15847748,"kb_used_omap":12,"kb_used_meta":2878771,"kb_avail":731234596,"statfs":{"total":767960285184,"available":748784226304,"internally_reserved":0,"allocated":16228093952,"data_stored":16223918406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2947862064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,3,4,6,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":399769600,"num_objects":6100,"num_object_clones":0,"num_object_copies":18300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6100,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6100,"num_write_kb":390400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.626044"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":483,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.541054+0000","last_change":"2024-03-26T05:09:39.468117+0000","last_active":"2024-03-26T05:11:59.541054+0000","last_peered":"2024-03-26T05:11:59.541054+0000","last_clean":"2024-03-26T05:11:59.541054+0000","last_became_active":"2024-03-26T05:07:59.996436+0000","last_became_peered":"2024-03-26T05:07:59.996436+0000","last_unstale":"2024-03-26T05:11:59.541054+0000","last_undegraded":"2024-03-26T05:11:59.541054+0000","last_fullsized":"2024-03-26T05:11:59.541054+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:39.468037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:39.468037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.364972+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":493,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.556357+0000","last_change":"2024-03-26T05:11:30.527893+0000","last_active":"2024-03-26T05:11:59.556357+0000","last_peered":"2024-03-26T05:11:59.556357+0000","last_clean":"2024-03-26T05:11:59.556357+0000","last_became_active":"2024-03-26T05:07:59.997089+0000","last_became_peered":"2024-03-26T05:07:59.997089+0000","last_unstale":"2024-03-26T05:11:59.556357+0000","last_undegraded":"2024-03-26T05:11:59.556357+0000","last_fullsized":"2024-03-26T05:11:59.556357+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:30.527764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:30.527764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:41.965653+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":504,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.556017+0000","last_change":"2024-03-26T05:11:24.102696+0000","last_active":"2024-03-26T05:11:59.556017+0000","last_peered":"2024-03-26T05:11:59.556017+0000","last_clean":"2024-03-26T05:11:59.556017+0000","last_became_active":"2024-03-26T05:07:59.994431+0000","last_became_peered":"2024-03-26T05:07:59.994431+0000","last_unstale":"2024-03-26T05:11:59.556017+0000","last_undegraded":"2024-03-26T05:11:59.556017+0000","last_fullsized":"2024-03-26T05:11:59.556017+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:24.102571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:24.102571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:51.047806+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"254'21","reported_seq":516,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.540885+0000","last_change":"2024-03-26T05:09:55.417464+0000","last_active":"2024-03-26T05:11:59.540885+0000","last_peered":"2024-03-26T05:11:59.540885+0000","last_clean":"2024-03-26T05:11:59.540885+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:11:59.540885+0000","last_undegraded":"2024-03-26T05:11:59.540885+0000","last_fullsized":"2024-03-26T05:11:59.540885+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-26T05:09:55.417325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.417325+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365159+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":461,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.557119+0000","last_change":"2024-03-26T05:09:59.528706+0000","last_active":"2024-03-26T05:11:59.557119+0000","last_peered":"2024-03-26T05:11:59.557119+0000","last_clean":"2024-03-26T05:11:59.557119+0000","last_became_active":"2024-03-26T05:00:29.462660+0000","last_became_peered":"2024-03-26T05:00:29.462660+0000","last_unstale":"2024-03-26T05:11:59.557119+0000","last_undegraded":"2024-03-26T05:11:59.557119+0000","last_fullsized":"2024-03-26T05:11:59.557119+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:59.528596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:59.528596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671487+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":475,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.540754+0000","last_change":"2024-03-26T05:10:29.404548+0000","last_active":"2024-03-26T05:11:59.540754+0000","last_peered":"2024-03-26T05:11:59.540754+0000","last_clean":"2024-03-26T05:11:59.540754+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:11:59.540754+0000","last_undegraded":"2024-03-26T05:11:59.540754+0000","last_fullsized":"2024-03-26T05:11:59.540754+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:29.404404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.404404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":482,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.556603+0000","last_change":"2024-03-26T05:09:40.635212+0000","last_active":"2024-03-26T05:11:59.556603+0000","last_peered":"2024-03-26T05:11:59.556603+0000","last_clean":"2024-03-26T05:11:59.556603+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:11:59.556603+0000","last_undegraded":"2024-03-26T05:11:59.556603+0000","last_fullsized":"2024-03-26T05:11:59.556603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:40.635129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:09:40.635129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"254'51","reported_seq":568,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.550131+0000","last_change":"2024-03-26T05:11:33.471634+0000","last_active":"2024-03-26T05:11:59.550131+0000","last_peered":"2024-03-26T05:11:59.550131+0000","last_clean":"2024-03-26T05:11:59.550131+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:11:59.550131+0000","last_undegraded":"2024-03-26T05:11:59.550131+0000","last_fullsized":"2024-03-26T05:11:59.550131+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'51","last_scrub_stamp":"2024-03-26T05:11:33.471534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:11:33.471534+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:37.511904+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":479,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.553921+0000","last_change":"2024-03-26T05:10:30.211104+0000","last_active":"2024-03-26T05:11:59.553921+0000","last_peered":"2024-03-26T05:11:59.553921+0000","last_clean":"2024-03-26T05:11:59.553921+0000","last_became_active":"2024-03-26T05:00:28.977223+0000","last_became_peered":"2024-03-26T05:00:28.977223+0000","last_unstale":"2024-03-26T05:11:59.553921+0000","last_undegraded":"2024-03-26T05:11:59.553921+0000","last_fullsized":"2024-03-26T05:11:59.553921+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:30.211007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.211007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":163,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.551859+0000","last_change":"2024-03-26T05:09:57.144879+0000","last_active":"2024-03-26T05:11:59.551859+0000","last_peered":"2024-03-26T05:11:59.551859+0000","last_clean":"2024-03-26T05:11:59.551859+0000","last_became_active":"2024-03-26T05:08:00.000128+0000","last_became_peered":"2024-03-26T05:08:00.000128+0000","last_unstale":"2024-03-26T05:11:59.551859+0000","last_undegraded":"2024-03-26T05:11:59.551859+0000","last_fullsized":"2024-03-26T05:11:59.551859+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:57.144806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:57.144806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":163,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.554988+0000","last_change":"2024-03-26T05:10:29.515852+0000","last_active":"2024-03-26T05:11:59.554988+0000","last_peered":"2024-03-26T05:11:59.554988+0000","last_clean":"2024-03-26T05:11:59.554988+0000","last_became_active":"2024-03-26T05:07:59.947557+0000","last_became_peered":"2024-03-26T05:07:59.947557+0000","last_unstale":"2024-03-26T05:11:59.554988+0000","last_undegraded":"2024-03-26T05:11:59.554988+0000","last_fullsized":"2024-03-26T05:11:59.554988+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:29.515767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.515767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.727213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"254'27","reported_seq":536,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.541409+0000","last_change":"2024-03-26T05:11:30.499748+0000","last_active":"2024-03-26T05:11:59.541409+0000","last_peered":"2024-03-26T05:11:59.541409+0000","last_clean":"2024-03-26T05:11:59.541409+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:11:59.541409+0000","last_undegraded":"2024-03-26T05:11:59.541409+0000","last_fullsized":"2024-03-26T05:11:59.541409+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:11:30.499663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:11:30.499663+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:33.755082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":163,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.556949+0000","last_change":"2024-03-26T05:09:48.186435+0000","last_active":"2024-03-26T05:11:59.556949+0000","last_peered":"2024-03-26T05:11:59.556949+0000","last_clean":"2024-03-26T05:11:59.556949+0000","last_became_active":"2024-03-26T05:07:59.998777+0000","last_became_peered":"2024-03-26T05:07:59.998777+0000","last_unstale":"2024-03-26T05:11:59.556949+0000","last_undegraded":"2024-03-26T05:11:59.556949+0000","last_fullsized":"2024-03-26T05:11:59.556949+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:48.186287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.186287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":487,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.548216+0000","last_change":"2024-03-26T05:10:41.797043+0000","last_active":"2024-03-26T05:11:59.548216+0000","last_peered":"2024-03-26T05:11:59.548216+0000","last_clean":"2024-03-26T05:11:59.548216+0000","last_became_active":"2024-03-26T05:00:26.960474+0000","last_became_peered":"2024-03-26T05:00:26.960474+0000","last_unstale":"2024-03-26T05:11:59.548216+0000","last_undegraded":"2024-03-26T05:11:59.548216+0000","last_fullsized":"2024-03-26T05:11:59.548216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:41.796930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.796930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.932914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":163,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.551860+0000","last_change":"2024-03-26T05:11:30.566220+0000","last_active":"2024-03-26T05:11:59.551860+0000","last_peered":"2024-03-26T05:11:59.551860+0000","last_clean":"2024-03-26T05:11:59.551860+0000","last_became_active":"2024-03-26T05:08:00.011262+0000","last_became_peered":"2024-03-26T05:08:00.011262+0000","last_unstale":"2024-03-26T05:11:59.551860+0000","last_undegraded":"2024-03-26T05:11:59.551860+0000","last_fullsized":"2024-03-26T05:11:59.551860+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:30.566108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:30.566108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:33.317879+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"254'13","reported_seq":504,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.541345+0000","last_change":"2024-03-26T05:10:32.425969+0000","last_active":"2024-03-26T05:11:59.541345+0000","last_peered":"2024-03-26T05:11:59.541345+0000","last_clean":"2024-03-26T05:11:59.541345+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:11:59.541345+0000","last_undegraded":"2024-03-26T05:11:59.541345+0000","last_fullsized":"2024-03-26T05:11:59.541345+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'13","last_scrub_stamp":"2024-03-26T05:10:32.425855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:32.425855+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":492,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.542463+0000","last_change":"2024-03-26T05:09:56.453730+0000","last_active":"2024-03-26T05:11:59.542463+0000","last_peered":"2024-03-26T05:11:59.542463+0000","last_clean":"2024-03-26T05:11:59.542463+0000","last_became_active":"2024-03-26T05:07:59.942829+0000","last_became_peered":"2024-03-26T05:07:59.942829+0000","last_unstale":"2024-03-26T05:11:59.542463+0000","last_undegraded":"2024-03-26T05:11:59.542463+0000","last_fullsized":"2024-03-26T05:11:59.542463+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:56.453608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:09:56.453608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365125+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":473,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.558761+0000","last_change":"2024-03-26T05:09:57.476795+0000","last_active":"2024-03-26T05:11:59.558761+0000","last_peered":"2024-03-26T05:11:59.558761+0000","last_clean":"2024-03-26T05:11:59.558761+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:11:59.558761+0000","last_undegraded":"2024-03-26T05:11:59.558761+0000","last_fullsized":"2024-03-26T05:11:59.558761+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:57.476689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:57.476689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.629614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":491,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.558043+0000","last_change":"2024-03-26T05:10:38.179665+0000","last_active":"2024-03-26T05:11:59.558043+0000","last_peered":"2024-03-26T05:11:59.558043+0000","last_clean":"2024-03-26T05:11:59.558043+0000","last_became_active":"2024-03-26T05:00:22.777536+0000","last_became_peered":"2024-03-26T05:00:22.777536+0000","last_unstale":"2024-03-26T05:11:59.558043+0000","last_undegraded":"2024-03-26T05:11:59.558043+0000","last_fullsized":"2024-03-26T05:11:59.558043+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:38.179521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:10:38.179521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"254'27","reported_seq":536,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.554313+0000","last_change":"2024-03-26T05:11:29.185267+0000","last_active":"2024-03-26T05:11:59.554313+0000","last_peered":"2024-03-26T05:11:59.554313+0000","last_clean":"2024-03-26T05:11:59.554313+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:11:59.554313+0000","last_undegraded":"2024-03-26T05:11:59.554313+0000","last_fullsized":"2024-03-26T05:11:59.554313+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:11:29.185118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.185118+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:30.826158+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":482,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.559326+0000","last_change":"2024-03-26T05:09:45.842503+0000","last_active":"2024-03-26T05:11:59.559326+0000","last_peered":"2024-03-26T05:11:59.559326+0000","last_clean":"2024-03-26T05:11:59.559326+0000","last_became_active":"2024-03-26T05:07:59.959705+0000","last_became_peered":"2024-03-26T05:07:59.959705+0000","last_unstale":"2024-03-26T05:11:59.559326+0000","last_undegraded":"2024-03-26T05:11:59.559326+0000","last_fullsized":"2024-03-26T05:11:59.559326+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:45.842392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.842392+0000","last_clean_scrub_stamp":"2024-03-26T05:09:45.842392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671255+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":475,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.542774+0000","last_change":"2024-03-26T05:09:54.412717+0000","last_active":"2024-03-26T05:11:59.542774+0000","last_peered":"2024-03-26T05:11:59.542774+0000","last_clean":"2024-03-26T05:11:59.542774+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:11:59.542774+0000","last_undegraded":"2024-03-26T05:11:59.542774+0000","last_fullsized":"2024-03-26T05:11:59.542774+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:54.412617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:54.412617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":499,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.541510+0000","last_change":"2024-03-26T05:10:45.477455+0000","last_active":"2024-03-26T05:11:59.541510+0000","last_peered":"2024-03-26T05:11:59.541510+0000","last_clean":"2024-03-26T05:11:59.541510+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:11:59.541510+0000","last_undegraded":"2024-03-26T05:11:59.541510+0000","last_fullsized":"2024-03-26T05:11:59.541510+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:45.477372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:10:45.477372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"254'26","reported_seq":523,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.557717+0000","last_change":"2024-03-26T05:11:20.328965+0000","last_active":"2024-03-26T05:11:59.557717+0000","last_peered":"2024-03-26T05:11:59.557717+0000","last_clean":"2024-03-26T05:11:59.557717+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:11:59.557717+0000","last_undegraded":"2024-03-26T05:11:59.557717+0000","last_fullsized":"2024-03-26T05:11:59.557717+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:11:20.328720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:11:20.328720+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:29.978823+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":496,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.549070+0000","last_change":"2024-03-26T05:10:32.752769+0000","last_active":"2024-03-26T05:11:59.549070+0000","last_peered":"2024-03-26T05:11:59.549070+0000","last_clean":"2024-03-26T05:11:59.549070+0000","last_became_active":"2024-03-26T05:07:59.955125+0000","last_became_peered":"2024-03-26T05:07:59.955125+0000","last_unstale":"2024-03-26T05:11:59.549070+0000","last_undegraded":"2024-03-26T05:11:59.549070+0000","last_fullsized":"2024-03-26T05:11:59.549070+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:10:32.752678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:32.752678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":648,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.553717+0000","last_change":"2024-03-26T05:11:19.698645+0000","last_active":"2024-03-26T05:11:59.553717+0000","last_peered":"2024-03-26T05:11:59.553717+0000","last_clean":"2024-03-26T05:11:59.553717+0000","last_became_active":"2024-03-26T05:00:26.954413+0000","last_became_peered":"2024-03-26T05:00:26.954413+0000","last_unstale":"2024-03-26T05:11:59.553717+0000","last_undegraded":"2024-03-26T05:11:59.553717+0000","last_fullsized":"2024-03-26T05:11:59.553717+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:11:19.698504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:19.698504+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:19.963186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":490,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.556432+0000","last_change":"2024-03-26T05:11:27.506774+0000","last_active":"2024-03-26T05:11:59.556432+0000","last_peered":"2024-03-26T05:11:59.556432+0000","last_clean":"2024-03-26T05:11:59.556432+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:11:59.556432+0000","last_undegraded":"2024-03-26T05:11:59.556432+0000","last_fullsized":"2024-03-26T05:11:59.556432+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:11:27.506646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:27.506646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:30.719232+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"254'26","reported_seq":530,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.555654+0000","last_change":"2024-03-26T05:10:41.134236+0000","last_active":"2024-03-26T05:11:59.555654+0000","last_peered":"2024-03-26T05:11:59.555654+0000","last_clean":"2024-03-26T05:11:59.555654+0000","last_became_active":"2024-03-26T05:00:25.648294+0000","last_became_peered":"2024-03-26T05:00:25.648294+0000","last_unstale":"2024-03-26T05:11:59.555654+0000","last_undegraded":"2024-03-26T05:11:59.555654+0000","last_fullsized":"2024-03-26T05:11:59.555654+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:10:41.134129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.134129+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.559878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":489,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.559199+0000","last_change":"2024-03-26T05:11:21.653534+0000","last_active":"2024-03-26T05:11:59.559199+0000","last_peered":"2024-03-26T05:11:59.559199+0000","last_clean":"2024-03-26T05:11:59.559199+0000","last_became_active":"2024-03-26T05:07:59.995261+0000","last_became_peered":"2024-03-26T05:07:59.995261+0000","last_unstale":"2024-03-26T05:11:59.559199+0000","last_undegraded":"2024-03-26T05:11:59.559199+0000","last_fullsized":"2024-03-26T05:11:59.559199+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:21.653418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:21.653418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:47.216314+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":484,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.555916+0000","last_change":"2024-03-26T05:11:27.081061+0000","last_active":"2024-03-26T05:11:59.555916+0000","last_peered":"2024-03-26T05:11:59.555916+0000","last_clean":"2024-03-26T05:11:59.555916+0000","last_became_active":"2024-03-26T05:00:26.962994+0000","last_became_peered":"2024-03-26T05:00:26.962994+0000","last_unstale":"2024-03-26T05:11:59.555916+0000","last_undegraded":"2024-03-26T05:11:59.555916+0000","last_fullsized":"2024-03-26T05:11:59.555916+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:27.080959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:27.080959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:49.679302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":481,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.549778+0000","last_change":"2024-03-26T05:09:56.528351+0000","last_active":"2024-03-26T05:11:59.549778+0000","last_peered":"2024-03-26T05:11:59.549778+0000","last_clean":"2024-03-26T05:11:59.549778+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:11:59.549778+0000","last_undegraded":"2024-03-26T05:11:59.549778+0000","last_fullsized":"2024-03-26T05:11:59.549778+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:56.528075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:56.528075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671462+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"254'25","reported_seq":596,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.553805+0000","last_change":"2024-03-26T05:10:39.649729+0000","last_active":"2024-03-26T05:11:59.553805+0000","last_peered":"2024-03-26T05:11:59.553805+0000","last_clean":"2024-03-26T05:11:59.553805+0000","last_became_active":"2024-03-26T05:07:59.962007+0000","last_became_peered":"2024-03-26T05:07:59.962007+0000","last_unstale":"2024-03-26T05:11:59.553805+0000","last_undegraded":"2024-03-26T05:11:59.553805+0000","last_fullsized":"2024-03-26T05:11:59.553805+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:10:39.649620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:39.649620+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.262212+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":475,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.548063+0000","last_change":"2024-03-26T05:10:43.824858+0000","last_active":"2024-03-26T05:11:59.548063+0000","last_peered":"2024-03-26T05:11:59.548063+0000","last_clean":"2024-03-26T05:11:59.548063+0000","last_became_active":"2024-03-26T05:00:28.978457+0000","last_became_peered":"2024-03-26T05:00:28.978457+0000","last_unstale":"2024-03-26T05:11:59.548063+0000","last_undegraded":"2024-03-26T05:11:59.548063+0000","last_fullsized":"2024-03-26T05:11:59.548063+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:43.824746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.824746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":488,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.552980+0000","last_change":"2024-03-26T05:11:25.693733+0000","last_active":"2024-03-26T05:11:59.552980+0000","last_peered":"2024-03-26T05:11:59.552980+0000","last_clean":"2024-03-26T05:11:59.552980+0000","last_became_active":"2024-03-26T05:00:26.957289+0000","last_became_peered":"2024-03-26T05:00:26.957289+0000","last_unstale":"2024-03-26T05:11:59.552980+0000","last_undegraded":"2024-03-26T05:11:59.552980+0000","last_fullsized":"2024-03-26T05:11:59.552980+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.693592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.693592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:48.442810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":503,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.556210+0000","last_change":"2024-03-26T05:10:46.108129+0000","last_active":"2024-03-26T05:11:59.556210+0000","last_peered":"2024-03-26T05:11:59.556210+0000","last_clean":"2024-03-26T05:11:59.556210+0000","last_became_active":"2024-03-26T05:07:59.993996+0000","last_became_peered":"2024-03-26T05:07:59.993996+0000","last_unstale":"2024-03-26T05:11:59.556210+0000","last_undegraded":"2024-03-26T05:11:59.556210+0000","last_fullsized":"2024-03-26T05:11:59.556210+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:46.108051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:46.108051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:12.619539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"254'10","reported_seq":491,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.549875+0000","last_change":"2024-03-26T05:10:29.575129+0000","last_active":"2024-03-26T05:11:59.549875+0000","last_peered":"2024-03-26T05:11:59.549875+0000","last_clean":"2024-03-26T05:11:59.549875+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:11:59.549875+0000","last_undegraded":"2024-03-26T05:11:59.549875+0000","last_fullsized":"2024-03-26T05:11:59.549875+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'6","last_scrub_stamp":"2024-03-26T05:10:29.575014+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.575014+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.644360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":469,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.557290+0000","last_change":"2024-03-26T05:10:41.595090+0000","last_active":"2024-03-26T05:11:59.557290+0000","last_peered":"2024-03-26T05:11:59.557290+0000","last_clean":"2024-03-26T05:11:59.557290+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:11:59.557290+0000","last_undegraded":"2024-03-26T05:11:59.557290+0000","last_fullsized":"2024-03-26T05:11:59.557290+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:41.594950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.594950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671568+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":487,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.548090+0000","last_change":"2024-03-26T05:11:23.957628+0000","last_active":"2024-03-26T05:11:59.548090+0000","last_peered":"2024-03-26T05:11:59.548090+0000","last_clean":"2024-03-26T05:11:59.548090+0000","last_became_active":"2024-03-26T05:00:26.959572+0000","last_became_peered":"2024-03-26T05:00:26.959572+0000","last_unstale":"2024-03-26T05:11:59.548090+0000","last_undegraded":"2024-03-26T05:11:59.548090+0000","last_fullsized":"2024-03-26T05:11:59.548090+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:23.957542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:11:23.957542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:45.371911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":163,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.552115+0000","last_change":"2024-03-26T05:11:19.513163+0000","last_active":"2024-03-26T05:11:59.552115+0000","last_peered":"2024-03-26T05:11:59.552115+0000","last_clean":"2024-03-26T05:11:59.552115+0000","last_became_active":"2024-03-26T05:07:59.997439+0000","last_became_peered":"2024-03-26T05:07:59.997439+0000","last_unstale":"2024-03-26T05:11:59.552115+0000","last_undegraded":"2024-03-26T05:11:59.552115+0000","last_fullsized":"2024-03-26T05:11:59.552115+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:19.513035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:11:19.513035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:21.802282+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"254'25","reported_seq":520,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.550545+0000","last_change":"2024-03-26T05:11:18.408613+0000","last_active":"2024-03-26T05:11:59.550545+0000","last_peered":"2024-03-26T05:11:59.550545+0000","last_clean":"2024-03-26T05:11:59.550545+0000","last_became_active":"2024-03-26T05:00:25.648089+0000","last_became_peered":"2024-03-26T05:00:25.648089+0000","last_unstale":"2024-03-26T05:11:59.550545+0000","last_undegraded":"2024-03-26T05:11:59.550545+0000","last_fullsized":"2024-03-26T05:11:59.550545+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:11:18.408463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:11:18.408463+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:25.165459+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":491,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.553482+0000","last_change":"2024-03-26T05:10:29.171886+0000","last_active":"2024-03-26T05:11:59.553482+0000","last_peered":"2024-03-26T05:11:59.553482+0000","last_clean":"2024-03-26T05:11:59.553482+0000","last_became_active":"2024-03-26T05:07:59.994210+0000","last_became_peered":"2024-03-26T05:07:59.994210+0000","last_unstale":"2024-03-26T05:11:59.553482+0000","last_undegraded":"2024-03-26T05:11:59.553482+0000","last_fullsized":"2024-03-26T05:11:59.553482+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:29.171780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.171780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":465,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.557034+0000","last_change":"2024-03-26T05:09:26.674360+0000","last_active":"2024-03-26T05:11:59.557034+0000","last_peered":"2024-03-26T05:11:59.557034+0000","last_clean":"2024-03-26T05:11:59.557034+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:11:59.557034+0000","last_undegraded":"2024-03-26T05:11:59.557034+0000","last_fullsized":"2024-03-26T05:11:59.557034+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:26.674245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:09:26.674245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671338+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":487,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.549303+0000","last_change":"2024-03-26T05:11:33.016175+0000","last_active":"2024-03-26T05:11:59.549303+0000","last_peered":"2024-03-26T05:11:59.549303+0000","last_clean":"2024-03-26T05:11:59.549303+0000","last_became_active":"2024-03-26T05:00:22.773135+0000","last_became_peered":"2024-03-26T05:00:22.773135+0000","last_unstale":"2024-03-26T05:11:59.549303+0000","last_undegraded":"2024-03-26T05:11:59.549303+0000","last_fullsized":"2024-03-26T05:11:59.549303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:33.016072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:11:33.016072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:48.671954+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"254'15","reported_seq":510,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.565422+0000","last_change":"2024-03-26T05:11:19.531695+0000","last_active":"2024-03-26T05:11:59.565422+0000","last_peered":"2024-03-26T05:11:59.565422+0000","last_clean":"2024-03-26T05:11:59.565422+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:11:59.565422+0000","last_undegraded":"2024-03-26T05:11:59.565422+0000","last_fullsized":"2024-03-26T05:11:59.565422+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'15","last_scrub_stamp":"2024-03-26T05:11:19.531525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:11:19.531525+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:21.003317+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":473,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.549845+0000","last_change":"2024-03-26T05:09:28.716469+0000","last_active":"2024-03-26T05:11:59.549845+0000","last_peered":"2024-03-26T05:11:59.549845+0000","last_clean":"2024-03-26T05:11:59.549845+0000","last_became_active":"2024-03-26T05:07:59.999703+0000","last_became_peered":"2024-03-26T05:07:59.999703+0000","last_unstale":"2024-03-26T05:11:59.549845+0000","last_undegraded":"2024-03-26T05:11:59.549845+0000","last_fullsized":"2024-03-26T05:11:59.549845+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:28.716348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:09:28.716348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671227+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":493,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.565309+0000","last_change":"2024-03-26T05:10:38.473529+0000","last_active":"2024-03-26T05:11:59.565309+0000","last_peered":"2024-03-26T05:11:59.565309+0000","last_clean":"2024-03-26T05:11:59.565309+0000","last_became_active":"2024-03-26T05:08:00.001601+0000","last_became_peered":"2024-03-26T05:08:00.001601+0000","last_unstale":"2024-03-26T05:11:59.565309+0000","last_undegraded":"2024-03-26T05:11:59.565309+0000","last_fullsized":"2024-03-26T05:11:59.565309+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:38.473446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:10:38.473446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.629934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":483,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.542514+0000","last_change":"2024-03-26T05:11:21.515404+0000","last_active":"2024-03-26T05:11:59.542514+0000","last_peered":"2024-03-26T05:11:59.542514+0000","last_clean":"2024-03-26T05:11:59.542514+0000","last_became_active":"2024-03-26T05:00:22.767559+0000","last_became_peered":"2024-03-26T05:00:22.767559+0000","last_unstale":"2024-03-26T05:11:59.542514+0000","last_undegraded":"2024-03-26T05:11:59.542514+0000","last_fullsized":"2024-03-26T05:11:59.542514+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:21.515294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:21.515294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:37.148774+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"254'24","reported_seq":523,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.541542+0000","last_change":"2024-03-26T05:11:24.487020+0000","last_active":"2024-03-26T05:11:59.541542+0000","last_peered":"2024-03-26T05:11:59.541542+0000","last_clean":"2024-03-26T05:11:59.541542+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:11:59.541542+0000","last_undegraded":"2024-03-26T05:11:59.541542+0000","last_fullsized":"2024-03-26T05:11:59.541542+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'24","last_scrub_stamp":"2024-03-26T05:11:24.486841+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:11:24.486841+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:49.008854+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":479,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.555356+0000","last_change":"2024-03-26T05:10:50.061225+0000","last_active":"2024-03-26T05:11:59.555356+0000","last_peered":"2024-03-26T05:11:59.555356+0000","last_clean":"2024-03-26T05:11:59.555356+0000","last_became_active":"2024-03-26T05:00:28.984457+0000","last_became_peered":"2024-03-26T05:00:28.984457+0000","last_unstale":"2024-03-26T05:11:59.555356+0000","last_undegraded":"2024-03-26T05:11:59.555356+0000","last_fullsized":"2024-03-26T05:11:59.555356+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:50.061128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_clean_scrub_stamp":"2024-03-26T05:10:50.061128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:08.890951+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"216'10","reported_seq":215,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.554892+0000","last_change":"2024-03-26T05:10:40.373554+0000","last_active":"2024-03-26T05:11:59.554892+0000","last_peered":"2024-03-26T05:11:59.554892+0000","last_clean":"2024-03-26T05:11:59.554892+0000","last_became_active":"2024-03-26T05:08:00.005648+0000","last_became_peered":"2024-03-26T05:08:00.005648+0000","last_unstale":"2024-03-26T05:11:59.554892+0000","last_undegraded":"2024-03-26T05:11:59.554892+0000","last_fullsized":"2024-03-26T05:11:59.554892+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:10:40.373470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.373470+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:08.010761+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":487,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.548341+0000","last_change":"2024-03-26T05:09:45.438723+0000","last_active":"2024-03-26T05:11:59.548341+0000","last_peered":"2024-03-26T05:11:59.548341+0000","last_clean":"2024-03-26T05:11:59.548341+0000","last_became_active":"2024-03-26T05:00:23.346839+0000","last_became_peered":"2024-03-26T05:00:23.346839+0000","last_unstale":"2024-03-26T05:11:59.548341+0000","last_undegraded":"2024-03-26T05:11:59.548341+0000","last_fullsized":"2024-03-26T05:11:59.548341+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:45.438615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:45.438615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"254'32","reported_seq":531,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.558192+0000","last_change":"2024-03-26T05:09:30.638211+0000","last_active":"2024-03-26T05:11:59.558192+0000","last_peered":"2024-03-26T05:11:59.558192+0000","last_clean":"2024-03-26T05:11:59.558192+0000","last_became_active":"2024-03-26T05:00:25.648353+0000","last_became_peered":"2024-03-26T05:00:25.648353+0000","last_unstale":"2024-03-26T05:11:59.558192+0000","last_undegraded":"2024-03-26T05:11:59.558192+0000","last_fullsized":"2024-03-26T05:11:59.558192+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'22","last_scrub_stamp":"2024-03-26T05:09:30.635624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:30.635624+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671314+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":483,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.542309+0000","last_change":"2024-03-26T05:10:40.548775+0000","last_active":"2024-03-26T05:11:59.542309+0000","last_peered":"2024-03-26T05:11:59.542309+0000","last_clean":"2024-03-26T05:11:59.542309+0000","last_became_active":"2024-03-26T05:07:59.942512+0000","last_became_peered":"2024-03-26T05:07:59.942512+0000","last_unstale":"2024-03-26T05:11:59.542309+0000","last_undegraded":"2024-03-26T05:11:59.542309+0000","last_fullsized":"2024-03-26T05:11:59.542309+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:40.548688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.548688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.537910+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"120'9","reported_seq":625,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.558565+0000","last_change":"2024-03-26T05:10:30.593032+0000","last_active":"2024-03-26T05:11:59.558565+0000","last_peered":"2024-03-26T05:11:59.558565+0000","last_clean":"2024-03-26T05:11:59.558565+0000","last_became_active":"2024-03-26T05:00:26.954645+0000","last_became_peered":"2024-03-26T05:00:26.954645+0000","last_unstale":"2024-03-26T05:11:59.558565+0000","last_undegraded":"2024-03-26T05:11:59.558565+0000","last_fullsized":"2024-03-26T05:11:59.558565+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'9","last_scrub_stamp":"2024-03-26T05:10:30.592948+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.592948+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":499,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.989425+0000","last_change":"2024-03-26T05:11:59.989459+0000","last_active":"2024-03-26T05:11:59.989425+0000","last_peered":"2024-03-26T05:11:59.989425+0000","last_clean":"2024-03-26T05:11:59.989425+0000","last_became_active":"2024-03-26T05:00:22.776919+0000","last_became_peered":"2024-03-26T05:00:22.776919+0000","last_unstale":"2024-03-26T05:11:59.989425+0000","last_undegraded":"2024-03-26T05:11:59.989425+0000","last_fullsized":"2024-03-26T05:11:59.989425+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:59.989341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:59.989341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:13:18.098300+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"254'26","reported_seq":184,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.554734+0000","last_change":"2024-03-26T05:10:33.467211+0000","last_active":"2024-03-26T05:11:59.554734+0000","last_peered":"2024-03-26T05:11:59.554734+0000","last_clean":"2024-03-26T05:11:59.554734+0000","last_became_active":"2024-03-26T05:07:59.960346+0000","last_became_peered":"2024-03-26T05:07:59.960346+0000","last_unstale":"2024-03-26T05:11:59.554734+0000","last_undegraded":"2024-03-26T05:11:59.554734+0000","last_fullsized":"2024-03-26T05:11:59.554734+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:10:33.467113+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.467113+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":485,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.554532+0000","last_change":"2024-03-26T05:10:41.140060+0000","last_active":"2024-03-26T05:11:59.554532+0000","last_peered":"2024-03-26T05:11:59.554532+0000","last_clean":"2024-03-26T05:11:59.554532+0000","last_became_active":"2024-03-26T05:00:28.980913+0000","last_became_peered":"2024-03-26T05:00:28.980913+0000","last_unstale":"2024-03-26T05:11:59.554532+0000","last_undegraded":"2024-03-26T05:11:59.554532+0000","last_fullsized":"2024-03-26T05:11:59.554532+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:41.139989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:41.139989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:04.392633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":473,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.560014+0000","last_change":"2024-03-26T05:11:19.364780+0000","last_active":"2024-03-26T05:11:59.560014+0000","last_peered":"2024-03-26T05:11:59.560014+0000","last_clean":"2024-03-26T05:11:59.560014+0000","last_became_active":"2024-03-26T05:00:26.955889+0000","last_became_peered":"2024-03-26T05:00:26.955889+0000","last_unstale":"2024-03-26T05:11:59.560014+0000","last_undegraded":"2024-03-26T05:11:59.560014+0000","last_fullsized":"2024-03-26T05:11:59.560014+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:19.364263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:11:19.364263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:31.115621+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":487,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.551144+0000","last_change":"2024-03-26T05:11:25.939682+0000","last_active":"2024-03-26T05:11:59.551144+0000","last_peered":"2024-03-26T05:11:59.551144+0000","last_clean":"2024-03-26T05:11:59.551144+0000","last_became_active":"2024-03-26T05:00:22.770452+0000","last_became_peered":"2024-03-26T05:00:22.770452+0000","last_unstale":"2024-03-26T05:11:59.551144+0000","last_undegraded":"2024-03-26T05:11:59.551144+0000","last_fullsized":"2024-03-26T05:11:59.551144+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.939576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.939576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:46.592227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"254'27","reported_seq":537,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.555754+0000","last_change":"2024-03-26T05:10:43.135431+0000","last_active":"2024-03-26T05:11:59.555754+0000","last_peered":"2024-03-26T05:11:59.555754+0000","last_clean":"2024-03-26T05:11:59.555754+0000","last_became_active":"2024-03-26T05:00:25.648661+0000","last_became_peered":"2024-03-26T05:00:25.648661+0000","last_unstale":"2024-03-26T05:11:59.555754+0000","last_undegraded":"2024-03-26T05:11:59.555754+0000","last_fullsized":"2024-03-26T05:11:59.555754+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:10:43.135325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.135325+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":475,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.549297+0000","last_change":"2024-03-26T05:10:35.847292+0000","last_active":"2024-03-26T05:11:59.549297+0000","last_peered":"2024-03-26T05:11:59.549297+0000","last_clean":"2024-03-26T05:11:59.549297+0000","last_became_active":"2024-03-26T05:00:29.462304+0000","last_became_peered":"2024-03-26T05:00:29.462304+0000","last_unstale":"2024-03-26T05:11:59.549297+0000","last_undegraded":"2024-03-26T05:11:59.549297+0000","last_fullsized":"2024-03-26T05:11:59.549297+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:35.847191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:35.847191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":163,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.552023+0000","last_change":"2024-03-26T05:09:52.050966+0000","last_active":"2024-03-26T05:11:59.552023+0000","last_peered":"2024-03-26T05:11:59.552023+0000","last_clean":"2024-03-26T05:11:59.552023+0000","last_became_active":"2024-03-26T05:07:59.997434+0000","last_became_peered":"2024-03-26T05:07:59.997434+0000","last_unstale":"2024-03-26T05:11:59.552023+0000","last_undegraded":"2024-03-26T05:11:59.552023+0000","last_fullsized":"2024-03-26T05:11:59.552023+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:52.050832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:52.050832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":523,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.552827+0000","last_change":"2024-03-26T05:11:21.665454+0000","last_active":"2024-03-26T05:11:59.552827+0000","last_peered":"2024-03-26T05:11:59.552827+0000","last_clean":"2024-03-26T05:11:59.552827+0000","last_became_active":"2024-03-26T05:07:59.992217+0000","last_became_peered":"2024-03-26T05:07:59.992217+0000","last_unstale":"2024-03-26T05:11:59.552827+0000","last_undegraded":"2024-03-26T05:11:59.552827+0000","last_fullsized":"2024-03-26T05:11:59.552827+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:11:21.665302+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_clean_scrub_stamp":"2024-03-26T05:11:21.665302+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:38.796997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"254'18","reported_seq":507,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.559484+0000","last_change":"2024-03-26T05:10:36.560789+0000","last_active":"2024-03-26T05:11:59.559484+0000","last_peered":"2024-03-26T05:11:59.559484+0000","last_clean":"2024-03-26T05:11:59.559484+0000","last_became_active":"2024-03-26T05:00:25.648068+0000","last_became_peered":"2024-03-26T05:00:25.648068+0000","last_unstale":"2024-03-26T05:11:59.559484+0000","last_undegraded":"2024-03-26T05:11:59.559484+0000","last_fullsized":"2024-03-26T05:11:59.559484+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'18","last_scrub_stamp":"2024-03-26T05:10:36.560689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:36.560689+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:04.290785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"286'5239","reported_seq":5292,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:12:00.844995+0000","last_change":"2024-03-26T05:10:55.650520+0000","last_active":"2024-03-26T05:12:00.844995+0000","last_peered":"2024-03-26T05:12:00.844995+0000","last_clean":"2024-03-26T05:12:00.844995+0000","last_became_active":"2024-03-26T05:10:55.650187+0000","last_became_peered":"2024-03-26T05:10:55.650187+0000","last_unstale":"2024-03-26T05:12:00.844995+0000","last_undegraded":"2024-03-26T05:12:00.844995+0000","last_fullsized":"2024-03-26T05:12:00.844995+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":5239,"log_dups_size":0,"ondisk_log_size":5239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:15.842844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343343104,"num_objects":5239,"num_object_clones":0,"num_object_copies":15717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5239,"num_write_kb":335296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":167,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.552360+0000","last_change":"2024-03-26T05:11:24.596604+0000","last_active":"2024-03-26T05:11:59.552360+0000","last_peered":"2024-03-26T05:11:59.552360+0000","last_clean":"2024-03-26T05:11:59.552360+0000","last_became_active":"2024-03-26T05:08:00.000166+0000","last_became_peered":"2024-03-26T05:08:00.000166+0000","last_unstale":"2024-03-26T05:11:59.552360+0000","last_undegraded":"2024-03-26T05:11:59.552360+0000","last_fullsized":"2024-03-26T05:11:59.552360+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:11:24.596481+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:11:24.596481+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:34.326458+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":517,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.549576+0000","last_change":"2024-03-26T05:11:29.959288+0000","last_active":"2024-03-26T05:11:59.549576+0000","last_peered":"2024-03-26T05:11:59.549576+0000","last_clean":"2024-03-26T05:11:59.549576+0000","last_became_active":"2024-03-26T05:00:22.774427+0000","last_became_peered":"2024-03-26T05:00:22.774427+0000","last_unstale":"2024-03-26T05:11:59.549576+0000","last_undegraded":"2024-03-26T05:11:59.549576+0000","last_fullsized":"2024-03-26T05:11:59.549576+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.959120+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:33.245239+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"254'48","reported_seq":210,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.551501+0000","last_change":"2024-03-26T05:09:59.156101+0000","last_active":"2024-03-26T05:11:59.551501+0000","last_peered":"2024-03-26T05:11:59.551501+0000","last_clean":"2024-03-26T05:11:59.551501+0000","last_became_active":"2024-03-26T05:07:59.947478+0000","last_became_peered":"2024-03-26T05:07:59.947478+0000","last_unstale":"2024-03-26T05:11:59.551501+0000","last_undegraded":"2024-03-26T05:11:59.551501+0000","last_fullsized":"2024-03-26T05:11:59.551501+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'36","last_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:09:59.155894+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":628,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.551429+0000","last_change":"2024-03-26T05:10:29.818124+0000","last_active":"2024-03-26T05:11:59.551429+0000","last_peered":"2024-03-26T05:11:59.551429+0000","last_clean":"2024-03-26T05:11:59.551429+0000","last_became_active":"2024-03-26T05:00:26.959738+0000","last_became_peered":"2024-03-26T05:00:26.959738+0000","last_unstale":"2024-03-26T05:11:59.551429+0000","last_undegraded":"2024-03-26T05:11:59.551429+0000","last_fullsized":"2024-03-26T05:11:59.551429+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:10:29.818015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:29.818015+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"286'5332","reported_seq":5383,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:12:00.429188+0000","last_change":"2024-03-26T05:10:55.653437+0000","last_active":"2024-03-26T05:12:00.429188+0000","last_peered":"2024-03-26T05:12:00.429188+0000","last_clean":"2024-03-26T05:12:00.429188+0000","last_became_active":"2024-03-26T05:10:55.652957+0000","last_became_peered":"2024-03-26T05:10:55.652957+0000","last_unstale":"2024-03-26T05:12:00.429188+0000","last_undegraded":"2024-03-26T05:12:00.429188+0000","last_fullsized":"2024-03-26T05:12:00.429188+0000","mapping_epoch":265,"log_start":"271'600","ondisk_log_start":"271'600","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":4732,"log_dups_size":0,"ondisk_log_size":4732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.640308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349437952,"num_objects":5332,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5332,"num_write_kb":341248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":476,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.557205+0000","last_change":"2024-03-26T05:10:33.604297+0000","last_active":"2024-03-26T05:11:59.557205+0000","last_peered":"2024-03-26T05:11:59.557205+0000","last_clean":"2024-03-26T05:11:59.557205+0000","last_became_active":"2024-03-26T05:08:55.556543+0000","last_became_peered":"2024-03-26T05:08:55.556543+0000","last_unstale":"2024-03-26T05:11:59.557205+0000","last_undegraded":"2024-03-26T05:11:59.557205+0000","last_fullsized":"2024-03-26T05:11:59.557205+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.604198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.604198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671512+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":488,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.548653+0000","last_change":"2024-03-26T05:10:38.869839+0000","last_active":"2024-03-26T05:11:59.548653+0000","last_peered":"2024-03-26T05:11:59.548653+0000","last_clean":"2024-03-26T05:11:59.548653+0000","last_became_active":"2024-03-26T05:00:22.775106+0000","last_became_peered":"2024-03-26T05:00:22.775106+0000","last_unstale":"2024-03-26T05:11:59.548653+0000","last_undegraded":"2024-03-26T05:11:59.548653+0000","last_fullsized":"2024-03-26T05:11:59.548653+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:38.869752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:38.869752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.797842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"254'37","reported_seq":543,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.540639+0000","last_change":"2024-03-26T05:09:53.374250+0000","last_active":"2024-03-26T05:11:59.540639+0000","last_peered":"2024-03-26T05:11:59.540639+0000","last_clean":"2024-03-26T05:11:59.540639+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:11:59.540639+0000","last_undegraded":"2024-03-26T05:11:59.540639+0000","last_fullsized":"2024-03-26T05:11:59.540639+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'31","last_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_clean_scrub_stamp":"2024-03-26T05:09:53.374102+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365092+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":482,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.553782+0000","last_change":"2024-03-26T05:10:37.078616+0000","last_active":"2024-03-26T05:11:59.553782+0000","last_peered":"2024-03-26T05:11:59.553782+0000","last_clean":"2024-03-26T05:11:59.553782+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:11:59.553782+0000","last_undegraded":"2024-03-26T05:11:59.553782+0000","last_fullsized":"2024-03-26T05:11:59.553782+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:37.078539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.432732+0000","last_clean_scrub_stamp":"2024-03-26T05:10:37.078539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.617367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"286'5238","reported_seq":5290,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.937409+0000","last_change":"2024-03-26T05:10:55.648932+0000","last_active":"2024-03-26T05:11:59.937409+0000","last_peered":"2024-03-26T05:11:59.937409+0000","last_clean":"2024-03-26T05:11:59.937409+0000","last_became_active":"2024-03-26T05:10:55.648385+0000","last_became_peered":"2024-03-26T05:10:55.648385+0000","last_unstale":"2024-03-26T05:11:59.937409+0000","last_undegraded":"2024-03-26T05:11:59.937409+0000","last_fullsized":"2024-03-26T05:11:59.937409+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:11.784317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343277568,"num_objects":5238,"num_object_clones":0,"num_object_copies":15714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5238,"num_write_kb":335232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":476,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.547458+0000","last_change":"2024-03-26T05:10:27.852306+0000","last_active":"2024-03-26T05:11:59.547458+0000","last_peered":"2024-03-26T05:11:59.547458+0000","last_clean":"2024-03-26T05:11:59.547458+0000","last_became_active":"2024-03-26T05:00:28.981486+0000","last_became_peered":"2024-03-26T05:00:28.981486+0000","last_unstale":"2024-03-26T05:11:59.547458+0000","last_undegraded":"2024-03-26T05:11:59.547458+0000","last_fullsized":"2024-03-26T05:11:59.547458+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:27.852200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_clean_scrub_stamp":"2024-03-26T05:10:27.852200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":502,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.554165+0000","last_change":"2024-03-26T05:11:26.081745+0000","last_active":"2024-03-26T05:11:59.554165+0000","last_peered":"2024-03-26T05:11:59.554165+0000","last_clean":"2024-03-26T05:11:59.554165+0000","last_became_active":"2024-03-26T05:08:00.000009+0000","last_became_peered":"2024-03-26T05:08:00.000009+0000","last_unstale":"2024-03-26T05:11:59.554165+0000","last_undegraded":"2024-03-26T05:11:59.554165+0000","last_fullsized":"2024-03-26T05:11:59.554165+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:26.081600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:26.081600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:28.011977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":500,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.549524+0000","last_change":"2024-03-26T05:11:27.963607+0000","last_active":"2024-03-26T05:11:59.549524+0000","last_peered":"2024-03-26T05:11:59.549524+0000","last_clean":"2024-03-26T05:11:59.549524+0000","last_became_active":"2024-03-26T05:07:59.945688+0000","last_became_peered":"2024-03-26T05:07:59.945688+0000","last_unstale":"2024-03-26T05:11:59.549524+0000","last_undegraded":"2024-03-26T05:11:59.549524+0000","last_fullsized":"2024-03-26T05:11:59.549524+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:27.963495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:27.963495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:35.174100+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"254'41","reported_seq":560,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.553020+0000","last_change":"2024-03-26T05:11:24.659018+0000","last_active":"2024-03-26T05:11:59.553020+0000","last_peered":"2024-03-26T05:11:59.553020+0000","last_clean":"2024-03-26T05:11:59.553020+0000","last_became_active":"2024-03-26T05:00:25.648597+0000","last_became_peered":"2024-03-26T05:00:25.648597+0000","last_unstale":"2024-03-26T05:11:59.553020+0000","last_undegraded":"2024-03-26T05:11:59.553020+0000","last_fullsized":"2024-03-26T05:11:59.553020+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'41","last_scrub_stamp":"2024-03-26T05:11:24.658818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:11:24.658818+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:38.713622+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"286'5044","reported_seq":5096,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.938375+0000","last_change":"2024-03-26T05:10:55.650295+0000","last_active":"2024-03-26T05:11:59.938375+0000","last_peered":"2024-03-26T05:11:59.938375+0000","last_clean":"2024-03-26T05:11:59.938375+0000","last_became_active":"2024-03-26T05:10:55.649911+0000","last_became_peered":"2024-03-26T05:10:55.649911+0000","last_unstale":"2024-03-26T05:11:59.938375+0000","last_undegraded":"2024-03-26T05:11:59.938375+0000","last_fullsized":"2024-03-26T05:11:59.938375+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:10.497962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330563584,"num_objects":5044,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":322816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":480,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.555381+0000","last_change":"2024-03-26T05:11:30.164221+0000","last_active":"2024-03-26T05:11:59.555381+0000","last_peered":"2024-03-26T05:11:59.555381+0000","last_clean":"2024-03-26T05:11:59.555381+0000","last_became_active":"2024-03-26T05:00:28.981090+0000","last_became_peered":"2024-03-26T05:00:28.981090+0000","last_unstale":"2024-03-26T05:11:59.555381+0000","last_undegraded":"2024-03-26T05:11:59.555381+0000","last_fullsized":"2024-03-26T05:11:59.555381+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:30.164093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:30.164093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:39.872048+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":482,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.557670+0000","last_change":"2024-03-26T05:09:58.378595+0000","last_active":"2024-03-26T05:11:59.557670+0000","last_peered":"2024-03-26T05:11:59.557670+0000","last_clean":"2024-03-26T05:11:59.557670+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:11:59.557670+0000","last_undegraded":"2024-03-26T05:11:59.557670+0000","last_fullsized":"2024-03-26T05:11:59.557670+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:58.378228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:58.378228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.083266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":171,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.589318+0000","last_change":"2024-03-26T05:11:59.589379+0000","last_active":"2024-03-26T05:11:59.589318+0000","last_peered":"2024-03-26T05:11:59.589318+0000","last_clean":"2024-03-26T05:11:59.589318+0000","last_became_active":"2024-03-26T05:07:59.959978+0000","last_became_peered":"2024-03-26T05:07:59.959978+0000","last_unstale":"2024-03-26T05:11:59.589318+0000","last_undegraded":"2024-03-26T05:11:59.589318+0000","last_fullsized":"2024-03-26T05:11:59.589318+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:59.589216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:59.589216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:13:03.188597+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"254'21","reported_seq":525,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.558179+0000","last_change":"2024-03-26T05:11:22.075860+0000","last_active":"2024-03-26T05:11:59.558179+0000","last_peered":"2024-03-26T05:11:59.558179+0000","last_clean":"2024-03-26T05:11:59.558179+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:11:59.558179+0000","last_undegraded":"2024-03-26T05:11:59.558179+0000","last_fullsized":"2024-03-26T05:11:59.558179+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:11:22.075687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:11:22.075687+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:25.455410+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"286'5299","reported_seq":5352,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:12:00.838733+0000","last_change":"2024-03-26T05:10:55.650442+0000","last_active":"2024-03-26T05:12:00.838733+0000","last_peered":"2024-03-26T05:12:00.838733+0000","last_clean":"2024-03-26T05:12:00.838733+0000","last_became_active":"2024-03-26T05:10:55.650103+0000","last_became_peered":"2024-03-26T05:10:55.650103+0000","last_unstale":"2024-03-26T05:12:00.838733+0000","last_undegraded":"2024-03-26T05:12:00.838733+0000","last_fullsized":"2024-03-26T05:12:00.838733+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":5299,"log_dups_size":0,"ondisk_log_size":5299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:18.480914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347275264,"num_objects":5299,"num_object_clones":0,"num_object_copies":15897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5299,"num_write_kb":339136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":470,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.556892+0000","last_change":"2024-03-26T05:11:31.518777+0000","last_active":"2024-03-26T05:11:59.556892+0000","last_peered":"2024-03-26T05:11:59.556892+0000","last_clean":"2024-03-26T05:11:59.556892+0000","last_became_active":"2024-03-26T05:00:28.983652+0000","last_became_peered":"2024-03-26T05:00:28.983652+0000","last_unstale":"2024-03-26T05:11:59.556892+0000","last_undegraded":"2024-03-26T05:11:59.556892+0000","last_fullsized":"2024-03-26T05:11:59.556892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:31.518660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:31.518660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:49.293805+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":167,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.552390+0000","last_change":"2024-03-26T05:11:20.563817+0000","last_active":"2024-03-26T05:11:59.552390+0000","last_peered":"2024-03-26T05:11:59.552390+0000","last_clean":"2024-03-26T05:11:59.552390+0000","last_became_active":"2024-03-26T05:07:59.998511+0000","last_became_peered":"2024-03-26T05:07:59.998511+0000","last_unstale":"2024-03-26T05:11:59.552390+0000","last_undegraded":"2024-03-26T05:11:59.552390+0000","last_fullsized":"2024-03-26T05:11:59.552390+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:11:20.563683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:11:20.563683+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:29.743222+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":476,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.541398+0000","last_change":"2024-03-26T05:09:31.499628+0000","last_active":"2024-03-26T05:11:59.541398+0000","last_peered":"2024-03-26T05:11:59.541398+0000","last_clean":"2024-03-26T05:11:59.541398+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:11:59.541398+0000","last_undegraded":"2024-03-26T05:11:59.541398+0000","last_fullsized":"2024-03-26T05:11:59.541398+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:31.499513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:31.499513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365026+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"254'46","reported_seq":704,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.560215+0000","last_change":"2024-03-26T05:10:48.344286+0000","last_active":"2024-03-26T05:11:59.560215+0000","last_peered":"2024-03-26T05:11:59.560215+0000","last_clean":"2024-03-26T05:11:59.560215+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:11:59.560215+0000","last_undegraded":"2024-03-26T05:11:59.560215+0000","last_fullsized":"2024-03-26T05:11:59.560215+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'46","last_scrub_stamp":"2024-03-26T05:10:48.344180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:48.344180+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":476,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.555638+0000","last_change":"2024-03-26T05:09:41.858043+0000","last_active":"2024-03-26T05:11:59.555638+0000","last_peered":"2024-03-26T05:11:59.555638+0000","last_clean":"2024-03-26T05:11:59.555638+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:11:59.555638+0000","last_undegraded":"2024-03-26T05:11:59.555638+0000","last_fullsized":"2024-03-26T05:11:59.555638+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:41.857888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:41.857888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"286'5169","reported_seq":5221,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.923288+0000","last_change":"2024-03-26T05:10:55.651127+0000","last_active":"2024-03-26T05:11:59.923288+0000","last_peered":"2024-03-26T05:11:59.923288+0000","last_clean":"2024-03-26T05:11:59.923288+0000","last_became_active":"2024-03-26T05:10:55.650612+0000","last_became_peered":"2024-03-26T05:10:55.650612+0000","last_unstale":"2024-03-26T05:11:59.923288+0000","last_undegraded":"2024-03-26T05:11:59.923288+0000","last_fullsized":"2024-03-26T05:11:59.923288+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":5169,"log_dups_size":0,"ondisk_log_size":5169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.833946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338755584,"num_objects":5169,"num_object_clones":0,"num_object_copies":15507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5169,"num_write_kb":330816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":484,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.549683+0000","last_change":"2024-03-26T05:11:20.054671+0000","last_active":"2024-03-26T05:11:59.549683+0000","last_peered":"2024-03-26T05:11:59.549683+0000","last_clean":"2024-03-26T05:11:59.549683+0000","last_became_active":"2024-03-26T05:00:29.461906+0000","last_became_peered":"2024-03-26T05:00:29.461906+0000","last_unstale":"2024-03-26T05:11:59.549683+0000","last_undegraded":"2024-03-26T05:11:59.549683+0000","last_fullsized":"2024-03-26T05:11:59.549683+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:20.054579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:20.054579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:20.943908+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":482,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.557725+0000","last_change":"2024-03-26T05:10:39.454264+0000","last_active":"2024-03-26T05:11:59.557725+0000","last_peered":"2024-03-26T05:11:59.557725+0000","last_clean":"2024-03-26T05:11:59.557725+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:11:59.557725+0000","last_undegraded":"2024-03-26T05:11:59.557725+0000","last_fullsized":"2024-03-26T05:11:59.557725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:39.454176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:39.454176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.629895+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":499,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.553496+0000","last_change":"2024-03-26T05:09:55.551716+0000","last_active":"2024-03-26T05:11:59.553496+0000","last_peered":"2024-03-26T05:11:59.553496+0000","last_clean":"2024-03-26T05:11:59.553496+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:11:59.553496+0000","last_undegraded":"2024-03-26T05:11:59.553496+0000","last_fullsized":"2024-03-26T05:11:59.553496+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:55.551595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.551595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.083297+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"254'23","reported_seq":304,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.555266+0000","last_change":"2024-03-26T05:11:24.162170+0000","last_active":"2024-03-26T05:11:59.555266+0000","last_peered":"2024-03-26T05:11:59.555266+0000","last_clean":"2024-03-26T05:11:59.555266+0000","last_became_active":"2024-03-26T05:08:00.001398+0000","last_became_peered":"2024-03-26T05:08:00.001398+0000","last_unstale":"2024-03-26T05:11:59.555266+0000","last_undegraded":"2024-03-26T05:11:59.555266+0000","last_fullsized":"2024-03-26T05:11:59.555266+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'23","last_scrub_stamp":"2024-03-26T05:11:24.161815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:11:24.161815+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:30.642740+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":502,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.555111+0000","last_change":"2024-03-26T05:10:43.076916+0000","last_active":"2024-03-26T05:11:59.555111+0000","last_peered":"2024-03-26T05:11:59.555111+0000","last_clean":"2024-03-26T05:11:59.555111+0000","last_became_active":"2024-03-26T05:07:59.955456+0000","last_became_peered":"2024-03-26T05:07:59.955456+0000","last_unstale":"2024-03-26T05:11:59.555111+0000","last_undegraded":"2024-03-26T05:11:59.555111+0000","last_fullsized":"2024-03-26T05:11:59.555111+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:43.076802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.076802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.083323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"286'5124","reported_seq":5177,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:12:00.674690+0000","last_change":"2024-03-26T05:10:55.651844+0000","last_active":"2024-03-26T05:12:00.674690+0000","last_peered":"2024-03-26T05:12:00.674690+0000","last_clean":"2024-03-26T05:12:00.674690+0000","last_became_active":"2024-03-26T05:10:55.651432+0000","last_became_peered":"2024-03-26T05:10:55.651432+0000","last_unstale":"2024-03-26T05:12:00.674690+0000","last_undegraded":"2024-03-26T05:12:00.674690+0000","last_fullsized":"2024-03-26T05:12:00.674690+0000","mapping_epoch":265,"log_start":"271'300","ondisk_log_start":"271'300","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":4824,"log_dups_size":0,"ondisk_log_size":4824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335806464,"num_objects":5124,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5124,"num_write_kb":327936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":171,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.552245+0000","last_change":"2024-03-26T05:11:31.654856+0000","last_active":"2024-03-26T05:11:59.552245+0000","last_peered":"2024-03-26T05:11:59.552245+0000","last_clean":"2024-03-26T05:11:59.552245+0000","last_became_active":"2024-03-26T05:07:59.959853+0000","last_became_peered":"2024-03-26T05:07:59.959853+0000","last_unstale":"2024-03-26T05:11:59.552245+0000","last_undegraded":"2024-03-26T05:11:59.552245+0000","last_fullsized":"2024-03-26T05:11:59.552245+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:31.654746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_clean_scrub_stamp":"2024-03-26T05:11:31.654746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:48.986484+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":743,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.547763+0000","last_change":"2024-03-26T05:10:39.866608+0000","last_active":"2024-03-26T05:11:59.547763+0000","last_peered":"2024-03-26T05:11:59.547763+0000","last_clean":"2024-03-26T05:11:59.547763+0000","last_became_active":"2024-03-26T05:00:13.519427+0000","last_became_peered":"2024-03-26T05:00:13.519427+0000","last_unstale":"2024-03-26T05:11:59.547763+0000","last_undegraded":"2024-03-26T05:11:59.547763+0000","last_fullsized":"2024-03-26T05:11:59.547763+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:10:39.866516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:10:39.866516+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.907367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":494,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.562879+0000","last_change":"2024-03-26T05:09:39.605190+0000","last_active":"2024-03-26T05:11:59.562879+0000","last_peered":"2024-03-26T05:11:59.562879+0000","last_clean":"2024-03-26T05:11:59.562879+0000","last_became_active":"2024-03-26T05:07:59.960061+0000","last_became_peered":"2024-03-26T05:07:59.960061+0000","last_unstale":"2024-03-26T05:11:59.562879+0000","last_undegraded":"2024-03-26T05:11:59.562879+0000","last_fullsized":"2024-03-26T05:11:59.562879+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:39.605053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:39.605053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"254'36","reported_seq":689,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.553555+0000","last_change":"2024-03-26T05:11:20.689790+0000","last_active":"2024-03-26T05:11:59.553555+0000","last_peered":"2024-03-26T05:11:59.553555+0000","last_clean":"2024-03-26T05:11:59.553555+0000","last_became_active":"2024-03-26T05:00:25.648174+0000","last_became_peered":"2024-03-26T05:00:25.648174+0000","last_unstale":"2024-03-26T05:11:59.553555+0000","last_undegraded":"2024-03-26T05:11:59.553555+0000","last_fullsized":"2024-03-26T05:11:59.553555+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'36","last_scrub_stamp":"2024-03-26T05:11:20.689640+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:11:20.689640+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:47.378697+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":500,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.549183+0000","last_change":"2024-03-26T05:10:36.850667+0000","last_active":"2024-03-26T05:11:59.549183+0000","last_peered":"2024-03-26T05:11:59.549183+0000","last_clean":"2024-03-26T05:11:59.549183+0000","last_became_active":"2024-03-26T05:07:59.956080+0000","last_became_peered":"2024-03-26T05:07:59.956080+0000","last_unstale":"2024-03-26T05:11:59.549183+0000","last_undegraded":"2024-03-26T05:11:59.549183+0000","last_fullsized":"2024-03-26T05:11:59.549183+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:36.850583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:36.850583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.001232+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"286'5374","reported_seq":5426,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:12:00.854542+0000","last_change":"2024-03-26T05:10:55.754958+0000","last_active":"2024-03-26T05:12:00.854542+0000","last_peered":"2024-03-26T05:12:00.854542+0000","last_clean":"2024-03-26T05:12:00.854542+0000","last_became_active":"2024-03-26T05:10:55.754590+0000","last_became_peered":"2024-03-26T05:10:55.754590+0000","last_unstale":"2024-03-26T05:12:00.854542+0000","last_undegraded":"2024-03-26T05:12:00.854542+0000","last_fullsized":"2024-03-26T05:12:00.854542+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":5374,"log_dups_size":0,"ondisk_log_size":5374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:13.783901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352190464,"num_objects":5374,"num_object_clones":0,"num_object_copies":16122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5374,"num_write_kb":343936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":490,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.558435+0000","last_change":"2024-03-26T05:11:18.433220+0000","last_active":"2024-03-26T05:11:59.558435+0000","last_peered":"2024-03-26T05:11:59.558435+0000","last_clean":"2024-03-26T05:11:59.558435+0000","last_became_active":"2024-03-26T05:08:00.001657+0000","last_became_peered":"2024-03-26T05:08:00.001657+0000","last_unstale":"2024-03-26T05:11:59.558435+0000","last_undegraded":"2024-03-26T05:11:59.558435+0000","last_fullsized":"2024-03-26T05:11:59.558435+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:18.429726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:18.429726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:27.846199+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":379,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.540547+0000","last_change":"2024-03-26T05:09:48.473285+0000","last_active":"2024-03-26T05:11:59.540547+0000","last_peered":"2024-03-26T05:11:59.540547+0000","last_clean":"2024-03-26T05:11:59.540547+0000","last_became_active":"2024-03-26T05:07:59.992489+0000","last_became_peered":"2024-03-26T05:07:59.992489+0000","last_unstale":"2024-03-26T05:11:59.540547+0000","last_undegraded":"2024-03-26T05:11:59.540547+0000","last_fullsized":"2024-03-26T05:11:59.540547+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_clean_scrub_stamp":"2024-03-26T05:09:48.473160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365042+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":1191,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.547564+0000","last_change":"2024-03-26T05:09:55.485007+0000","last_active":"2024-03-26T05:11:59.547564+0000","last_peered":"2024-03-26T05:11:59.547564+0000","last_clean":"2024-03-26T05:11:59.547564+0000","last_became_active":"2024-03-26T05:00:21.764668+0000","last_became_peered":"2024-03-26T05:00:21.764668+0000","last_unstale":"2024-03-26T05:11:59.547564+0000","last_undegraded":"2024-03-26T05:11:59.547564+0000","last_fullsized":"2024-03-26T05:11:59.547564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:09:55.484790+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.484790+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":690,"num_read_kb":685,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"254'30","reported_seq":570,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.555457+0000","last_change":"2024-03-26T05:11:18.125189+0000","last_active":"2024-03-26T05:11:59.555457+0000","last_peered":"2024-03-26T05:11:59.555457+0000","last_clean":"2024-03-26T05:11:59.555457+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:11:59.555457+0000","last_undegraded":"2024-03-26T05:11:59.555457+0000","last_fullsized":"2024-03-26T05:11:59.555457+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'30","last_scrub_stamp":"2024-03-26T05:11:18.125091+0000","last_deep_scrub":"187'18","last_deep_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_clean_scrub_stamp":"2024-03-26T05:11:18.125091+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:25.474415+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":163,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.552169+0000","last_change":"2024-03-26T05:10:44.404350+0000","last_active":"2024-03-26T05:11:59.552169+0000","last_peered":"2024-03-26T05:11:59.552169+0000","last_clean":"2024-03-26T05:11:59.552169+0000","last_became_active":"2024-03-26T05:07:59.999990+0000","last_became_peered":"2024-03-26T05:07:59.999990+0000","last_unstale":"2024-03-26T05:11:59.552169+0000","last_undegraded":"2024-03-26T05:11:59.552169+0000","last_fullsized":"2024-03-26T05:11:59.552169+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:44.404261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:44.404261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.651981+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"286'5133","reported_seq":5185,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.928886+0000","last_change":"2024-03-26T05:10:55.652321+0000","last_active":"2024-03-26T05:11:59.928886+0000","last_peered":"2024-03-26T05:11:59.928886+0000","last_clean":"2024-03-26T05:11:59.928886+0000","last_became_active":"2024-03-26T05:10:55.651931+0000","last_became_peered":"2024-03-26T05:10:55.651931+0000","last_unstale":"2024-03-26T05:11:59.928886+0000","last_undegraded":"2024-03-26T05:11:59.928886+0000","last_fullsized":"2024-03-26T05:11:59.928886+0000","mapping_epoch":265,"log_start":"271'200","ondisk_log_start":"271'200","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:04.080657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336396288,"num_objects":5133,"num_object_clones":0,"num_object_copies":15399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5133,"num_write_kb":328512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":470,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.556266+0000","last_change":"2024-03-26T05:10:34.408142+0000","last_active":"2024-03-26T05:11:59.556266+0000","last_peered":"2024-03-26T05:11:59.556266+0000","last_clean":"2024-03-26T05:11:59.556266+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:11:59.556266+0000","last_undegraded":"2024-03-26T05:11:59.556266+0000","last_fullsized":"2024-03-26T05:11:59.556266+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:34.408045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:10:34.408045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.629719+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":502,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.560115+0000","last_change":"2024-03-26T05:11:29.515091+0000","last_active":"2024-03-26T05:11:59.560115+0000","last_peered":"2024-03-26T05:11:59.560115+0000","last_clean":"2024-03-26T05:11:59.560115+0000","last_became_active":"2024-03-26T05:08:00.003099+0000","last_became_peered":"2024-03-26T05:08:00.003099+0000","last_unstale":"2024-03-26T05:11:59.560115+0000","last_undegraded":"2024-03-26T05:11:59.560115+0000","last_fullsized":"2024-03-26T05:11:59.560115+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:29.514915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:11:29.514915+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.514915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:40.247013+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"254'44","reported_seq":620,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.557948+0000","last_change":"2024-03-26T05:11:29.126279+0000","last_active":"2024-03-26T05:11:59.557948+0000","last_peered":"2024-03-26T05:11:59.557948+0000","last_clean":"2024-03-26T05:11:59.557948+0000","last_became_active":"2024-03-26T05:07:59.994226+0000","last_became_peered":"2024-03-26T05:07:59.994226+0000","last_unstale":"2024-03-26T05:11:59.557948+0000","last_undegraded":"2024-03-26T05:11:59.557948+0000","last_fullsized":"2024-03-26T05:11:59.557948+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'44","last_scrub_stamp":"2024-03-26T05:11:29.126183+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.126183+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:49.423429+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":501,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.553114+0000","last_change":"2024-03-26T05:10:42.594397+0000","last_active":"2024-03-26T05:11:59.553114+0000","last_peered":"2024-03-26T05:11:59.553114+0000","last_clean":"2024-03-26T05:11:59.553114+0000","last_became_active":"2024-03-26T05:07:59.991484+0000","last_became_peered":"2024-03-26T05:07:59.991484+0000","last_unstale":"2024-03-26T05:11:59.553114+0000","last_undegraded":"2024-03-26T05:11:59.553114+0000","last_fullsized":"2024-03-26T05:11:59.553114+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:42.594296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:10:42.594296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.820010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"286'5276","reported_seq":5328,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.932181+0000","last_change":"2024-03-26T05:10:55.648891+0000","last_active":"2024-03-26T05:11:59.932181+0000","last_peered":"2024-03-26T05:11:59.932181+0000","last_clean":"2024-03-26T05:11:59.932181+0000","last_became_active":"2024-03-26T05:10:55.648533+0000","last_became_peered":"2024-03-26T05:10:55.648533+0000","last_unstale":"2024-03-26T05:11:59.932181+0000","last_undegraded":"2024-03-26T05:11:59.932181+0000","last_fullsized":"2024-03-26T05:11:59.932181+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":5276,"log_dups_size":0,"ondisk_log_size":5276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345767936,"num_objects":5276,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":337664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":480,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.553598+0000","last_change":"2024-03-26T05:11:21.090707+0000","last_active":"2024-03-26T05:11:59.553598+0000","last_peered":"2024-03-26T05:11:59.553598+0000","last_clean":"2024-03-26T05:11:59.553598+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:11:59.553598+0000","last_undegraded":"2024-03-26T05:11:59.553598+0000","last_fullsized":"2024-03-26T05:11:59.553598+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:21.090576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:21.090576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:31.977600+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":163,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.554816+0000","last_change":"2024-03-26T05:09:42.163715+0000","last_active":"2024-03-26T05:11:59.554816+0000","last_peered":"2024-03-26T05:11:59.554816+0000","last_clean":"2024-03-26T05:11:59.554816+0000","last_became_active":"2024-03-26T05:07:59.960768+0000","last_became_peered":"2024-03-26T05:07:59.960768+0000","last_unstale":"2024-03-26T05:11:59.554816+0000","last_undegraded":"2024-03-26T05:11:59.554816+0000","last_fullsized":"2024-03-26T05:11:59.554816+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:42.163609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:42.163609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"254'26","reported_seq":539,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.557931+0000","last_change":"2024-03-26T05:11:28.134373+0000","last_active":"2024-03-26T05:11:59.557931+0000","last_peered":"2024-03-26T05:11:59.557931+0000","last_clean":"2024-03-26T05:11:59.557931+0000","last_became_active":"2024-03-26T05:07:59.960351+0000","last_became_peered":"2024-03-26T05:07:59.960351+0000","last_unstale":"2024-03-26T05:11:59.557931+0000","last_undegraded":"2024-03-26T05:11:59.557931+0000","last_fullsized":"2024-03-26T05:11:59.557931+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:11:28.134056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:11:28.134056+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:33.025191+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"216'11","reported_seq":654,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.558275+0000","last_change":"2024-03-26T05:10:34.204907+0000","last_active":"2024-03-26T05:11:59.558275+0000","last_peered":"2024-03-26T05:11:59.558275+0000","last_clean":"2024-03-26T05:11:59.558275+0000","last_became_active":"2024-03-26T05:07:59.994682+0000","last_became_peered":"2024-03-26T05:07:59.994682+0000","last_unstale":"2024-03-26T05:11:59.558275+0000","last_undegraded":"2024-03-26T05:11:59.558275+0000","last_fullsized":"2024-03-26T05:11:59.558275+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:10:34.204798+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:10:34.204798+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.783859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"286'5178","reported_seq":5231,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:12:00.731989+0000","last_change":"2024-03-26T05:10:55.648350+0000","last_active":"2024-03-26T05:12:00.731989+0000","last_peered":"2024-03-26T05:12:00.731989+0000","last_clean":"2024-03-26T05:12:00.731989+0000","last_became_active":"2024-03-26T05:10:55.647949+0000","last_became_peered":"2024-03-26T05:10:55.647949+0000","last_unstale":"2024-03-26T05:12:00.731989+0000","last_undegraded":"2024-03-26T05:12:00.731989+0000","last_fullsized":"2024-03-26T05:12:00.731989+0000","mapping_epoch":265,"log_start":"271'400","ondisk_log_start":"271'400","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:04.820875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339345408,"num_objects":5178,"num_object_clones":0,"num_object_copies":15534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5178,"num_write_kb":331392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":476,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.550089+0000","last_change":"2024-03-26T05:10:34.839543+0000","last_active":"2024-03-26T05:11:59.550089+0000","last_peered":"2024-03-26T05:11:59.550089+0000","last_clean":"2024-03-26T05:11:59.550089+0000","last_became_active":"2024-03-26T05:00:29.461100+0000","last_became_peered":"2024-03-26T05:00:29.461100+0000","last_unstale":"2024-03-26T05:11:59.550089+0000","last_undegraded":"2024-03-26T05:11:59.550089+0000","last_fullsized":"2024-03-26T05:11:59.550089+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:34.837732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:34.837732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.929082+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":501,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.555559+0000","last_change":"2024-03-26T05:11:23.098540+0000","last_active":"2024-03-26T05:11:59.555559+0000","last_peered":"2024-03-26T05:11:59.555559+0000","last_clean":"2024-03-26T05:11:59.555559+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:11:59.555559+0000","last_undegraded":"2024-03-26T05:11:59.555559+0000","last_fullsized":"2024-03-26T05:11:59.555559+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:11:23.098414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:23.098414+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:45.120158+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"254'34","reported_seq":582,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.540394+0000","last_change":"2024-03-26T05:09:47.484748+0000","last_active":"2024-03-26T05:11:59.540394+0000","last_peered":"2024-03-26T05:11:59.540394+0000","last_clean":"2024-03-26T05:11:59.540394+0000","last_became_active":"2024-03-26T05:07:59.942510+0000","last_became_peered":"2024-03-26T05:07:59.942510+0000","last_unstale":"2024-03-26T05:11:59.540394+0000","last_undegraded":"2024-03-26T05:11:59.540394+0000","last_fullsized":"2024-03-26T05:11:59.540394+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'20","last_scrub_stamp":"2024-03-26T05:09:47.484585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:47.484585+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":622,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.550463+0000","last_change":"2024-03-26T05:11:26.393095+0000","last_active":"2024-03-26T05:11:59.550463+0000","last_peered":"2024-03-26T05:11:59.550463+0000","last_clean":"2024-03-26T05:11:59.550463+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:11:59.550463+0000","last_undegraded":"2024-03-26T05:11:59.550463+0000","last_fullsized":"2024-03-26T05:11:59.550463+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:11:26.392974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:26.392974+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:32.171757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"286'5181","reported_seq":5233,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.932795+0000","last_change":"2024-03-26T05:10:55.652074+0000","last_active":"2024-03-26T05:11:59.932795+0000","last_peered":"2024-03-26T05:11:59.932795+0000","last_clean":"2024-03-26T05:11:59.932795+0000","last_became_active":"2024-03-26T05:10:55.651719+0000","last_became_peered":"2024-03-26T05:10:55.651719+0000","last_unstale":"2024-03-26T05:11:59.932795+0000","last_undegraded":"2024-03-26T05:11:59.932795+0000","last_fullsized":"2024-03-26T05:11:59.932795+0000","mapping_epoch":265,"log_start":"271'200","ondisk_log_start":"271'200","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:18.177643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339542016,"num_objects":5181,"num_object_clones":0,"num_object_copies":15543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5181,"num_write_kb":331584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":480,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.541298+0000","last_change":"2024-03-26T05:11:26.503047+0000","last_active":"2024-03-26T05:11:59.541298+0000","last_peered":"2024-03-26T05:11:59.541298+0000","last_clean":"2024-03-26T05:11:59.541298+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:11:59.541298+0000","last_undegraded":"2024-03-26T05:11:59.541298+0000","last_fullsized":"2024-03-26T05:11:59.541298+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:26.502923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:26.502923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:34.287702+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":474,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.559035+0000","last_change":"2024-03-26T05:09:38.690272+0000","last_active":"2024-03-26T05:11:59.559035+0000","last_peered":"2024-03-26T05:11:59.559035+0000","last_clean":"2024-03-26T05:11:59.559035+0000","last_became_active":"2024-03-26T05:00:22.778716+0000","last_became_peered":"2024-03-26T05:00:22.778716+0000","last_unstale":"2024-03-26T05:11:59.559035+0000","last_undegraded":"2024-03-26T05:11:59.559035+0000","last_fullsized":"2024-03-26T05:11:59.559035+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:38.690131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:38.690131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671388+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"254'52","reported_seq":562,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.550122+0000","last_change":"2024-03-26T05:09:34.644416+0000","last_active":"2024-03-26T05:11:59.550122+0000","last_peered":"2024-03-26T05:11:59.550122+0000","last_clean":"2024-03-26T05:11:59.550122+0000","last_became_active":"2024-03-26T05:00:25.650767+0000","last_became_peered":"2024-03-26T05:00:25.650767+0000","last_unstale":"2024-03-26T05:11:59.550122+0000","last_undegraded":"2024-03-26T05:11:59.550122+0000","last_fullsized":"2024-03-26T05:11:59.550122+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'40","last_scrub_stamp":"2024-03-26T05:09:34.643076+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:09:34.643076+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671285+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":476,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.540161+0000","last_change":"2024-03-26T05:10:33.411222+0000","last_active":"2024-03-26T05:11:59.540161+0000","last_peered":"2024-03-26T05:11:59.540161+0000","last_clean":"2024-03-26T05:11:59.540161+0000","last_became_active":"2024-03-26T05:00:26.958601+0000","last_became_peered":"2024-03-26T05:00:26.958601+0000","last_unstale":"2024-03-26T05:11:59.540161+0000","last_undegraded":"2024-03-26T05:11:59.540161+0000","last_fullsized":"2024-03-26T05:11:59.540161+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.411139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.411139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"286'5148","reported_seq":5201,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:12:00.709929+0000","last_change":"2024-03-26T05:10:55.652478+0000","last_active":"2024-03-26T05:12:00.709929+0000","last_peered":"2024-03-26T05:12:00.709929+0000","last_clean":"2024-03-26T05:12:00.709929+0000","last_became_active":"2024-03-26T05:10:55.652104+0000","last_became_peered":"2024-03-26T05:10:55.652104+0000","last_unstale":"2024-03-26T05:12:00.709929+0000","last_undegraded":"2024-03-26T05:12:00.709929+0000","last_fullsized":"2024-03-26T05:12:00.709929+0000","mapping_epoch":265,"log_start":"271'300","ondisk_log_start":"271'300","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":4848,"log_dups_size":0,"ondisk_log_size":4848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337379328,"num_objects":5148,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":329472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":163,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.554649+0000","last_change":"2024-03-26T05:11:22.658281+0000","last_active":"2024-03-26T05:11:59.554649+0000","last_peered":"2024-03-26T05:11:59.554649+0000","last_clean":"2024-03-26T05:11:59.554649+0000","last_became_active":"2024-03-26T05:08:00.005676+0000","last_became_peered":"2024-03-26T05:08:00.005676+0000","last_unstale":"2024-03-26T05:11:59.554649+0000","last_undegraded":"2024-03-26T05:11:59.554649+0000","last_fullsized":"2024-03-26T05:11:59.554649+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:22.658167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_clean_scrub_stamp":"2024-03-26T05:11:22.658167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:36.946170+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"216'10","reported_seq":645,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.542577+0000","last_change":"2024-03-26T05:09:36.472026+0000","last_active":"2024-03-26T05:11:59.542577+0000","last_peered":"2024-03-26T05:11:59.542577+0000","last_clean":"2024-03-26T05:11:59.542577+0000","last_became_active":"2024-03-26T05:07:59.997429+0000","last_became_peered":"2024-03-26T05:07:59.997429+0000","last_unstale":"2024-03-26T05:11:59.542577+0000","last_undegraded":"2024-03-26T05:11:59.542577+0000","last_fullsized":"2024-03-26T05:11:59.542577+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:09:36.471890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:09:36.471890+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365009+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":482,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.549961+0000","last_change":"2024-03-26T05:10:40.541468+0000","last_active":"2024-03-26T05:11:59.549961+0000","last_peered":"2024-03-26T05:11:59.549961+0000","last_clean":"2024-03-26T05:11:59.549961+0000","last_became_active":"2024-03-26T05:00:22.774709+0000","last_became_peered":"2024-03-26T05:00:22.774709+0000","last_unstale":"2024-03-26T05:11:59.549961+0000","last_undegraded":"2024-03-26T05:11:59.549961+0000","last_fullsized":"2024-03-26T05:11:59.549961+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:40.541340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.541340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.162836+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"254'27","reported_seq":529,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.564582+0000","last_change":"2024-03-26T05:10:43.423183+0000","last_active":"2024-03-26T05:11:59.564582+0000","last_peered":"2024-03-26T05:11:59.564582+0000","last_clean":"2024-03-26T05:11:59.564582+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:11:59.564582+0000","last_undegraded":"2024-03-26T05:11:59.564582+0000","last_fullsized":"2024-03-26T05:11:59.564582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:10:43.423040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:43.423040+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:00.629967+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"286'5215","reported_seq":5267,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.939226+0000","last_change":"2024-03-26T05:10:55.651677+0000","last_active":"2024-03-26T05:11:59.939226+0000","last_peered":"2024-03-26T05:11:59.939226+0000","last_clean":"2024-03-26T05:11:59.939226+0000","last_became_active":"2024-03-26T05:10:55.651348+0000","last_became_peered":"2024-03-26T05:10:55.651348+0000","last_unstale":"2024-03-26T05:11:59.939226+0000","last_undegraded":"2024-03-26T05:11:59.939226+0000","last_fullsized":"2024-03-26T05:11:59.939226+0000","mapping_epoch":265,"log_start":"271'200","ondisk_log_start":"271'200","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:03.135363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341770240,"num_objects":5215,"num_object_clones":0,"num_object_copies":15645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5215,"num_write_kb":333760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":482,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.558089+0000","last_change":"2024-03-26T05:10:44.608828+0000","last_active":"2024-03-26T05:11:59.558089+0000","last_peered":"2024-03-26T05:11:59.558089+0000","last_clean":"2024-03-26T05:11:59.558089+0000","last_became_active":"2024-03-26T05:07:59.957995+0000","last_became_peered":"2024-03-26T05:07:59.957995+0000","last_unstale":"2024-03-26T05:11:59.558089+0000","last_undegraded":"2024-03-26T05:11:59.558089+0000","last_fullsized":"2024-03-26T05:11:59.558089+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:44.608717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:10:44.608717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":163,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.552093+0000","last_change":"2024-03-26T05:10:48.413966+0000","last_active":"2024-03-26T05:11:59.552093+0000","last_peered":"2024-03-26T05:11:59.552093+0000","last_clean":"2024-03-26T05:11:59.552093+0000","last_became_active":"2024-03-26T05:07:59.947747+0000","last_became_peered":"2024-03-26T05:07:59.947747+0000","last_unstale":"2024-03-26T05:11:59.552093+0000","last_undegraded":"2024-03-26T05:11:59.552093+0000","last_fullsized":"2024-03-26T05:11:59.552093+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:48.413868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.446060+0000","last_clean_scrub_stamp":"2024-03-26T05:10:48.413868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:13.855843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":492,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.540254+0000","last_change":"2024-03-26T05:11:29.466096+0000","last_active":"2024-03-26T05:11:59.540254+0000","last_peered":"2024-03-26T05:11:59.540254+0000","last_clean":"2024-03-26T05:11:59.540254+0000","last_became_active":"2024-03-26T05:00:23.346753+0000","last_became_peered":"2024-03-26T05:00:23.346753+0000","last_unstale":"2024-03-26T05:11:59.540254+0000","last_undegraded":"2024-03-26T05:11:59.540254+0000","last_fullsized":"2024-03-26T05:11:59.540254+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:29.465975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.465975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:37.362979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"254'25","reported_seq":531,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.542337+0000","last_change":"2024-03-26T05:10:48.381843+0000","last_active":"2024-03-26T05:11:59.542337+0000","last_peered":"2024-03-26T05:11:59.542337+0000","last_clean":"2024-03-26T05:11:59.542337+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:11:59.542337+0000","last_undegraded":"2024-03-26T05:11:59.542337+0000","last_fullsized":"2024-03-26T05:11:59.542337+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:10:48.381743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:48.381743+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"286'5183","reported_seq":5235,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:12:00.430092+0000","last_change":"2024-03-26T05:10:55.754903+0000","last_active":"2024-03-26T05:12:00.430092+0000","last_peered":"2024-03-26T05:12:00.430092+0000","last_clean":"2024-03-26T05:12:00.430092+0000","last_became_active":"2024-03-26T05:10:55.754440+0000","last_became_peered":"2024-03-26T05:10:55.754440+0000","last_unstale":"2024-03-26T05:12:00.430092+0000","last_undegraded":"2024-03-26T05:12:00.430092+0000","last_fullsized":"2024-03-26T05:12:00.430092+0000","mapping_epoch":265,"log_start":"271'400","ondisk_log_start":"271'400","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339673088,"num_objects":5183,"num_object_clones":0,"num_object_copies":15549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5183,"num_write_kb":331712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":478,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.555068+0000","last_change":"2024-03-26T05:11:20.139860+0000","last_active":"2024-03-26T05:11:59.555068+0000","last_peered":"2024-03-26T05:11:59.555068+0000","last_clean":"2024-03-26T05:11:59.555068+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:11:59.555068+0000","last_undegraded":"2024-03-26T05:11:59.555068+0000","last_fullsized":"2024-03-26T05:11:59.555068+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:20.139734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:20.139734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:50.114639+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":482,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.560452+0000","last_change":"2024-03-26T05:11:25.504226+0000","last_active":"2024-03-26T05:11:59.560452+0000","last_peered":"2024-03-26T05:11:59.560452+0000","last_clean":"2024-03-26T05:11:59.560452+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:11:59.560452+0000","last_undegraded":"2024-03-26T05:11:59.560452+0000","last_fullsized":"2024-03-26T05:11:59.560452+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.503148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.503148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:33.732846+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":511,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.554067+0000","last_change":"2024-03-26T05:11:28.780032+0000","last_active":"2024-03-26T05:11:59.554067+0000","last_peered":"2024-03-26T05:11:59.554067+0000","last_clean":"2024-03-26T05:11:59.554067+0000","last_became_active":"2024-03-26T05:06:06.521150+0000","last_became_peered":"2024-03-26T05:06:06.521150+0000","last_unstale":"2024-03-26T05:11:59.554067+0000","last_undegraded":"2024-03-26T05:11:59.554067+0000","last_fullsized":"2024-03-26T05:11:59.554067+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:28.779949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:11:28.779949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:39.065599+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"254'31","reported_seq":529,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.542214+0000","last_change":"2024-03-26T05:09:58.433286+0000","last_active":"2024-03-26T05:11:59.542214+0000","last_peered":"2024-03-26T05:11:59.542214+0000","last_clean":"2024-03-26T05:11:59.542214+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:11:59.542214+0000","last_undegraded":"2024-03-26T05:11:59.542214+0000","last_fullsized":"2024-03-26T05:11:59.542214+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-26T05:09:58.433038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:58.433038+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365059+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"286'5140","reported_seq":5193,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:12:00.724416+0000","last_change":"2024-03-26T05:10:55.755077+0000","last_active":"2024-03-26T05:12:00.724416+0000","last_peered":"2024-03-26T05:12:00.724416+0000","last_clean":"2024-03-26T05:12:00.724416+0000","last_became_active":"2024-03-26T05:10:55.754445+0000","last_became_peered":"2024-03-26T05:10:55.754445+0000","last_unstale":"2024-03-26T05:12:00.724416+0000","last_undegraded":"2024-03-26T05:12:00.724416+0000","last_fullsized":"2024-03-26T05:12:00.724416+0000","mapping_epoch":265,"log_start":"271'300","ondisk_log_start":"271'300","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:10:54.621176+0000","last_clean_scrub_stamp":"2024-03-26T05:10:54.621176+0000","objects_scrubbed":0,"log_size":4840,"log_dups_size":0,"ondisk_log_size":4840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.644393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336855040,"num_objects":5140,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5140,"num_write_kb":328960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":472,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.542596+0000","last_change":"2024-03-26T05:09:38.506191+0000","last_active":"2024-03-26T05:11:59.542596+0000","last_peered":"2024-03-26T05:11:59.542596+0000","last_clean":"2024-03-26T05:11:59.542596+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:11:59.542596+0000","last_undegraded":"2024-03-26T05:11:59.542596+0000","last_fullsized":"2024-03-26T05:11:59.542596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:38.506070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:38.506070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.364991+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"216'11","reported_seq":652,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.553707+0000","last_change":"2024-03-26T05:10:30.100941+0000","last_active":"2024-03-26T05:11:59.553707+0000","last_peered":"2024-03-26T05:11:59.553707+0000","last_clean":"2024-03-26T05:11:59.553707+0000","last_became_active":"2024-03-26T05:07:59.960519+0000","last_became_peered":"2024-03-26T05:07:59.960519+0000","last_unstale":"2024-03-26T05:11:59.553707+0000","last_undegraded":"2024-03-26T05:11:59.553707+0000","last_fullsized":"2024-03-26T05:11:59.553707+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:10:30.100859+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.100859+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:02.133197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":488,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.556950+0000","last_change":"2024-03-26T05:09:55.525744+0000","last_active":"2024-03-26T05:11:59.556950+0000","last_peered":"2024-03-26T05:11:59.556950+0000","last_clean":"2024-03-26T05:11:59.556950+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:11:59.556950+0000","last_undegraded":"2024-03-26T05:11:59.556950+0000","last_fullsized":"2024-03-26T05:11:59.556950+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:55.525633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:09:55.525633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:59.671363+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"254'20","reported_seq":510,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-26T05:11:59.542680+0000","last_change":"2024-03-26T05:09:49.460933+0000","last_active":"2024-03-26T05:11:59.542680+0000","last_peered":"2024-03-26T05:11:59.542680+0000","last_clean":"2024-03-26T05:11:59.542680+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:11:59.542680+0000","last_undegraded":"2024-03-26T05:11:59.542680+0000","last_fullsized":"2024-03-26T05:11:59.542680+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-26T05:09:49.460784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:09:49.460784+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:12:01.365142+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5457379328,"num_objects":83273,"num_object_clones":0,"num_object_copies":249819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83273,"num_write_kb":5329472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16349921280,"data_stored":16349921280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80373,"ondisk_log_size":80373,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":690,"num_read_kb":685,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1413,"num_read_kb":1138,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":1870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":934,"ondisk_log_size":934,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005200,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3255728,"kb_used_data":2935784,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":90489424,"statfs":{"total":95995035648,"available":92661170176,"internally_reserved":0,"allocated":3006242816,"data_stored":3005724493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":327612874},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332245,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1281392,"kb_used_data":993380,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92463760,"statfs":{"total":95995035648,"available":94682890240,"internally_reserved":0,"allocated":1017221120,"data_stored":1016713607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":294910406},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3329684,"kb_used_data":2969480,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":90415468,"statfs":{"total":95995035648,"available":92585439232,"internally_reserved":0,"allocated":3040747520,"data_stored":3040203533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738548,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1706188,"kb_used_data":1332096,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92038964,"statfs":{"total":95995035648,"available":94247899136,"internally_reserved":0,"allocated":1364066304,"data_stored":1363531788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411498,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1305136,"kb_used_data":981028,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92440016,"statfs":{"total":95995035648,"available":94658576384,"internally_reserved":0,"allocated":1004572672,"data_stored":1004064759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117156,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3118536,"kb_used_data":2676412,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":90626616,"statfs":{"total":95995035648,"available":92801654784,"internally_reserved":0,"allocated":2740645888,"data_stored":2740113735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":452721095},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920928,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1254260,"kb_used_data":988460,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92490892,"statfs":{"total":95995035648,"available":94710673408,"internally_reserved":0,"allocated":1012183040,"data_stored":1011659630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659290,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3475696,"kb_used_data":2971108,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":90269456,"statfs":{"total":95995035648,"available":92435922944,"internally_reserved":0,"allocated":3042414592,"data_stored":3041906861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1379008512,"data_stored":1379008512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3073966080,"data_stored":3073966080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1007616000,"data_stored":1007616000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2743336960,"data_stored":2743336960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1017839616,"data_stored":1017839616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3060269056,"data_stored":3060269056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1018167296,"data_stored":1018167296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3049717760,"data_stored":3049717760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:12:05.376 INFO:tasks.thrashosds.thrasher:Setting 5.10 to [0, 4, 6, 3, 5, 1] 2024-03-26T05:12:05.376 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.10', '0', '4', '6', '3', '5', '1'] 2024-03-26T05:12:05.376 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pg-upmap-items 5.10 0 4 6 3 5 1 2024-03-26T05:12:05.377 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:05.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:05.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:05.479 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:05.520 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:05.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:05.622 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:05.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:05.655 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:05.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:05.675 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:12:05.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[40404]: pgmap v533: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.02k op/s 2024-03-26T05:12:05.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[40404]: 5.a scrub starts 2024-03-26T05:12:05.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[40404]: 5.a scrub ok 2024-03-26T05:12:05.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[40404]: 5.1b scrub starts 2024-03-26T05:12:05.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[40404]: 5.1b scrub ok 2024-03-26T05:12:05.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[40404]: 3.1b scrub starts 2024-03-26T05:12:05.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[40404]: 3.1b scrub ok 2024-03-26T05:12:05.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:12:05.677 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[29691]: pgmap v533: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.02k op/s 2024-03-26T05:12:05.677 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[29691]: 5.a scrub starts 2024-03-26T05:12:05.677 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[29691]: 5.a scrub ok 2024-03-26T05:12:05.677 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[29691]: 5.1b scrub starts 2024-03-26T05:12:05.677 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[29691]: 5.1b scrub ok 2024-03-26T05:12:05.677 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[29691]: 3.1b scrub starts 2024-03-26T05:12:05.677 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[29691]: 3.1b scrub ok 2024-03-26T05:12:05.677 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:12:05.721 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:12:05.745 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 57 16 89540 89524 98.1444 69.3125 0.0250128 0.0101836 2024-03-26T05:12:05.756 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:05.761 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:12:05.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:05 smithi132 ceph-mon[33500]: pgmap v533: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.02k op/s 2024-03-26T05:12:05.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:05 smithi132 ceph-mon[33500]: 5.a scrub starts 2024-03-26T05:12:05.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:05 smithi132 ceph-mon[33500]: 5.a scrub ok 2024-03-26T05:12:05.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:05 smithi132 ceph-mon[33500]: 5.1b scrub starts 2024-03-26T05:12:05.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:05 smithi132 ceph-mon[33500]: 5.1b scrub ok 2024-03-26T05:12:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:05 smithi132 ceph-mon[33500]: 3.1b scrub starts 2024-03-26T05:12:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:05 smithi132 ceph-mon[33500]: 3.1b scrub ok 2024-03-26T05:12:05.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:12:05.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:12:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:12:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:05.810 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:05.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:05.852 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:12:05.891 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:12:05.912 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:05.933 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:12:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:05.949 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:05.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:05.975 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:12:06.017 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:12:06.020 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:12:06.021 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:12:06.021 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:12:06.021 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:12:06.021 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:12:06.021 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:12:06.021 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:12:06.022 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:12:06.050 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:06.082 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:06.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:06.183 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:06.219 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:06.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:06.321 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:06.355 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:06.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:06.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:06.457 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:06.493 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:06.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:06.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:06.595 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:06.636 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:06.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:06.685 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:06.686 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: 5.16 scrub starts 2024-03-26T05:12:06.686 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: 5.16 scrub ok 2024-03-26T05:12:06.686 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: 5.1d scrub starts 2024-03-26T05:12:06.686 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: 6.1f scrub starts 2024-03-26T05:12:06.686 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/183860326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:06.686 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: 5.1d scrub ok 2024-03-26T05:12:06.686 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:12:06.686 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:12:06.686 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: 6.1f scrub ok 2024-03-26T05:12:06.686 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: 6.8 scrub starts 2024-03-26T05:12:06.686 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: 6.8 scrub ok 2024-03-26T05:12:06.686 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[29691]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T05:12:06.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:06.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: 5.16 scrub starts 2024-03-26T05:12:06.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: 5.16 scrub ok 2024-03-26T05:12:06.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: 5.1d scrub starts 2024-03-26T05:12:06.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: 6.1f scrub starts 2024-03-26T05:12:06.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/183860326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:06.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: 5.1d scrub ok 2024-03-26T05:12:06.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:12:06.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:12:06.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: 6.1f scrub ok 2024-03-26T05:12:06.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: 6.8 scrub starts 2024-03-26T05:12:06.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: 6.8 scrub ok 2024-03-26T05:12:06.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:06 smithi080 ceph-mon[40404]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T05:12:06.738 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:06.744 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 58 16 90724 90708 97.7281 74 0.0165006 0.0102275 2024-03-26T05:12:06.774 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:06.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:06.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:06.876 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:06.916 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:06.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:07.018 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: 5.16 scrub starts 2024-03-26T05:12:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: 5.16 scrub ok 2024-03-26T05:12:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: 5.1d scrub starts 2024-03-26T05:12:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: 6.1f scrub starts 2024-03-26T05:12:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/183860326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: 5.1d scrub ok 2024-03-26T05:12:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:12:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:12:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: 6.1f scrub ok 2024-03-26T05:12:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: 6.8 scrub starts 2024-03-26T05:12:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: 6.8 scrub ok 2024-03-26T05:12:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:06 smithi132 ceph-mon[33500]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T05:12:07.055 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:07.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:07.157 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:07.192 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:07.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:07.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:07.197 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:07.294 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:07.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:07.348 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:07.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:07.449 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:07.483 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:07.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:07.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:07.626 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:07.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:07.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:07.643 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[29691]: pgmap v534: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.6 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-26T05:12:07.643 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[29691]: 2.1 scrub starts 2024-03-26T05:12:07.643 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[29691]: 2.1 scrub ok 2024-03-26T05:12:07.644 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[29691]: 4.0 scrub starts 2024-03-26T05:12:07.644 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[29691]: 4.0 scrub ok 2024-03-26T05:12:07.644 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[29691]: 6.15 scrub starts 2024-03-26T05:12:07.644 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[29691]: 6.15 scrub ok 2024-03-26T05:12:07.644 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[40404]: pgmap v534: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.6 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-26T05:12:07.644 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[40404]: 2.1 scrub starts 2024-03-26T05:12:07.644 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[40404]: 2.1 scrub ok 2024-03-26T05:12:07.644 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[40404]: 4.0 scrub starts 2024-03-26T05:12:07.644 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[40404]: 4.0 scrub ok 2024-03-26T05:12:07.644 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[40404]: 6.15 scrub starts 2024-03-26T05:12:07.644 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:07 smithi080 ceph-mon[40404]: 6.15 scrub ok 2024-03-26T05:12:07.728 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:07.745 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 59 16 91962 91946 97.3824 77.375 0.0102255 0.0102641 2024-03-26T05:12:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:07.775 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:07.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:07.876 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:07.913 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:07.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:08.015 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:07 smithi132 ceph-mon[33500]: pgmap v534: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.6 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-26T05:12:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:07 smithi132 ceph-mon[33500]: 2.1 scrub starts 2024-03-26T05:12:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:07 smithi132 ceph-mon[33500]: 2.1 scrub ok 2024-03-26T05:12:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:07 smithi132 ceph-mon[33500]: 4.0 scrub starts 2024-03-26T05:12:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:07 smithi132 ceph-mon[33500]: 4.0 scrub ok 2024-03-26T05:12:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:07 smithi132 ceph-mon[33500]: 6.15 scrub starts 2024-03-26T05:12:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:07 smithi132 ceph-mon[33500]: 6.15 scrub ok 2024-03-26T05:12:08.064 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:08.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:08.166 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:08.199 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:08.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:08.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:08.213 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:12:08.213 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:12:08.304 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:08.345 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:08.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:08.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:08.447 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:08.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:08.489 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:08.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:08.590 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:08.631 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:08.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:08.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:08.732 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[40404]: 5.6 scrub starts 2024-03-26T05:12:08.732 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[40404]: 5.6 scrub ok 2024-03-26T05:12:08.732 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[40404]: 4.14 scrub starts 2024-03-26T05:12:08.732 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[40404]: 4.14 scrub ok 2024-03-26T05:12:08.732 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[40404]: 4.15 scrub starts 2024-03-26T05:12:08.732 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[40404]: 4.15 scrub ok 2024-03-26T05:12:08.732 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[40404]: pgmap v536: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 797 B/s rd, 57 MiB/s wr, 916 op/s 2024-03-26T05:12:08.732 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[40404]: 6.1a scrub starts 2024-03-26T05:12:08.732 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[40404]: 6.1a scrub ok 2024-03-26T05:12:08.733 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[29691]: 5.6 scrub starts 2024-03-26T05:12:08.733 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[29691]: 5.6 scrub ok 2024-03-26T05:12:08.733 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[29691]: 4.14 scrub starts 2024-03-26T05:12:08.733 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[29691]: 4.14 scrub ok 2024-03-26T05:12:08.733 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[29691]: 4.15 scrub starts 2024-03-26T05:12:08.733 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[29691]: 4.15 scrub ok 2024-03-26T05:12:08.733 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[29691]: pgmap v536: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 797 B/s rd, 57 MiB/s wr, 916 op/s 2024-03-26T05:12:08.733 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[29691]: 6.1a scrub starts 2024-03-26T05:12:08.733 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:08 smithi080 ceph-mon[29691]: 6.1a scrub ok 2024-03-26T05:12:08.733 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:08.740 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:12:08.745 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:12:08.755381+0000 min lat: 0.00115197 max lat: 0.222181 avg lat: 0.0103154 2024-03-26T05:12:08.745 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:12:08.745 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 60 16 93051 93035 96.8937 68.0625 0.0090177 0.0103154 2024-03-26T05:12:08.765 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:08.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:08.867 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:08.895 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:08.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:08.997 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:09.026 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:09.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:09.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:08 smithi132 ceph-mon[33500]: 5.6 scrub starts 2024-03-26T05:12:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:08 smithi132 ceph-mon[33500]: 5.6 scrub ok 2024-03-26T05:12:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:08 smithi132 ceph-mon[33500]: 4.14 scrub starts 2024-03-26T05:12:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:08 smithi132 ceph-mon[33500]: 4.14 scrub ok 2024-03-26T05:12:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:08 smithi132 ceph-mon[33500]: 4.15 scrub starts 2024-03-26T05:12:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:08 smithi132 ceph-mon[33500]: 4.15 scrub ok 2024-03-26T05:12:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:08 smithi132 ceph-mon[33500]: pgmap v536: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 797 B/s rd, 57 MiB/s wr, 916 op/s 2024-03-26T05:12:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:08 smithi132 ceph-mon[33500]: 6.1a scrub starts 2024-03-26T05:12:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:08 smithi132 ceph-mon[33500]: 6.1a scrub ok 2024-03-26T05:12:09.129 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:09.152 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:09.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:09.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:09.254 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:09.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:09.297 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:09.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:09.398 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:09.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:09.439 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:09.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:09.540 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:09.586 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:09.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:09.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:09.689 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:09.725 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:09.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:09.745 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 61 16 93980 93964 96.2571 58.0625 0.0107071 0.0103837 2024-03-26T05:12:09.826 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:09.864 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:09.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:09.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:09.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:09 smithi080 ceph-mon[40404]: 5.d scrub starts 2024-03-26T05:12:09.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:09 smithi080 ceph-mon[40404]: 5.d scrub ok 2024-03-26T05:12:09.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:09 smithi080 ceph-mon[40404]: 6.1 scrub starts 2024-03-26T05:12:09.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:09 smithi080 ceph-mon[40404]: 6.1 scrub ok 2024-03-26T05:12:09.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:09 smithi080 ceph-mon[40404]: 6.e scrub starts 2024-03-26T05:12:09.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:09 smithi080 ceph-mon[40404]: 6.e scrub ok 2024-03-26T05:12:09.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:09 smithi080 ceph-mon[29691]: 5.d scrub starts 2024-03-26T05:12:09.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:09 smithi080 ceph-mon[29691]: 5.d scrub ok 2024-03-26T05:12:09.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:09 smithi080 ceph-mon[29691]: 6.1 scrub starts 2024-03-26T05:12:09.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:09 smithi080 ceph-mon[29691]: 6.1 scrub ok 2024-03-26T05:12:09.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:09 smithi080 ceph-mon[29691]: 6.e scrub starts 2024-03-26T05:12:09.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:09 smithi080 ceph-mon[29691]: 6.e scrub ok 2024-03-26T05:12:09.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:09.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:09.999 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:09.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:10.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:09 smithi132 ceph-mon[33500]: 5.d scrub starts 2024-03-26T05:12:10.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:09 smithi132 ceph-mon[33500]: 5.d scrub ok 2024-03-26T05:12:10.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:09 smithi132 ceph-mon[33500]: 6.1 scrub starts 2024-03-26T05:12:10.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:09 smithi132 ceph-mon[33500]: 6.1 scrub ok 2024-03-26T05:12:10.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:09 smithi132 ceph-mon[33500]: 6.e scrub starts 2024-03-26T05:12:10.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:09 smithi132 ceph-mon[33500]: 6.e scrub ok 2024-03-26T05:12:10.100 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:10.140 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:10.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:10.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:10.242 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:10.266 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:10.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:10.367 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:10.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:10.403 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:10.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:10.504 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:10.540 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:10.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:10.641 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:10.679 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:10.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:10.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:10.745 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 62 16 94995 94979 95.7277 63.4375 0.00325561 0.0104395 2024-03-26T05:12:10.782 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:10.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:10.806 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:10.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:10.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[29691]: 5.8 scrub starts 2024-03-26T05:12:10.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[29691]: 5.8 scrub ok 2024-03-26T05:12:10.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[29691]: 3.1d scrub starts 2024-03-26T05:12:10.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[29691]: 3.1d scrub ok 2024-03-26T05:12:10.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[29691]: pgmap v537: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 868 op/s 2024-03-26T05:12:10.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[29691]: 6.12 scrub starts 2024-03-26T05:12:10.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[29691]: 6.12 scrub ok 2024-03-26T05:12:10.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[29691]: 5.e scrub starts 2024-03-26T05:12:10.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[29691]: 5.e scrub ok 2024-03-26T05:12:10.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[40404]: 5.8 scrub starts 2024-03-26T05:12:10.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[40404]: 5.8 scrub ok 2024-03-26T05:12:10.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[40404]: 3.1d scrub starts 2024-03-26T05:12:10.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[40404]: 3.1d scrub ok 2024-03-26T05:12:10.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[40404]: pgmap v537: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 868 op/s 2024-03-26T05:12:10.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[40404]: 6.12 scrub starts 2024-03-26T05:12:10.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[40404]: 6.12 scrub ok 2024-03-26T05:12:10.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[40404]: 5.e scrub starts 2024-03-26T05:12:10.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:10 smithi080 ceph-mon[40404]: 5.e scrub ok 2024-03-26T05:12:10.908 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:10.937 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:10.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:10.956 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:11.038 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:11.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:10 smithi132 ceph-mon[33500]: 5.8 scrub starts 2024-03-26T05:12:11.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:10 smithi132 ceph-mon[33500]: 5.8 scrub ok 2024-03-26T05:12:11.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:10 smithi132 ceph-mon[33500]: 3.1d scrub starts 2024-03-26T05:12:11.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:10 smithi132 ceph-mon[33500]: 3.1d scrub ok 2024-03-26T05:12:11.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:10 smithi132 ceph-mon[33500]: pgmap v537: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 868 op/s 2024-03-26T05:12:11.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:10 smithi132 ceph-mon[33500]: 6.12 scrub starts 2024-03-26T05:12:11.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:10 smithi132 ceph-mon[33500]: 6.12 scrub ok 2024-03-26T05:12:11.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:10 smithi132 ceph-mon[33500]: 5.e scrub starts 2024-03-26T05:12:11.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:10 smithi132 ceph-mon[33500]: 5.e scrub ok 2024-03-26T05:12:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:11.077 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:11.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:11.178 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:11.221 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:11.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:11.322 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:11.359 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:11.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:11.460 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:11.500 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:11.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:11.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T05:12:11.601 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:11.635 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:11.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:11.736 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:11.746 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 63 16 96039 96023 95.2436 65.25 0.00866433 0.0104948 2024-03-26T05:12:11.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:11.769 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:11.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:11.835 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:11 smithi080 ceph-mon[40404]: 5.13 scrub starts 2024-03-26T05:12:11.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:11 smithi080 ceph-mon[40404]: 5.13 scrub ok 2024-03-26T05:12:11.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:11 smithi080 ceph-mon[40404]: 6.10 scrub starts 2024-03-26T05:12:11.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:11 smithi080 ceph-mon[40404]: 6.10 scrub ok 2024-03-26T05:12:11.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:11 smithi080 ceph-mon[40404]: 3.3 scrub starts 2024-03-26T05:12:11.836 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:11 smithi080 ceph-mon[40404]: 3.3 scrub ok 2024-03-26T05:12:11.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:11 smithi080 ceph-mon[29691]: 5.13 scrub starts 2024-03-26T05:12:11.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:11 smithi080 ceph-mon[29691]: 5.13 scrub ok 2024-03-26T05:12:11.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:11 smithi080 ceph-mon[29691]: 6.10 scrub starts 2024-03-26T05:12:11.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:11 smithi080 ceph-mon[29691]: 6.10 scrub ok 2024-03-26T05:12:11.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:11 smithi080 ceph-mon[29691]: 3.3 scrub starts 2024-03-26T05:12:11.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:11 smithi080 ceph-mon[29691]: 3.3 scrub ok 2024-03-26T05:12:11.870 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:11.924 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:11.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:11.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:12.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:11 smithi132 ceph-mon[33500]: 5.13 scrub starts 2024-03-26T05:12:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:11 smithi132 ceph-mon[33500]: 5.13 scrub ok 2024-03-26T05:12:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:11 smithi132 ceph-mon[33500]: 6.10 scrub starts 2024-03-26T05:12:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:11 smithi132 ceph-mon[33500]: 6.10 scrub ok 2024-03-26T05:12:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:11 smithi132 ceph-mon[33500]: 3.3 scrub starts 2024-03-26T05:12:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:11 smithi132 ceph-mon[33500]: 3.3 scrub ok 2024-03-26T05:12:12.062 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:12.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:12.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:12.199 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:12.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:12.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:12.301 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:12.337 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:12.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:12.439 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:12.476 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:12.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:12.578 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:12.605 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:12.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:12.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:12 smithi080 ceph-mon[29691]: 3.b scrub starts 2024-03-26T05:12:12.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:12 smithi080 ceph-mon[29691]: 3.b scrub ok 2024-03-26T05:12:12.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:12 smithi080 ceph-mon[29691]: 4.16 scrub starts 2024-03-26T05:12:12.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:12 smithi080 ceph-mon[29691]: 4.16 scrub ok 2024-03-26T05:12:12.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:12 smithi080 ceph-mon[29691]: pgmap v538: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-26T05:12:12.663 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:12 smithi080 ceph-mon[40404]: 3.b scrub starts 2024-03-26T05:12:12.664 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:12 smithi080 ceph-mon[40404]: 3.b scrub ok 2024-03-26T05:12:12.664 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:12 smithi080 ceph-mon[40404]: 4.16 scrub starts 2024-03-26T05:12:12.664 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:12 smithi080 ceph-mon[40404]: 4.16 scrub ok 2024-03-26T05:12:12.664 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:12 smithi080 ceph-mon[40404]: pgmap v538: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-26T05:12:12.707 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:12.745 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:12.746 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 64 16 97243 97227 94.9311 75.25 0.011885 0.0105296 2024-03-26T05:12:12.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:12.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:12.847 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:12.884 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:12.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:12.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:12.986 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:13.026 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:13.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:12 smithi132 ceph-mon[33500]: 3.b scrub starts 2024-03-26T05:12:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:12 smithi132 ceph-mon[33500]: 3.b scrub ok 2024-03-26T05:12:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:12 smithi132 ceph-mon[33500]: 4.16 scrub starts 2024-03-26T05:12:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:12 smithi132 ceph-mon[33500]: 4.16 scrub ok 2024-03-26T05:12:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:12 smithi132 ceph-mon[33500]: pgmap v538: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-26T05:12:13.127 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:13.160 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:13.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:13.262 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:13.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:13.300 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:13.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:13.401 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:13.437 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:13.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:13.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:13.528 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:13.539 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:13.547 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:13.608 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:13.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:13.647 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:12:13.709 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:13.746 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 65 16 98361 98345 94.5455 69.875 0.020211 0.010572 2024-03-26T05:12:13.759 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:13.759 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[29691]: 3.0 scrub starts 2024-03-26T05:12:13.759 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[29691]: 3.0 scrub ok 2024-03-26T05:12:13.759 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[29691]: 5.3 scrub starts 2024-03-26T05:12:13.759 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[29691]: 5.3 scrub ok 2024-03-26T05:12:13.759 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/623662248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:12:13.759 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[29691]: 3.1 scrub starts 2024-03-26T05:12:13.760 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[29691]: 3.1 scrub ok 2024-03-26T05:12:13.760 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[40404]: 3.0 scrub starts 2024-03-26T05:12:13.760 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[40404]: 3.0 scrub ok 2024-03-26T05:12:13.760 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[40404]: 5.3 scrub starts 2024-03-26T05:12:13.760 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[40404]: 5.3 scrub ok 2024-03-26T05:12:13.760 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/623662248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:12:13.760 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[40404]: 3.1 scrub starts 2024-03-26T05:12:13.760 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:13 smithi080 ceph-mon[40404]: 3.1 scrub ok 2024-03-26T05:12:13.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:13.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:13.864 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:13.893 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:13.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:13.995 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:14.035 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:14.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:14.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:13 smithi132 ceph-mon[33500]: 3.0 scrub starts 2024-03-26T05:12:14.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:13 smithi132 ceph-mon[33500]: 3.0 scrub ok 2024-03-26T05:12:14.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:13 smithi132 ceph-mon[33500]: 5.3 scrub starts 2024-03-26T05:12:14.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:13 smithi132 ceph-mon[33500]: 5.3 scrub ok 2024-03-26T05:12:14.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:13 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/623662248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:12:14.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:13 smithi132 ceph-mon[33500]: 3.1 scrub starts 2024-03-26T05:12:14.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:13 smithi132 ceph-mon[33500]: 3.1 scrub ok 2024-03-26T05:12:14.137 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:14.178 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:14.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:14.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:14.281 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:14.322 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:14.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:14.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:12:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:12:14.424 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:14.463 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:14.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:14.564 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:14.596 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:14.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:14.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:14.698 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:14.731 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:14.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:14.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:14.745 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 66 16 99637 99621 94.3213 79.75 0.0145915 0.0105978 2024-03-26T05:12:14.833 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:14.868 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:14.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:14.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:14.966 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: 6.14 scrub starts 2024-03-26T05:12:14.966 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: 6.14 scrub ok 2024-03-26T05:12:14.966 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:12:14.966 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:12:14.966 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: 5.7 scrub starts 2024-03-26T05:12:14.966 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: 5.7 scrub ok 2024-03-26T05:12:14.966 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:12:14.966 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: Cluster is now healthy 2024-03-26T05:12:14.966 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/623662248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:12:14.967 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T05:12:14.967 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: pgmap v540: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 677 op/s 2024-03-26T05:12:14.967 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: 6.1c scrub starts 2024-03-26T05:12:14.967 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: 6.1c scrub ok 2024-03-26T05:12:14.967 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: 4.d scrub starts 2024-03-26T05:12:14.967 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[40404]: 4.d scrub ok 2024-03-26T05:12:14.967 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: 6.14 scrub starts 2024-03-26T05:12:14.967 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: 6.14 scrub ok 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: 5.7 scrub starts 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: 5.7 scrub ok 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: Cluster is now healthy 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/623662248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: pgmap v540: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 677 op/s 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: 6.1c scrub starts 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: 6.1c scrub ok 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: 4.d scrub starts 2024-03-26T05:12:14.968 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:14 smithi080 ceph-mon[29691]: 4.d scrub ok 2024-03-26T05:12:14.971 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:15.010 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:15.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:15.039 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: 6.14 scrub starts 2024-03-26T05:12:15.039 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: 6.14 scrub ok 2024-03-26T05:12:15.039 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:12:15.039 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:12:15.039 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: 5.7 scrub starts 2024-03-26T05:12:15.039 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: 5.7 scrub ok 2024-03-26T05:12:15.039 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:12:15.039 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: Cluster is now healthy 2024-03-26T05:12:15.039 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/623662248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:12:15.039 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T05:12:15.039 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: pgmap v540: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 677 op/s 2024-03-26T05:12:15.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: 6.1c scrub starts 2024-03-26T05:12:15.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: 6.1c scrub ok 2024-03-26T05:12:15.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: 4.d scrub starts 2024-03-26T05:12:15.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:14 smithi132 ceph-mon[33500]: 4.d scrub ok 2024-03-26T05:12:15.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:15.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:15.151 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:15.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:15.252 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:15.297 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:15.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:15.399 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:15.433 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:15.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:15.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:15.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:15.569 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:15.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:15.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:15.660 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:12:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:12:15.671 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:15.706 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:15.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:15.746 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 67 16 100923 100907 94.1131 80.375 0.0114484 0.010621 2024-03-26T05:12:15.808 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:15.849 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:15.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:15.927 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:15 smithi132 ceph-mon[33500]: 5.4 deep-scrub starts 2024-03-26T05:12:15.927 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:15 smithi132 ceph-mon[33500]: 5.4 deep-scrub ok 2024-03-26T05:12:15.927 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:15 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:15.951 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:15.991 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:15.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:16.092 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:15 smithi080 ceph-mon[40404]: 5.4 deep-scrub starts 2024-03-26T05:12:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:15 smithi080 ceph-mon[40404]: 5.4 deep-scrub ok 2024-03-26T05:12:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:15 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:16.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:15 smithi080 ceph-mon[29691]: 5.4 deep-scrub starts 2024-03-26T05:12:16.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:15 smithi080 ceph-mon[29691]: 5.4 deep-scrub ok 2024-03-26T05:12:16.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:15 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:16.127 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:16.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:16.229 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:16.264 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:16.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:16.366 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:16.404 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:16.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:16.505 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:16.546 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:16.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:16.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:16.648 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:16.685 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:16.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:16.744 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:12:16.746 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 68 15 102104 102089 93.8154 73.875 0.0621744 0.0106525 2024-03-26T05:12:16.787 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:16.819 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:16.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:16.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:16.921 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:16.974 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:16.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:16.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:16 smithi132 ceph-mon[33500]: pgmap v541: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T05:12:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:16 smithi132 ceph-mon[33500]: 4.1e deep-scrub starts 2024-03-26T05:12:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:16 smithi132 ceph-mon[33500]: 4.1e deep-scrub ok 2024-03-26T05:12:17.076 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:17.115 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:17.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:17.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:16 smithi080 ceph-mon[40404]: pgmap v541: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T05:12:17.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:16 smithi080 ceph-mon[40404]: 4.1e deep-scrub starts 2024-03-26T05:12:17.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:16 smithi080 ceph-mon[40404]: 4.1e deep-scrub ok 2024-03-26T05:12:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:16 smithi080 ceph-mon[29691]: pgmap v541: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T05:12:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:16 smithi080 ceph-mon[29691]: 4.1e deep-scrub starts 2024-03-26T05:12:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:16 smithi080 ceph-mon[29691]: 4.1e deep-scrub ok 2024-03-26T05:12:17.217 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:17.246 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:17.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:17.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:17.348 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:17.384 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:17.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:17.486 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:17.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:17.517 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:17.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:17.618 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:17.647 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:17.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:17.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:17.746 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 69 16 103062 103046 93.3225 59.8125 0.0193704 0.0107093 2024-03-26T05:12:17.749 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:17.789 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:17.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:17 smithi080 ceph-mon[29691]: 9.f deep-scrub starts 2024-03-26T05:12:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:17 smithi080 ceph-mon[29691]: 5.1c scrub starts 2024-03-26T05:12:17.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:17 smithi080 ceph-mon[29691]: 5.1c scrub ok 2024-03-26T05:12:17.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:17 smithi080 ceph-mon[40404]: 9.f deep-scrub starts 2024-03-26T05:12:17.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:17 smithi080 ceph-mon[40404]: 5.1c scrub starts 2024-03-26T05:12:17.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:17 smithi080 ceph-mon[40404]: 5.1c scrub ok 2024-03-26T05:12:17.891 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:17.942 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:17.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:17 smithi132 ceph-mon[33500]: 9.f deep-scrub starts 2024-03-26T05:12:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:17 smithi132 ceph-mon[33500]: 5.1c scrub starts 2024-03-26T05:12:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:17 smithi132 ceph-mon[33500]: 5.1c scrub ok 2024-03-26T05:12:18.043 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:18.088 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:18.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:18.190 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:18.220 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:18.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:18.322 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:18.359 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:18.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:18.461 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:18.497 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:18.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:18.598 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:18.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:18.647 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:18.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:18.724 INFO:teuthology.orchestra.run.smithi080.stderr:set 5.10 pg_upmap_items mapping to [0->4,6->3,5->1] 2024-03-26T05:12:18.746 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 70 16 104001 103985 92.8274 58.6875 0.0121877 0.0107684 2024-03-26T05:12:18.748 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:18.797 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:18.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:18.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[40404]: 3.7 scrub starts 2024-03-26T05:12:18.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[40404]: 3.7 scrub ok 2024-03-26T05:12:18.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[40404]: 6.16 scrub starts 2024-03-26T05:12:18.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[40404]: pgmap v542: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-26T05:12:18.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[40404]: 6.16 scrub ok 2024-03-26T05:12:18.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[40404]: 4.9 scrub starts 2024-03-26T05:12:18.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[40404]: 4.9 scrub ok 2024-03-26T05:12:18.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/235880283' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [0, 4, 6, 3, 5, 1]}]: dispatch 2024-03-26T05:12:18.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[29691]: 3.7 scrub starts 2024-03-26T05:12:18.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[29691]: 3.7 scrub ok 2024-03-26T05:12:18.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[29691]: 6.16 scrub starts 2024-03-26T05:12:18.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[29691]: pgmap v542: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-26T05:12:18.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[29691]: 6.16 scrub ok 2024-03-26T05:12:18.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[29691]: 4.9 scrub starts 2024-03-26T05:12:18.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[29691]: 4.9 scrub ok 2024-03-26T05:12:18.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:18 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/235880283' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [0, 4, 6, 3, 5, 1]}]: dispatch 2024-03-26T05:12:18.899 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:18.932 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:18.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:19.033 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:19.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:18 smithi132 ceph-mon[33500]: 3.7 scrub starts 2024-03-26T05:12:19.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:18 smithi132 ceph-mon[33500]: 3.7 scrub ok 2024-03-26T05:12:19.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:18 smithi132 ceph-mon[33500]: 6.16 scrub starts 2024-03-26T05:12:19.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:18 smithi132 ceph-mon[33500]: pgmap v542: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-26T05:12:19.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:18 smithi132 ceph-mon[33500]: 6.16 scrub ok 2024-03-26T05:12:19.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:18 smithi132 ceph-mon[33500]: 4.9 scrub starts 2024-03-26T05:12:19.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:18 smithi132 ceph-mon[33500]: 4.9 scrub ok 2024-03-26T05:12:19.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:18 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/235880283' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [0, 4, 6, 3, 5, 1]}]: dispatch 2024-03-26T05:12:19.067 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:19.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:19.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:19.170 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:19.198 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:19.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:19.300 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:19.350 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:19.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:19.452 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:19.463 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:19.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:19.501 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:19.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:19.602 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:19.648 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:19.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:19.746 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 71 16 105002 104986 92.401 62.5625 0.0092011 0.0108159 2024-03-26T05:12:19.750 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:19.786 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:19.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:19.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[29691]: 5.12 scrub starts 2024-03-26T05:12:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[29691]: 3.2 scrub starts 2024-03-26T05:12:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[29691]: 5.12 scrub ok 2024-03-26T05:12:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[29691]: 3.2 scrub ok 2024-03-26T05:12:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/235880283' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [0, 4, 6, 3, 5, 1]}]': finished 2024-03-26T05:12:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[29691]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T05:12:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[29691]: 5.1a deep-scrub starts 2024-03-26T05:12:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[29691]: 5.1a deep-scrub ok 2024-03-26T05:12:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[29691]: 4.1 scrub starts 2024-03-26T05:12:19.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[29691]: 4.1 scrub ok 2024-03-26T05:12:19.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[40404]: 5.12 scrub starts 2024-03-26T05:12:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[40404]: 3.2 scrub starts 2024-03-26T05:12:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[40404]: 5.12 scrub ok 2024-03-26T05:12:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[40404]: 3.2 scrub ok 2024-03-26T05:12:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/235880283' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [0, 4, 6, 3, 5, 1]}]': finished 2024-03-26T05:12:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[40404]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T05:12:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[40404]: 5.1a deep-scrub starts 2024-03-26T05:12:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[40404]: 5.1a deep-scrub ok 2024-03-26T05:12:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[40404]: 4.1 scrub starts 2024-03-26T05:12:19.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:19 smithi080 ceph-mon[40404]: 4.1 scrub ok 2024-03-26T05:12:19.889 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:19.920 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:19.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:20.021 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:19 smithi132 ceph-mon[33500]: 5.12 scrub starts 2024-03-26T05:12:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:19 smithi132 ceph-mon[33500]: 3.2 scrub starts 2024-03-26T05:12:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:19 smithi132 ceph-mon[33500]: 5.12 scrub ok 2024-03-26T05:12:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:19 smithi132 ceph-mon[33500]: 3.2 scrub ok 2024-03-26T05:12:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:19 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/235880283' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [0, 4, 6, 3, 5, 1]}]': finished 2024-03-26T05:12:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:19 smithi132 ceph-mon[33500]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T05:12:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:19 smithi132 ceph-mon[33500]: 5.1a deep-scrub starts 2024-03-26T05:12:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:19 smithi132 ceph-mon[33500]: 5.1a deep-scrub ok 2024-03-26T05:12:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:19 smithi132 ceph-mon[33500]: 4.1 scrub starts 2024-03-26T05:12:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:19 smithi132 ceph-mon[33500]: 4.1 scrub ok 2024-03-26T05:12:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:20.071 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:20.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:20.172 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:20.213 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:20.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:20.314 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:20.350 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:20.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:20.452 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:20.483 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:20.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:20.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:20.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:20.623 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:20.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:20.725 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:20.746 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 72 15 106033 106018 92.0135 64.5 0.00898261 0.010862 2024-03-26T05:12:20.760 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:20.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:20.862 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:20.898 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:20.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:20.999 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:21.018 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:20 smithi080 ceph-mon[29691]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T05:12:21.018 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:20 smithi080 ceph-mon[29691]: 3.17 scrub starts 2024-03-26T05:12:21.019 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:20 smithi080 ceph-mon[29691]: pgmap v545: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 811 B/s rd, 58 MiB/s wr, 935 op/s 2024-03-26T05:12:21.019 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:20 smithi080 ceph-mon[29691]: 3.17 scrub ok 2024-03-26T05:12:21.019 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:12:21.019 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:12:21.019 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:20 smithi080 ceph-mon[40404]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T05:12:21.019 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:20 smithi080 ceph-mon[40404]: 3.17 scrub starts 2024-03-26T05:12:21.019 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:20 smithi080 ceph-mon[40404]: pgmap v545: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 811 B/s rd, 58 MiB/s wr, 935 op/s 2024-03-26T05:12:21.019 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:20 smithi080 ceph-mon[40404]: 3.17 scrub ok 2024-03-26T05:12:21.019 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:12:21.019 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:12:21.033 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:21.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:21.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:20 smithi132 ceph-mon[33500]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T05:12:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:20 smithi132 ceph-mon[33500]: 3.17 scrub starts 2024-03-26T05:12:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:20 smithi132 ceph-mon[33500]: pgmap v545: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 811 B/s rd, 58 MiB/s wr, 935 op/s 2024-03-26T05:12:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:20 smithi132 ceph-mon[33500]: 3.17 scrub ok 2024-03-26T05:12:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:12:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:12:21.135 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:21.173 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:21.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:21.274 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:21.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:21.319 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:21.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:21.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:21.454 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:21.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:21.556 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:21.583 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:21.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:21.685 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:21.724 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:21.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:21.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:21.746 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 73 16 106886 106870 91.4824 53.25 0.00656903 0.0109218 2024-03-26T05:12:21.826 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:21.875 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:21.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:21.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:21.977 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:22.014 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:22.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:21 smithi132 ceph-mon[33500]: 3.e scrub starts 2024-03-26T05:12:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:21 smithi132 ceph-mon[33500]: 3.e scrub ok 2024-03-26T05:12:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:21 smithi132 ceph-mon[33500]: 9.b scrub starts 2024-03-26T05:12:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:12:22.091 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:21 smithi080 ceph-mon[29691]: 3.e scrub starts 2024-03-26T05:12:22.092 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:21 smithi080 ceph-mon[29691]: 3.e scrub ok 2024-03-26T05:12:22.092 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:21 smithi080 ceph-mon[29691]: 9.b scrub starts 2024-03-26T05:12:22.092 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:12:22.092 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:21 smithi080 ceph-mon[40404]: 3.e scrub starts 2024-03-26T05:12:22.092 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:21 smithi080 ceph-mon[40404]: 3.e scrub ok 2024-03-26T05:12:22.092 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:21 smithi080 ceph-mon[40404]: 9.b scrub starts 2024-03-26T05:12:22.092 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:12:22.115 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:12:22.144 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:22.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:12:22.246 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:22.292 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:22.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:22.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:22.394 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:22.436 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:22.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:22.448 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:12:22.448 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:12:22.538 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:22.579 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:22.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:22.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:22.665 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:12:22.681 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:22.704 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:12:22.712 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:22.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:22.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:22.747 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 74 16 107562 107546 90.8171 42.25 0.0232271 0.0110034 2024-03-26T05:12:22.747 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:12:22.802 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:12:22.813 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:22.844 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:12:22.852 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:22.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:22.885 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:12:22.926 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:12:22.954 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:22.969 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: 3.1e scrub starts 2024-03-26T05:12:22.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: 3.1e scrub ok 2024-03-26T05:12:22.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: 1.0 scrub starts 2024-03-26T05:12:22.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: 1.0 scrub ok 2024-03-26T05:12:22.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: pgmap v546: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-26T05:12:22.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: 9.5 scrub starts 2024-03-26T05:12:22.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:12:22.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:12:22.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:12:22.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:12:22.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: Adjusting osd_memory_target on smithi080 to 3248M 2024-03-26T05:12:22.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:12:22.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:12:22.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:12:22.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:12:22.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3637978554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:22.973 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: 3.1e scrub starts 2024-03-26T05:12:22.973 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: 3.1e scrub ok 2024-03-26T05:12:22.973 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: 1.0 scrub starts 2024-03-26T05:12:22.973 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: 1.0 scrub ok 2024-03-26T05:12:22.973 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: pgmap v546: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-26T05:12:22.974 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: 9.5 scrub starts 2024-03-26T05:12:22.974 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:12:22.974 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:12:22.974 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:12:22.974 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:12:22.974 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: Adjusting osd_memory_target on smithi080 to 3248M 2024-03-26T05:12:22.975 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:12:22.975 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:12:22.975 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:12:22.975 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:12:22.975 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:22 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3637978554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:22.976 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:12:22.977 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:12:22.977 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:12:22.977 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:12:22.978 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:12:22.978 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:12:22.978 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:12:22.978 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:12:22.978 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:12:22.995 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:22.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:22.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: 3.1e scrub starts 2024-03-26T05:12:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: 3.1e scrub ok 2024-03-26T05:12:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: 1.0 scrub starts 2024-03-26T05:12:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: 1.0 scrub ok 2024-03-26T05:12:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: pgmap v546: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-26T05:12:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: 9.5 scrub starts 2024-03-26T05:12:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:12:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:12:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:12:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:12:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: Adjusting osd_memory_target on smithi080 to 3248M 2024-03-26T05:12:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:12:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:12:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:12:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:12:23.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:22 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3637978554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:23.097 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:23.133 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:23.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:23.235 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:23.267 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:23.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:23.368 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:23.419 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:23.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:23.521 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:23.562 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:23.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:23.663 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:23.695 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:23.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:23.747 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 75 16 108473 108457 90.3654 56.9375 0.0128246 0.0110608 2024-03-26T05:12:23.796 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:23.828 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:23.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:23.930 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:23.958 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:23.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:23.994 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[40404]: 3.18 scrub starts 2024-03-26T05:12:23.994 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[40404]: 3.18 scrub ok 2024-03-26T05:12:23.994 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[40404]: 6.3 scrub starts 2024-03-26T05:12:23.994 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[40404]: 9.b scrub ok 2024-03-26T05:12:23.994 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[40404]: 6.3 scrub ok 2024-03-26T05:12:23.994 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[40404]: 6.1e scrub starts 2024-03-26T05:12:23.995 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[40404]: 6.1e scrub ok 2024-03-26T05:12:23.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[29691]: 3.18 scrub starts 2024-03-26T05:12:23.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[29691]: 3.18 scrub ok 2024-03-26T05:12:23.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[29691]: 6.3 scrub starts 2024-03-26T05:12:23.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[29691]: 9.b scrub ok 2024-03-26T05:12:23.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[29691]: 6.3 scrub ok 2024-03-26T05:12:23.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[29691]: 6.1e scrub starts 2024-03-26T05:12:23.999 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:23 smithi080 ceph-mon[29691]: 6.1e scrub ok 2024-03-26T05:12:24.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:23 smithi132 ceph-mon[33500]: 3.18 scrub starts 2024-03-26T05:12:24.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:23 smithi132 ceph-mon[33500]: 3.18 scrub ok 2024-03-26T05:12:24.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:23 smithi132 ceph-mon[33500]: 6.3 scrub starts 2024-03-26T05:12:24.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:23 smithi132 ceph-mon[33500]: 9.b scrub ok 2024-03-26T05:12:24.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:23 smithi132 ceph-mon[33500]: 6.3 scrub ok 2024-03-26T05:12:24.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:23 smithi132 ceph-mon[33500]: 6.1e scrub starts 2024-03-26T05:12:24.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:23 smithi132 ceph-mon[33500]: 6.1e scrub ok 2024-03-26T05:12:24.061 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:24.101 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:24.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:24.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:24.252 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:24.260 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:12:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:12:24.281 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:24.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:24.382 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:24.420 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:24.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:24.522 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:24.553 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:24.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:24.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:24.655 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:24.686 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:24.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:24.747 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 76 16 109397 109381 89.9362 57.75 0.0102589 0.0111124 2024-03-26T05:12:24.787 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:24.794 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T05:12:24.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:24.819 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:24.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:24.920 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:24.953 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:24.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:24.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:24.973 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:12:24.973 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:12:24.974 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:12:24.974 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T05:12:24.974 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.7 2024-03-26T05:12:24.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-mon[29691]: 4.f deep-scrub starts 2024-03-26T05:12:24.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-mon[29691]: 4.f deep-scrub ok 2024-03-26T05:12:24.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-mon[29691]: 9.5 scrub ok 2024-03-26T05:12:24.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-mon[29691]: pgmap v547: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 651 op/s 2024-03-26T05:12:24.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-mon[29691]: 4.c scrub starts 2024-03-26T05:12:24.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-mon[29691]: 4.c scrub ok 2024-03-26T05:12:24.996 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-mon[40404]: 4.f deep-scrub starts 2024-03-26T05:12:24.996 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-mon[40404]: 4.f deep-scrub ok 2024-03-26T05:12:24.996 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-mon[40404]: 9.5 scrub ok 2024-03-26T05:12:24.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-mon[40404]: pgmap v547: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 651 op/s 2024-03-26T05:12:24.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-mon[40404]: 4.c scrub starts 2024-03-26T05:12:24.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:24 smithi080 ceph-mon[40404]: 4.c scrub ok 2024-03-26T05:12:25.023 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:24 smithi132 ceph-mon[33500]: 4.f deep-scrub starts 2024-03-26T05:12:25.024 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:24 smithi132 ceph-mon[33500]: 4.f deep-scrub ok 2024-03-26T05:12:25.024 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:24 smithi132 ceph-mon[33500]: 9.5 scrub ok 2024-03-26T05:12:25.024 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:24 smithi132 ceph-mon[33500]: pgmap v547: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 651 op/s 2024-03-26T05:12:25.024 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:24 smithi132 ceph-mon[33500]: 4.c scrub starts 2024-03-26T05:12:25.024 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:24 smithi132 ceph-mon[33500]: 4.c scrub ok 2024-03-26T05:12:25.055 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:25.092 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:25.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:25.194 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:25.266 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:25.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:25.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:25.280 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:25.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:25 smithi132 systemd[1]: Stopping Ceph osd.7 for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:12:25.368 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:25.413 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:25.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:25.516 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:25.555 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:25.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:25.656 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:25.688 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:25.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:25.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:25.747 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 77 16 109982 109966 89.243 36.5625 0.029484 0.0112002 2024-03-26T05:12:25.789 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[55659]: 2024-03-26T05:12:25.425+0000 7ff44554a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:12:25.789 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[55659]: 2024-03-26T05:12:25.425+0000 7ff44554a700 -1 osd.7 292 *** Got signal Terminated *** 2024-03-26T05:12:25.789 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[55659]: 2024-03-26T05:12:25.425+0000 7ff44554a700 -1 osd.7 292 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:12:25.789 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:12:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:12:25.790 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:25.793 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:12:25.834 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:25.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:25.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:25.960 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:25.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:26.013 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[40404]: 9.4 scrub starts 2024-03-26T05:12:26.013 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[40404]: 9.a deep-scrub starts 2024-03-26T05:12:26.013 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[40404]: 9.3 scrub starts 2024-03-26T05:12:26.013 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:26.013 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[40404]: osd.7 marked itself down and dead 2024-03-26T05:12:26.013 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3627120841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:12:26.014 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:12:26.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[29691]: 9.4 scrub starts 2024-03-26T05:12:26.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[29691]: 9.a deep-scrub starts 2024-03-26T05:12:26.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[29691]: 9.3 scrub starts 2024-03-26T05:12:26.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:26.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[29691]: osd.7 marked itself down and dead 2024-03-26T05:12:26.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3627120841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:12:26.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:25 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:12:26.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:25 smithi132 ceph-mon[33500]: 9.4 scrub starts 2024-03-26T05:12:26.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:25 smithi132 ceph-mon[33500]: 9.a deep-scrub starts 2024-03-26T05:12:26.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:25 smithi132 ceph-mon[33500]: 9.3 scrub starts 2024-03-26T05:12:26.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:25 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:26.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:25 smithi132 ceph-mon[33500]: osd.7 marked itself down and dead 2024-03-26T05:12:26.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:25 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3627120841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:12:26.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:25 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:12:26.044 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:12:26.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:26.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:26.087 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:26.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:26.188 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:26.215 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:26.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:26.317 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:26.344 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:26.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:26.446 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:26.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:26.470 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:26.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:26.571 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:26.597 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:26.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:26.617 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:26 smithi132 podman[109576]: 2024-03-26 05:12:26.364336442 +0000 UTC m=+1.083643158 container died e176eec6c1548c8e2382671937cc4e377cf3b434f5b3b9aa612378f7ebc08033 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, ceph=True, GIT_CLEAN=True) 2024-03-26T05:12:26.698 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:26.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:26.723 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:26.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:26.747 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 78 16 110014 109998 88.1246 2 0.0441825 0.0112081 2024-03-26T05:12:26.823 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:26.860 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:26.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:26.962 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:27.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:27.005 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:27.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:27.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:26 smithi132 podman[109576]: 2024-03-26 05:12:26.717756502 +0000 UTC m=+1.437063205 container cleanup e176eec6c1548c8e2382671937cc4e377cf3b434f5b3b9aa612378f7ebc08033 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319) 2024-03-26T05:12:27.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:26 smithi132 bash[109576]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7 2024-03-26T05:12:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:26 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:12:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:26 smithi132 ceph-mon[33500]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T05:12:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:26 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:12:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:26 smithi132 ceph-mon[33500]: osdmap e293: 8 total, 7 up, 8 in 2024-03-26T05:12:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:26 smithi132 ceph-mon[33500]: 4.4 scrub starts 2024-03-26T05:12:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:26 smithi132 ceph-mon[33500]: pgmap v549: 148 pgs: 26 peering, 1 active+clean+scrubbing, 18 stale+active+clean, 2 active+clean+scrubbing+deep, 101 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-03-26T05:12:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:26 smithi132 ceph-mon[33500]: osdmap e294: 8 total, 7 up, 8 in 2024-03-26T05:12:27.106 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:27.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:12:27.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[29691]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T05:12:27.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:12:27.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[29691]: osdmap e293: 8 total, 7 up, 8 in 2024-03-26T05:12:27.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[29691]: 4.4 scrub starts 2024-03-26T05:12:27.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[29691]: pgmap v549: 148 pgs: 26 peering, 1 active+clean+scrubbing, 18 stale+active+clean, 2 active+clean+scrubbing+deep, 101 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-03-26T05:12:27.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[29691]: osdmap e294: 8 total, 7 up, 8 in 2024-03-26T05:12:27.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:12:27.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[40404]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T05:12:27.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:12:27.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[40404]: osdmap e293: 8 total, 7 up, 8 in 2024-03-26T05:12:27.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[40404]: 4.4 scrub starts 2024-03-26T05:12:27.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[40404]: pgmap v549: 148 pgs: 26 peering, 1 active+clean+scrubbing, 18 stale+active+clean, 2 active+clean+scrubbing+deep, 101 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-03-26T05:12:27.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:26 smithi080 ceph-mon[40404]: osdmap e294: 8 total, 7 up, 8 in 2024-03-26T05:12:27.137 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:27.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:27.239 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:27.274 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:27.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:27.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:27.376 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:27.400 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:27.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:27.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:27.485 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:27 smithi132 podman[109644]: 2024-03-26 05:12:27.053450885 +0000 UTC m=+0.683489751 container remove e176eec6c1548c8e2382671937cc4e377cf3b434f5b3b9aa612378f7ebc08033 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:12:27.502 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:27.539 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:27.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:27.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:27.641 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:27.666 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:27.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:27.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:27.747 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 79 16 110971 110955 87.7662 59.8125 0.00426162 0.0113802 2024-03-26T05:12:27.768 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:27.791 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:27 smithi132 podman[109765]: 2024-03-26 05:12:27.399005388 +0000 UTC m=+0.019999674 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:12:27.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:27.810 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:27.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:27.911 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:27.938 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:27.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:27.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:28.040 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:28.056 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:27 smithi132 ceph-mon[33500]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T05:12:28.057 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:27 smithi132 podman[109765]: 2024-03-26 05:12:27.821737083 +0000 UTC m=+0.442731405 container create fc49e446c93fc7ceccbeeb5626343fbbfbc9b99cb0dd208d12d6b60185d2dc47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_BRANCH=HEAD) 2024-03-26T05:12:28.072 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:28.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:28.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:27 smithi080 ceph-mon[29691]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T05:12:28.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:27 smithi080 ceph-mon[40404]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T05:12:28.174 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:28.212 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:28.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:28.314 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:28.341 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:28.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:28.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:28.439 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:28 smithi132 podman[109765]: 2024-03-26 05:12:28.066702744 +0000 UTC m=+0.687697023 container init fc49e446c93fc7ceccbeeb5626343fbbfbc9b99cb0dd208d12d6b60185d2dc47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:12:28.439 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:28 smithi132 podman[109765]: 2024-03-26 05:12:28.073365499 +0000 UTC m=+0.694359777 container start fc49e446c93fc7ceccbeeb5626343fbbfbc9b99cb0dd208d12d6b60185d2dc47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-03-26T05:12:28.439 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:28 smithi132 podman[109765]: 2024-03-26 05:12:28.175394039 +0000 UTC m=+0.796388325 container attach fc49e446c93fc7ceccbeeb5626343fbbfbc9b99cb0dd208d12d6b60185d2dc47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0) 2024-03-26T05:12:28.442 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:28.479 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:28.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:28.581 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:28.620 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:28.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:28.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:28.721 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:28.745 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:28.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:28.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:28.747 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:12:28.757726+0000 min lat: 0.00115197 max lat: 1.08124 avg lat: 0.0114336 2024-03-26T05:12:28.747 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:12:28.747 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 80 16 111888 111872 87.3854 57.3125 0.0033461 0.0114336 2024-03-26T05:12:28.792 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:12:28 smithi132 podman[109765]: 2024-03-26 05:12:28.492497275 +0000 UTC m=+1.113491557 container died fc49e446c93fc7ceccbeeb5626343fbbfbc9b99cb0dd208d12d6b60185d2dc47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T05:12:28.846 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:28.883 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:28.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:28.985 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:29.015 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:29.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:29.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:29.088 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:28 smithi132 ceph-mon[33500]: pgmap v551: 148 pgs: 26 peering, 1 active+clean+scrubbing, 18 stale+active+clean, 2 active+clean+scrubbing+deep, 101 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-03-26T05:12:29.106 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T05:12:29.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:28 smithi080 ceph-mon[40404]: pgmap v551: 148 pgs: 26 peering, 1 active+clean+scrubbing, 18 stale+active+clean, 2 active+clean+scrubbing+deep, 101 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-03-26T05:12:29.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:28 smithi080 ceph-mon[29691]: pgmap v551: 148 pgs: 26 peering, 1 active+clean+scrubbing, 18 stale+active+clean, 2 active+clean+scrubbing+deep, 101 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-03-26T05:12:29.117 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:29.160 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:29.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:29.166 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:12:29.167 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T05:12:29.167 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-03-26T05:12:29.167 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi132.front.sepia.ceph.com 2024-03-26T05:12:29.167 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -n osd.7 --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-03-26T05:12:29.261 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:29.315 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:29.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:29.415 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:29.417 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/osd.7/config 2024-03-26T05:12:29.445 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:29.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:29.546 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:29.579 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:29.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:29.680 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:29.720 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:29.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:29.747 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 81 16 112880 112864 87.072 62 0.016213 0.0114796 2024-03-26T05:12:29.822 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:29.865 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:29.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:29.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:29.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:29.998 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:29.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:29.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:30.100 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:30.130 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:30.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:30.231 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:30.272 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:30.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:30.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:30 smithi132 ceph-mon[33500]: Health check failed: Degraded data redundancy: 7088/330690 objects degraded (2.143%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:30.314 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:30 smithi080 ceph-mon[29691]: Health check failed: Degraded data redundancy: 7088/330690 objects degraded (2.143%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:30.314 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:30 smithi080 ceph-mon[40404]: Health check failed: Degraded data redundancy: 7088/330690 objects degraded (2.143%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:30.373 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:30.397 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:30.420 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:30.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:30.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:30.432 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:12:30.432 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:12:30.432 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.432 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:113530 9.d 9:bca041ea:::benchmark_data_smithi080_136801_object113529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-03-26T05:12:30.432 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:30.424957+0000", 2024-03-26T05:12:30.432 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.014501896, 2024-03-26T05:12:30.432 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014529823000000001, 2024-03-26T05:12:30.432 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:30.432 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:12:30.432 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:30.432 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 113530 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.424957+0000", 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.424957+0000", 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:30.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.424958+0000", 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3680000000000001e-06 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.424982+0000", 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3915999999999999e-05 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:30.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.424997+0000", 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5373e-05 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.425004+0000", 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0899999999999999e-06 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.425380+0000", 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00037596899999999998 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.425511+0000", 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013039900000000001 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5", 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.425674+0000", 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016311900000000001 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.428766+0000", 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0030916160000000002 2024-03-26T05:12:30.436 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:113531 9.d 9:b43e909c:::benchmark_data_smithi080_136801_object113530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:30.425019+0000", 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.014439868999999999, 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014525549, 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:12:30.437 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 113531 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.425019+0000", 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.425019+0000", 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.425019+0000", 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4799999999999999e-07 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.425042+0000", 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2498e-05 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.425049+0000", 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5020000000000003e-06 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.425055+0000", 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0100000000000001e-06 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.428829+0000", 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0037739929999999998 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.428989+0000", 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000160023 2024-03-26T05:12:30.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5", 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.429189+0000", 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000199636 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.438988+0000", 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0097990760000000003 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.441 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:113535 9.4 9:2e22b4a9:::benchmark_data_smithi080_136801_object113534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:30.430332+0000", 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0091264889999999998, 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0092699779999999999, 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 113535 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.442 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430332+0000", 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430332+0000", 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430334+0000", 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.708e-06 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:30.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430356+0000", 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2263e-05 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430372+0000", 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5659e-05 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430450+0000", 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8487999999999998e-05 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:30.444 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430506+0000", 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5614e-05 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430624+0000", 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118212 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,4", 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430757+0000", 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013273199999999999 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:30.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.436519+0000", 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0057627329999999999 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.438902+0000", 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0023829319999999999 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.25268.0:113537 9.6 e294/265 9:66fd8a7d:::benchmark_data_smithi080_136801_object113536:head v 294'6993, mlcod=294'6991)", 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:30.430583+0000", 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0088758640000000007, 2024-03-26T05:12:30.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0090809830000000008, 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430583+0000", 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430583+0000", 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430584+0000", 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.16e-07 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430814+0000", 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00023097800000000001 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430855+0000", 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0976999999999997e-05 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:30.449 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.430865+0000", 2024-03-26T05:12:30.449 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2390000000000008e-06 2024-03-26T05:12:30.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:30.449 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.431026+0000", 2024-03-26T05:12:30.449 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016134600000000001 2024-03-26T05:12:30.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:30.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:30.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:30.449 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:30.431046+0000", 2024-03-26T05:12:30.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9896000000000001e-05 2024-03-26T05:12:30.450 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:30.450 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:30.450 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:30.450 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:30.450 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:12:30.450 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 4 2024-03-26T05:12:30.450 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:12:30.523 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:30.553 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:30.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:30.654 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:30.688 INFO:teuthology.orchestra.run.smithi132.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T05:12:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:30.703 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:30.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:30.747 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 82 16 113878 113862 86.7708 62.375 0.0048111 0.0115199 2024-03-26T05:12:30.804 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:30.835 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:30.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:30.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:30.966 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:30.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:30.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:31.067 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:31.111 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:31.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:31.161 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T05:12:31.213 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:31.253 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:31 smithi132 ceph-mon[33500]: pgmap v552: 148 pgs: 98 active+clean, 8 active+undersized, 2 active+undersized+degraded, 2 active+clean+scrubbing+deep, 11 stale+active+clean, 1 active+clean+scrubbing, 26 peering; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 34 MiB/s wr, 539 op/s; 7088/330690 objects degraded (2.143%) 2024-03-26T05:12:31.263 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:31.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:31.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:31.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:31 smithi080 ceph-mon[29691]: pgmap v552: 148 pgs: 98 active+clean, 8 active+undersized, 2 active+undersized+degraded, 2 active+clean+scrubbing+deep, 11 stale+active+clean, 1 active+clean+scrubbing, 26 peering; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 34 MiB/s wr, 539 op/s; 7088/330690 objects degraded (2.143%) 2024-03-26T05:12:31.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:31 smithi080 ceph-mon[40404]: pgmap v552: 148 pgs: 98 active+clean, 8 active+undersized, 2 active+undersized+degraded, 2 active+clean+scrubbing+deep, 11 stale+active+clean, 1 active+clean+scrubbing, 26 peering; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 34 MiB/s wr, 539 op/s; 7088/330690 objects degraded (2.143%) 2024-03-26T05:12:31.394 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:31.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:31.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:31.496 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:31.523 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:31.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:31.625 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:31.657 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:31.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:31.747 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 83 16 114858 114842 86.4632 61.25 0.00477858 0.0115585 2024-03-26T05:12:31.758 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:12:31.759 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:31.790 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:31.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:31.892 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:31.931 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:31.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:32.033 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:32.075 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:32.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:32.177 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:32.215 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:32.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:32.317 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:32.346 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:32.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:32.447 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:32.476 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:32.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:32.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:32.560 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:32.577 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:32.578 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:32 smithi080 ceph-mon[40404]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-26T05:12:32.604 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:32.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:32.707 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:32.749 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 84 16 115800 115784 86.1338 58.875 0.00549419 0.0116023 2024-03-26T05:12:32.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:32.750 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:32.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:32.753 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:32.852 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:32.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:32 smithi080 ceph-mon[29691]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-26T05:12:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:32.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:32.982 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:33.012 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:33.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:33.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:32 smithi132 ceph-mon[33500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-26T05:12:33.114 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:33.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:33.153 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:33.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:33.254 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:33.286 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:33.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:33.388 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:33.419 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:12:33.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:33.426 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:33.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:33.476 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:12:33.513 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:12:33.528 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:33.554 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:12:33.564 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:33.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:33.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:33.600 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:12:33.635 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:12:33.667 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:33.670 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:12:33.672 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:12:33.672 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:12:33.672 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:12:33.672 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:12:33.672 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:12:33.672 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:12:33.672 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:12:33.672 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:12:33.698 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:33.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:33.749 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 85 16 116767 116751 85.8314 60.4375 0.00593593 0.0116447 2024-03-26T05:12:33.800 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:33.830 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:33.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:33.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:33.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:33 smithi080 ceph-mon[40404]: pgmap v553: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 999 op/s; 64039/341727 objects degraded (18.740%); 7.9 KiB/s, 0 objects/s recovering 2024-03-26T05:12:33.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:33 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1435891254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:33.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:33 smithi080 ceph-mon[29691]: pgmap v553: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 999 op/s; 64039/341727 objects degraded (18.740%); 7.9 KiB/s, 0 objects/s recovering 2024-03-26T05:12:33.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:33 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1435891254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:33.932 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:33.975 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:33.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:34.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:33 smithi132 ceph-mon[33500]: pgmap v553: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 999 op/s; 64039/341727 objects degraded (18.740%); 7.9 KiB/s, 0 objects/s recovering 2024-03-26T05:12:34.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:33 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1435891254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:34.077 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:34.110 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:34.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:34.212 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:34.241 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:34.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:34.302 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:12:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:12:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:12:34.342 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:34.370 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:34.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:34.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:34.472 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:34.503 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:34.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:12:34.511 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:12:34.603 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:34.643 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:34.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:34.745 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:34.748 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 86 16 117832 117816 85.6073 66.5625 0.0133728 0.0116759 2024-03-26T05:12:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:34.782 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:34.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:34.883 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:34.918 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:34.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:35.020 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:35.051 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:35.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:35.083 INFO:teuthology.orchestra.run.smithi132.stdout:9.3 2024-03-26T05:12:35.083 INFO:teuthology.orchestra.run.smithi132.stdout:6.9 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:6.6 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:6.3 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:9.a 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:6.1c 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:6.18 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:6.16 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:6.12 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:5.9 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:5.6 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:6.14 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:5.4 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:4.0 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:6.15 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:4.a 2024-03-26T05:12:35.084 INFO:teuthology.orchestra.run.smithi132.stdout:3.f 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:4.8 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:3.9 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:4.15 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:3.11 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:3.8 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:6.1b 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:3.7 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:6.5 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:3.e 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:4.16 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:4.1e 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:5.10 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:5.11 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:3.16 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:3.1b 2024-03-26T05:12:35.085 INFO:teuthology.orchestra.run.smithi132.stdout:3.15 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:3.13 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:4.5 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:5.1f 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:1.0 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:2.0 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:3.d 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:3.12 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:9.e 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:4.3 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:4.11 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:5.19 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:9.5 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:5.d 2024-03-26T05:12:35.086 INFO:teuthology.orchestra.run.smithi132.stdout:4.17 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:3.14 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:4.14 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:5.1e 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:5.14 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:9.d 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:9.7 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:5.17 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:5.1 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:9.9 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:9.1 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:5.1b 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:9.c 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:6.8 2024-03-26T05:12:35.087 INFO:teuthology.orchestra.run.smithi132.stdout:5.1a 2024-03-26T05:12:35.153 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:35.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:35.200 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:35.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:35.301 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:35.331 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:35.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:35.432 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:35.457 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:35.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:35.558 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:35.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:35.599 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:35.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:35.700 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:35.728 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:35.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:35.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:35.749 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 87 16 118831 118815 85.341 62.4375 0.0194549 0.0117122 2024-03-26T05:12:35.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:35 smithi132 ceph-mon[33500]: pgmap v554: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 43 MiB/s wr, 680 op/s; 64039/341727 objects degraded (18.740%); 7.8 KiB/s, 0 objects/s recovering 2024-03-26T05:12:35.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:12:35.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:12:35.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:12:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:12:35.830 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:35.851 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:35 smithi080 ceph-mon[40404]: pgmap v554: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 43 MiB/s wr, 680 op/s; 64039/341727 objects degraded (18.740%); 7.8 KiB/s, 0 objects/s recovering 2024-03-26T05:12:35.851 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:12:35.851 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:12:35.851 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:35 smithi080 ceph-mon[29691]: pgmap v554: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 43 MiB/s wr, 680 op/s; 64039/341727 objects degraded (18.740%); 7.8 KiB/s, 0 objects/s recovering 2024-03-26T05:12:35.852 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:12:35.852 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:12:35.859 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:35.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:35.961 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:36.009 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:36.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:36.110 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:36.155 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:36.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:36.257 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:36.286 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:36.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:36.372 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -n osd.7 --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 5.1a --file /var/log/ceph/exp.5.1a.7 2024-03-26T05:12:36.388 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:36.444 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:36.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:36.512 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:12:36.545 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:36.575 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:36.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:36.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:36.613 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/osd.7/config 2024-03-26T05:12:36.677 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:36.681 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:36.681 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:36 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 64039/341727 objects degraded (18.740%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:36.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:36.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:36 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 64039/341727 objects degraded (18.740%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:36.733 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:36.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:36.749 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 88 16 119808 119792 85.0647 61.0625 0.0257244 0.0117486 2024-03-26T05:12:36.835 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:36.865 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:36.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:36.917 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:36.918 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:36 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 64039/341727 objects degraded (18.740%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:36.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:36.993 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:36.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:36.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:37.095 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:37.143 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:37.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:37.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:37.232 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:37.245 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:37.268 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:37.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:37.270 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:12:37.270 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:12:37.371 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:37.400 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:37.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:37.502 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:37.547 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:37.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:37.622 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:12:37.649 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:37.695 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:37.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:37.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:37.749 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 89 16 120605 120589 84.6686 49.8125 0.0101129 0.0118059 2024-03-26T05:12:37.797 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:37.840 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:37.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:37.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:37.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:37 smithi080 ceph-mon[40404]: pgmap v555: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 55 MiB/s wr, 881 op/s; 66840/356565 objects degraded (18.746%); 6.4 KiB/s, 0 objects/s recovering 2024-03-26T05:12:37.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:37 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1819881280' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:12:37.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:37 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:12:37.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:37 smithi080 ceph-mon[29691]: pgmap v555: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 55 MiB/s wr, 881 op/s; 66840/356565 objects degraded (18.746%); 6.4 KiB/s, 0 objects/s recovering 2024-03-26T05:12:37.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:37 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1819881280' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:12:37.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:37 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:12:37.942 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:37.983 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:37.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:38.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:37 smithi132 ceph-mon[33500]: pgmap v555: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 55 MiB/s wr, 881 op/s; 66840/356565 objects degraded (18.746%); 6.4 KiB/s, 0 objects/s recovering 2024-03-26T05:12:38.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:37 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1819881280' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:12:38.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:37 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:12:38.085 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:38.120 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:38.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:38.222 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:38.249 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:38.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:38.351 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:38.379 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:38.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:38.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:38.480 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:38.523 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:38.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:38.625 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:38.674 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:38.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:38.750 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Total time run: 90.0092 2024-03-26T05:12:38.750 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Total writes made: 121580 2024-03-26T05:12:38.750 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Write size: 65536 2024-03-26T05:12:38.750 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Object size: 65536 2024-03-26T05:12:38.750 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Bandwidth (MB/sec): 84.4219 2024-03-26T05:12:38.750 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev Bandwidth: 34.5739 2024-03-26T05:12:38.750 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max bandwidth (MB/sec): 211.25 2024-03-26T05:12:38.750 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min bandwidth (MB/sec): 2 2024-03-26T05:12:38.751 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Average IOPS: 1350 2024-03-26T05:12:38.751 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev IOPS: 553.183 2024-03-26T05:12:38.751 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max IOPS: 3380 2024-03-26T05:12:38.751 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min IOPS: 32 2024-03-26T05:12:38.751 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Average Latency(s): 0.0118411 2024-03-26T05:12:38.751 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev Latency(s): 0.0168574 2024-03-26T05:12:38.751 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max latency(s): 1.08124 2024-03-26T05:12:38.751 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min latency(s): 0.00115197 2024-03-26T05:12:38.754 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T05:12:38.775 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:38.811 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:38.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:38.913 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:38.951 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:38.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:38.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:39.002 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:38 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:12:39.003 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:38 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:12:39.003 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:38 smithi080 ceph-mon[40404]: osdmap e295: 8 total, 7 up, 8 in 2024-03-26T05:12:39.003 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:38 smithi080 ceph-mon[40404]: pgmap v557: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 55 MiB/s wr, 881 op/s; 66840/356565 objects degraded (18.746%); 6.4 KiB/s, 0 objects/s recovering 2024-03-26T05:12:39.003 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:38 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:12:39.003 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:38 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:12:39.003 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:38 smithi080 ceph-mon[29691]: osdmap e295: 8 total, 7 up, 8 in 2024-03-26T05:12:39.003 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:38 smithi080 ceph-mon[29691]: pgmap v557: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 55 MiB/s wr, 881 op/s; 66840/356565 objects degraded (18.746%); 6.4 KiB/s, 0 objects/s recovering 2024-03-26T05:12:39.003 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:38 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:12:39.004 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:38 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:12:39.004 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:38 smithi132 ceph-mon[33500]: osdmap e295: 8 total, 7 up, 8 in 2024-03-26T05:12:39.004 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:38 smithi132 ceph-mon[33500]: pgmap v557: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 55 MiB/s wr, 881 op/s; 66840/356565 objects degraded (18.746%); 6.4 KiB/s, 0 objects/s recovering 2024-03-26T05:12:39.050 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T05:12:39.052 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:39.061 INFO:teuthology.orchestra.run.smithi132.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T05:12:39.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:39.091 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:39.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:39.192 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:39.217 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:39.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:39.319 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:39.348 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:39.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:39.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:39.450 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:39.504 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:39.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:39.606 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:39.648 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:39.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:39.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:39.750 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:39.796 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:39.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:39.809 INFO:teuthology.orchestra.run.smithi132.stderr:Exporting 5.1a info 5.1a( empty local-lis/les=71/72 n=0 ec=71/71 lis/c=71/71 les/c/f=72/72/0 sis=71) 2024-03-26T05:12:39.812 INFO:teuthology.orchestra.run.smithi132.stderr:Export successful 2024-03-26T05:12:39.898 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:39.933 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:39.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:40.035 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:40.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:39 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:12:40.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:39 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:12:40.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:39 smithi132 ceph-mon[33500]: osdmap e296: 8 total, 7 up, 8 in 2024-03-26T05:12:40.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:39 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:12:40.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:39 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:12:40.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:40.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:40.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:40.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:39 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:12:40.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:39 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:12:40.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:39 smithi080 ceph-mon[40404]: osdmap e296: 8 total, 7 up, 8 in 2024-03-26T05:12:40.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:39 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:12:40.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:39 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:12:40.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:39 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:12:40.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:39 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:12:40.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:39 smithi080 ceph-mon[29691]: osdmap e296: 8 total, 7 up, 8 in 2024-03-26T05:12:40.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:39 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:12:40.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:39 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:12:40.170 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:40.207 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:40.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:40.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:40.309 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:40.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:40.351 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:40.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:40.452 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:40.486 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:40.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:40.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:40.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:40.616 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:40.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:40.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:40.718 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:40.753 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:40.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:40.853 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:40.855 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:40.883 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:40.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:40.985 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:40.991 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:40 smithi080 ceph-mon[29691]: osdmap e297: 8 total, 7 up, 8 in 2024-03-26T05:12:40.991 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:40 smithi080 ceph-mon[29691]: pgmap v560: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 54 MiB/s wr, 876 op/s; 66979/356982 objects degraded (18.763%) 2024-03-26T05:12:40.991 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:40 smithi080 ceph-mon[40404]: osdmap e297: 8 total, 7 up, 8 in 2024-03-26T05:12:40.991 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:40 smithi080 ceph-mon[40404]: pgmap v560: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 54 MiB/s wr, 876 op/s; 66979/356982 objects degraded (18.763%) 2024-03-26T05:12:41.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:41.028 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:41.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:41.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:40 smithi132 ceph-mon[33500]: osdmap e297: 8 total, 7 up, 8 in 2024-03-26T05:12:41.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:40 smithi132 ceph-mon[33500]: pgmap v560: 148 pgs: 86 active+clean, 33 active+undersized, 1 active+clean+scrubbing+deep, 28 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 54 MiB/s wr, 876 op/s; 66979/356982 objects degraded (18.763%) 2024-03-26T05:12:41.128 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:41.164 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:41.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:41.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:41.266 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:41.313 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:41.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:41.414 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:41.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:41.445 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:41.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:41.546 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:41.574 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:41.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:41.675 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:41.689 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -n osd.7 --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 5.1a 2024-03-26T05:12:41.706 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:41.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:41.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:41.808 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:41.847 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:41.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:41.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:41.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:41 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 66979/356982 objects degraded (18.763%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:41.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:41 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 66979/356982 objects degraded (18.763%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:41.949 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:41.951 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/osd.7/config 2024-03-26T05:12:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:41.995 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:41.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:42.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:41 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 66979/356982 objects degraded (18.763%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:42.096 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:42.140 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:42.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:42.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:42.186 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:12:42.229 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:12:42.243 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:42.271 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:12:42.281 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:42.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:42.312 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:12:42.345 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:12:42.355 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:12:42.383 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:42.400 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:12:42.431 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:42.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:42.438 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:12:42.445 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:12:42.445 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:12:42.445 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:12:42.445 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:12:42.445 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:12:42.445 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:12:42.445 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:12:42.445 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:12:42.534 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:42.582 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:42.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:42.683 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:42.730 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:42.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:42.832 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:42.861 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:42.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:42.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:42 smithi080 ceph-mon[29691]: pgmap v561: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 7.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 31 MiB/s wr, 888 op/s; 67156/358317 objects degraded (18.742%) 2024-03-26T05:12:42.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:42 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2960273919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:42.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:42 smithi080 ceph-mon[40404]: pgmap v561: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 7.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 31 MiB/s wr, 888 op/s; 67156/358317 objects degraded (18.742%) 2024-03-26T05:12:42.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:42 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2960273919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:42.963 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:43.012 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:43.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:43.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:43.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:42 smithi132 ceph-mon[33500]: pgmap v561: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 7.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 31 MiB/s wr, 888 op/s; 67156/358317 objects degraded (18.742%) 2024-03-26T05:12:43.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:42 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2960273919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:43.114 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:43.151 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:43.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:12:43.185 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:12:43.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:43.283 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:43.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:43.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:43.385 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:43.413 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:43.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:43.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:43.477 INFO:teuthology.orchestra.run.smithi132.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T05:12:43.516 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:43.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:43.547 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:43.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:43.648 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:43.689 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:43.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:43.791 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:43.815 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:43.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:43.916 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:43.952 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:43.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:44.054 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:44.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:44.091 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:44.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:44.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:44.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:44.227 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:44.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:44.227 INFO:teuthology.orchestra.run.smithi132.stdout: marking collection for removal 2024-03-26T05:12:44.227 INFO:teuthology.orchestra.run.smithi132.stdout:setting '_remove' omap key 2024-03-26T05:12:44.227 INFO:teuthology.orchestra.run.smithi132.stdout:finish_remove_pgs 5.1a_head removing 5.1a 2024-03-26T05:12:44.228 INFO:teuthology.orchestra.run.smithi132.stdout:Remove successful 2024-03-26T05:12:44.327 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:44.364 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:44.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:44.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:44.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:12:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:12:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:12:44.466 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:44.504 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:44.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:44.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:44.606 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:44.665 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:44.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:44.768 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:44.802 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:44.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:44.904 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:44.934 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:44.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:45.036 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:45.070 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:45.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:45.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:45.171 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:45.186 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:12:45.200 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:45.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:45.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:45.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:45 smithi132 ceph-mon[33500]: pgmap v562: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 7.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 966 B/s rd, 29 MiB/s wr, 839 op/s; 67156/358317 objects degraded (18.742%) 2024-03-26T05:12:45.302 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:45.344 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:45.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:45.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:45 smithi080 ceph-mon[29691]: pgmap v562: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 7.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 966 B/s rd, 29 MiB/s wr, 839 op/s; 67156/358317 objects degraded (18.742%) 2024-03-26T05:12:45.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:45 smithi080 ceph-mon[40404]: pgmap v562: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 7.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 966 B/s rd, 29 MiB/s wr, 839 op/s; 67156/358317 objects degraded (18.742%) 2024-03-26T05:12:45.436 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -n osd.7 --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.1a.7 2024-03-26T05:12:45.444 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:45.498 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:45.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:45.601 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:45.656 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:45.657 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/osd.7/config 2024-03-26T05:12:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:45.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:45.759 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:45.784 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:12:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:12:45.786 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:45.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:45.888 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:45.907 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:45.916 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:45.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:45.994 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:12:45.994 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:12:45.994 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:12:45.994 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:12:45.994 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:45.994 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206018 7.b 7:d4295363:::benchmark_data_smithi080_80848_object86519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.872555+0000", 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 430.12652361200003, 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.670752888, 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206018 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:45.995 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872555+0000", 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872555+0000", 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872556+0000", 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3629999999999999e-06 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:45.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872562+0000", 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5949999999999998e-06 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872577+0000", 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5183e-05 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872587+0000", 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4620000000000002e-06 2024-03-26T05:12:45.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:45.998 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:45.998 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:45.998 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872681+0000", 2024-03-26T05:12:45.998 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4724999999999999e-05 2024-03-26T05:12:45.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:45.998 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:45.998 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:45.998 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.873453+0000", 2024-03-26T05:12:45.998 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00077158499999999998 2024-03-26T05:12:45.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:45.998 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:45.998 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.873565+0000", 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011171900000000001 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.900679+0000", 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.027114513 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.900691+0000", 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2386e-05 2024-03-26T05:12:45.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392310+0000", 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.491618318 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392323+0000", 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2852e-05 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.543230+0000", 2024-03-26T05:12:46.001 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1509076469999999 2024-03-26T05:12:46.001 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.001 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.001 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.001 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.543308+0000", 2024-03-26T05:12:46.001 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7540000000000006e-05 2024-03-26T05:12:46.001 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.001 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.001 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.001 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.001 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.001 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206009 7.0 7:0aae643b:::benchmark_data_smithi080_80848_object86510:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.886333+0000", 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 430.11274541699999, 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6852875250000001, 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206009 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.002 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886333+0000", 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886333+0000", 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886335+0000", 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2790000000000001e-06 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.003 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886341+0000", 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6509999999999998e-06 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886354+0000", 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2736e-05 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.886364+0000", 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0159999999999999e-05 2024-03-26T05:12:46.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.887501+0000", 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0011366639999999999 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.887510+0000", 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8609999999999993e-06 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.394783+0000", 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.50727263899999997 2024-03-26T05:12:46.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.394988+0000", 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020464800000000001 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.569805+0000", 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1748175009999999 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.006 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.571621+0000", 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001815386 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206029 7.0 7:0428147a:::benchmark_data_smithi080_80848_object86530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.897565+0000", 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 430.10151339200002, 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7163614869999999, 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.007 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206029 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897565+0000", 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897565+0000", 2024-03-26T05:12:46.008 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897566+0000", 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.319e-06 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897569+0000", 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.7920000000000001e-06 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897578+0000", 2024-03-26T05:12:46.009 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4840000000000004e-06 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897585+0000", 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5750000000000003e-06 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.898208+0000", 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00062269599999999995 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.010 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.898218+0000", 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0395e-05 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395017+0000", 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.496798144 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395026+0000", 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6509999999999996e-06 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.011 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.571919+0000", 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1768926829999999 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.572211+0000", 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00029205600000000001 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.572422+0000", 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021075299999999999 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.012 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591634+0000", 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.019212002999999998 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596999+0000", 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0053655129999999997 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597046+0000", 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6810000000000001e-05 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.013 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613723+0000", 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016677112000000001 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613791+0000", 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8207999999999995e-05 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613927+0000", 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013529300000000001 2024-03-26T05:12:46.014 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206032 7.b 7:db699848:::benchmark_data_smithi080_80848_object86533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.907812+0000", 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 430.09126608600002, 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.654457345, 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.015 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206032 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907812+0000", 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907812+0000", 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.016 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907814+0000", 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7710000000000001e-06 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907821+0000", 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0049999999999998e-06 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907845+0000", 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4151e-05 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.017 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.907854+0000", 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8279999999999992e-06 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.909278+0000", 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014236120000000001 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.909289+0000", 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1054e-05 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.018 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392352+0000", 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.48306306100000002 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392361+0000", 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9190000000000005e-06 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.543561+0000", 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.151200486 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.019 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.543686+0000", 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012433600000000001 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,5,6", 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.544119+0000", 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00043299000000000001 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558280+0000", 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014161366999999999 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.020 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558330+0000", 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0148000000000002e-05 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558377+0000", 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6708000000000001e-05 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.561932+0000", 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0035554779999999999 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.022 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:46.023 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.023 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562152+0000", 2024-03-26T05:12:46.023 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021959599999999999 2024-03-26T05:12:46.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.023 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.023 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.023 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562270+0000", 2024-03-26T05:12:46.023 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117835 2024-03-26T05:12:46.023 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.024 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.024 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.024 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.024 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.024 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206042 7.0 7:0cf7d89a:::benchmark_data_smithi080_80848_object86543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:12:46.024 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.917369+0000", 2024-03-26T05:12:46.024 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 430.08170958800002, 2024-03-26T05:12:46.024 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.696765531, 2024-03-26T05:12:46.024 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.024 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.024 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.024 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206042 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917369+0000", 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917369+0000", 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.026 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.026 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917371+0000", 2024-03-26T05:12:46.026 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.232e-06 2024-03-26T05:12:46.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.026 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.026 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.026 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917379+0000", 2024-03-26T05:12:46.026 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2479999999999996e-06 2024-03-26T05:12:46.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.026 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.026 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.026 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917402+0000", 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2521999999999999e-05 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917444+0000", 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1974999999999997e-05 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917648+0000", 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020391800000000001 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.027 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917662+0000", 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4052e-05 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395054+0000", 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.47739227899999997 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395062+0000", 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2689999999999999e-06 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.586328+0000", 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.191265663 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.587220+0000", 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00089233699999999999 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.587549+0000", 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00032828700000000002 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597064+0000", 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0095153470000000004 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597117+0000", 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2428e-05 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613547+0000", 2024-03-26T05:12:46.030 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016430177000000001 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613969+0000", 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00042261700000000002 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614022+0000", 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2505e-05 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.031 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614134+0000", 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000112675 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206043 7.0 7:0c4ab3a4:::benchmark_data_smithi080_80848_object86544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.917482+0000", 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 430.08159689299998, 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6968406790000001, 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.032 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206043 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917482+0000", 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.033 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917482+0000", 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917482+0000", 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7299999999999996e-07 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917487+0000", 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5020000000000003e-06 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.034 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917495+0000", 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3049999999999993e-06 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917501+0000", 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1340000000000001e-06 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917687+0000", 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018563299999999999 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917694+0000", 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.708e-06 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395094+0000", 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.47740021100000002 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395102+0000", 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2160000000000006e-06 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.036 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591668+0000", 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1965658779999999 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591783+0000", 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011553 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591918+0000", 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000134303 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.037 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597070+0000", 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051523330000000003 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613593+0000", 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016523013 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613637+0000", 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4057999999999997e-05 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614180+0000", 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00054313899999999999 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614232+0000", 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2014e-05 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614322+0000", 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0129000000000004e-05 2024-03-26T05:12:46.039 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206045 7.0 7:0d078525:::benchmark_data_smithi080_80848_object86546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.919727+0000", 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 430.079351159, 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.707372347, 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.040 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206045 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919727+0000", 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919727+0000", 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.041 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919729+0000", 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9149999999999999e-06 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919737+0000", 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.926e-06 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919757+0000", 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0190000000000002e-05 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.042 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919767+0000", 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9899999999999992e-06 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919857+0000", 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9819000000000002e-05 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.919869+0000", 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2241e-05 2024-03-26T05:12:46.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395136+0000", 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.47526629399999998 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395144+0000", 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1950000000000003e-06 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.594264+0000", 2024-03-26T05:12:46.044 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.199119764 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.594373+0000", 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010950699999999999 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.594495+0000", 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000122044 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597075+0000", 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002579951 2024-03-26T05:12:46.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.613680+0000", 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016604739 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614358+0000", 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00067832500000000002 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.626740+0000", 2024-03-26T05:12:46.046 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.012381614000000001 2024-03-26T05:12:46.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.047 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.047 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.626798+0000", 2024-03-26T05:12:46.047 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8226000000000001e-05 2024-03-26T05:12:46.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.047 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.047 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.627100+0000", 2024-03-26T05:12:46.047 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00030160700000000001 2024-03-26T05:12:46.047 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206049 7.b 7:d42fcd17:::benchmark_data_smithi080_80848_object86550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.923737+0000", 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 430.07534116800002, 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.638929509, 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206049 2024-03-26T05:12:46.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923737+0000", 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923737+0000", 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923739+0000", 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9750000000000001e-06 2024-03-26T05:12:46.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.050 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.050 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.050 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923747+0000", 2024-03-26T05:12:46.050 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0439999999999996e-06 2024-03-26T05:12:46.050 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.050 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.050 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.050 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923781+0000", 2024-03-26T05:12:46.050 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4029000000000002e-05 2024-03-26T05:12:46.050 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.050 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.050 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923793+0000", 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1834999999999999e-05 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923882+0000", 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8941000000000001e-05 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.923895+0000", 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2989e-05 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.051 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392396+0000", 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.46850114399999998 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.392404+0000", 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9589999999999995e-06 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.554545+0000", 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.162140891 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.052 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.554683+0000", 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013746099999999999 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,5,6", 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.555704+0000", 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001020927 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558652+0000", 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0029487879999999999 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.053 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.054 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558667+0000", 2024-03-26T05:12:46.054 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4798e-05 2024-03-26T05:12:46.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.054 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.054 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.054 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.558710+0000", 2024-03-26T05:12:46.054 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2973999999999999e-05 2024-03-26T05:12:46.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.054 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.054 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.054 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562312+0000", 2024-03-26T05:12:46.054 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036020919999999999 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562556+0000", 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000244057 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562667+0000", 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000110605 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206053 7.0 7:08b24a04:::benchmark_data_smithi080_80848_object86554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.926223+0000", 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 430.07285515500001, 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7011446530000001, 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206053 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926223+0000", 2024-03-26T05:12:46.056 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926223+0000", 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926225+0000", 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5409999999999999e-06 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926230+0000", 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5219999999999998e-06 2024-03-26T05:12:46.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926246+0000", 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5339999999999999e-05 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926255+0000", 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7110000000000001e-06 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926346+0000", 2024-03-26T05:12:46.058 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1246e-05 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.926354+0000", 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0260000000000007e-06 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395168+0000", 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.46881425399999999 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.395176+0000", 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0439999999999996e-06 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.059 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597154+0000", 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2019782939999999 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597287+0000", 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000132848 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597420+0000", 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000132917 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.060 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614394+0000", 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016974182000000001 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614560+0000", 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016532100000000001 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.619983+0000", 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0054232200000000003 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.061 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.627148+0000", 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0071655720000000003 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.627250+0000", 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010138599999999999 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.627368+0000", 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118229 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.062 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.063 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.063 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.063 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236543 7.0 7:02068409:::benchmark_data_smithi080_80848_object117044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:12:46.064 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:46 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.452286+0000", 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 394.54679205799999, 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.001309335, 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236543 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.065 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452286+0000", 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452286+0000", 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452288+0000", 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4249999999999999e-06 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.066 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452293+0000", 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8969999999999999e-06 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452305+0000", 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2170999999999999e-05 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452313+0000", 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6329999999999996e-06 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.067 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452604+0000", 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000290943 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.452613+0000", 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0860000000000008e-06 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.447196+0000", 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99458339399999995 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.068 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.447355+0000", 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015935600000000001 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.447444+0000", 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8975999999999996e-05 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.449713+0000", 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022690480000000001 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.069 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.449794+0000", 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1021999999999998e-05 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453503+0000", 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0037082650000000001 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453557+0000", 2024-03-26T05:12:46.070 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4512000000000003e-05 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453596+0000", 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8606999999999998e-05 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236550 7.0 7:0e84af01:::benchmark_data_smithi080_80848_object117051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.460859+0000", 2024-03-26T05:12:46.071 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 394.53821991299998, 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99296332399999998, 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236550 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460859+0000", 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.072 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460859+0000", 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460860+0000", 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1090000000000001e-06 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.073 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460865+0000", 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5389999999999998e-06 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460881+0000", 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6152999999999999e-05 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460889+0000", 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7100000000000007e-06 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.074 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460950+0000", 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1356e-05 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.460959+0000", 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6209999999999993e-06 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.449848+0000", 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98888869099999999 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.075 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.449965+0000", 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117137 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.450042+0000", 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7497999999999996e-05 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453686+0000", 2024-03-26T05:12:46.076 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036440470000000001 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453744+0000", 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7787000000000001e-05 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453769+0000", 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4307e-05 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.077 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.078 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453799+0000", 2024-03-26T05:12:46.078 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.0151999999999999e-05 2024-03-26T05:12:46.078 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.078 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.078 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.078 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453822+0000", 2024-03-26T05:12:46.078 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3217000000000001e-05 2024-03-26T05:12:46.078 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.078 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.078 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.078 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.078 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236558 7.0 7:0d5d8d1f:::benchmark_data_smithi080_80848_object117059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.474483+0000", 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 394.52459519799999, 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97948798000000004, 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236558 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.079 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474483+0000", 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474483+0000", 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474485+0000", 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.412e-06 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.080 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474488+0000", 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6160000000000002e-06 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474497+0000", 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9619999999999999e-06 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474504+0000", 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4899999999999997e-06 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.081 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474602+0000", 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8004999999999994e-05 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474611+0000", 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6840000000000002e-06 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.451610+0000", 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97699924800000004 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.082 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.451703+0000", 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2864999999999999e-05 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.451781+0000", 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8761e-05 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453735+0000", 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0019537589999999998 2024-03-26T05:12:46.083 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453749+0000", 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4021e-05 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453879+0000", 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000129605 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453931+0000", 2024-03-26T05:12:46.084 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1987999999999998e-05 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453971+0000", 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0564e-05 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236560 7.0 7:0afff4a7:::benchmark_data_smithi080_80848_object117061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.474521+0000", 2024-03-26T05:12:46.085 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 394.52455745999998, 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98418719899999996, 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236560 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474521+0000", 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.086 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474521+0000", 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474521+0000", 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1300000000000001e-07 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474524+0000", 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.289e-06 2024-03-26T05:12:46.087 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474530+0000", 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2160000000000003e-06 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474535+0000", 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.011e-06 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474632+0000", 2024-03-26T05:12:46.088 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6506000000000001e-05 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.474637+0000", 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3539999999999999e-06 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.454019+0000", 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97938261800000004 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.089 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.454151+0000", 2024-03-26T05:12:46.090 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013146800000000001 2024-03-26T05:12:46.090 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.090 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.090 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:12:46.090 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.454275+0000", 2024-03-26T05:12:46.090 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000123635 2024-03-26T05:12:46.090 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.090 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.090 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.090 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458530+0000", 2024-03-26T05:12:46.090 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0042555229999999998 2024-03-26T05:12:46.090 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458574+0000", 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4110999999999998e-05 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458589+0000", 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.505e-05 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458635+0000", 2024-03-26T05:12:46.091 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5636000000000001e-05 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458708+0000", 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3368999999999995e-05 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236566 7.0 7:0070f26d:::benchmark_data_smithi080_80848_object117067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.480345+0000", 2024-03-26T05:12:46.092 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 394.51873371900001, 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98387816699999997, 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236566 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480345+0000", 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.093 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480345+0000", 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480347+0000", 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6750000000000001e-06 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.094 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480351+0000", 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9309999999999999e-06 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480367+0000", 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5599000000000001e-05 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480376+0000", 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7949999999999996e-06 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.095 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480438+0000", 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2456e-05 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480448+0000", 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4320000000000005e-06 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.456042+0000", 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97559416799999998 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.456124+0000", 2024-03-26T05:12:46.096 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2108999999999996e-05 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.456202+0000", 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8367000000000004e-05 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458755+0000", 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0025522359999999998 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458765+0000", 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0862e-05 2024-03-26T05:12:46.097 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464122+0000", 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0053569200000000003 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464175+0000", 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2556999999999998e-05 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.098 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464223+0000", 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8059999999999997e-05 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236568 7.0 7:07c96239:::benchmark_data_smithi080_80848_object117069:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.480947+0000", 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 394.51813115099998, 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98352315300000004, 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.099 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236568 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480947+0000", 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480947+0000", 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.100 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480948+0000", 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0470000000000001e-06 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480953+0000", 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7589999999999997e-06 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480969+0000", 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5812e-05 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.101 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.480977+0000", 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7659999999999998e-06 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.481028+0000", 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0794999999999997e-05 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.481035+0000", 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3629999999999996e-06 2024-03-26T05:12:46.102 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.103 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.103 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.103 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458791+0000", 2024-03-26T05:12:46.103 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97775615100000002 2024-03-26T05:12:46.103 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.103 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.103 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.103 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.458955+0000", 2024-03-26T05:12:46.103 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000164411 2024-03-26T05:12:46.103 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.103 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.103 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.459033+0000", 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7306999999999997e-05 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464267+0000", 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0052343420000000003 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464329+0000", 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1607999999999997e-05 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.104 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.105 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.105 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464386+0000", 2024-03-26T05:12:46.105 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7380000000000003e-05 2024-03-26T05:12:46.105 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.105 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.105 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.105 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464431+0000", 2024-03-26T05:12:46.105 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5321999999999997e-05 2024-03-26T05:12:46.105 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.105 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.105 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.105 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464471+0000", 2024-03-26T05:12:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:46.106 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:46.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:46.107 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.909e-05 2024-03-26T05:12:46.107 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.107 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.107 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.107 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.107 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236570 7.0 7:03c0468e:::benchmark_data_smithi080_80848_object117071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.482326+0000", 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 394.51675282600002, 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98740077800000003, 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236570 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.108 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482326+0000", 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482326+0000", 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482327+0000", 2024-03-26T05:12:46.109 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.522e-06 2024-03-26T05:12:46.110 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.110 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.110 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.110 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482332+0000", 2024-03-26T05:12:46.110 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8600000000000001e-06 2024-03-26T05:12:46.110 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.110 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.110 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.110 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482347+0000", 2024-03-26T05:12:46.110 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4577e-05 2024-03-26T05:12:46.110 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.110 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482354+0000", 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7160000000000003e-06 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482406+0000", 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1476e-05 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.482415+0000", 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2310000000000002e-06 2024-03-26T05:12:46.111 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.461304+0000", 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97888937600000003 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.461434+0000", 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012907000000000001 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:12:46.112 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.461601+0000", 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000167589 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464375+0000", 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027743030000000001 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.464479+0000", 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103106 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.113 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.114 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469623+0000", 2024-03-26T05:12:46.114 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051441489999999998 2024-03-26T05:12:46.114 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.114 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.114 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.114 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469676+0000", 2024-03-26T05:12:46.114 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3794e-05 2024-03-26T05:12:46.114 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.114 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.114 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.114 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.469726+0000", 2024-03-26T05:12:46.114 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0009000000000002e-05 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236654 7.0 7:028e4924:::benchmark_data_smithi080_80848_object117155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:11.555854+0000", 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 394.44322463100002, 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93320793999999996, 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:12:46.115 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236654 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555854+0000", 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555854+0000", 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.116 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555856+0000", 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6899999999999999e-06 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555861+0000", 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3449999999999996e-06 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.117 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555877+0000", 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5977e-05 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555885+0000", 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9959999999999993e-06 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555943+0000", 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8131999999999998e-05 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.118 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:11.555951+0000", 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9440000000000005e-06 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.472491+0000", 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.91653987999999997 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.472604+0000", 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000113354 2024-03-26T05:12:46.119 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.472720+0000", 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115812 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.483540+0000", 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010820412 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.483703+0000", 2024-03-26T05:12:46.120 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000162645 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.488951+0000", 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0052477599999999998 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489011+0000", 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9954000000000002e-05 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.121 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489062+0000", 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1038999999999997e-05 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142102 8.a 8:5d371889:::benchmark_data_smithi080_109213_object27585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:54.540155+0000", 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 231.45892315899999, 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.032435765, 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.122 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142102 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540155+0000", 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540155+0000", 2024-03-26T05:12:46.123 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540157+0000", 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.505e-06 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540162+0000", 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4240000000000001e-06 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.124 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540173+0000", 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1156e-05 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540181+0000", 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2230000000000001e-06 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540234+0000", 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2961000000000003e-05 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540370+0000", 2024-03-26T05:12:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000136611 2024-03-26T05:12:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:12:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540527+0000", 2024-03-26T05:12:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015644199999999999 2024-03-26T05:12:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.549579+0000", 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0090522169999999996 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.550582+0000", 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010026220000000001 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.555385+0000", 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0048038550000000001 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.576371+0000", 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.020985875000000001 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.576390+0000", 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8377e-05 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.572512+0000", 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99612238600000003 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.572591+0000", 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9110999999999995e-05 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142135 8.a 8:57959c30:::benchmark_data_smithi080_109213_object27618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:54.600498+0000", 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 231.398580713, 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.981086244, 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142135 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600498+0000", 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600498+0000", 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600499+0000", 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4529999999999999e-06 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600506+0000", 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4169999999999997e-06 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600527+0000", 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0822000000000002e-05 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600535+0000", 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9379999999999998e-06 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600632+0000", 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6347000000000004e-05 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.600641+0000", 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6239999999999997e-06 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.573005+0000", 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97236343199999997 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.573102+0000", 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.666e-05 2024-03-26T05:12:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,5", 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.573212+0000", 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011006999999999999 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.578723+0000", 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.005511719 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.578760+0000", 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7172000000000003e-05 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.581485+0000", 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027241370000000002 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.581539+0000", 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4712000000000001e-05 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.581584+0000", 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4740999999999997e-05 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142148 8.a 8:58078de1:::benchmark_data_smithi080_109213_object27631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:54.627220+0000", 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 231.37185842900001, 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95987994300000001, 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:12:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142148 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627220+0000", 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627220+0000", 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627222+0000", 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.392e-06 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627227+0000", 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9580000000000004e-06 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627248+0000", 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0429e-05 2024-03-26T05:12:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627258+0000", 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0284e-05 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627341+0000", 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2446000000000003e-05 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.627350+0000", 2024-03-26T05:12:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8460000000000003e-06 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.575891+0000", 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.94854066800000003 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.576039+0000", 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000148193 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,5", 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.576169+0000", 2024-03-26T05:12:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012968100000000001 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.578794+0000", 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002625334 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.581412+0000", 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0026176149999999998 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:12:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.586988+0000", 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.005576335 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.587046+0000", 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7314e-05 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.587100+0000", 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4447999999999998e-05 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.142 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.143 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:12:46.143 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:12:46.143 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:12:46.207 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:46.234 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:46.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:46.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:46.336 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:46.365 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:46.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:46 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:46.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:46 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:46.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:46.468 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:46.507 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:46.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:46.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:46.609 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:46.635 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:46.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:46.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:46.737 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:46.773 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:46.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:46.875 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:46.914 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:46.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:46.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:47.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:47.043 INFO:teuthology.orchestra.run.smithi132.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T05:12:47.055 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:47.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:47.157 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:47.172 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:47 smithi132 ceph-mon[33500]: pgmap v563: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 1.44k op/s; 63712/339774 objects degraded (18.751%) 2024-03-26T05:12:47.172 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:47 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 63712/339774 objects degraded (18.751%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:47.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:47.182 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:47.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:47.283 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:47.310 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:47.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:47.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:47 smithi080 ceph-mon[29691]: pgmap v563: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 1.44k op/s; 63712/339774 objects degraded (18.751%) 2024-03-26T05:12:47.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:47 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 63712/339774 objects degraded (18.751%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:47.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:47 smithi080 ceph-mon[40404]: pgmap v563: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 1.44k op/s; 63712/339774 objects degraded (18.751%) 2024-03-26T05:12:47.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:47 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 63712/339774 objects degraded (18.751%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:47.412 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:47.437 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:47.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:47.539 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:47.579 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:47.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:47.681 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:47.714 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:47.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:47.817 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:47.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:47.870 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:47.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:47.971 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:48.011 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:48.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:48.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:48.159 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:48.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:48.186 INFO:teuthology.orchestra.run.smithi132.stdout:get_pg_num_history pg_num_history pg_num_history(e292 pg_nums {1={20=1},2={66=3},3={67=32},4={69=32},5={71=32},6={73=32},8={184=16},9={265=16}} deleted_pools 256,8) 2024-03-26T05:12:48.186 INFO:teuthology.orchestra.run.smithi132.stdout:Importing pgid 5.1a 2024-03-26T05:12:48.186 INFO:teuthology.orchestra.run.smithi132.stdout:write_pg epoch 291 info 5.1a( empty local-lis/les=71/72 n=0 ec=71/71 lis/c=71/71 les/c/f=72/72/0 sis=71) 2024-03-26T05:12:48.187 INFO:teuthology.orchestra.run.smithi132.stdout:Import successful 2024-03-26T05:12:48.261 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:48.293 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T05:12:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:48.312 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:48.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:48.413 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:48.441 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:48.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:48.484 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:48.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:48.569 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:48.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:48.671 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:48.716 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:48.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:48.818 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:48.875 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:48.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:48.977 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:49.004 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:49.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:49.106 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:49.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:49 smithi080 ceph-mon[40404]: pgmap v564: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 20 MiB/s wr, 1.23k op/s; 63712/339774 objects degraded (18.751%) 2024-03-26T05:12:49.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:49 smithi080 ceph-mon[29691]: pgmap v564: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 20 MiB/s wr, 1.23k op/s; 63712/339774 objects degraded (18.751%) 2024-03-26T05:12:49.150 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:49.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:49.168 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:49.251 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:49 smithi132 ceph-mon[33500]: pgmap v564: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 20 MiB/s wr, 1.23k op/s; 63712/339774 objects degraded (18.751%) 2024-03-26T05:12:49.307 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:49.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:49.340 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/exp.5.1a.7 2024-03-26T05:12:49.409 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:49.444 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:49.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:49.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:49.546 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:49.573 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:49.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:49.675 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:49.716 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:49.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:49.818 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:49.844 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:49.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:49.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:49.946 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:49.985 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:49.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:49.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:50.056 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:12:50.087 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:50.125 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:50.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:50.227 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:50.268 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:50.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:50.307 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:50 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1516500852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:12:50.308 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:50 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:12:50.308 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:50 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1516500852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:12:50.308 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:50 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:12:50.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:50.398 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:50.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:50.500 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:50.528 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:50.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:50.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:50.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:50 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1516500852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:12:50.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:50 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:12:50.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:50.656 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:50.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:50.758 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:50.786 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:50.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:50.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:50.888 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:50.927 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:50.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:50.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:51.029 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:51.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:51.066 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:51.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:51.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:51.199 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:51.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:51.301 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:51.328 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:51.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:51.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[40404]: pgmap v565: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 18 MiB/s wr, 1.15k op/s; 63327/338619 objects degraded (18.702%) 2024-03-26T05:12:51.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:12:51.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:12:51.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[40404]: osdmap e298: 8 total, 7 up, 8 in 2024-03-26T05:12:51.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:12:51.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[40404]: 5.b scrub starts 2024-03-26T05:12:51.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[40404]: 5.b scrub ok 2024-03-26T05:12:51.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[29691]: pgmap v565: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 18 MiB/s wr, 1.15k op/s; 63327/338619 objects degraded (18.702%) 2024-03-26T05:12:51.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:12:51.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:12:51.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[29691]: osdmap e298: 8 total, 7 up, 8 in 2024-03-26T05:12:51.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:12:51.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[29691]: 5.b scrub starts 2024-03-26T05:12:51.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:51 smithi080 ceph-mon[29691]: 5.b scrub ok 2024-03-26T05:12:51.430 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:51.453 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:51.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:51.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:51.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:51 smithi132 ceph-mon[33500]: pgmap v565: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 18 MiB/s wr, 1.15k op/s; 63327/338619 objects degraded (18.702%) 2024-03-26T05:12:51.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:51 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:12:51.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:51 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:12:51.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:51 smithi132 ceph-mon[33500]: osdmap e298: 8 total, 7 up, 8 in 2024-03-26T05:12:51.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:12:51.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:51 smithi132 ceph-mon[33500]: 5.b scrub starts 2024-03-26T05:12:51.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:51 smithi132 ceph-mon[33500]: 5.b scrub ok 2024-03-26T05:12:51.554 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:51.591 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:51.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:51.693 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:51.722 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:51.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:51.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:51.823 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:51.858 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:51.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:51.961 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:51.985 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:51.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:51.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:52.087 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:52.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:52.130 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:52.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:52.231 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:52.256 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:52.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:52.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:52.357 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:52.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:52 smithi080 ceph-mon[40404]: 6.b scrub starts 2024-03-26T05:12:52.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:52 smithi080 ceph-mon[40404]: 6.b scrub ok 2024-03-26T05:12:52.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:52 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 63327/338619 objects degraded (18.702%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:52.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:52 smithi080 ceph-mon[29691]: 6.b scrub starts 2024-03-26T05:12:52.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:52 smithi080 ceph-mon[29691]: 6.b scrub ok 2024-03-26T05:12:52.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:52 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 63327/338619 objects degraded (18.702%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:52.385 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:52.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:52.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:52.487 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:52.513 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:52.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:52.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:52 smithi132 ceph-mon[33500]: 6.b scrub starts 2024-03-26T05:12:52.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:52 smithi132 ceph-mon[33500]: 6.b scrub ok 2024-03-26T05:12:52.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:52 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 63327/338619 objects degraded (18.702%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:52.614 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:52.654 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:52.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:52.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:52.787 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:52.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:52.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:52.889 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:52.916 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:52.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:53.017 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:53.062 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:53.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:53.164 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:53.222 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:53.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:53.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:53.324 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:53.364 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:53.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:53.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:53 smithi080 ceph-mon[40404]: pgmap v567: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 60132/320667 objects degraded (18.752%) 2024-03-26T05:12:53.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:53 smithi080 ceph-mon[40404]: 9.4 scrub starts 2024-03-26T05:12:53.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:53 smithi080 ceph-mon[29691]: pgmap v567: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 60132/320667 objects degraded (18.752%) 2024-03-26T05:12:53.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:53 smithi080 ceph-mon[29691]: 9.4 scrub starts 2024-03-26T05:12:53.466 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:53.510 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:53.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:53.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:53.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:53 smithi132 ceph-mon[33500]: pgmap v567: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 60132/320667 objects degraded (18.752%) 2024-03-26T05:12:53.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:53 smithi132 ceph-mon[33500]: 9.4 scrub starts 2024-03-26T05:12:53.612 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:53.661 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:53.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:53.763 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:53.800 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:53.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:53.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:53.902 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:53.940 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:53.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:54.041 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:54.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:54.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:54.170 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:54.197 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:54.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:54.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:54.332 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-mon[29691]: 4.b scrub starts 2024-03-26T05:12:54.332 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-mon[29691]: 4.b scrub ok 2024-03-26T05:12:54.332 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-mon[29691]: 3.10 scrub starts 2024-03-26T05:12:54.332 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-mon[29691]: 3.10 scrub ok 2024-03-26T05:12:54.332 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-mon[29691]: 4.6 scrub starts 2024-03-26T05:12:54.332 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-mon[29691]: 4.6 scrub ok 2024-03-26T05:12:54.332 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:12:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:12:54.332 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-mon[40404]: 4.b scrub starts 2024-03-26T05:12:54.333 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-mon[40404]: 4.b scrub ok 2024-03-26T05:12:54.333 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-mon[40404]: 3.10 scrub starts 2024-03-26T05:12:54.333 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-mon[40404]: 3.10 scrub ok 2024-03-26T05:12:54.333 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-mon[40404]: 4.6 scrub starts 2024-03-26T05:12:54.333 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:54 smithi080 ceph-mon[40404]: 4.6 scrub ok 2024-03-26T05:12:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:54.340 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:54.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:54.376 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-03-26T05:12:54.377 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:12:54.377 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-26T05:12:54.377 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-26T05:12:54.441 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:54.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:54.510 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:54.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:54.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:54 smithi132 ceph-mon[33500]: 4.b scrub starts 2024-03-26T05:12:54.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:54 smithi132 ceph-mon[33500]: 4.b scrub ok 2024-03-26T05:12:54.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:54 smithi132 ceph-mon[33500]: 3.10 scrub starts 2024-03-26T05:12:54.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:54 smithi132 ceph-mon[33500]: 3.10 scrub ok 2024-03-26T05:12:54.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:54 smithi132 ceph-mon[33500]: 4.6 scrub starts 2024-03-26T05:12:54.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:54 smithi132 ceph-mon[33500]: 4.6 scrub ok 2024-03-26T05:12:54.611 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:54.637 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:54.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:54.738 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:54.812 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:54.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:54.914 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:54.939 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:54.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:55.041 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:55.052 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:12:55.052 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:12:55.052 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:12:55.079 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:12:55.089 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:55.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:55.123 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:12:55.162 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:12:55.191 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:55.207 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:12:55.209 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:12:55.209 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:12:55.209 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:12:55.209 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:12:55.210 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:12:55.210 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:12:55.210 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:12:55.210 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:12:55.217 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:55.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:55.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[40404]: pgmap v568: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 60132/320667 objects degraded (18.752%) 2024-03-26T05:12:55.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:12:55.271 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:12:55.272 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[40404]: osdmap e299: 8 total, 7 up, 8 in 2024-03-26T05:12:55.272 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[40404]: 4.18 scrub starts 2024-03-26T05:12:55.272 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[40404]: 4.18 scrub ok 2024-03-26T05:12:55.272 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3222072182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:55.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[29691]: pgmap v568: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 60132/320667 objects degraded (18.752%) 2024-03-26T05:12:55.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:12:55.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:12:55.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[29691]: osdmap e299: 8 total, 7 up, 8 in 2024-03-26T05:12:55.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[29691]: 4.18 scrub starts 2024-03-26T05:12:55.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[29691]: 4.18 scrub ok 2024-03-26T05:12:55.272 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:55 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3222072182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:55.318 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:55.345 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:55.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:55.385 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:12:55.385 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:12:55.445 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:55 smithi132 ceph-mon[33500]: pgmap v568: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 60132/320667 objects degraded (18.752%) 2024-03-26T05:12:55.445 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:55 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:12:55.445 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:55 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:12:55.445 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:55 smithi132 ceph-mon[33500]: osdmap e299: 8 total, 7 up, 8 in 2024-03-26T05:12:55.445 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:55 smithi132 ceph-mon[33500]: 4.18 scrub starts 2024-03-26T05:12:55.445 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:55 smithi132 ceph-mon[33500]: 4.18 scrub ok 2024-03-26T05:12:55.445 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:55 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3222072182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:12:55.447 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:55.483 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:55.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:55.570 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:12:55.585 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:55.610 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:55.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:55.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:55.743 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:55.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:55.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:12:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:12:55.844 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:55.873 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:55.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:55.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:55.974 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:56.001 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:56.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:56.103 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:56.131 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:56.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:56.233 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:56.335 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:56.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:56.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:56 smithi080 ceph-mon[29691]: 9.4 scrub ok 2024-03-26T05:12:56.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:56 smithi080 ceph-mon[29691]: 4.1b scrub starts 2024-03-26T05:12:56.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:56 smithi080 ceph-mon[29691]: 4.7 scrub starts 2024-03-26T05:12:56.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:56 smithi080 ceph-mon[29691]: 4.7 scrub ok 2024-03-26T05:12:56.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:56 smithi080 ceph-mon[29691]: 4.1b scrub ok 2024-03-26T05:12:56.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:56 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:56.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:56 smithi080 ceph-mon[40404]: 9.4 scrub ok 2024-03-26T05:12:56.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:56 smithi080 ceph-mon[40404]: 4.1b scrub starts 2024-03-26T05:12:56.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:56 smithi080 ceph-mon[40404]: 4.7 scrub starts 2024-03-26T05:12:56.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:56 smithi080 ceph-mon[40404]: 4.7 scrub ok 2024-03-26T05:12:56.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:56 smithi080 ceph-mon[40404]: 4.1b scrub ok 2024-03-26T05:12:56.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:56 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:56.437 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:56.471 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:56.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:56.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:56.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:56 smithi132 ceph-mon[33500]: 9.4 scrub ok 2024-03-26T05:12:56.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:56 smithi132 ceph-mon[33500]: 4.1b scrub starts 2024-03-26T05:12:56.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:56 smithi132 ceph-mon[33500]: 4.7 scrub starts 2024-03-26T05:12:56.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:56 smithi132 ceph-mon[33500]: 4.7 scrub ok 2024-03-26T05:12:56.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:56 smithi132 ceph-mon[33500]: 4.1b scrub ok 2024-03-26T05:12:56.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:56 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:12:56.572 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:56.599 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T05:12:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:56.600 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:56.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:56.701 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:56.728 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:56.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:56.830 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:56.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:56.885 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:56.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:56.986 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:57.020 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:57.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:57.110 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:12:57.111 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:12:57.121 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:57.148 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:57.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:57.153 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:57.177 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:57.226 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:12:57.249 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:57.296 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:57.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:57.304 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:57 smithi080 ceph-mon[40404]: pgmap v570: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 56957/303759 objects degraded (18.751%) 2024-03-26T05:12:57.304 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:57 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 56957/303759 objects degraded (18.751%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:57.304 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:57 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:12:57.305 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:57 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:12:57.305 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:57 smithi080 ceph-mon[29691]: pgmap v570: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 56957/303759 objects degraded (18.751%) 2024-03-26T05:12:57.305 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:57 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 56957/303759 objects degraded (18.751%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:57.305 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:57 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:12:57.305 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:57 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:12:57.397 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:57.442 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:57.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:57.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:57 smithi132 ceph-mon[33500]: pgmap v570: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 56957/303759 objects degraded (18.751%) 2024-03-26T05:12:57.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:57 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 56957/303759 objects degraded (18.751%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:12:57.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:57 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:12:57.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:57 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:12:57.543 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:57.580 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:57.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:57.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:57.682 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:57.723 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:57.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:57.827 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:57.866 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:57.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:57.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:57.968 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:12:58.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:58.015 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:58.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:12:58.116 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:58.155 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:58.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:58.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:58.257 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:58.285 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:58.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:58.387 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:58.415 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:58.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:58.516 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:58.532 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:58 smithi080 ceph-mon[29691]: 4.4 scrub starts 2024-03-26T05:12:58.532 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:58 smithi080 ceph-mon[29691]: 4.4 scrub ok 2024-03-26T05:12:58.533 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:58 smithi080 ceph-mon[29691]: 6.0 scrub starts 2024-03-26T05:12:58.533 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:58 smithi080 ceph-mon[29691]: 6.0 scrub ok 2024-03-26T05:12:58.533 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:58 smithi080 ceph-mon[40404]: 4.4 scrub starts 2024-03-26T05:12:58.533 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:58 smithi080 ceph-mon[40404]: 4.4 scrub ok 2024-03-26T05:12:58.533 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:58 smithi080 ceph-mon[40404]: 6.0 scrub starts 2024-03-26T05:12:58.533 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:58 smithi080 ceph-mon[40404]: 6.0 scrub ok 2024-03-26T05:12:58.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:58 smithi132 ceph-mon[33500]: 4.4 scrub starts 2024-03-26T05:12:58.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:58 smithi132 ceph-mon[33500]: 4.4 scrub ok 2024-03-26T05:12:58.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:58 smithi132 ceph-mon[33500]: 6.0 scrub starts 2024-03-26T05:12:58.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:58 smithi132 ceph-mon[33500]: 6.0 scrub ok 2024-03-26T05:12:58.557 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:58.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:58.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:12:58.659 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:58.699 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:58.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:58.800 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:58.842 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:58.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:58.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:58.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:58.976 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:58.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:59.077 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:12:59.105 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:59.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:59.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:12:59.111 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:12:59.206 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:12:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:59.240 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:59.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:12:59.252 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:59 smithi080 ceph-mon[40404]: 6.13 scrub starts 2024-03-26T05:12:59.252 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:59 smithi080 ceph-mon[40404]: 6.13 scrub ok 2024-03-26T05:12:59.253 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:59 smithi080 ceph-mon[40404]: pgmap v571: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 56957/303759 objects degraded (18.751%) 2024-03-26T05:12:59.253 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:12:59 smithi080 ceph-mon[40404]: 9.f scrub starts 2024-03-26T05:12:59.253 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:59 smithi080 ceph-mon[29691]: 6.13 scrub starts 2024-03-26T05:12:59.253 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:59 smithi080 ceph-mon[29691]: 6.13 scrub ok 2024-03-26T05:12:59.253 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:59 smithi080 ceph-mon[29691]: pgmap v571: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 56957/303759 objects degraded (18.751%) 2024-03-26T05:12:59.253 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:12:59 smithi080 ceph-mon[29691]: 9.f scrub starts 2024-03-26T05:12:59.341 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:59.368 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:59.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:59.469 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:12:59.532 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:59.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:12:59.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:59 smithi132 ceph-mon[33500]: 6.13 scrub starts 2024-03-26T05:12:59.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:59 smithi132 ceph-mon[33500]: 6.13 scrub ok 2024-03-26T05:12:59.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:59 smithi132 ceph-mon[33500]: pgmap v571: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 56957/303759 objects degraded (18.751%) 2024-03-26T05:12:59.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:12:59 smithi132 ceph-mon[33500]: 9.f scrub starts 2024-03-26T05:12:59.634 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:12:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:59.667 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:59.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:12:59.768 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:12:59.797 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:12:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:59.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:12:59.899 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:12:59.935 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:12:59.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:12:59.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:00.037 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:00.066 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:00.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:00.168 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:00.213 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:00.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:00.315 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:00.347 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:00.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:00.449 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:00.488 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:00.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:00.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:00 smithi132 ceph-mon[33500]: 9.0 scrub starts 2024-03-26T05:13:00.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:00 smithi132 ceph-mon[33500]: 4.1c scrub starts 2024-03-26T05:13:00.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:00 smithi132 ceph-mon[33500]: 4.1c scrub ok 2024-03-26T05:13:00.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:00 smithi132 ceph-mon[33500]: 5.15 scrub starts 2024-03-26T05:13:00.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:00 smithi132 ceph-mon[33500]: 5.15 scrub ok 2024-03-26T05:13:00.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:00 smithi132 ceph-mon[33500]: 3.6 scrub starts 2024-03-26T05:13:00.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:00 smithi132 ceph-mon[33500]: 3.6 scrub ok 2024-03-26T05:13:00.590 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:00.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[29691]: 9.0 scrub starts 2024-03-26T05:13:00.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[29691]: 4.1c scrub starts 2024-03-26T05:13:00.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[29691]: 4.1c scrub ok 2024-03-26T05:13:00.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[29691]: 5.15 scrub starts 2024-03-26T05:13:00.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[29691]: 5.15 scrub ok 2024-03-26T05:13:00.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[29691]: 3.6 scrub starts 2024-03-26T05:13:00.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[29691]: 3.6 scrub ok 2024-03-26T05:13:00.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[40404]: 9.0 scrub starts 2024-03-26T05:13:00.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[40404]: 4.1c scrub starts 2024-03-26T05:13:00.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[40404]: 4.1c scrub ok 2024-03-26T05:13:00.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[40404]: 5.15 scrub starts 2024-03-26T05:13:00.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[40404]: 5.15 scrub ok 2024-03-26T05:13:00.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[40404]: 3.6 scrub starts 2024-03-26T05:13:00.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:00 smithi080 ceph-mon[40404]: 3.6 scrub ok 2024-03-26T05:13:00.619 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:00.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:00.721 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:00.762 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:00.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:00.864 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:00.890 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:00.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:00.991 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:01.040 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:01.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:01.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:01.141 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:01.179 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:01.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:01.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:01.281 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:01.308 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:01.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:01.410 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:01.439 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:01.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:01.541 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:01.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:01 smithi132 ceph-mon[33500]: pgmap v572: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.21k op/s; 56550/302538 objects degraded (18.692%) 2024-03-26T05:13:01.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:01 smithi132 ceph-mon[33500]: 4.2 scrub starts 2024-03-26T05:13:01.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:01 smithi132 ceph-mon[33500]: 4.2 scrub ok 2024-03-26T05:13:01.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:01 smithi132 ceph-mon[33500]: 9.f scrub ok 2024-03-26T05:13:01.571 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:01.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:01.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:01 smithi080 ceph-mon[29691]: pgmap v572: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.21k op/s; 56550/302538 objects degraded (18.692%) 2024-03-26T05:13:01.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:01 smithi080 ceph-mon[29691]: 4.2 scrub starts 2024-03-26T05:13:01.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:01 smithi080 ceph-mon[29691]: 4.2 scrub ok 2024-03-26T05:13:01.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:01 smithi080 ceph-mon[29691]: 9.f scrub ok 2024-03-26T05:13:01.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:01 smithi080 ceph-mon[40404]: pgmap v572: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.21k op/s; 56550/302538 objects degraded (18.692%) 2024-03-26T05:13:01.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:01 smithi080 ceph-mon[40404]: 4.2 scrub starts 2024-03-26T05:13:01.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:01 smithi080 ceph-mon[40404]: 4.2 scrub ok 2024-03-26T05:13:01.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:01 smithi080 ceph-mon[40404]: 9.f scrub ok 2024-03-26T05:13:01.673 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:01.697 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:01.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:01.798 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:01.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:01.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:01.940 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:01.982 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:01.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:02.084 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:02.121 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:02.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:02.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:02.223 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:02.250 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:02.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:02.352 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:02.390 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:02.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:02.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:02.492 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:02.527 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:02.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:02.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:02 smithi132 ceph-mon[33500]: 9.0 scrub ok 2024-03-26T05:13:02.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:02 smithi132 ceph-mon[33500]: 3.1a scrub starts 2024-03-26T05:13:02.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:02 smithi132 ceph-mon[33500]: 3.1a scrub ok 2024-03-26T05:13:02.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:02 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 56550/302538 objects degraded (18.692%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:02.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:02 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:13:02.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:02 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:13:02.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:02 smithi132 ceph-mon[33500]: 3.1f scrub starts 2024-03-26T05:13:02.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:02 smithi132 ceph-mon[33500]: 3.1f scrub ok 2024-03-26T05:13:02.583 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[40404]: 9.0 scrub ok 2024-03-26T05:13:02.583 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[40404]: 3.1a scrub starts 2024-03-26T05:13:02.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[40404]: 3.1a scrub ok 2024-03-26T05:13:02.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 56550/302538 objects degraded (18.692%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:02.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:13:02.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:13:02.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[40404]: 3.1f scrub starts 2024-03-26T05:13:02.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[40404]: 3.1f scrub ok 2024-03-26T05:13:02.584 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[29691]: 9.0 scrub ok 2024-03-26T05:13:02.584 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[29691]: 3.1a scrub starts 2024-03-26T05:13:02.584 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[29691]: 3.1a scrub ok 2024-03-26T05:13:02.584 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 56550/302538 objects degraded (18.692%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:02.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:13:02.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:13:02.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[29691]: 3.1f scrub starts 2024-03-26T05:13:02.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:02 smithi080 ceph-mon[29691]: 3.1f scrub ok 2024-03-26T05:13:02.629 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:02.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:02.695 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:02.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:02.796 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:02.834 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:02.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:02.894 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:02.937 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:02.993 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:02.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:03.095 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:03.125 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:03.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:03.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:03.226 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:03.252 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:03.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:03.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:03.354 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:03.382 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:03.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:03.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:03.483 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:03.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:03.519 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:03.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:03.621 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:03.664 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:03.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:03.765 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:03.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:03.805 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:03.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:03.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:03 smithi080 ceph-mon[40404]: pgmap v573: 148 pgs: 2 active+clean+scrubbing, 85 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 53507/285336 objects degraded (18.752%) 2024-03-26T05:13:03.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:03 smithi080 ceph-mon[40404]: osdmap e300: 8 total, 7 up, 8 in 2024-03-26T05:13:03.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:03 smithi080 ceph-mon[40404]: 4.12 scrub starts 2024-03-26T05:13:03.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:03 smithi080 ceph-mon[40404]: 4.12 scrub ok 2024-03-26T05:13:03.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:03 smithi080 ceph-mon[29691]: pgmap v573: 148 pgs: 2 active+clean+scrubbing, 85 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 53507/285336 objects degraded (18.752%) 2024-03-26T05:13:03.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:03 smithi080 ceph-mon[29691]: osdmap e300: 8 total, 7 up, 8 in 2024-03-26T05:13:03.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:03 smithi080 ceph-mon[29691]: 4.12 scrub starts 2024-03-26T05:13:03.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:03 smithi080 ceph-mon[29691]: 4.12 scrub ok 2024-03-26T05:13:03.906 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:03.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:03.944 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:03.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:04.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:03 smithi132 ceph-mon[33500]: pgmap v573: 148 pgs: 2 active+clean+scrubbing, 85 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 53507/285336 objects degraded (18.752%) 2024-03-26T05:13:04.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:03 smithi132 ceph-mon[33500]: osdmap e300: 8 total, 7 up, 8 in 2024-03-26T05:13:04.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:03 smithi132 ceph-mon[33500]: 4.12 scrub starts 2024-03-26T05:13:04.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:03 smithi132 ceph-mon[33500]: 4.12 scrub ok 2024-03-26T05:13:04.045 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:04.074 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:04.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:04.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:04.176 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:04.219 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:04.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:04.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:04.320 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:04.324 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:13:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:13:04.349 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:04.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:04.451 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:04.479 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:04.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:04.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:04.582 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:04.613 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:04.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:04.715 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:04.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:04 smithi132 ceph-mon[33500]: 6.f scrub starts 2024-03-26T05:13:04.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:04 smithi132 ceph-mon[33500]: 6.f scrub ok 2024-03-26T05:13:04.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:04 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:13:04.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:04 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:13:04.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:04 smithi132 ceph-mon[33500]: osdmap e301: 8 total, 7 up, 8 in 2024-03-26T05:13:04.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:04 smithi132 ceph-mon[33500]: 4.e scrub starts 2024-03-26T05:13:04.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:04 smithi132 ceph-mon[33500]: 4.e scrub ok 2024-03-26T05:13:04.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:04.821 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:04.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:04.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[29691]: 6.f scrub starts 2024-03-26T05:13:04.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[29691]: 6.f scrub ok 2024-03-26T05:13:04.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:13:04.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:13:04.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[29691]: osdmap e301: 8 total, 7 up, 8 in 2024-03-26T05:13:04.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[29691]: 4.e scrub starts 2024-03-26T05:13:04.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[29691]: 4.e scrub ok 2024-03-26T05:13:04.841 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[40404]: 6.f scrub starts 2024-03-26T05:13:04.841 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[40404]: 6.f scrub ok 2024-03-26T05:13:04.841 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:13:04.841 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:13:04.841 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[40404]: osdmap e301: 8 total, 7 up, 8 in 2024-03-26T05:13:04.841 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[40404]: 4.e scrub starts 2024-03-26T05:13:04.841 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:04 smithi080 ceph-mon[40404]: 4.e scrub ok 2024-03-26T05:13:04.922 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:04.924 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:13:04.924 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:13:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:04.954 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:04.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:05.055 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:05.089 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:05.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:05.099 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:13:05.100 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:13:05.100 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:13:05.100 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:13:05.100 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:13:05.100 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:13:05.100 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:13:05.100 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:13:05.100 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:13:05.100 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:13:05.100 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:13:05.100 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:13:05.191 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:05.229 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:05.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:05.331 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:05.374 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:05.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:05.476 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:05.519 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:05.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:05.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:05.620 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:05.660 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:05.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:05.721 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:05 smithi080 ceph-mon[40404]: pgmap v576: 148 pgs: 2 active+clean+scrubbing, 85 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 767 op/s; 53507/285336 objects degraded (18.752%) 2024-03-26T05:13:05.721 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:05 smithi080 ceph-mon[40404]: 9.6 scrub starts 2024-03-26T05:13:05.721 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:13:05.721 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:05 smithi080 ceph-mon[29691]: pgmap v576: 148 pgs: 2 active+clean+scrubbing, 85 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 767 op/s; 53507/285336 objects degraded (18.752%) 2024-03-26T05:13:05.721 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:05 smithi080 ceph-mon[29691]: 9.6 scrub starts 2024-03-26T05:13:05.721 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:13:05.762 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:05.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:05 smithi132 ceph-mon[33500]: pgmap v576: 148 pgs: 2 active+clean+scrubbing, 85 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 767 op/s; 53507/285336 objects degraded (18.752%) 2024-03-26T05:13:05.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:05 smithi132 ceph-mon[33500]: 9.6 scrub starts 2024-03-26T05:13:05.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:13:05.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:13:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:13:05.804 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:05.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:05.908 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:05.947 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:05.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:06.049 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:06.079 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:06.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:06.180 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:06.210 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:06.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:06.311 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:06.338 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:06.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:06.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:06.441 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:06.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:06.488 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:06.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:13:06.505 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:13:06.506 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-26T05:13:06.589 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:06.635 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:06.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:06.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:06.724 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:06.724 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[29691]: 4.10 scrub starts 2024-03-26T05:13:06.724 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[29691]: 4.10 scrub ok 2024-03-26T05:13:06.724 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[29691]: 6.4 scrub starts 2024-03-26T05:13:06.724 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[29691]: 6.4 scrub ok 2024-03-26T05:13:06.724 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3659549419' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:13:06.724 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 53507/285336 objects degraded (18.752%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:06.724 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:06.725 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[40404]: 4.10 scrub starts 2024-03-26T05:13:06.725 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[40404]: 4.10 scrub ok 2024-03-26T05:13:06.725 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[40404]: 6.4 scrub starts 2024-03-26T05:13:06.725 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[40404]: 6.4 scrub ok 2024-03-26T05:13:06.725 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3659549419' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:13:06.725 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:06 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 53507/285336 objects degraded (18.752%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:06.737 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:06.778 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:06.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:06.843 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:13:06.879 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:06.924 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:06.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:07.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:06 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:06 smithi132 ceph-mon[33500]: 4.10 scrub starts 2024-03-26T05:13:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:06 smithi132 ceph-mon[33500]: 4.10 scrub ok 2024-03-26T05:13:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:06 smithi132 ceph-mon[33500]: 6.4 scrub starts 2024-03-26T05:13:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:06 smithi132 ceph-mon[33500]: 6.4 scrub ok 2024-03-26T05:13:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:06 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3659549419' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:13:07.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:06 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 53507/285336 objects degraded (18.752%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:07.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:07.055 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:07.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:07.156 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:07.196 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:07.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:07.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:07.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:07.338 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:07.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:07.438 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:07.463 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:07.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:07.565 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:07.591 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:07.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:07.693 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:07.721 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:07.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:07.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:07.823 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:07.862 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:07.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[40404]: pgmap v577: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 51221/272625 objects degraded (18.788%) 2024-03-26T05:13:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[40404]: 9.6 scrub ok 2024-03-26T05:13:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:13:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3659549419' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:13:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[40404]: osdmap e302: 8 total, 7 up, 8 in 2024-03-26T05:13:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[40404]: 3.19 scrub starts 2024-03-26T05:13:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[40404]: 3.19 scrub ok 2024-03-26T05:13:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[29691]: pgmap v577: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 51221/272625 objects degraded (18.788%) 2024-03-26T05:13:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[29691]: 9.6 scrub ok 2024-03-26T05:13:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:13:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3659549419' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:13:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[29691]: osdmap e302: 8 total, 7 up, 8 in 2024-03-26T05:13:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[29691]: 3.19 scrub starts 2024-03-26T05:13:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:07 smithi080 ceph-mon[29691]: 3.19 scrub ok 2024-03-26T05:13:07.964 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:07.991 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:07.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:07.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:07 smithi132 ceph-mon[33500]: pgmap v577: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 51221/272625 objects degraded (18.788%) 2024-03-26T05:13:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:07 smithi132 ceph-mon[33500]: 9.6 scrub ok 2024-03-26T05:13:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:07 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:13:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:07 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3659549419' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:13:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:07 smithi132 ceph-mon[33500]: osdmap e302: 8 total, 7 up, 8 in 2024-03-26T05:13:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:07 smithi132 ceph-mon[33500]: 3.19 scrub starts 2024-03-26T05:13:08.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:07 smithi132 ceph-mon[33500]: 3.19 scrub ok 2024-03-26T05:13:08.093 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:08.128 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:08.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:08.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:08.229 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:08.269 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:08.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:08.370 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:08.409 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:08.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:08.510 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:08.534 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:08.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:08.636 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:08.672 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:08.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:08.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:08.774 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:08.812 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:08.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:08.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:08 smithi080 ceph-mon[40404]: 5.f deep-scrub starts 2024-03-26T05:13:08.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:08 smithi080 ceph-mon[40404]: 5.f deep-scrub ok 2024-03-26T05:13:08.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:08 smithi080 ceph-mon[40404]: 5.c scrub starts 2024-03-26T05:13:08.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:08 smithi080 ceph-mon[40404]: 5.c scrub ok 2024-03-26T05:13:08.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:08 smithi080 ceph-mon[29691]: 5.f deep-scrub starts 2024-03-26T05:13:08.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:08 smithi080 ceph-mon[29691]: 5.f deep-scrub ok 2024-03-26T05:13:08.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:08 smithi080 ceph-mon[29691]: 5.c scrub starts 2024-03-26T05:13:08.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:08 smithi080 ceph-mon[29691]: 5.c scrub ok 2024-03-26T05:13:08.914 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:08.951 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:08.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:08 smithi132 ceph-mon[33500]: 5.f deep-scrub starts 2024-03-26T05:13:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:08 smithi132 ceph-mon[33500]: 5.f deep-scrub ok 2024-03-26T05:13:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:08 smithi132 ceph-mon[33500]: 5.c scrub starts 2024-03-26T05:13:09.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:08 smithi132 ceph-mon[33500]: 5.c scrub ok 2024-03-26T05:13:09.052 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:09.079 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:09.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:09.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:09.181 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:09.209 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:09.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:09.311 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:09.352 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:09.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:09.454 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:09.489 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:09.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:09.591 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:09.620 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:09.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:09.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:09.722 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:09.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:09.749 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:09.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:09.850 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:09.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:09 smithi080 ceph-mon[40404]: pgmap v579: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 50052/266844 objects degraded (18.757%) 2024-03-26T05:13:09.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:09 smithi080 ceph-mon[40404]: 5.a scrub starts 2024-03-26T05:13:09.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:09 smithi080 ceph-mon[40404]: 5.a scrub ok 2024-03-26T05:13:09.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:09 smithi080 ceph-mon[29691]: pgmap v579: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 50052/266844 objects degraded (18.757%) 2024-03-26T05:13:09.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:09 smithi080 ceph-mon[29691]: 5.a scrub starts 2024-03-26T05:13:09.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:09 smithi080 ceph-mon[29691]: 5.a scrub ok 2024-03-26T05:13:09.878 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:09.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:09.980 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:10.007 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:10.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:10.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:09 smithi132 ceph-mon[33500]: pgmap v579: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 50052/266844 objects degraded (18.757%) 2024-03-26T05:13:10.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:09 smithi132 ceph-mon[33500]: 5.a scrub starts 2024-03-26T05:13:10.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:09 smithi132 ceph-mon[33500]: 5.a scrub ok 2024-03-26T05:13:10.109 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:10.149 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:10.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:10.251 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:10.286 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:10.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:10.388 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:10.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:10.519 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:10.546 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:10.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:10.648 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:10.699 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:10.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:10.801 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:10.831 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:10.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:10.932 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:10.969 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:10.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:11.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:10 smithi132 ceph-mon[33500]: 2.2 scrub starts 2024-03-26T05:13:11.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:10 smithi132 ceph-mon[33500]: 2.2 scrub ok 2024-03-26T05:13:11.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:10 smithi132 ceph-mon[33500]: pgmap v580: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 690 B/s rd, 0 B/s wr, 881 op/s; 49683/265737 objects degraded (18.696%) 2024-03-26T05:13:11.070 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:11.088 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:10 smithi080 ceph-mon[40404]: 2.2 scrub starts 2024-03-26T05:13:11.088 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:10 smithi080 ceph-mon[40404]: 2.2 scrub ok 2024-03-26T05:13:11.088 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:10 smithi080 ceph-mon[40404]: pgmap v580: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 690 B/s rd, 0 B/s wr, 881 op/s; 49683/265737 objects degraded (18.696%) 2024-03-26T05:13:11.089 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:10 smithi080 ceph-mon[29691]: 2.2 scrub starts 2024-03-26T05:13:11.089 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:10 smithi080 ceph-mon[29691]: 2.2 scrub ok 2024-03-26T05:13:11.089 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:10 smithi080 ceph-mon[29691]: pgmap v580: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 690 B/s rd, 0 B/s wr, 881 op/s; 49683/265737 objects degraded (18.696%) 2024-03-26T05:13:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:11.112 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:11.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:11.213 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:11.240 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:11.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:11.342 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:11.381 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:11.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:11.456 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T05:13:11.476 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:11.482 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:11.526 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:11.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:11.626 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:11.664 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:11.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:11.710 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:11 smithi080 ceph-mon[40404]: 3.4 scrub starts 2024-03-26T05:13:11.710 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:11 smithi080 ceph-mon[40404]: 3.4 scrub ok 2024-03-26T05:13:11.710 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:11 smithi080 ceph-mon[40404]: 5.5 scrub starts 2024-03-26T05:13:11.710 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:11 smithi080 ceph-mon[40404]: 5.5 scrub ok 2024-03-26T05:13:11.710 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:11 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 49683/265737 objects degraded (18.696%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:11.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:11 smithi080 ceph-mon[29691]: 3.4 scrub starts 2024-03-26T05:13:11.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:11 smithi080 ceph-mon[29691]: 3.4 scrub ok 2024-03-26T05:13:11.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:11 smithi080 ceph-mon[29691]: 5.5 scrub starts 2024-03-26T05:13:11.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:11 smithi080 ceph-mon[29691]: 5.5 scrub ok 2024-03-26T05:13:11.711 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:11 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 49683/265737 objects degraded (18.696%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:11.765 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:11.792 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:11.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:11.894 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:11.935 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:11.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:12.036 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:11 smithi132 ceph-mon[33500]: 3.4 scrub starts 2024-03-26T05:13:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:11 smithi132 ceph-mon[33500]: 3.4 scrub ok 2024-03-26T05:13:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:11 smithi132 ceph-mon[33500]: 5.5 scrub starts 2024-03-26T05:13:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:11 smithi132 ceph-mon[33500]: 5.5 scrub ok 2024-03-26T05:13:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:11 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 49683/265737 objects degraded (18.696%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:12.090 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:12.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:12.161 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:13:12.192 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:12.206 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:13:12.219 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:12.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:12.240 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:13:12.243 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:13:12.286 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:13:12.320 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:12.333 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:13:12.346 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:12.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:12.382 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:13:12.423 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:13:12.424 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:13:12.425 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:13:12.425 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:13:12.425 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:13:12.425 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:13:12.425 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:13:12.425 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:13:12.425 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:13:12.447 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:12.472 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:12.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:12.574 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:12.627 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:12.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:12.729 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:12.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:12.759 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:12.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:12.860 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:12.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:12 smithi080 ceph-mon[29691]: 3.1c scrub starts 2024-03-26T05:13:12.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:12 smithi080 ceph-mon[29691]: 3.1c scrub ok 2024-03-26T05:13:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:12 smithi080 ceph-mon[29691]: pgmap v581: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 46540/248424 objects degraded (18.734%) 2024-03-26T05:13:12.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:12 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2899812606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:13:12.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:12 smithi080 ceph-mon[40404]: 3.1c scrub starts 2024-03-26T05:13:12.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:12 smithi080 ceph-mon[40404]: 3.1c scrub ok 2024-03-26T05:13:12.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:12 smithi080 ceph-mon[40404]: pgmap v581: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 46540/248424 objects degraded (18.734%) 2024-03-26T05:13:12.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:12 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2899812606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:13:12.907 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:12.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:13.009 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:13.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:12 smithi132 ceph-mon[33500]: 3.1c scrub starts 2024-03-26T05:13:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:12 smithi132 ceph-mon[33500]: 3.1c scrub ok 2024-03-26T05:13:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:12 smithi132 ceph-mon[33500]: pgmap v581: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 46540/248424 objects degraded (18.734%) 2024-03-26T05:13:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:12 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2899812606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:13:13.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:13.051 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:13.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:13.152 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:13.190 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:13.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:13.291 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:13.335 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:13.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:13.436 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:13.462 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:13.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:13.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:13.563 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:13.610 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:13.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:13.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:13.712 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:13.749 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:13.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:13.850 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:13.877 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:13.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:13.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:13.907 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:13 smithi080 ceph-mon[40404]: 9.8 scrub starts 2024-03-26T05:13:13.908 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:13 smithi080 ceph-mon[29691]: 9.8 scrub starts 2024-03-26T05:13:13.979 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:14.021 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:14.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:14.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:13 smithi132 ceph-mon[33500]: 9.8 scrub starts 2024-03-26T05:13:14.123 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:14.160 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:14.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:14.261 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:14.288 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:14.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:14.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:14.364 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:13:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:13:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:13:14.389 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:14.417 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:13:14.417 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:13:14.440 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:14.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:14.542 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:14.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:14.578 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:14.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:14.679 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:14.706 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:14.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:14.808 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:14.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:14.884 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:14.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:14.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:14 smithi080 ceph-mon[29691]: pgmap v582: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 46540/248424 objects degraded (18.734%) 2024-03-26T05:13:14.928 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:14 smithi080 ceph-mon[40404]: pgmap v582: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 46540/248424 objects degraded (18.734%) 2024-03-26T05:13:14.947 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:14 smithi132 ceph-mon[33500]: pgmap v582: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 46540/248424 objects degraded (18.734%) 2024-03-26T05:13:14.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:15.011 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:15.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:15.084 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:15.104 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:15.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:15.187 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:15.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:15.201 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:13:15.201 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:13:15.201 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:13:15.201 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:13:15.202 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:13:15.202 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:13:15.202 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:13:15.202 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:13:15.202 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:13:15.202 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:13:15.202 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:13:15.202 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:13:15.290 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:15.318 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:15.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:15.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:15.446 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:15.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:15.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:15.548 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:15.575 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:15.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:15.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:15.678 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:15.709 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:15.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:15.744 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:13:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:13:15.744 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:15 smithi132 ceph-mon[33500]: 9.8 scrub ok 2024-03-26T05:13:15.745 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:15 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:15.759 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:15 smithi080 ceph-mon[40404]: 9.8 scrub ok 2024-03-26T05:13:15.759 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:15 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:15.759 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:15 smithi080 ceph-mon[29691]: 9.8 scrub ok 2024-03-26T05:13:15.759 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:15 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:15.812 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:15.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:15.855 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:15.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:15.956 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:15.994 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:15.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:16.095 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:16.122 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:16.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:16.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:16.251 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:16.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:16.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:16.353 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:16.390 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:16.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:16.418 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:13:16.491 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:16.527 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:16.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:16.629 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:16.657 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:16.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:16.759 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:16.777 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:16 smithi080 ceph-mon[40404]: 6.1d scrub starts 2024-03-26T05:13:16.777 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:16 smithi080 ceph-mon[40404]: 6.1d scrub ok 2024-03-26T05:13:16.777 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:16 smithi080 ceph-mon[40404]: pgmap v583: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 44318/236073 objects degraded (18.773%) 2024-03-26T05:13:16.778 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:16 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 46540/248424 objects degraded (18.734%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:16.778 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:16 smithi080 ceph-mon[29691]: 6.1d scrub starts 2024-03-26T05:13:16.778 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:16 smithi080 ceph-mon[29691]: 6.1d scrub ok 2024-03-26T05:13:16.778 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:16 smithi080 ceph-mon[29691]: pgmap v583: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 44318/236073 objects degraded (18.773%) 2024-03-26T05:13:16.778 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:16 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 46540/248424 objects degraded (18.734%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:16.785 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:16.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:16.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:16.887 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:16.914 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:16.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:16.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:17.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:17.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:16 smithi132 ceph-mon[33500]: 6.1d scrub starts 2024-03-26T05:13:17.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:16 smithi132 ceph-mon[33500]: 6.1d scrub ok 2024-03-26T05:13:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:16 smithi132 ceph-mon[33500]: pgmap v583: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 44318/236073 objects degraded (18.773%) 2024-03-26T05:13:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:16 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 46540/248424 objects degraded (18.734%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:17.066 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:17.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:17.168 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:17.212 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:17.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:17.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:17.314 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:17.365 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:17.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:13:17.467 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:13:17.467 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-26T05:13:17.467 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:17.513 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:17.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:17.615 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:17.662 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:17.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:17.763 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:17.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:17.821 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:17.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:17.923 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:17.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:17.966 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:17.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:17 smithi132 ceph-mon[33500]: 6.17 scrub starts 2024-03-26T05:13:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:17 smithi132 ceph-mon[33500]: 6.17 scrub ok 2024-03-26T05:13:18.068 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:18.108 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:18.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:18.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:17 smithi080 ceph-mon[29691]: 6.17 scrub starts 2024-03-26T05:13:18.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:17 smithi080 ceph-mon[29691]: 6.17 scrub ok 2024-03-26T05:13:18.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:17 smithi080 ceph-mon[40404]: 6.17 scrub starts 2024-03-26T05:13:18.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:17 smithi080 ceph-mon[40404]: 6.17 scrub ok 2024-03-26T05:13:18.210 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:18.238 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:18.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:18.339 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:18.366 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:18.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:18.468 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:18.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:18.514 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:18.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:18.615 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:18.656 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:18.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:18.758 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:18.786 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:18.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:18.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:18.848 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:18.888 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:18.932 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:18.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:18 smithi080 ceph-mon[29691]: 6.a scrub starts 2024-03-26T05:13:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:18 smithi080 ceph-mon[29691]: 6.a scrub ok 2024-03-26T05:13:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:18 smithi080 ceph-mon[29691]: pgmap v584: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 900 B/s rd, 0 B/s wr, 1.06k op/s; 43235/230766 objects degraded (18.735%) 2024-03-26T05:13:18.943 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:18 smithi080 ceph-mon[40404]: 6.a scrub starts 2024-03-26T05:13:18.943 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:18 smithi080 ceph-mon[40404]: 6.a scrub ok 2024-03-26T05:13:18.943 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:18 smithi080 ceph-mon[40404]: pgmap v584: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 900 B/s rd, 0 B/s wr, 1.06k op/s; 43235/230766 objects degraded (18.735%) 2024-03-26T05:13:19.035 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:19.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:18 smithi132 ceph-mon[33500]: 6.a scrub starts 2024-03-26T05:13:19.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:18 smithi132 ceph-mon[33500]: 6.a scrub ok 2024-03-26T05:13:19.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:18 smithi132 ceph-mon[33500]: pgmap v584: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 900 B/s rd, 0 B/s wr, 1.06k op/s; 43235/230766 objects degraded (18.735%) 2024-03-26T05:13:19.062 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:19.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:19.164 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:19.191 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:19.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:19.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:19.293 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:19.321 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:19.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:19.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:19.423 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:19.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:19.464 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:19.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:19.561 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:13:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:13:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:13:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:13:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206041 7.e 7:7f2653d5:::benchmark_data_smithi080_80848_object86542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:13:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.917013+0000", 2024-03-26T05:13:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 463.648156202, 2024-03-26T05:13:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.690052211, 2024-03-26T05:13:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206041 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917013+0000", 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917013+0000", 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917015+0000", 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5099999999999999e-06 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917020+0000", 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5049999999999998e-06 2024-03-26T05:13:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917031+0000", 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0971e-05 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917040+0000", 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7690000000000002e-06 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917195+0000", 2024-03-26T05:13:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015542099999999999 2024-03-26T05:13:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.917210+0000", 2024-03-26T05:13:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4626e-05 2024-03-26T05:13:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.389350+0000", 2024-03-26T05:13:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.47213961900000001 2024-03-26T05:13:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.389365+0000", 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.556e-05 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562507+0000", 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1731419940000001 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.562650+0000", 2024-03-26T05:13:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014278399999999999 2024-03-26T05:13:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3,5", 2024-03-26T05:13:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.563216+0000", 2024-03-26T05:13:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00056574299999999995 2024-03-26T05:13:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596626+0000", 2024-03-26T05:13:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.033409844000000001 2024-03-26T05:13:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596672+0000", 2024-03-26T05:13:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6075999999999997e-05 2024-03-26T05:13:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596719+0000", 2024-03-26T05:13:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7132000000000002e-05 2024-03-26T05:13:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.606905+0000", 2024-03-26T05:13:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010185886 2024-03-26T05:13:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.606964+0000", 2024-03-26T05:13:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9617999999999997e-05 2024-03-26T05:13:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.607065+0000", 2024-03-26T05:13:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000101153 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206060 7.e 7:75f147c1:::benchmark_data_smithi080_80848_object86561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.940957+0000", 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 463.62421209600001, 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.666353937, 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206060 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.940957+0000", 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.940957+0000", 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.940959+0000", 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5400000000000001e-06 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.940965+0000", 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.947e-06 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.940979+0000", 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4008e-05 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.940988+0000", 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1970000000000002e-06 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.941053+0000", 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4642999999999998e-05 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.941061+0000", 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5329999999999994e-06 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.389945+0000", 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.44888396000000003 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.389955+0000", 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7759999999999992e-06 2024-03-26T05:13:19.575 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.582041+0000", 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.192085727 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.582172+0000", 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013096399999999999 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3,5", 2024-03-26T05:13:19.576 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.582308+0000", 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013676500000000001 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596685+0000", 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014376751 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.596756+0000", 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1322999999999995e-05 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.577 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.607116+0000", 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010359653999999999 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.607166+0000", 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9573000000000001e-05 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.607212+0000", 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6544e-05 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.578 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.607311+0000", 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9031999999999996e-05 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206066 7.e 7:7516dda5:::benchmark_data_smithi080_80848_object86567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.959389+0000", 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 463.60578040600001, 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6554198369999999, 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.579 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206066 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959389+0000", 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.580 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959389+0000", 2024-03-26T05:13:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959390+0000", 2024-03-26T05:13:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3499999999999995e-07 2024-03-26T05:13:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959394+0000", 2024-03-26T05:13:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9049999999999999e-06 2024-03-26T05:13:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959402+0000", 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7930000000000003e-06 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959408+0000", 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7150000000000003e-06 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959469+0000", 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0435e-05 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.959481+0000", 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2065000000000001e-05 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.390011+0000", 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.43053056200000001 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.583 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.390020+0000", 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8130000000000002e-06 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.590994+0000", 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.200974006 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591097+0000", 2024-03-26T05:13:19.584 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000102939 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3,5", 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.591211+0000", 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011416700000000001 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.597026+0000", 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0058151940000000001 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.585 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.586 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.598631+0000", 2024-03-26T05:13:19.586 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001604496 2024-03-26T05:13:19.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.586 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.586 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.586 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.607489+0000", 2024-03-26T05:13:19.586 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0088579139999999997 2024-03-26T05:13:19.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.586 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.586 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.586 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614504+0000", 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0070156089999999999 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614687+0000", 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018228299999999999 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.614809+0000", 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000122206 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.587 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236812 7.e 7:703f6da5:::benchmark_data_smithi080_80848_object117313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.453857+0000", 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 427.111312844, 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99701717899999998, 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236812 2024-03-26T05:13:19.588 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453857+0000", 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453857+0000", 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.589 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453858+0000", 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5969999999999999e-06 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453864+0000", 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2169999999999999e-06 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453879+0000", 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4384e-05 2024-03-26T05:13:19.590 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.591 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.591 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.591 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453888+0000", 2024-03-26T05:13:19.591 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8170000000000005e-06 2024-03-26T05:13:19.591 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.591 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.591 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:19.592 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.592 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453955+0000", 2024-03-26T05:13:19.592 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7786000000000005e-05 2024-03-26T05:13:19.592 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.592 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.592 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.593 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.453968+0000", 2024-03-26T05:13:19.593 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2860999999999999e-05 2024-03-26T05:13:19.593 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.593 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.593 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.593 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.443220+0000", 2024-03-26T05:13:19.593 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.989251518 2024-03-26T05:13:19.593 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.593 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.593 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.593 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.443311+0000", 2024-03-26T05:13:19.593 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0847999999999995e-05 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.443402+0000", 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1513000000000003e-05 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.445956+0000", 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0025534899999999998 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.594 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.448724+0000", 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027681440000000002 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.450797+0000", 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0020734920000000001 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.450839+0000", 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1789000000000003e-05 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.595 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.596 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.596 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.450874+0000", 2024-03-26T05:13:19.596 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4722999999999999e-05 2024-03-26T05:13:19.596 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.596 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.596 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.596 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.596 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.596 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236839 7.e 7:7f30ee9a:::benchmark_data_smithi080_80848_object117340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:13:19.596 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.489276+0000", 2024-03-26T05:13:19.596 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 427.07589322699999, 2024-03-26T05:13:19.596 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96184769999999997, 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236839 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489276+0000", 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.597 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489276+0000", 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489278+0000", 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.73e-06 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489284+0000", 2024-03-26T05:13:19.598 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5010000000000001e-06 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489298+0000", 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3339000000000001e-05 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489307+0000", 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7219999999999994e-06 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489373+0000", 2024-03-26T05:13:19.599 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5878999999999997e-05 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.489382+0000", 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.098e-06 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.446003+0000", 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95662086599999996 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.446115+0000", 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011173800000000001 2024-03-26T05:13:19.600 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.446193+0000", 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8335000000000001e-05 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451000+0000", 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0048067680000000003 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451047+0000", 2024-03-26T05:13:19.601 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7169999999999997e-05 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451056+0000", 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9190000000000005e-06 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451086+0000", 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.9490000000000001e-05 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.602 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451124+0000", 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8145000000000002e-05 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236853 7.e 7:7a866308:::benchmark_data_smithi080_80848_object117354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.506895+0000", 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 427.05827408300001, 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.94437797099999998, 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.603 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236853 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506895+0000", 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506895+0000", 2024-03-26T05:13:19.604 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506897+0000", 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.579e-06 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506902+0000", 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8670000000000002e-06 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506914+0000", 2024-03-26T05:13:19.605 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2329e-05 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506922+0000", 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0989999999999999e-06 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506978+0000", 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5912000000000003e-05 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.606 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.506991+0000", 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3091000000000001e-05 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.448760+0000", 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.94176837199999996 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.448849+0000", 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9405000000000002e-05 2024-03-26T05:13:19.607 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.448923+0000", 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4035000000000005e-05 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451132+0000", 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022092420000000002 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451167+0000", 2024-03-26T05:13:19.608 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5067999999999997e-05 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451212+0000", 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4963000000000003e-05 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451253+0000", 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0608999999999997e-05 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.609 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451273+0000", 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0400000000000001e-05 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236855 7.e 7:7174933d:::benchmark_data_smithi080_80848_object117356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.507103+0000", 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 427.05806603000002, 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95728903099999996, 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.610 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236855 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507103+0000", 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.611 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507103+0000", 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507104+0000", 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1399999999999996e-07 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507108+0000", 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0890000000000002e-06 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.612 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507118+0000", 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8430000000000005e-06 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507125+0000", 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0609999999999998e-06 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507184+0000", 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8700999999999999e-05 2024-03-26T05:13:19.613 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.614 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.614 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.614 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.507191+0000", 2024-03-26T05:13:19.614 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4089999999999997e-06 2024-03-26T05:13:19.614 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.614 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.614 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.614 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451291+0000", 2024-03-26T05:13:19.614 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.944100151 2024-03-26T05:13:19.614 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.614 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.614 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451359+0000", 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7286999999999995e-05 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.451420+0000", 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0949999999999998e-05 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464151+0000", 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.012730844 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.615 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464218+0000", 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7546999999999996e-05 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464288+0000", 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0036000000000005e-05 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464347+0000", 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8801999999999997e-05 2024-03-26T05:13:19.616 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464392+0000", 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5497000000000002e-05 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236870 7.e 7:7a7ed7ca:::benchmark_data_smithi080_80848_object117371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.520180+0000", 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 427.04498905999998, 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.94435229899999995, 2024-03-26T05:13:19.617 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.618 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.618 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.618 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.618 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.618 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236870 2024-03-26T05:13:19.618 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.618 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.618 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.618 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.618 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520180+0000", 2024-03-26T05:13:19.618 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.618 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520180+0000", 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520182+0000", 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.468e-06 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520187+0000", 2024-03-26T05:13:19.619 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0900000000000004e-06 2024-03-26T05:13:19.620 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.620 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.620 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.620 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520204+0000", 2024-03-26T05:13:19.620 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7142e-05 2024-03-26T05:13:19.620 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.620 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.620 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.620 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520213+0000", 2024-03-26T05:13:19.620 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.3049999999999998e-06 2024-03-26T05:13:19.620 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.620 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.621 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.621 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520350+0000", 2024-03-26T05:13:19.621 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000136978 2024-03-26T05:13:19.621 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.621 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.621 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.621 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.520358+0000", 2024-03-26T05:13:19.621 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6490000000000008e-06 2024-03-26T05:13:19.621 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.621 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.621 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.622 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.453989+0000", 2024-03-26T05:13:19.622 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.933631447 2024-03-26T05:13:19.622 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.622 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.622 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.622 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.454089+0000", 2024-03-26T05:13:19.622 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010003 2024-03-26T05:13:19.622 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.622 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.622 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.454198+0000", 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000108436 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464202+0000", 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010004413 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464226+0000", 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3917000000000002e-05 2024-03-26T05:13:19.623 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464437+0000", 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000210645 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464494+0000", 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7138000000000001e-05 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464533+0000", 2024-03-26T05:13:19.624 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8640999999999998e-05 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236882 7.e 7:74c18bc4:::benchmark_data_smithi080_80848_object117383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.532839+0000", 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 427.03232999400001, 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93196347400000001, 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.625 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236882 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532839+0000", 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532839+0000", 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.626 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532841+0000", 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5770000000000001e-06 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532847+0000", 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7300000000000002e-06 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532860+0000", 2024-03-26T05:13:19.627 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2891000000000001e-05 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532868+0000", 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.259e-06 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532976+0000", 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010864799999999999 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.628 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.532985+0000", 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0700000000000007e-06 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.456804+0000", 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.92381993200000001 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.457000+0000", 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019507699999999999 2024-03-26T05:13:19.629 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.457116+0000", 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000116297 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464232+0000", 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0071157690000000001 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.630 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464576+0000", 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000344295 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464618+0000", 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2042999999999997e-05 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464765+0000", 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014673299999999999 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.631 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464803+0000", 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8152999999999999e-05 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236902 7.e 7:7eec6e5e:::benchmark_data_smithi080_80848_object117403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.551741+0000", 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 427.01342876299998, 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93036452400000003, 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.632 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236902 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551741+0000", 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551741+0000", 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.633 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551742+0000", 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4920000000000001e-06 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551748+0000", 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5790000000000003e-06 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.634 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551765+0000", 2024-03-26T05:13:19.636 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7620000000000001e-05 2024-03-26T05:13:19.636 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.636 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.636 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.636 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551775+0000", 2024-03-26T05:13:19.636 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2359999999999993e-06 2024-03-26T05:13:19.636 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.636 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.636 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.636 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551846+0000", 2024-03-26T05:13:19.636 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1070999999999998e-05 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.551854+0000", 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5469999999999997e-06 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.460854+0000", 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.90899986499999996 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.460962+0000", 2024-03-26T05:13:19.637 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000107797 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.461367+0000", 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00040546499999999998 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.471563+0000", 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010195895999999999 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.638 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.471608+0000", 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4672999999999997e-05 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482013+0000", 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010404665 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482062+0000", 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9329000000000002e-05 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482105+0000", 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3288999999999998e-05 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236911 7.e 7:77b891a1:::benchmark_data_smithi080_80848_object117412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.556123+0000", 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 427.00904671400002, 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.92610570000000003, 2024-03-26T05:13:19.640 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236911 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.556123+0000", 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.641 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.556123+0000", 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.556124+0000", 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.677e-06 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.556130+0000", 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.169e-06 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.642 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.556148+0000", 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8705000000000001e-05 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.556158+0000", 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0017000000000001e-05 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.558482+0000", 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002324156 2024-03-26T05:13:19.643 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.558495+0000", 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2238e-05 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464848+0000", 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.90635296700000001 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.464957+0000", 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010899399999999999 2024-03-26T05:13:19.644 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.465070+0000", 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000112959 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.471649+0000", 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0065793639999999999 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.475930+0000", 2024-03-26T05:13:19.645 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0042813010000000004 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482152+0000", 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0062221480000000003 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482194+0000", 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1829999999999998e-05 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.646 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482228+0000", 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4174999999999997e-05 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236941 7.e 7:756847e5:::benchmark_data_smithi080_80848_object117442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.586338+0000", 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 426.97883177300002, 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.89596953999999995, 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.647 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236941 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586338+0000", 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.648 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586338+0000", 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586339+0000", 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.685e-06 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586346+0000", 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5629999999999997e-06 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.649 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586365+0000", 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8972999999999999e-05 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586375+0000", 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0319999999999999e-05 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586454+0000", 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9178000000000007e-05 2024-03-26T05:13:19.650 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.651 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.651 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.651 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.586464+0000", 2024-03-26T05:13:19.651 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5370000000000003e-06 2024-03-26T05:13:19.651 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.651 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.651 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.651 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.468430+0000", 2024-03-26T05:13:19.651 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.88196650799999998 2024-03-26T05:13:19.651 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.651 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.651 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.652 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.468546+0000", 2024-03-26T05:13:19.652 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115884 2024-03-26T05:13:19.652 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.652 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.652 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:13:19.652 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.468660+0000", 2024-03-26T05:13:19.652 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000113459 2024-03-26T05:13:19.652 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.652 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.652 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.652 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.481992+0000", 2024-03-26T05:13:19.652 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.013331989000000001 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482142+0000", 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000149932 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482234+0000", 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2281999999999995e-05 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.653 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482274+0000", 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0361000000000002e-05 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482307+0000", 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2869e-05 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236962 7.e 7:7543a6bb:::benchmark_data_smithi080_80848_object117463:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:13:19.654 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.607100+0000", 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 426.95806919099999, 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.890074221, 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236962 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.655 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607100+0000", 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607100+0000", 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607102+0000", 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5540000000000001e-06 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.656 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607107+0000", 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8760000000000004e-06 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607120+0000", 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3003000000000001e-05 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607128+0000", 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9079999999999995e-06 2024-03-26T05:13:19.657 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607185+0000", 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.749e-05 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.607193+0000", 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8709999999999997e-06 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.658 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.471679+0000", 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.86448656899999998 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.471781+0000", 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000101777 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.471901+0000", 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000119647 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.659 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482312+0000", 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010411521 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482350+0000", 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7594e-05 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.496765+0000", 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014415193999999999 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.660 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497123+0000", 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000357513 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497174+0000", 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1703999999999998e-05 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.661 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236970 7.e 7:73842dff:::benchmark_data_smithi080_80848_object117471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.617214+0000", 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 426.94795523800002, 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.88018738900000004, 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236970 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.662 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617214+0000", 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617214+0000", 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617216+0000", 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8390000000000001e-06 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.663 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617221+0000", 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4419999999999997e-06 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617235+0000", 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3132999999999999e-05 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617243+0000", 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.755e-06 2024-03-26T05:13:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617308+0000", 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4640000000000005e-05 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.617318+0000", 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8840000000000006e-06 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.475962+0000", 2024-03-26T05:13:19.665 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.85864369799999996 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.476087+0000", 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012535899999999999 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.476209+0000", 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000122397 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.666 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482317+0000", 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0061080370000000002 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.482387+0000", 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9219000000000003e-05 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497219+0000", 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014832451999999999 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.667 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497363+0000", 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014368700000000001 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497402+0000", 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8847000000000003e-05 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.668 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:236991 7.e 7:79f2f371:::benchmark_data_smithi080_80848_object117492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:06:12.641178+0000", 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 426.92399116899998, 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.85638282899999996, 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 236991 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.669 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641178+0000", 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641178+0000", 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641180+0000", 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.39e-06 2024-03-26T05:13:19.670 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641186+0000", 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7009999999999999e-06 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641197+0000", 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.047e-05 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641204+0000", 2024-03-26T05:13:19.671 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.759e-06 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641255+0000", 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1249999999999999e-05 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:12.641265+0000", 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8260000000000005e-06 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.478969+0000", 2024-03-26T05:13:19.672 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.83770405299999995 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.479078+0000", 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010932700000000001 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.479192+0000", 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000114244 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.488347+0000", 2024-03-26T05:13:19.673 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0091552439999999999 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.491459+0000", 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0031117340000000001 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497460+0000", 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.006000521 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.674 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497518+0000", 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8733999999999997e-05 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:06:13.497561+0000", 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2576e-05 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:94673 8.4 8:27890670:::benchmark_data_smithi080_109213_object94672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T05:13:19.675 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:59.948789+0000", 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 319.61638008300002, 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2924905719999999, 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 94673 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.948789+0000", 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.676 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.948789+0000", 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.948791+0000", 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7770000000000001e-06 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.948829+0000", 2024-03-26T05:13:19.677 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7764e-05 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.948847+0000", 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.77e-05 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.948858+0000", 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1610000000000001e-05 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.678 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953737+0000", 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.004879146 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953746+0000", 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.149e-06 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.227982+0000", 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.274235945 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.679 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.229412+0000", 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014291989999999999 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,5,6", 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.229601+0000", 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018986599999999999 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.240892+0000", 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.011290134 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.680 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.240913+0000", 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1382999999999998e-05 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241019+0000", 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010596500000000001 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241107+0000", 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7849000000000005e-05 2024-03-26T05:13:19.681 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241170+0000", 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2898000000000007e-05 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241280+0000", 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011018699999999999 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.682 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:94680 8.d 8:b14d91ac:::benchmark_data_smithi080_109213_object94679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:59.950462+0000", 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 319.61470783499999, 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.286237412, 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 94680 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.683 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.950462+0000", 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.950462+0000", 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.950463+0000", 2024-03-26T05:13:19.684 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.004e-06 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.950547+0000", 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4522000000000002e-05 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.952906+0000", 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0023584489999999999 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.685 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.952925+0000", 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9752999999999999e-05 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953683+0000", 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00075799099999999996 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953695+0000", 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1571e-05 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.686 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.225855+0000", 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2721605570000001 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.227764+0000", 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001908428 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6,7", 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.227964+0000", 2024-03-26T05:13:19.687 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000200306 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.233395+0000", 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0054312329999999997 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.235301+0000", 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001906105 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236142+0000", 2024-03-26T05:13:19.688 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00084030800000000001 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236510+0000", 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00036848300000000002 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236573+0000", 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2756999999999996e-05 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.689 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236699+0000", 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000125945 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:94684 8.4 8:22a13d96:::benchmark_data_smithi080_109213_object94683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:59.953511+0000", 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 319.611658213, 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2881801740000001, 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.690 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:13:19.691 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:13:19.691 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 94684 2024-03-26T05:13:19.691 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.691 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.691 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.691 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.692 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953511+0000", 2024-03-26T05:13:19.692 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.692 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.692 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953511+0000", 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953513+0000", 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.373e-06 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953551+0000", 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8426000000000001e-05 2024-03-26T05:13:19.693 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.694 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.694 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.694 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953563+0000", 2024-03-26T05:13:19.694 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1569999999999999e-05 2024-03-26T05:13:19.694 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.694 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.694 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.694 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953572+0000", 2024-03-26T05:13:19.694 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5889999999999992e-06 2024-03-26T05:13:19.694 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.694 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.694 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953772+0000", 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000200027 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953779+0000", 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3189999999999997e-06 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.232106+0000", 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.278326329 2024-03-26T05:13:19.695 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.232238+0000", 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013240300000000001 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,5,6", 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.232399+0000", 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000161082 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.240960+0000", 2024-03-26T05:13:19.696 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.008560916 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.240973+0000", 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3251000000000001e-05 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241061+0000", 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7934000000000002e-05 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.697 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241532+0000", 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00047092299999999999 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241590+0000", 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8105e-05 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241691+0000", 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000100927 2024-03-26T05:13:19.698 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:94695 8.d 8:bd3743b0:::benchmark_data_smithi080_109213_object94694:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:59.953586+0000", 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 319.61158371599998, 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2833069830000001, 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:13:19.699 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 94695 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953586+0000", 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953586+0000", 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953586+0000", 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7999999999999995e-07 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953614+0000", 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.8221e-05 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953625+0000", 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0037000000000001e-05 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953631+0000", 2024-03-26T05:13:19.701 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8029999999999999e-06 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.954047+0000", 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00041608900000000002 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.954056+0000", 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6899999999999998e-06 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.230573+0000", 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2765164849999999 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.230722+0000", 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014921300000000001 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6,7", 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.230890+0000", 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016784600000000001 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.233965+0000", 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0030752990000000001 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.235525+0000", 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0015597969999999999 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236186+0000", 2024-03-26T05:13:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00066131299999999996 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236736+0000", 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00055022600000000001 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236786+0000", 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0059999999999998e-05 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236893+0000", 2024-03-26T05:13:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010632399999999999 2024-03-26T05:13:19.706 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.706 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.706 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.706 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.706 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.706 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:94700 8.4 8:245b8f55:::benchmark_data_smithi080_109213_object94699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T05:13:19.706 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:07:59.953642+0000", 2024-03-26T05:13:19.706 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 319.611527282, 2024-03-26T05:13:19.706 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2889140180000001, 2024-03-26T05:13:19.706 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:19.706 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 94700 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953642+0000", 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953642+0000", 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953643+0000", 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9699999999999996e-07 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953671+0000", 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.8181999999999999e-05 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953678+0000", 2024-03-26T05:13:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5839999999999997e-06 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953685+0000", 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2450000000000003e-06 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953804+0000", 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000119354 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:07:59.953811+0000", 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1810000000000004e-06 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.235579+0000", 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2817674969999999 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.235799+0000", 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00022075099999999999 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,5,6", 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.236011+0000", 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021142099999999999 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.240979+0000", 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.004968175 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.241892+0000", 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00091299600000000001 2024-03-26T05:13:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.242287+0000", 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00039454099999999999 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.242324+0000", 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7509999999999998e-05 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.242378+0000", 2024-03-26T05:13:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3770000000000002e-05 2024-03-26T05:13:19.713 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:19.713 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:19.713 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:19.713 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:01.242556+0000", 2024-03-26T05:13:19.713 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000178314 2024-03-26T05:13:19.713 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.713 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.713 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.713 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:19.713 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:19.713 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:13:19.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:19.714 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:19.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:19.815 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:19.856 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:19.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:19.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:19 smithi080 ceph-mon[29691]: 6.d scrub starts 2024-03-26T05:13:19.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:19 smithi080 ceph-mon[29691]: 6.d scrub ok 2024-03-26T05:13:19.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:19 smithi080 ceph-mon[40404]: 6.d scrub starts 2024-03-26T05:13:19.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:19 smithi080 ceph-mon[40404]: 6.d scrub ok 2024-03-26T05:13:19.958 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:19.997 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:19.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:20.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:19 smithi132 ceph-mon[33500]: 6.d scrub starts 2024-03-26T05:13:20.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:19 smithi132 ceph-mon[33500]: 6.d scrub ok 2024-03-26T05:13:20.099 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:20.126 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:20.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:20.228 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:20.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:20.269 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:20.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:20.370 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:20.411 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:20.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:20.512 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:20.552 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:20.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:20.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:20.654 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:20.692 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:20.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:20.794 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:20.821 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:20.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:20.923 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:20.951 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:20.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:20 smithi132 ceph-mon[33500]: 6.10 scrub starts 2024-03-26T05:13:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:20 smithi132 ceph-mon[33500]: pgmap v585: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 42857/229632 objects degraded (18.663%) 2024-03-26T05:13:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:20 smithi132 ceph-mon[33500]: 6.10 scrub ok 2024-03-26T05:13:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:13:21.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:13:21.053 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:21.079 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:21.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:21.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:21.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:20 smithi080 ceph-mon[40404]: 6.10 scrub starts 2024-03-26T05:13:21.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:20 smithi080 ceph-mon[40404]: pgmap v585: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 42857/229632 objects degraded (18.663%) 2024-03-26T05:13:21.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:20 smithi080 ceph-mon[40404]: 6.10 scrub ok 2024-03-26T05:13:21.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:13:21.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:13:21.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:20 smithi080 ceph-mon[29691]: 6.10 scrub starts 2024-03-26T05:13:21.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:20 smithi080 ceph-mon[29691]: pgmap v585: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 42857/229632 objects degraded (18.663%) 2024-03-26T05:13:21.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:20 smithi080 ceph-mon[29691]: 6.10 scrub ok 2024-03-26T05:13:21.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:13:21.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:13:21.181 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:21.208 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:21.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:21.310 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:21.336 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:21.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:21.428 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:21.438 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:21.486 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:21.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:21.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:21.614 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:21.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:21.716 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:21.759 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:21.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:21.861 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:21.887 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:21.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:21.990 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:22.015 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:22.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:21 smithi132 ceph-mon[33500]: 9.2 scrub starts 2024-03-26T05:13:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:21 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:13:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:21 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:13:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:21 smithi132 ceph-mon[33500]: 5.3 scrub starts 2024-03-26T05:13:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:21 smithi132 ceph-mon[33500]: 5.3 scrub ok 2024-03-26T05:13:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:21 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 42857/229632 objects degraded (18.663%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:22.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:21 smithi080 ceph-mon[40404]: 9.2 scrub starts 2024-03-26T05:13:22.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:21 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:13:22.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:21 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:13:22.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:21 smithi080 ceph-mon[40404]: 5.3 scrub starts 2024-03-26T05:13:22.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:21 smithi080 ceph-mon[40404]: 5.3 scrub ok 2024-03-26T05:13:22.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:21 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 42857/229632 objects degraded (18.663%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:22.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:21 smithi080 ceph-mon[29691]: 9.2 scrub starts 2024-03-26T05:13:22.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:21 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:13:22.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:21 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:13:22.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:21 smithi080 ceph-mon[29691]: 5.3 scrub starts 2024-03-26T05:13:22.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:21 smithi080 ceph-mon[29691]: 5.3 scrub ok 2024-03-26T05:13:22.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:21 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 42857/229632 objects degraded (18.663%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:22.118 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:22.158 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:22.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:22.259 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:22.287 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:22.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:22.389 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:22.428 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:22.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:22.530 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:22.570 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:22.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:22.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:22.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:22.700 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:22.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:22.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:22.745 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:13:22.801 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:22.845 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:22.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:22.947 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:22.982 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:22.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:23.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:22 smithi132 ceph-mon[33500]: osdmap e303: 8 total, 7 up, 8 in 2024-03-26T05:13:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:22 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:13:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:22 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:13:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:22 smithi132 ceph-mon[33500]: 6.c scrub starts 2024-03-26T05:13:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:22 smithi132 ceph-mon[33500]: 6.c scrub ok 2024-03-26T05:13:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:22 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/316239097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:13:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:22 smithi132 ceph-mon[33500]: 4.1a scrub starts 2024-03-26T05:13:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:22 smithi132 ceph-mon[33500]: pgmap v587: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 39628/211620 objects degraded (18.726%) 2024-03-26T05:13:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:22 smithi132 ceph-mon[33500]: 4.1a scrub ok 2024-03-26T05:13:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:22 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:13:23.083 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:23.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[29691]: osdmap e303: 8 total, 7 up, 8 in 2024-03-26T05:13:23.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:13:23.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:13:23.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[29691]: 6.c scrub starts 2024-03-26T05:13:23.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[29691]: 6.c scrub ok 2024-03-26T05:13:23.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/316239097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:13:23.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[29691]: 4.1a scrub starts 2024-03-26T05:13:23.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[29691]: pgmap v587: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 39628/211620 objects degraded (18.726%) 2024-03-26T05:13:23.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[29691]: 4.1a scrub ok 2024-03-26T05:13:23.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:13:23.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[40404]: osdmap e303: 8 total, 7 up, 8 in 2024-03-26T05:13:23.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:13:23.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:13:23.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[40404]: 6.c scrub starts 2024-03-26T05:13:23.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[40404]: 6.c scrub ok 2024-03-26T05:13:23.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/316239097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:13:23.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[40404]: 4.1a scrub starts 2024-03-26T05:13:23.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[40404]: pgmap v587: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 39628/211620 objects degraded (18.726%) 2024-03-26T05:13:23.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[40404]: 4.1a scrub ok 2024-03-26T05:13:23.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:22 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:13:23.134 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:23.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:23.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:23.236 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:23.285 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:23.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:23.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:23.341 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T05:13:23.388 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:23.447 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:23.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:23.548 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:23.574 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:23.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:23.676 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:23.703 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:23.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:23.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:23.805 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:23.862 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:23.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:23.963 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:24.003 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:24.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:24.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:23 smithi080 ceph-mon[40404]: 9.2 scrub ok 2024-03-26T05:13:24.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:23 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:13:24.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:23 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/316239097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:13:24.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:23 smithi080 ceph-mon[40404]: osdmap e304: 8 total, 7 up, 8 in 2024-03-26T05:13:24.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:23 smithi080 ceph-mon[29691]: 9.2 scrub ok 2024-03-26T05:13:24.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:23 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:13:24.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:23 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/316239097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:13:24.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:23 smithi080 ceph-mon[29691]: osdmap e304: 8 total, 7 up, 8 in 2024-03-26T05:13:24.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:23 smithi132 ceph-mon[33500]: 9.2 scrub ok 2024-03-26T05:13:24.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:23 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:13:24.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:23 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/316239097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:13:24.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:23 smithi132 ceph-mon[33500]: osdmap e304: 8 total, 7 up, 8 in 2024-03-26T05:13:24.105 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:24.131 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:24.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:24.233 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:24.279 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:24.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:24.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:24.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:13:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:13:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:13:24.385 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:24.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:24.435 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:24.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:24.536 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:24.575 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:24.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:24.677 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:24.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:24.718 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:24.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:24.819 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:24.858 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:24.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:24.960 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:24.989 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:24 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:13:24.989 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:24 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:13:24.989 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:24 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:13:24.989 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:24 smithi080 ceph-mon[40404]: pgmap v589: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 39628/211620 objects degraded (18.726%) 2024-03-26T05:13:24.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:24 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:13:24.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:24 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:13:24.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:24 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:13:24.991 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:24 smithi080 ceph-mon[29691]: pgmap v589: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 39628/211620 objects degraded (18.726%) 2024-03-26T05:13:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:25.017 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:25.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:24 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:13:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:24 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:13:25.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:24 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:13:25.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:24 smithi132 ceph-mon[33500]: pgmap v589: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 39628/211620 objects degraded (18.726%) 2024-03-26T05:13:25.118 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:25.146 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:25.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:25.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:25.248 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:25.284 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:25.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:25.386 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:25.410 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:25.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:25.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:25.512 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:25.539 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:25.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:25.640 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:25.679 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:25.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:25.781 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:25.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:13:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:13:25.808 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:25.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:25.910 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:25.952 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:25.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:26.054 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:26.094 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:26.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:26.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:26.145 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:13:26.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:25 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:26.184 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:25 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:26.196 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:26.251 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:26.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:26.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:25 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:26.353 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:26.400 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:26.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:26.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:26.502 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:26.530 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:26.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:26.631 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:26.658 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:26.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:26.759 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:26.797 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:26.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:26.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:26.899 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:26.926 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:26.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:27.028 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:27.070 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:27.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:27.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:27.172 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:27.208 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:27.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:26 smithi132 ceph-mon[33500]: pgmap v590: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 37338/199137 objects degraded (18.750%) 2024-03-26T05:13:27.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:26 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 39628/211620 objects degraded (18.726%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:27.308 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:27.337 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:27.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:27.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:26 smithi080 ceph-mon[29691]: pgmap v590: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 37338/199137 objects degraded (18.750%) 2024-03-26T05:13:27.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:26 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 39628/211620 objects degraded (18.726%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:27.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:26 smithi080 ceph-mon[40404]: pgmap v590: 148 pgs: 1 active+clean+scrubbing, 86 active+clean, 33 active+undersized, 28 active+undersized+degraded; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 37338/199137 objects degraded (18.750%) 2024-03-26T05:13:27.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:26 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 39628/211620 objects degraded (18.726%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T05:13:27.439 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:28.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:28.714 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:28.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:28.815 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:28.844 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:28.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:28.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:28.946 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:28.974 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:28.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:28.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:29.076 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:29.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:29.118 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:29.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:29.219 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:29.233 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:29.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:29.249 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:29.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:29.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:29 smithi132 ceph-mon[33500]: pgmap v591: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 36190/193491 objects degraded (18.704%) 2024-03-26T05:13:29.350 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:29.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:29 smithi080 ceph-mon[40404]: pgmap v591: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 36190/193491 objects degraded (18.704%) 2024-03-26T05:13:29.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:29 smithi080 ceph-mon[29691]: pgmap v591: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 36190/193491 objects degraded (18.704%) 2024-03-26T05:13:29.378 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:29.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:29.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:29.480 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:29.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:29.537 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:29.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:29.638 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:29.674 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:29.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:29.776 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:29.814 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:29.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:29.915 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:29.961 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:29.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:30.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:30.089 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:30.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:30.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:30.115 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:13:30.155 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:13:30.190 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:30.198 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:13:30.231 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:30.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:30.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:30.242 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:13:30.282 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:13:30.324 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:13:30.333 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:30.357 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:13:30.360 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:30.360 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:13:30.360 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:13:30.361 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:13:30.361 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:13:30.361 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:13:30.361 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:13:30.361 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:13:30.361 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:13:30.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:30.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:30.463 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:30.491 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:30.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:30.593 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:30.620 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:30.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:30.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:30.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:30.754 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:30.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:30.855 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:30.886 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:30.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:30.987 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:31.017 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:31.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:31.119 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:31.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:31.161 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:31.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:31.262 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:31.292 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:31.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:31.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:31 smithi080 ceph-mon[40404]: pgmap v592: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 620 B/s rd, 0 B/s wr, 779 op/s; 35801/192324 objects degraded (18.615%) 2024-03-26T05:13:31.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:31 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3399919014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:13:31.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:31 smithi080 ceph-mon[29691]: pgmap v592: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 620 B/s rd, 0 B/s wr, 779 op/s; 35801/192324 objects degraded (18.615%) 2024-03-26T05:13:31.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:31 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3399919014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:13:31.394 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:31.447 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:31.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:31.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:31.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:31 smithi132 ceph-mon[33500]: pgmap v592: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 620 B/s rd, 0 B/s wr, 779 op/s; 35801/192324 objects degraded (18.615%) 2024-03-26T05:13:31.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:31 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3399919014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:13:31.550 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:31.577 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:31.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:31.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:31.718 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:31.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:31.820 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:31.859 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:31.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:31.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:31.961 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:31.988 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:31.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:32.090 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:32.129 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:32.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:32.230 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:32.259 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:32.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:32.361 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:32.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:32 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 35801/192324 objects degraded (18.615%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:32.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:32 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 35801/192324 objects degraded (18.615%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:32.399 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:32.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:32.501 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:32.530 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:32.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:32.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:32 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 35801/192324 objects degraded (18.615%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:32.631 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:32.658 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:32.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:32.760 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:32.787 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:32.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:32.890 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:32.925 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:32.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:33.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:33.055 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:33.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:33.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:33.156 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:33.186 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:33.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:33.288 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:33.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:33.341 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:33.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:33.344 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:33 smithi080 ceph-mon[40404]: pgmap v593: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 32630/174117 objects degraded (18.740%) 2024-03-26T05:13:33.344 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:33 smithi080 ceph-mon[29691]: pgmap v593: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 32630/174117 objects degraded (18.740%) 2024-03-26T05:13:33.360 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:33.442 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:33.481 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:13:33.481 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:13:33.481 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:13:33.481 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:13:33.481 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:13:33.482 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:13:33.482 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:13:33.482 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:13:33.482 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:13:33.482 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:13:33.482 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:13:33.482 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:13:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:33.488 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:33.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:33.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:33 smithi132 ceph-mon[33500]: pgmap v593: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 32630/174117 objects degraded (18.740%) 2024-03-26T05:13:33.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:33.617 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:33.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:33.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:33.719 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:33.757 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:33.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:33.858 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:33.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:33.888 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:33.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:33.989 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:34.012 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:34.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:34.114 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:34.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:34.155 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:34.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:34.256 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:34.295 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:34.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:34.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:13:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:13:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:13:34.397 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:34.436 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:34.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:34.538 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:34.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:34.567 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:34.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:34.668 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:34.696 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:34.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:34.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:34.823 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:34.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:34.925 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:34.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:34.963 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:34.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:35.064 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:35.093 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:35.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:35.194 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:35.226 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:35.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:35.297 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:35 smithi080 ceph-mon[29691]: pgmap v594: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 910 B/s rd, 0 B/s wr, 1.11k op/s; 32630/174117 objects degraded (18.740%) 2024-03-26T05:13:35.297 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:35 smithi080 ceph-mon[40404]: pgmap v594: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 910 B/s rd, 0 B/s wr, 1.11k op/s; 32630/174117 objects degraded (18.740%) 2024-03-26T05:13:35.327 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:35.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:35 smithi132 ceph-mon[33500]: pgmap v594: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 910 B/s rd, 0 B/s wr, 1.11k op/s; 32630/174117 objects degraded (18.740%) 2024-03-26T05:13:35.367 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:35.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:13:35.453 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:13:35.469 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:35.507 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:35.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:35.510 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:13:35.510 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:13:35.610 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:35.621 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:13:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:13:35.645 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:35.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:35.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:35.747 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:35.775 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:35.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:35.876 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:35.907 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:35.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:36.009 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:36.039 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:36.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:36.141 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:36.180 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:36.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:36.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:36.282 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:36.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:36 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:13:36.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:36 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:13:36.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:36.317 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:36.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:36.418 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:13:36.437 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:13:36.437 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-26T05:13:36.438 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:36 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:13:36.438 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:36 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:13:36.438 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:36.438 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:36 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:13:36.439 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:36 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:13:36.439 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:36.464 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:36.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:36.566 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:36.594 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:36.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:36.696 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:36.740 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:36.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:36.842 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:36.856 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T05:13:36.903 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:36.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:36.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:37.005 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:37.036 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:37.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:37.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:37.138 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:37.168 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:37.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:37.269 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:37.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:37.325 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:37.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:37.426 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:37.454 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:13:37.457 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:37.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:37.498 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:37 smithi080 ceph-mon[40404]: pgmap v595: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 31026/164592 objects degraded (18.850%) 2024-03-26T05:13:37.498 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:37 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 31026/164592 objects degraded (18.850%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:37.498 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:37 smithi080 ceph-mon[29691]: pgmap v595: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 31026/164592 objects degraded (18.850%) 2024-03-26T05:13:37.498 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:37 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 31026/164592 objects degraded (18.850%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:37.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:37 smithi132 ceph-mon[33500]: pgmap v595: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 31026/164592 objects degraded (18.850%) 2024-03-26T05:13:37.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:37 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 31026/164592 objects degraded (18.850%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:37.559 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:37.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:37.608 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:37.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:37.709 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:37.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:37.768 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:37.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:37.851 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:37.869 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:37.936 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:37.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:37.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:38.025 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:38.038 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:38.093 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:38.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:38.194 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:38.220 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:38.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:38.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:38.322 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:38.363 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:38.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:38.465 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:38.494 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:38.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:38.595 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:38.622 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:38.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:38.724 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:38.750 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:38.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:38.852 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:38.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:38.894 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:38.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:38.995 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:39.039 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:39.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:39.140 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:39.178 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:39.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:39.188 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:13:39.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:39.307 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:39.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:39.409 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:39.439 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:39.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:39.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:39.541 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:39.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:39 smithi132 ceph-mon[33500]: pgmap v596: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s; 29044/155019 objects degraded (18.736%) 2024-03-26T05:13:39.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:39 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1820837583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:13:39.578 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:39.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:39.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:39.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:39 smithi080 ceph-mon[40404]: pgmap v596: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s; 29044/155019 objects degraded (18.736%) 2024-03-26T05:13:39.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:39 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1820837583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:13:39.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:39 smithi080 ceph-mon[29691]: pgmap v596: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s; 29044/155019 objects degraded (18.736%) 2024-03-26T05:13:39.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:39 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1820837583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:13:39.680 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:39.721 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:39.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:39.822 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:39.849 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:39.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:39.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:39.950 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:39.979 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:39.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:40.081 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:40.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:40.119 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:40.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:40.220 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:40.253 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:40.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:40.356 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:40.379 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:40.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:40.481 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:40.509 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:40.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:40.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:40 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:13:40.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:40 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1820837583' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:13:40.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:40 smithi132 ceph-mon[33500]: osdmap e305: 8 total, 7 up, 8 in 2024-03-26T05:13:40.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:40 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:13:40.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:40 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:13:40.611 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:40.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:40 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:13:40.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:40 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1820837583' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:13:40.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:40 smithi080 ceph-mon[40404]: osdmap e305: 8 total, 7 up, 8 in 2024-03-26T05:13:40.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:40 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:13:40.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:40 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:13:40.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:40 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:13:40.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:40 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1820837583' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:13:40.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:40 smithi080 ceph-mon[29691]: osdmap e305: 8 total, 7 up, 8 in 2024-03-26T05:13:40.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:40 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:13:40.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:40 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:13:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:40.640 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:40.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:40.741 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:40.780 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:40.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:40.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:40.882 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:40.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:40.944 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:40.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:41.045 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:41.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:41.091 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:41.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:41.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:41.238 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:41.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:41.339 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:41 smithi080 ceph-mon[40404]: pgmap v598: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s; 28591/153660 objects degraded (18.607%) 2024-03-26T05:13:41.339 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:41 smithi080 ceph-mon[40404]: osdmap e306: 8 total, 7 up, 8 in 2024-03-26T05:13:41.339 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:41 smithi080 ceph-mon[29691]: pgmap v598: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s; 28591/153660 objects degraded (18.607%) 2024-03-26T05:13:41.339 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:41 smithi080 ceph-mon[29691]: osdmap e306: 8 total, 7 up, 8 in 2024-03-26T05:13:41.340 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:41.367 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:41.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:41.468 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:41.510 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:41.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:41.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:41 smithi132 ceph-mon[33500]: pgmap v598: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 3.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s; 28591/153660 objects degraded (18.607%) 2024-03-26T05:13:41.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:41 smithi132 ceph-mon[33500]: osdmap e306: 8 total, 7 up, 8 in 2024-03-26T05:13:41.611 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:41.637 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:41.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:41.738 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:41.786 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:41.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:41.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:41.888 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:41.918 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:41.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:42.020 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:42.057 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:42.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:42.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:42.196 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:42.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:42.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:42.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:42.335 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:42.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:42.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:42.437 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:42.464 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:42.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:42.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:42.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:42 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 28591/153660 objects degraded (18.607%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:42.566 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:42.582 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:42 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 28591/153660 objects degraded (18.607%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:42.582 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:42 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 28591/153660 objects degraded (18.607%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:42.619 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:42.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:42.721 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:42.776 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:42.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:42.877 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:42.905 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:42.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:42.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:42.917 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:13:42.917 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:13:43.007 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:43.036 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:43.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:43.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:43.138 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:43.176 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:43.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:43.277 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:43.304 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:43.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:43.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:43.406 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:43.431 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:43.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:43.533 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:43.561 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:43.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:43.663 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:43.704 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:43.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:43.806 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:43.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:43.835 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:43.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:43.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:43 smithi080 ceph-mon[40404]: pgmap v600: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 25456/135789 objects degraded (18.747%) 2024-03-26T05:13:43.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:43 smithi080 ceph-mon[29691]: pgmap v600: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 25456/135789 objects degraded (18.747%) 2024-03-26T05:13:43.935 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:43.965 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:43.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:44.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:43 smithi132 ceph-mon[33500]: pgmap v600: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 25456/135789 objects degraded (18.747%) 2024-03-26T05:13:44.066 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:44.093 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:44.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:44.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:44.195 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:44.224 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:44.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:44.325 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:44.354 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:44.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:44.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:44.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:13:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:13:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:13:44.456 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:44.485 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:44.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:44.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:44.626 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:44.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:44.727 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:44.758 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:44.768 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:44.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:44.844 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:13:44.844 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:13:44.844 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:13:44.845 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:13:44.845 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:13:44.845 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:13:44.845 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:13:44.845 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:13:44.845 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:13:44.845 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:13:44.845 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:13:44.845 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:13:44.870 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:44.899 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:44.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:45.001 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:45.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:45.045 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:45.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:45.146 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:45.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:45.182 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:45.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:45.283 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:45.311 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:45.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:45.414 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:45.442 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:45.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:45.544 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:45.580 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:45.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:45.653 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T05:13:45.681 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:45.709 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:45.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:45.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:45.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:45 smithi132 ceph-mon[33500]: pgmap v601: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.20k op/s; 25456/135789 objects degraded (18.747%) 2024-03-26T05:13:45.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:13:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:13:45.811 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:45.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:45.839 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:45.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:45.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:45 smithi080 ceph-mon[40404]: pgmap v601: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.20k op/s; 25456/135789 objects degraded (18.747%) 2024-03-26T05:13:45.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:45 smithi080 ceph-mon[29691]: pgmap v601: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.20k op/s; 25456/135789 objects degraded (18.747%) 2024-03-26T05:13:45.939 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:45.965 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:45.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:46.067 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:46.095 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:46.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:46.197 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:46.226 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:46.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:46.328 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:46.371 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:46.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:46.472 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:46.502 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:46.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:46.520 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:13:46.603 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:13:46.609 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:13:46.609 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-26T05:13:46.652 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:46.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:46.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:46.755 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:46.845 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:46.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:46.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:46 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:46.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:46 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 25456/135789 objects degraded (18.747%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:46.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:46 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:46.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:46 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 25456/135789 objects degraded (18.747%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:46.947 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:47.010 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:47.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:46 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:47.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:46 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 25456/135789 objects degraded (18.747%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:47.111 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:47.168 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:47.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:47.269 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:47.316 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:47.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:47.417 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:47.454 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:47.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:47.556 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:47.586 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:47.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:47.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:47.688 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:47.724 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:47.775 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:47.784 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:47.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:47.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:47 smithi080 ceph-mon[29691]: pgmap v602: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 23797/125859 objects degraded (18.908%) 2024-03-26T05:13:47.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:47 smithi080 ceph-mon[40404]: pgmap v602: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 23797/125859 objects degraded (18.908%) 2024-03-26T05:13:47.886 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:47.915 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:47.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:48.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:47 smithi132 ceph-mon[33500]: pgmap v602: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 23797/125859 objects degraded (18.908%) 2024-03-26T05:13:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:48.082 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:48.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:48.183 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:48.214 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:48.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:48.316 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:48.354 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:48.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:48.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:48.456 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:48.482 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:48.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:48.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:48.584 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:48.612 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:48.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:48.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:48.714 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:48.755 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:48.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:48.856 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:48.895 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:48.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:48.900 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:48.997 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:49.023 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:49.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:49.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:49.125 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:49.164 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:49.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:49.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:49.266 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:49.291 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:49.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:49.387 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:13:49.393 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:49.431 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:13:49.441 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:49.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:49.470 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:13:49.523 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:13:49.543 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:49.564 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:13:49.584 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:49.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:49.606 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:13:49.649 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:13:49.649 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:49 smithi080 ceph-mon[40404]: pgmap v603: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.48k op/s; 21744/115992 objects degraded (18.746%) 2024-03-26T05:13:49.649 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:49 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/218539350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:13:49.650 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:49 smithi080 ceph-mon[29691]: pgmap v603: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.48k op/s; 21744/115992 objects degraded (18.746%) 2024-03-26T05:13:49.650 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:49 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/218539350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:13:49.650 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:13:49.650 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:13:49.650 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:13:49.650 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:13:49.650 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:13:49.651 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:13:49.651 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:13:49.651 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:13:49.687 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:49.717 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:49.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:49.819 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:49.843 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:49.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:49.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:49.945 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:49.983 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:49.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:49 smithi132 ceph-mon[33500]: pgmap v603: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.48k op/s; 21744/115992 objects degraded (18.746%) 2024-03-26T05:13:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:49 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/218539350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:13:50.085 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:50.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:50.124 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:50.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:50.225 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:50.264 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:50.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:50.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:50.393 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:50.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:50.495 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:50.525 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:50.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:50.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:50.626 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:50.653 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:50.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:50.755 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:50.782 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:50.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:50.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:50.884 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:50.911 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:50.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:50.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:50.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:13:50.972 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:13:51.012 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:51.018 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:13:51.019 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:13:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:13:51.052 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:51.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:51.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:51.154 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:51.194 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:51.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:51.297 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:51.325 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:51.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:51.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:51.427 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:51.478 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:51.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:51.580 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:51.608 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:51.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:51.710 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:51.737 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:51.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:51.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:51.839 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:51.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:51 smithi080 ceph-mon[29691]: pgmap v604: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 21318/114714 objects degraded (18.584%) 2024-03-26T05:13:51.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:51 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:13:51.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:51 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:13:51.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:51 smithi080 ceph-mon[29691]: osdmap e307: 8 total, 7 up, 8 in 2024-03-26T05:13:51.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:51 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 21318/114714 objects degraded (18.584%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:51.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:51 smithi080 ceph-mon[40404]: pgmap v604: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 21318/114714 objects degraded (18.584%) 2024-03-26T05:13:51.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:51 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:13:51.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:51 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:13:51.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:51 smithi080 ceph-mon[40404]: osdmap e307: 8 total, 7 up, 8 in 2024-03-26T05:13:51.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:51 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 21318/114714 objects degraded (18.584%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:51.882 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:51.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:51.985 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:52.034 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:52.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:51 smithi132 ceph-mon[33500]: pgmap v604: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 21318/114714 objects degraded (18.584%) 2024-03-26T05:13:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:51 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:13:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:51 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:13:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:51 smithi132 ceph-mon[33500]: osdmap e307: 8 total, 7 up, 8 in 2024-03-26T05:13:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:51 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 21318/114714 objects degraded (18.584%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:52.136 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:52.163 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:52.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:52.265 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:52.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:52.293 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:52.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:52.393 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:52.423 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:52.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:52.524 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:52.552 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:52.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:52.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:52.654 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:52.692 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:52.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:52.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:52.791 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:52 smithi080 ceph-mon[29691]: pgmap v606: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s; 18075/96498 objects degraded (18.731%) 2024-03-26T05:13:52.792 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:52 smithi080 ceph-mon[40404]: pgmap v606: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s; 18075/96498 objects degraded (18.731%) 2024-03-26T05:13:52.794 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:52.838 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:52.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:52.939 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:52.987 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:52.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:53.020 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:13:53.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:52 smithi132 ceph-mon[33500]: pgmap v606: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s; 18075/96498 objects degraded (18.731%) 2024-03-26T05:13:53.088 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:53.114 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:53.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:53.216 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:53.243 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:53.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:53.344 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:53.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:53.399 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:53.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:53.500 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:53.541 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:53.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:53.643 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:53.670 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:53.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:53.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:53.700 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:13:53.772 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:53.798 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:53.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:53.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:53.834 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206027 7.a 7:5c2c0985:::benchmark_data_smithi080_80848_object86528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.896706+0000", 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 497.94312130399999, 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.622309515, 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206027 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.896706+0000", 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.896706+0000", 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.896708+0000", 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5379999999999999e-06 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.896713+0000", 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2059999999999998e-06 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897284+0000", 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00057081500000000002 2024-03-26T05:13:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.897296+0000", 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2602999999999999e-05 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.901530+0000", 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0042335389999999997 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.901542+0000", 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1963999999999999e-05 2024-03-26T05:13:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.415809+0000", 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.51426740500000001 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.415825+0000", 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5913000000000002e-05 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.499412+0000", 2024-03-26T05:13:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.083586833 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.499564+0000", 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015167199999999999 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,2,5", 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.500614+0000", 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010504080000000001 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.508388+0000", 2024-03-26T05:13:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0077737520000000001 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.513624+0000", 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0052356499999999997 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.513772+0000", 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000148851 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.842 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.518794+0000", 2024-03-26T05:13:53.842 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0050213829999999999 2024-03-26T05:13:53.842 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.842 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.842 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.842 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.518881+0000", 2024-03-26T05:13:53.842 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7582000000000003e-05 2024-03-26T05:13:53.842 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.843 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.843 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.843 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.519016+0000", 2024-03-26T05:13:53.843 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000134401 2024-03-26T05:13:53.843 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.843 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.843 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206030 7.a 7:5ba88963:::benchmark_data_smithi080_80848_object86531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:13:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.904298+0000", 2024-03-26T05:13:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 497.93552928700001, 2024-03-26T05:13:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6149776980000001, 2024-03-26T05:13:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206030 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.904298+0000", 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.904298+0000", 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.904300+0000", 2024-03-26T05:13:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.406e-06 2024-03-26T05:13:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.904304+0000", 2024-03-26T05:13:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8970000000000001e-06 2024-03-26T05:13:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.904314+0000", 2024-03-26T05:13:53.847 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0521e-05 2024-03-26T05:13:53.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.847 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.847 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.847 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.904324+0000", 2024-03-26T05:13:53.847 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4979999999999995e-06 2024-03-26T05:13:53.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.848 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.848 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.848 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.904403+0000", 2024-03-26T05:13:53.848 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9746000000000005e-05 2024-03-26T05:13:53.848 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.848 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.849 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.849 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.904417+0000", 2024-03-26T05:13:53.849 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3494e-05 2024-03-26T05:13:53.849 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.849 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.849 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.850 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.415858+0000", 2024-03-26T05:13:53.850 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.51144128899999997 2024-03-26T05:13:53.850 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.850 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.850 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.850 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.415866+0000", 2024-03-26T05:13:53.851 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4329999999999998e-06 2024-03-26T05:13:53.851 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.851 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.851 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.851 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.508425+0000", 2024-03-26T05:13:53.852 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.092559778 2024-03-26T05:13:53.852 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.852 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.852 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.852 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.508550+0000", 2024-03-26T05:13:53.852 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012423600000000001 2024-03-26T05:13:53.852 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.853 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.853 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,2,5", 2024-03-26T05:13:53.853 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.508644+0000", 2024-03-26T05:13:53.853 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4817999999999999e-05 2024-03-26T05:13:53.853 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.513693+0000", 2024-03-26T05:13:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00504909 2024-03-26T05:13:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.516714+0000", 2024-03-26T05:13:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0030207139999999999 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.518262+0000", 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001548006 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.519101+0000", 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00083859600000000005 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.519162+0000", 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0949000000000002e-05 2024-03-26T05:13:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.519276+0000", 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000114227 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206046 7.a 7:57dcd937:::benchmark_data_smithi080_80848_object86547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.922047+0000", 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 497.91778034200001, 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5974842840000001, 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206046 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.922047+0000", 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.922047+0000", 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.922049+0000", 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6330000000000001e-06 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.922055+0000", 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.942e-06 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.922070+0000", 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4773999999999999e-05 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.922078+0000", 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7940000000000008e-06 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.922157+0000", 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8658999999999993e-05 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.922168+0000", 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1222e-05 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.415889+0000", 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.49372051 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.415896+0000", 2024-03-26T05:13:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.131e-06 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.511153+0000", 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.095257296 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.511304+0000", 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000151099 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,2,5", 2024-03-26T05:13:53.862 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.511464+0000", 2024-03-26T05:13:53.862 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015995399999999999 2024-03-26T05:13:53.862 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.862 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.862 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.513754+0000", 2024-03-26T05:13:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002289981 2024-03-26T05:13:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.516721+0000", 2024-03-26T05:13:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0029672819999999999 2024-03-26T05:13:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.864 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.864 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.519323+0000", 2024-03-26T05:13:53.864 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0026014929999999999 2024-03-26T05:13:53.864 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.864 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.864 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.519368+0000", 2024-03-26T05:13:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4761000000000001e-05 2024-03-26T05:13:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.519421+0000", 2024-03-26T05:13:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3519e-05 2024-03-26T05:13:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.519531+0000", 2024-03-26T05:13:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000110234 2024-03-26T05:13:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24701.0:206050 7.a 7:588af59a:::benchmark_data_smithi080_80848_object86551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:13:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.925667+0000", 2024-03-26T05:13:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 497.91416022800001, 2024-03-26T05:13:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5944858310000001, 2024-03-26T05:13:53.867 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.867 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.867 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.867 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24701", 2024-03-26T05:13:53.867 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/572455885", 2024-03-26T05:13:53.867 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 206050 2024-03-26T05:13:53.868 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.868 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.868 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.868 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.868 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.925667+0000", 2024-03-26T05:13:53.869 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.869 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.869 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.869 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.869 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.925667+0000", 2024-03-26T05:13:53.869 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.870 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.870 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.870 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.870 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.925669+0000", 2024-03-26T05:13:53.870 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7990000000000001e-06 2024-03-26T05:13:53.871 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.871 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.871 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.871 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.925675+0000", 2024-03-26T05:13:53.871 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.981e-06 2024-03-26T05:13:53.871 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.872 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.872 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.872 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.925689+0000", 2024-03-26T05:13:53.872 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3779e-05 2024-03-26T05:13:53.872 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.872 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.873 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.873 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.925699+0000", 2024-03-26T05:13:53.873 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0234999999999999e-05 2024-03-26T05:13:53.873 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.873 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.873 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.874 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.925770+0000", 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0482999999999996e-05 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.925782+0000", 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2074e-05 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.415919+0000", 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.49013746000000002 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.415927+0000", 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4420000000000001e-06 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.514035+0000", 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.098108131 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.514198+0000", 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000163611 2024-03-26T05:13:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,2,5", 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.514375+0000", 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017663600000000001 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.517154+0000", 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027794109999999999 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.518199+0000", 2024-03-26T05:13:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010445210000000001 2024-03-26T05:13:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.519575+0000", 2024-03-26T05:13:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0013757109999999999 2024-03-26T05:13:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.519780+0000", 2024-03-26T05:13:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020568299999999999 2024-03-26T05:13:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.519849+0000", 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8850000000000007e-05 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:37.520153+0000", 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00030402400000000002 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142549 8.0 8:0f39baff:::benchmark_data_smithi080_109213_object28032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.504592+0000", 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 297.33523548199997, 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98156162899999999, 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142549 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504592+0000", 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504592+0000", 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504593+0000", 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7100000000000002e-07 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504595+0000", 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4679999999999999e-06 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504602+0000", 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5119999999999997e-06 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504607+0000", 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2379999999999997e-06 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504822+0000", 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021480699999999999 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504829+0000", 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4590000000000001e-06 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.483043+0000", 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97821355600000004 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.483154+0000", 2024-03-26T05:13:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000111266 2024-03-26T05:13:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:13:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.483263+0000", 2024-03-26T05:13:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000109616 2024-03-26T05:13:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.485976+0000", 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027129250000000001 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.485989+0000", 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2797e-05 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.486051+0000", 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1488999999999994e-05 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.486108+0000", 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7441000000000001e-05 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.486154+0000", 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5584000000000003e-05 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142557 8.0 8:07e8ac51:::benchmark_data_smithi080_109213_object28040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.514964+0000", 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 297.32486372699998, 2024-03-26T05:13:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97959498, 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142557 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.514964+0000", 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.514964+0000", 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.514965+0000", 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5549999999999999e-06 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.514970+0000", 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8829999999999997e-06 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.514985+0000", 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4725999999999999e-05 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.514994+0000", 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8850000000000005e-06 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515051+0000", 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7119e-05 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515062+0000", 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1157e-05 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.486193+0000", 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97113100100000005 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.486297+0000", 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103472 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.486394+0000", 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7063000000000002e-05 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494287+0000", 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0078930370000000003 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494351+0000", 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4397e-05 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494448+0000", 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6775999999999996e-05 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494509+0000", 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0908e-05 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494559+0000", 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0000999999999998e-05 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.892 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142560 8.0 8:03494147:::benchmark_data_smithi080_109213_object28043:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.515785+0000", 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 297.32404224300001, 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97893898300000004, 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142560 2024-03-26T05:13:53.893 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515785+0000", 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515785+0000", 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.894 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515786+0000", 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1289999999999999e-06 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515791+0000", 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6670000000000004e-06 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515807+0000", 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5560999999999999e-05 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.895 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515815+0000", 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5019999999999992e-06 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515877+0000", 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1439000000000006e-05 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515885+0000", 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1200000000000002e-06 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.896 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.489108+0000", 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97322330599999995 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.489231+0000", 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012295399999999999 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.489350+0000", 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118609 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.897 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494397+0000", 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0050477040000000001 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494406+0000", 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3999999999999992e-06 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494622+0000", 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021623700000000001 2024-03-26T05:13:53.898 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494681+0000", 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8885999999999998e-05 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494724+0000", 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3469e-05 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142575 8.0 8:0d7e4a70:::benchmark_data_smithi080_109213_object28058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:13:53.899 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.530049+0000", 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 297.30977884599997, 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97616341699999998, 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142575 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530049+0000", 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.900 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530049+0000", 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530050+0000", 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.42e-06 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530054+0000", 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.641e-06 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.901 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530063+0000", 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6889999999999999e-06 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530070+0000", 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8210000000000004e-06 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530137+0000", 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7057000000000005e-05 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.902 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530146+0000", 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1349999999999998e-06 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.491950+0000", 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96180365499999998 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.492068+0000", 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117935 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.903 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.492192+0000", 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012364400000000001 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494734+0000", 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0025423289999999999 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.500311+0000", 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055767949999999998 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.904 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506113+0000", 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0058020199999999997 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506164+0000", 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1505000000000002e-05 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506212+0000", 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7771000000000001e-05 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.905 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142577 8.0 8:0fb818fc:::benchmark_data_smithi080_109213_object28060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.533454+0000", 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 297.30637323799999, 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97287799699999999, 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142577 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.906 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533454+0000", 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533454+0000", 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.907 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533456+0000", 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.753e-06 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533461+0000", 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1309999999999997e-06 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533476+0000", 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5241e-05 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.908 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533485+0000", 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2749999999999995e-06 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533534+0000", 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9351999999999998e-05 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533540+0000", 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8109999999999999e-06 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.909 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494773+0000", 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96123300899999997 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494899+0000", 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012616099999999999 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.495016+0000", 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117179 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.910 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.500360+0000", 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0053439079999999996 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.500372+0000", 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1649e-05 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506261+0000", 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0058893499999999998 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.911 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506301+0000", 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9548000000000002e-05 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506332+0000", 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.163e-05 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142588 8.0 8:0e33f004:::benchmark_data_smithi080_109213_object28071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.543025+0000", 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 297.29680291099999, 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96893715599999997, 2024-03-26T05:13:53.912 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142588 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543025+0000", 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.913 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.914 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.914 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543025+0000", 2024-03-26T05:13:53.914 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.914 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.914 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.914 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.915 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543026+0000", 2024-03-26T05:13:53.915 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.144e-06 2024-03-26T05:13:53.915 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.915 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.915 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.915 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543030+0000", 2024-03-26T05:13:53.915 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9949999999999999e-06 2024-03-26T05:13:53.915 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.916 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.916 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.916 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543041+0000", 2024-03-26T05:13:53.916 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1202999999999999e-05 2024-03-26T05:13:53.916 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.916 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.916 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.916 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543048+0000", 2024-03-26T05:13:53.916 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4460000000000004e-06 2024-03-26T05:13:53.916 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.916 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.916 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543303+0000", 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000254221 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543312+0000", 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0389999999999993e-06 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.497590+0000", 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95427875200000001 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.917 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.497727+0000", 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000136747 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.497849+0000", 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000121468 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506224+0000", 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0083755749999999997 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.918 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506364+0000", 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000140121 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.511871+0000", 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055066919999999997 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.511923+0000", 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1759999999999999e-05 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.919 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.511962+0000", 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8992999999999997e-05 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142592 8.0 8:02c66466:::benchmark_data_smithi080_109213_object28075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.544864+0000", 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 297.29496343699998, 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96723239800000005, 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.920 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142592 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544864+0000", 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544864+0000", 2024-03-26T05:13:53.921 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544865+0000", 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.234e-06 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544871+0000", 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4009999999999996e-06 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.922 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544887+0000", 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5928000000000001e-05 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544895+0000", 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3590000000000008e-06 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544950+0000", 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4487000000000002e-05 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.923 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544957+0000", 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8560000000000007e-06 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.500399+0000", 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95544189700000004 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.500508+0000", 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000108707 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.924 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.500609+0000", 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000100936 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506229+0000", 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0056203420000000004 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506411+0000", 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018178500000000001 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.925 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.512004+0000", 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055931770000000004 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.512057+0000", 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2735000000000002e-05 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.512096+0000", 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9554000000000001e-05 2024-03-26T05:13:53.926 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109994 9.c 9:34e37923:::benchmark_data_smithi080_136801_object109993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.792060+0000", 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 88.047767800000003, 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0406930759999999, 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109994 2024-03-26T05:13:53.927 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.928 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.928 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.928 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.928 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792060+0000", 2024-03-26T05:13:53.928 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.928 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.928 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.928 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.928 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:53.930 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792060+0000", 2024-03-26T05:13:53.930 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.930 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.930 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.930 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.930 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792062+0000", 2024-03-26T05:13:53.930 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7889999999999999e-06 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792085+0000", 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3445999999999999e-05 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792099+0000", 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4372000000000001e-05 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792110+0000", 2024-03-26T05:13:53.931 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0739e-05 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792816+0000", 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00070612300000000002 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792992+0000", 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017620600000000001 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,7", 2024-03-26T05:13:53.932 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.793149+0000", 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015638199999999999 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.809680+0000", 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016531033000000001 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.809693+0000", 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2781e-05 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.933 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.821562+0000", 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.01186924 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for degraded object", 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.822506+0000", 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00094436699999999999 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.832661+0000", 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.01015492 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.934 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.832753+0000", 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1677999999999999e-05 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109996 9.c 9:318dcee9:::benchmark_data_smithi080_136801_object109995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.817916+0000", 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 88.021911396999997, 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.016490052, 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:13:53.935 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109996 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817916+0000", 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817916+0000", 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.936 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817917+0000", 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1600000000000001e-07 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817942+0000", 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5422000000000002e-05 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817950+0000", 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.977e-06 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.937 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817956+0000", 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.263e-06 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822354+0000", 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0043976070000000004 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822365+0000", 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.098e-05 2024-03-26T05:13:53.938 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.822966+0000", 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0006012339999999 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.823106+0000", 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013994699999999999 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4", 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.823382+0000", 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00027621999999999998 2024-03-26T05:13:53.939 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.832803+0000", 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0094210860000000004 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.834257+0000", 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014535419999999999 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.834309+0000", 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2330999999999998e-05 2024-03-26T05:13:53.940 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.834406+0000", 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6927000000000002e-05 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109989 9.3 9:c1ef3764:::benchmark_data_smithi080_136801_object109988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.822224+0000", 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 88.017603416, 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.002023533, 2024-03-26T05:13:53.941 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109989 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822224+0000", 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.942 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822224+0000", 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822227+0000", 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.8040000000000002e-06 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822261+0000", 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3615000000000003e-05 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.943 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822275+0000", 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4759e-05 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822287+0000", 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1419e-05 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822922+0000", 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00063540799999999996 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.944 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822932+0000", 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5510000000000005e-06 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.822466+0000", 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99953455999999996 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.824248+0000", 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001781417 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.945 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109993 9.3 9:c281200c:::benchmark_data_smithi080_136801_object109992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.822314+0000", 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 88.017513953000005, 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0022707790000001, 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109993 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.946 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822314+0000", 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822314+0000", 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822314+0000", 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6000000000000003e-07 2024-03-26T05:13:53.947 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822356+0000", 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1310999999999998e-05 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822363+0000", 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9940000000000003e-06 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822368+0000", 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7200000000000003e-06 2024-03-26T05:13:53.948 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822959+0000", 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00059071500000000001 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822965+0000", 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0800000000000002e-06 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.824517+0000", 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0015518800000001 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.949 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.824584+0000", 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7418999999999999e-05 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109998 9.3 9:cd2a8d52:::benchmark_data_smithi080_136801_object109997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.822378+0000", 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 88.017449560000003, 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0183832850000001, 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.950 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109998 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822378+0000", 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822378+0000", 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822378+0000", 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6699999999999999e-07 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822401+0000", 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2480000000000002e-05 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822407+0000", 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5139999999999998e-06 2024-03-26T05:13:53.952 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822413+0000", 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9549999999999997e-06 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822988+0000", 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00057432800000000003 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822994+0000", 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9800000000000003e-06 2024-03-26T05:13:53.953 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.824610+0000", 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.001616391 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.826015+0000", 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014048019999999999 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1", 2024-03-26T05:13:53.954 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.826423+0000", 2024-03-26T05:13:53.955 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00040815099999999999 2024-03-26T05:13:53.955 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.955 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.955 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.955 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840598+0000", 2024-03-26T05:13:53.955 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014175045000000001 2024-03-26T05:13:53.955 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.955 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.955 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.956 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840625+0000", 2024-03-26T05:13:53.956 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6840000000000001e-05 2024-03-26T05:13:53.956 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.956 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.956 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.956 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840672+0000", 2024-03-26T05:13:53.957 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6598999999999999e-05 2024-03-26T05:13:53.957 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.957 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.957 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.957 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840761+0000", 2024-03-26T05:13:53.957 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9733000000000003e-05 2024-03-26T05:13:53.957 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.958 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.958 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.958 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.958 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.958 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109999 9.3 9:c8cd8b0a:::benchmark_data_smithi080_136801_object109998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:13:53.958 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.829248+0000", 2024-03-26T05:13:53.959 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 88.010579340999996, 2024-03-26T05:13:53.959 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0177361709999999, 2024-03-26T05:13:53.959 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.959 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.959 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.959 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:13:53.959 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:13:53.960 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109999 2024-03-26T05:13:53.960 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.960 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.960 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.960 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.960 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829248+0000", 2024-03-26T05:13:53.960 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.961 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.961 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.961 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.961 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829248+0000", 2024-03-26T05:13:53.961 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.961 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.961 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.962 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.962 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829251+0000", 2024-03-26T05:13:53.962 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.706e-06 2024-03-26T05:13:53.962 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.962 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.963 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.963 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829301+0000", 2024-03-26T05:13:53.963 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0448000000000002e-05 2024-03-26T05:13:53.963 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.963 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.963 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.963 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829327+0000", 2024-03-26T05:13:53.964 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6041999999999999e-05 2024-03-26T05:13:53.964 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.964 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.964 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.964 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829340+0000", 2024-03-26T05:13:53.964 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2177999999999999e-05 2024-03-26T05:13:53.965 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.965 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.965 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.965 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829560+0000", 2024-03-26T05:13:53.965 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00022066499999999999 2024-03-26T05:13:53.965 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.965 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.966 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.966 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829571+0000", 2024-03-26T05:13:53.966 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1049999999999999e-05 2024-03-26T05:13:53.966 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.966 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.966 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.967 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.830957+0000", 2024-03-26T05:13:53.967 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.00138595 2024-03-26T05:13:53.967 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.967 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.967 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.967 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.831076+0000", 2024-03-26T05:13:53.968 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118507 2024-03-26T05:13:53.968 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.968 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.968 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1", 2024-03-26T05:13:53.968 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.831217+0000", 2024-03-26T05:13:53.968 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014082499999999999 2024-03-26T05:13:53.969 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.969 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.969 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.969 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840769+0000", 2024-03-26T05:13:53.969 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0095528499999999999 2024-03-26T05:13:53.969 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.970 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.970 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.970 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.846840+0000", 2024-03-26T05:13:53.970 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0060710260000000002 2024-03-26T05:13:53.970 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.970 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.971 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.971 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.846896+0000", 2024-03-26T05:13:53.971 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5721999999999999e-05 2024-03-26T05:13:53.971 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.971 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.971 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.972 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.846984+0000", 2024-03-26T05:13:53.972 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8201999999999993e-05 2024-03-26T05:13:53.972 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.972 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.972 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.972 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.973 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.973 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:110007 9.1 9:894e6b0c:::benchmark_data_smithi080_136801_object110006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:13:53.973 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.844428+0000", 2024-03-26T05:13:53.973 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 87.995399586999994, 2024-03-26T05:13:53.973 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97646292999999995, 2024-03-26T05:13:53.973 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.974 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.974 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.974 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:13:53.974 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:13:53.974 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 110007 2024-03-26T05:13:53.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.975 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.975 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.975 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.975 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844428+0000", 2024-03-26T05:13:53.975 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.975 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.976 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.976 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.976 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844428+0000", 2024-03-26T05:13:53.976 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.976 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.977 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.977 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844430+0000", 2024-03-26T05:13:53.977 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.68e-06 2024-03-26T05:13:53.977 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.977 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.977 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.977 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844464+0000", 2024-03-26T05:13:53.978 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4585000000000002e-05 2024-03-26T05:13:53.978 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.978 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.978 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.978 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844488+0000", 2024-03-26T05:13:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3312e-05 2024-03-26T05:13:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844497+0000", 2024-03-26T05:13:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4709999999999996e-06 2024-03-26T05:13:53.980 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.980 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.980 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.980 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844569+0000", 2024-03-26T05:13:53.980 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1805999999999997e-05 2024-03-26T05:13:53.980 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.981 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.981 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.981 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844578+0000", 2024-03-26T05:13:53.981 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5799999999999998e-06 2024-03-26T05:13:53.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.981 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.981 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.982 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.811307+0000", 2024-03-26T05:13:53.982 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96672887299999999 2024-03-26T05:13:53.982 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.982 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.982 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.982 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.811467+0000", 2024-03-26T05:13:53.983 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016000899999999999 2024-03-26T05:13:53.983 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.983 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.983 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 6", 2024-03-26T05:13:53.983 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.813451+0000", 2024-03-26T05:13:53.983 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0019838109999999998 2024-03-26T05:13:53.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.984 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.984 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.984 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817336+0000", 2024-03-26T05:13:53.984 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0038850600000000001 2024-03-26T05:13:53.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.985 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.985 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.985 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820757+0000", 2024-03-26T05:13:53.985 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0034208289999999998 2024-03-26T05:13:53.985 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.985 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.985 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.986 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820805+0000", 2024-03-26T05:13:53.986 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7953e-05 2024-03-26T05:13:53.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.986 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.986 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.986 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820891+0000", 2024-03-26T05:13:53.987 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5960999999999999e-05 2024-03-26T05:13:53.987 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.987 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.987 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.987 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.987 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.987 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:110010 9.3 9:cd2ac3f8:::benchmark_data_smithi080_136801_object110009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:13:53.987 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.848965+0000", 2024-03-26T05:13:53.987 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 87.990862535999995, 2024-03-26T05:13:53.987 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99831434500000005, 2024-03-26T05:13:53.988 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.988 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.988 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.988 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:13:53.988 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:13:53.988 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 110010 2024-03-26T05:13:53.988 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.988 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.988 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.989 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.989 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848965+0000", 2024-03-26T05:13:53.989 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.989 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.989 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.989 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848965+0000", 2024-03-26T05:13:53.989 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.990 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.990 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.990 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848966+0000", 2024-03-26T05:13:53.990 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.381e-06 2024-03-26T05:13:53.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.990 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.990 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848987+0000", 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0312000000000001e-05 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848999+0000", 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2170999999999999e-05 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.849009+0000", 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6700000000000006e-06 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.849085+0000", 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6884000000000005e-05 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.849096+0000", 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0179e-05 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.833721+0000", 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98462529899999995 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.992 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.833851+0000", 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012981200000000001 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1", 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.834023+0000", 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017234300000000001 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840776+0000", 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0067526269999999998 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.993 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.847155+0000", 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0063788459999999996 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.847201+0000", 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6702000000000001e-05 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.847279+0000", 2024-03-26T05:13:53.994 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8119000000000002e-05 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:110013 9.1 9:8edffc06:::benchmark_data_smithi080_136801_object110012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.857433+0000", 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 87.982394717000005, 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96361645399999996, 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:13:53.995 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 110013 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857433+0000", 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857433+0000", 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:13:53.996 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857434+0000", 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.178e-06 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857452+0000", 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7561e-05 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857463+0000", 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1099e-05 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.997 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857470+0000", 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1999999999999997e-06 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857537+0000", 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6946999999999994e-05 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857547+0000", 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0669000000000001e-05 2024-03-26T05:13:53.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817405+0000", 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95985719199999997 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817539+0000", 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013459600000000001 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 6", 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817693+0000", 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015364900000000001 2024-03-26T05:13:53.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820443+0000", 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002750438 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820935+0000", 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00049171399999999997 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820979+0000", 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3495000000000003e-05 2024-03-26T05:13:54.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:13:54.001 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:13:54.001 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:13:54.001 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.821049+0000", 2024-03-26T05:13:54.001 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0716000000000005e-05 2024-03-26T05:13:54.001 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:54.001 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:54.001 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:54.001 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:13:54.001 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:13:54.001 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:13:54.002 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:53 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3952282000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:13:54.002 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:53 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:13:54.003 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:53 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3952282000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:13:54.003 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:53 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:13:54.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:54.004 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:54.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:53 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3952282000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:13:54.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:53 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:13:54.105 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:54.133 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:54.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:54.235 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:54.275 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:54.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:54.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:13:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:13:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:13:54.376 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:54.415 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:54.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:54.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:54.517 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:54.560 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:54.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:54.661 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:54.690 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:54.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:54.791 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:54.828 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:54.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:54.929 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:54.957 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:54.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:54.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:55.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:54 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:13:55.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:54 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:13:55.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:54 smithi132 ceph-mon[33500]: osdmap e308: 8 total, 7 up, 8 in 2024-03-26T05:13:55.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:54 smithi132 ceph-mon[33500]: pgmap v608: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.22k op/s; 18075/96498 objects degraded (18.731%) 2024-03-26T05:13:55.060 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:55.087 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:55.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:55.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:55.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:54 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:13:55.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:54 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:13:55.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:54 smithi080 ceph-mon[29691]: osdmap e308: 8 total, 7 up, 8 in 2024-03-26T05:13:55.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:54 smithi080 ceph-mon[29691]: pgmap v608: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.22k op/s; 18075/96498 objects degraded (18.731%) 2024-03-26T05:13:55.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:54 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:13:55.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:54 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:13:55.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:54 smithi080 ceph-mon[40404]: osdmap e308: 8 total, 7 up, 8 in 2024-03-26T05:13:55.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:54 smithi080 ceph-mon[40404]: pgmap v608: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.22k op/s; 18075/96498 objects degraded (18.731%) 2024-03-26T05:13:55.189 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:55.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:55.232 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:55.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:55.332 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:55.362 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:55.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:55.464 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:55.492 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:55.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:55.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:55.594 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:55.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:55.634 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:55.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:55.735 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:55.776 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:55.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:55.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:13:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:13:55.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:55 smithi132 ceph-mon[33500]: 5.1d scrub starts 2024-03-26T05:13:55.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:55 smithi132 ceph-mon[33500]: 5.1d scrub ok 2024-03-26T05:13:55.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:55 smithi132 ceph-mon[33500]: 6.2 scrub starts 2024-03-26T05:13:55.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:55 smithi132 ceph-mon[33500]: 6.2 scrub ok 2024-03-26T05:13:55.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:55 smithi132 ceph-mon[33500]: 9.b scrub starts 2024-03-26T05:13:55.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:55 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:55.877 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:55.921 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:55.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:56.012 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:13:56.012 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:13:56.012 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:13:56.012 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:13:56.012 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:13:56.012 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:13:56.012 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:13:56.012 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:13:56.012 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:13:56.012 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:13:56.013 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:13:56.013 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:13:56.013 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:55 smithi080 ceph-mon[29691]: 5.1d scrub starts 2024-03-26T05:13:56.013 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:55 smithi080 ceph-mon[29691]: 5.1d scrub ok 2024-03-26T05:13:56.013 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:55 smithi080 ceph-mon[29691]: 6.2 scrub starts 2024-03-26T05:13:56.013 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:55 smithi080 ceph-mon[29691]: 6.2 scrub ok 2024-03-26T05:13:56.013 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:55 smithi080 ceph-mon[29691]: 9.b scrub starts 2024-03-26T05:13:56.013 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:55 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:56.014 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:55 smithi080 ceph-mon[40404]: 5.1d scrub starts 2024-03-26T05:13:56.014 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:55 smithi080 ceph-mon[40404]: 5.1d scrub ok 2024-03-26T05:13:56.014 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:55 smithi080 ceph-mon[40404]: 6.2 scrub starts 2024-03-26T05:13:56.014 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:55 smithi080 ceph-mon[40404]: 6.2 scrub ok 2024-03-26T05:13:56.014 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:55 smithi080 ceph-mon[40404]: 9.b scrub starts 2024-03-26T05:13:56.014 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:55 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:13:56.023 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:56.061 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:56.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:56.164 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:56.196 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:56.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:56.297 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:56.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:56.321 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:56.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:56.422 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:56.465 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:56.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:56.567 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:56.600 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:56.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:56.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:56.702 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:56.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:56.742 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:56.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:56.843 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:56.883 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:56.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:56.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:57.008 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:57.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:57.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:56 smithi132 ceph-mon[33500]: 9.b scrub ok 2024-03-26T05:13:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:56 smithi132 ceph-mon[33500]: 4.1d scrub starts 2024-03-26T05:13:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:56 smithi132 ceph-mon[33500]: 4.1d scrub ok 2024-03-26T05:13:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:56 smithi132 ceph-mon[33500]: 3.17 scrub starts 2024-03-26T05:13:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:56 smithi132 ceph-mon[33500]: 3.17 scrub ok 2024-03-26T05:13:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:56 smithi132 ceph-mon[33500]: pgmap v609: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 16423/86640 objects degraded (18.955%) 2024-03-26T05:13:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:56 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 18075/96498 objects degraded (18.731%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:57.110 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[40404]: 9.b scrub ok 2024-03-26T05:13:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[40404]: 4.1d scrub starts 2024-03-26T05:13:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[40404]: 4.1d scrub ok 2024-03-26T05:13:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[40404]: 3.17 scrub starts 2024-03-26T05:13:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[40404]: 3.17 scrub ok 2024-03-26T05:13:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[40404]: pgmap v609: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 16423/86640 objects degraded (18.955%) 2024-03-26T05:13:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 18075/96498 objects degraded (18.731%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:57.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[29691]: 9.b scrub ok 2024-03-26T05:13:57.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[29691]: 4.1d scrub starts 2024-03-26T05:13:57.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[29691]: 4.1d scrub ok 2024-03-26T05:13:57.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[29691]: 3.17 scrub starts 2024-03-26T05:13:57.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[29691]: 3.17 scrub ok 2024-03-26T05:13:57.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[29691]: pgmap v609: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 16423/86640 objects degraded (18.955%) 2024-03-26T05:13:57.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:56 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 18075/96498 objects degraded (18.731%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:13:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:57.149 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:57.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:57.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:57.285 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:57.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:57.387 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:57.430 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:57.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:57.532 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:13:57.561 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:57.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:13:57.663 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:57.691 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:57.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:57.793 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:57.820 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:57.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:13:57.912 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:13:57.913 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-26T05:13:57.922 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:57.949 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:57.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:58.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:57 smithi132 ceph-mon[33500]: 5.0 scrub starts 2024-03-26T05:13:58.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:57 smithi132 ceph-mon[33500]: 5.0 scrub ok 2024-03-26T05:13:58.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:57 smithi132 ceph-mon[33500]: 4.13 scrub starts 2024-03-26T05:13:58.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:57 smithi132 ceph-mon[33500]: 4.13 scrub ok 2024-03-26T05:13:58.051 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:58.093 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:58.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:58.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:57 smithi080 ceph-mon[40404]: 5.0 scrub starts 2024-03-26T05:13:58.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:57 smithi080 ceph-mon[40404]: 5.0 scrub ok 2024-03-26T05:13:58.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:57 smithi080 ceph-mon[40404]: 4.13 scrub starts 2024-03-26T05:13:58.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:57 smithi080 ceph-mon[40404]: 4.13 scrub ok 2024-03-26T05:13:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:57 smithi080 ceph-mon[29691]: 5.0 scrub starts 2024-03-26T05:13:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:57 smithi080 ceph-mon[29691]: 5.0 scrub ok 2024-03-26T05:13:58.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:57 smithi080 ceph-mon[29691]: 4.13 scrub starts 2024-03-26T05:13:58.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:57 smithi080 ceph-mon[29691]: 4.13 scrub ok 2024-03-26T05:13:58.194 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:13:58.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:58.244 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:58.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:13:58.345 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:58.373 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:58.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:58.475 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:58.523 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:58.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:58.540 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T05:13:58.556 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:58.623 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:58.655 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:58.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:58.757 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:58.789 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:58.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:58.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:58.854 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:13:58.891 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:13:58.927 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:58.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:13:58.981 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:58 smithi132 ceph-mon[33500]: 2.1 scrub starts 2024-03-26T05:13:58.981 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:58 smithi132 ceph-mon[33500]: 2.1 scrub ok 2024-03-26T05:13:58.981 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:58 smithi132 ceph-mon[33500]: 6.7 scrub starts 2024-03-26T05:13:58.981 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:58 smithi132 ceph-mon[33500]: 6.7 scrub ok 2024-03-26T05:13:58.981 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:58 smithi132 ceph-mon[33500]: pgmap v610: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 14489/77295 objects degraded (18.745%) 2024-03-26T05:13:59.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:58 smithi080 ceph-mon[29691]: 2.1 scrub starts 2024-03-26T05:13:59.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:58 smithi080 ceph-mon[29691]: 2.1 scrub ok 2024-03-26T05:13:59.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:58 smithi080 ceph-mon[29691]: 6.7 scrub starts 2024-03-26T05:13:59.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:58 smithi080 ceph-mon[29691]: 6.7 scrub ok 2024-03-26T05:13:59.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:58 smithi080 ceph-mon[29691]: pgmap v610: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 14489/77295 objects degraded (18.745%) 2024-03-26T05:13:59.011 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:58 smithi080 ceph-mon[40404]: 2.1 scrub starts 2024-03-26T05:13:59.011 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:58 smithi080 ceph-mon[40404]: 2.1 scrub ok 2024-03-26T05:13:59.011 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:58 smithi080 ceph-mon[40404]: 6.7 scrub starts 2024-03-26T05:13:59.011 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:58 smithi080 ceph-mon[40404]: 6.7 scrub ok 2024-03-26T05:13:59.011 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:58 smithi080 ceph-mon[40404]: pgmap v610: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 14489/77295 objects degraded (18.745%) 2024-03-26T05:13:59.029 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:59.057 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:59.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:59.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:59.159 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:59.192 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:59.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:59.294 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:13:59.323 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:13:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:59.335 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:59.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:13:59.405 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:13:59.436 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:13:59.464 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:59.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:13:59.566 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:59.594 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:59.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:59.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:59.696 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:13:59.724 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:13:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:59.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:13:59.826 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:13:59.863 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:13:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:13:59.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:13:59.965 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:00.001 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:00.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:00.029 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:59 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:14:00.030 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:13:59 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:14:00.103 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:00.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:59 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:14:00.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:13:59 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:14:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:59 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:14:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:13:59 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:14:00.148 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:00.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:00.250 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:00.278 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:00.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:00.379 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:00.419 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:00.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:00.521 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:00.557 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:00.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:00.658 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:00.688 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:00.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:00.789 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:00.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:00.923 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:00.962 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:00.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:00.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:01.024 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:00 smithi132 ceph-mon[33500]: 4.19 scrub starts 2024-03-26T05:14:01.024 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:00 smithi132 ceph-mon[33500]: 4.19 scrub ok 2024-03-26T05:14:01.024 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:00 smithi132 ceph-mon[33500]: 5.18 scrub starts 2024-03-26T05:14:01.024 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:00 smithi132 ceph-mon[33500]: 5.18 scrub ok 2024-03-26T05:14:01.024 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:00 smithi132 ceph-mon[33500]: pgmap v611: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s; 14087/76089 objects degraded (18.514%) 2024-03-26T05:14:01.032 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:00 smithi080 ceph-mon[40404]: 4.19 scrub starts 2024-03-26T05:14:01.032 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:00 smithi080 ceph-mon[40404]: 4.19 scrub ok 2024-03-26T05:14:01.032 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:00 smithi080 ceph-mon[40404]: 5.18 scrub starts 2024-03-26T05:14:01.032 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:00 smithi080 ceph-mon[40404]: 5.18 scrub ok 2024-03-26T05:14:01.032 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:00 smithi080 ceph-mon[40404]: pgmap v611: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s; 14087/76089 objects degraded (18.514%) 2024-03-26T05:14:01.032 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:00 smithi080 ceph-mon[29691]: 4.19 scrub starts 2024-03-26T05:14:01.032 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:00 smithi080 ceph-mon[29691]: 4.19 scrub ok 2024-03-26T05:14:01.032 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:00 smithi080 ceph-mon[29691]: 5.18 scrub starts 2024-03-26T05:14:01.033 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:00 smithi080 ceph-mon[29691]: 5.18 scrub ok 2024-03-26T05:14:01.033 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:00 smithi080 ceph-mon[29691]: pgmap v611: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s; 14087/76089 objects degraded (18.514%) 2024-03-26T05:14:01.064 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:01.072 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:14:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.25268.0:224386 9.e 9:7417df5e:::benchmark_data_smithi080_136801_object102803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:14:01.067865+0000", 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.014186320000000001, 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014210989, 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.25268", 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 224386 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.067865+0000", 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.067865+0000", 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.067866+0000", 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.71e-07 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.067870+0000", 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7869999999999999e-06 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.067878+0000", 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6920000000000002e-06 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.067883+0000", 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1599999999999997e-06 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.067932+0000", 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9486e-05 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.068043+0000", 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011118800000000001 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:14:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.068168+0000", 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012459299999999999 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.070801+0000", 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0026326840000000002 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.25268.0:224397 9.e 9:7d1cd91f:::benchmark_data_smithi080_136801_object102814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:14:01.074449+0000", 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0076027079999999997, 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076810539999999997, 2024-03-26T05:14:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:14:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.25268", 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 224397 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.074449+0000", 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.074449+0000", 2024-03-26T05:14:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.074450+0000", 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.339e-06 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.074456+0000", 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4389999999999999e-06 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.074467+0000", 2024-03-26T05:14:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1059e-05 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.074474+0000", 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7409999999999992e-06 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.074532+0000", 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7229999999999999e-05 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.074608+0000", 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5897999999999997e-05 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.074676+0000", 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8685999999999999e-05 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:14:01.077381+0000", 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0027050160000000002 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:01.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:01.083 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-03-26T05:14:01.083 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 2 2024-03-26T05:14:01.083 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:14:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:01.128 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:01.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:01.230 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:01.277 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:01.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:01.380 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:01.419 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:01.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:01.520 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:01.553 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:01.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:01.574 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:01.655 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:01.706 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:01.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:01.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:01.748 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[29691]: 3.c scrub starts 2024-03-26T05:14:01.748 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[29691]: 3.c scrub ok 2024-03-26T05:14:01.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[29691]: 6.11 scrub starts 2024-03-26T05:14:01.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[29691]: 6.11 scrub ok 2024-03-26T05:14:01.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[29691]: 5.12 scrub starts 2024-03-26T05:14:01.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[29691]: 5.12 scrub ok 2024-03-26T05:14:01.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 14087/76089 objects degraded (18.514%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:01.771 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T05:14:01.809 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:01.857 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:01.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:01.958 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:01.999 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:01.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:02.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:01 smithi132 ceph-mon[33500]: 3.c scrub starts 2024-03-26T05:14:02.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:01 smithi132 ceph-mon[33500]: 3.c scrub ok 2024-03-26T05:14:02.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:01 smithi132 ceph-mon[33500]: 6.11 scrub starts 2024-03-26T05:14:02.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:01 smithi132 ceph-mon[33500]: 6.11 scrub ok 2024-03-26T05:14:02.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:01 smithi132 ceph-mon[33500]: 5.12 scrub starts 2024-03-26T05:14:02.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:01 smithi132 ceph-mon[33500]: 5.12 scrub ok 2024-03-26T05:14:02.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:01 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 14087/76089 objects degraded (18.514%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:02.100 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:02.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[40404]: 3.c scrub starts 2024-03-26T05:14:02.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[40404]: 3.c scrub ok 2024-03-26T05:14:02.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[40404]: 6.11 scrub starts 2024-03-26T05:14:02.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[40404]: 6.11 scrub ok 2024-03-26T05:14:02.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[40404]: 5.12 scrub starts 2024-03-26T05:14:02.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[40404]: 5.12 scrub ok 2024-03-26T05:14:02.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:01 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 14087/76089 objects degraded (18.514%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:02.118 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:14:02.140 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:02.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:02.242 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:02.282 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:02.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:02.383 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:02.422 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:02.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:02.523 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:02.559 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:02.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:02.661 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:02.692 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:02.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:02.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:02 smithi132 ceph-mon[33500]: 5.8 scrub starts 2024-03-26T05:14:02.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:02 smithi132 ceph-mon[33500]: 5.8 scrub ok 2024-03-26T05:14:02.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:02 smithi132 ceph-mon[33500]: pgmap v612: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 10939/58434 objects degraded (18.720%) 2024-03-26T05:14:02.794 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:02.835 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:02.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:02.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:02.970 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:02.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:03.071 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:03.096 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:03.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:03.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:02 smithi080 ceph-mon[40404]: 5.8 scrub starts 2024-03-26T05:14:03.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:02 smithi080 ceph-mon[40404]: 5.8 scrub ok 2024-03-26T05:14:03.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:02 smithi080 ceph-mon[40404]: pgmap v612: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 10939/58434 objects degraded (18.720%) 2024-03-26T05:14:03.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:02 smithi080 ceph-mon[29691]: 5.8 scrub starts 2024-03-26T05:14:03.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:02 smithi080 ceph-mon[29691]: 5.8 scrub ok 2024-03-26T05:14:03.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:02 smithi080 ceph-mon[29691]: pgmap v612: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 10939/58434 objects degraded (18.720%) 2024-03-26T05:14:03.197 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:03.227 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:03.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:03.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:03.329 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:03.362 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:03.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:03.463 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:03.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:03.501 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:03.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:03.602 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:03.633 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:03.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:03.735 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:03.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:03.766 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:03.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:03.867 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:03.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:03.913 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:03.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:03.997 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:03 smithi132 ceph-mon[33500]: 4.f scrub starts 2024-03-26T05:14:03.997 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:03 smithi132 ceph-mon[33500]: 4.f scrub ok 2024-03-26T05:14:03.997 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:03 smithi132 ceph-mon[33500]: 3.3 scrub starts 2024-03-26T05:14:03.997 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:03 smithi132 ceph-mon[33500]: 3.3 scrub ok 2024-03-26T05:14:03.997 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:03 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1658661430' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:04.014 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:04.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:03 smithi080 ceph-mon[29691]: 4.f scrub starts 2024-03-26T05:14:04.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:03 smithi080 ceph-mon[29691]: 4.f scrub ok 2024-03-26T05:14:04.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:03 smithi080 ceph-mon[29691]: 3.3 scrub starts 2024-03-26T05:14:04.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:03 smithi080 ceph-mon[29691]: 3.3 scrub ok 2024-03-26T05:14:04.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:03 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1658661430' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:04.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:03 smithi080 ceph-mon[40404]: 4.f scrub starts 2024-03-26T05:14:04.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:03 smithi080 ceph-mon[40404]: 4.f scrub ok 2024-03-26T05:14:04.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:03 smithi080 ceph-mon[40404]: 3.3 scrub starts 2024-03-26T05:14:04.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:03 smithi080 ceph-mon[40404]: 3.3 scrub ok 2024-03-26T05:14:04.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:03 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1658661430' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:04.050 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:04.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:04.152 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:04.179 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:04.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:04.280 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:04.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:04.317 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:04.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:04.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:14:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:14:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:14:04.419 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:04.450 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:04.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:04.531 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:14:04.531 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:14:04.531 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:14:04.531 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:14:04.531 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:14:04.531 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:14:04.531 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:14:04.531 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:14:04.531 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:14:04.532 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:14:04.532 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:14:04.532 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:14:04.532 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:14:04.532 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:14:04.532 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:14:04.551 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:04.588 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:04.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:04.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:04.651 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:14:04.651 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:14:04.690 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:04.727 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:04.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:04.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:04.817 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:04 smithi080 ceph-mon[29691]: 5.e scrub starts 2024-03-26T05:14:04.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:04 smithi080 ceph-mon[40404]: 5.e scrub starts 2024-03-26T05:14:04.829 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:04.859 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:04.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:04.961 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:04.991 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:04 smithi132 ceph-mon[33500]: 5.e scrub starts 2024-03-26T05:14:04.991 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:04 smithi132 ceph-mon[33500]: 5.e scrub ok 2024-03-26T05:14:04.991 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:04 smithi132 ceph-mon[33500]: pgmap v613: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 992 B/s rd, 0 B/s wr, 1.23k op/s; 10939/58434 objects degraded (18.720%) 2024-03-26T05:14:04.995 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:04.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:04.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:05.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:05.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:04 smithi080 ceph-mon[29691]: 5.e scrub ok 2024-03-26T05:14:05.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:04 smithi080 ceph-mon[29691]: pgmap v613: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 992 B/s rd, 0 B/s wr, 1.23k op/s; 10939/58434 objects degraded (18.720%) 2024-03-26T05:14:05.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:04 smithi080 ceph-mon[40404]: 5.e scrub ok 2024-03-26T05:14:05.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:04 smithi080 ceph-mon[40404]: pgmap v613: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 992 B/s rd, 0 B/s wr, 1.23k op/s; 10939/58434 objects degraded (18.720%) 2024-03-26T05:14:05.135 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:05.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:05.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:05.237 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:05.279 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:05.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:05.380 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:05.423 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:05.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:05.524 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:05.567 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:05.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:05.570 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T05:14:05.668 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:05.692 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:05.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:05.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:05.773 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:14:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:14:05.775 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:05 smithi132 ceph-mon[33500]: 5.16 scrub starts 2024-03-26T05:14:05.775 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:05 smithi132 ceph-mon[33500]: 5.16 scrub ok 2024-03-26T05:14:05.775 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:05 smithi132 ceph-mon[33500]: 5.1c scrub starts 2024-03-26T05:14:05.775 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:05 smithi132 ceph-mon[33500]: 5.1c scrub ok 2024-03-26T05:14:05.775 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:14:05.775 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:05 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:05.793 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:05.832 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:05.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:05.868 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:14:05.933 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:05.971 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:05.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:06.072 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:06.107 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:06.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:06.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:05 smithi080 ceph-mon[29691]: 5.16 scrub starts 2024-03-26T05:14:06.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:05 smithi080 ceph-mon[29691]: 5.16 scrub ok 2024-03-26T05:14:06.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:05 smithi080 ceph-mon[29691]: 5.1c scrub starts 2024-03-26T05:14:06.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:05 smithi080 ceph-mon[29691]: 5.1c scrub ok 2024-03-26T05:14:06.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:14:06.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:05 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:06.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:05 smithi080 ceph-mon[40404]: 5.16 scrub starts 2024-03-26T05:14:06.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:05 smithi080 ceph-mon[40404]: 5.16 scrub ok 2024-03-26T05:14:06.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:05 smithi080 ceph-mon[40404]: 5.1c scrub starts 2024-03-26T05:14:06.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:05 smithi080 ceph-mon[40404]: 5.1c scrub ok 2024-03-26T05:14:06.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:14:06.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:05 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:06.208 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:06.251 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:06.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:06.352 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:06.396 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:06.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:06.497 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:06.535 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:06.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:06.636 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:06.679 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:06.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:06.780 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:06.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:06.811 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:06.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:06.912 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:06.949 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:06.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:06.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:06 smithi132 ceph-mon[33500]: 4.2 scrub starts 2024-03-26T05:14:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:06 smithi132 ceph-mon[33500]: 4.2 scrub ok 2024-03-26T05:14:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:06 smithi132 ceph-mon[33500]: 5.13 scrub starts 2024-03-26T05:14:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:06 smithi132 ceph-mon[33500]: 5.13 scrub ok 2024-03-26T05:14:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:06 smithi132 ceph-mon[33500]: 6.1e scrub starts 2024-03-26T05:14:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:06 smithi132 ceph-mon[33500]: 6.1e scrub ok 2024-03-26T05:14:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:06 smithi132 ceph-mon[33500]: pgmap v614: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1016 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 9430/49410 objects degraded (19.085%) 2024-03-26T05:14:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:06 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 10939/58434 objects degraded (18.720%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:07.051 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:07.081 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:07.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:07.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[29691]: 4.2 scrub starts 2024-03-26T05:14:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[29691]: 4.2 scrub ok 2024-03-26T05:14:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[29691]: 5.13 scrub starts 2024-03-26T05:14:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[29691]: 5.13 scrub ok 2024-03-26T05:14:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[29691]: 6.1e scrub starts 2024-03-26T05:14:07.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[29691]: 6.1e scrub ok 2024-03-26T05:14:07.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[29691]: pgmap v614: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1016 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 9430/49410 objects degraded (19.085%) 2024-03-26T05:14:07.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 10939/58434 objects degraded (18.720%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[40404]: 4.2 scrub starts 2024-03-26T05:14:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[40404]: 4.2 scrub ok 2024-03-26T05:14:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[40404]: 5.13 scrub starts 2024-03-26T05:14:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[40404]: 5.13 scrub ok 2024-03-26T05:14:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[40404]: 6.1e scrub starts 2024-03-26T05:14:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[40404]: 6.1e scrub ok 2024-03-26T05:14:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[40404]: pgmap v614: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 1016 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 9430/49410 objects degraded (19.085%) 2024-03-26T05:14:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:06 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 10939/58434 objects degraded (18.720%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:07.183 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:07.208 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:07.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:07.310 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:07.340 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:07.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:07.443 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:07.473 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:07.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:07.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:07.575 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:07.612 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:07.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:07.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:07.714 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:07.753 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:07.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:07.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:07.856 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:07.891 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:07.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:07.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:07.993 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:08.031 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:08.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:07 smithi132 ceph-mon[33500]: 6.1 scrub starts 2024-03-26T05:14:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:07 smithi132 ceph-mon[33500]: 6.1 scrub ok 2024-03-26T05:14:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:07 smithi132 ceph-mon[33500]: 6.0 scrub starts 2024-03-26T05:14:08.088 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:07 smithi080 ceph-mon[40404]: 6.1 scrub starts 2024-03-26T05:14:08.088 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:07 smithi080 ceph-mon[40404]: 6.1 scrub ok 2024-03-26T05:14:08.088 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:07 smithi080 ceph-mon[40404]: 6.0 scrub starts 2024-03-26T05:14:08.089 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:07 smithi080 ceph-mon[29691]: 6.1 scrub starts 2024-03-26T05:14:08.089 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:07 smithi080 ceph-mon[29691]: 6.1 scrub ok 2024-03-26T05:14:08.089 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:07 smithi080 ceph-mon[29691]: 6.0 scrub starts 2024-03-26T05:14:08.133 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:08.161 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:08.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:08.263 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:08.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:08.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:08.394 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:08.421 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:08.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:08.523 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:08.554 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:08.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:08.657 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:08.694 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:08.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:08.796 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:08.842 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:08.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:08.902 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:08 smithi132 ceph-mon[33500]: 6.0 scrub ok 2024-03-26T05:14:08.902 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:08 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:14:08.902 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:08 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:14:08.902 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:08 smithi132 ceph-mon[33500]: pgmap v615: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 831 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s; 7558/40542 objects degraded (18.642%) 2024-03-26T05:14:08.902 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:08 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:14:08.902 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:08 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:14:08.944 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:14:08.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:14:09.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:09.027 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:09.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:09.082 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:08 smithi080 ceph-mon[40404]: 6.0 scrub ok 2024-03-26T05:14:09.082 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:08 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:14:09.082 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:08 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:14:09.082 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:08 smithi080 ceph-mon[40404]: pgmap v615: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 831 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s; 7558/40542 objects degraded (18.642%) 2024-03-26T05:14:09.082 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:08 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:14:09.082 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:08 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:14:09.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:08 smithi080 ceph-mon[29691]: 6.0 scrub ok 2024-03-26T05:14:09.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:08 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:14:09.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:08 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:14:09.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:08 smithi080 ceph-mon[29691]: pgmap v615: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 831 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s; 7558/40542 objects degraded (18.642%) 2024-03-26T05:14:09.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:08 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:14:09.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:08 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:14:09.128 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:09.157 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:09.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:09.259 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:09.302 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:09.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:09.404 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:09.450 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:09.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:09.520 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:14:09.520 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:14:09.552 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:09.611 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:09.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:09.713 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:09.735 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:14:09.735 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:14:09.735 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:14:09.735 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:14:09.735 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:14:09.735 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:14:09.735 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:14:09.735 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:14:09.735 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:14:09.735 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:14:09.735 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:14:09.735 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:14:09.750 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:09.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:09.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:09.851 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:09.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:09.893 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:09.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:09.994 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:10.032 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:10.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:10.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:09 smithi080 ceph-mon[40404]: osdmap e309: 8 total, 7 up, 8 in 2024-03-26T05:14:10.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:09 smithi080 ceph-mon[40404]: 3.b scrub starts 2024-03-26T05:14:10.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:09 smithi080 ceph-mon[40404]: 3.b scrub ok 2024-03-26T05:14:10.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:09 smithi080 ceph-mon[29691]: osdmap e309: 8 total, 7 up, 8 in 2024-03-26T05:14:10.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:09 smithi080 ceph-mon[29691]: 3.b scrub starts 2024-03-26T05:14:10.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:09 smithi080 ceph-mon[29691]: 3.b scrub ok 2024-03-26T05:14:10.132 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:09 smithi132 ceph-mon[33500]: osdmap e309: 8 total, 7 up, 8 in 2024-03-26T05:14:10.133 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:09 smithi132 ceph-mon[33500]: 3.b scrub starts 2024-03-26T05:14:10.133 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:09 smithi132 ceph-mon[33500]: 3.b scrub ok 2024-03-26T05:14:10.135 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:10.168 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:10.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:10.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:10.210 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T05:14:10.270 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:10.304 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:10.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:10.406 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:10.439 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:10.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:10.540 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:10.550 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:14:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:14:10.561 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:14:10.561 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-26T05:14:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:10.577 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:10.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:10.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:10.723 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:10.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:10.825 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:10.827 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:14:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:10.889 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:10.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:10.948 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:14:10.958 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:10 smithi132 ceph-mon[33500]: 3.2 scrub starts 2024-03-26T05:14:10.959 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:10 smithi132 ceph-mon[33500]: 3.2 scrub ok 2024-03-26T05:14:10.959 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:10 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3193203701' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:10.959 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:10 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:10.959 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:10 smithi132 ceph-mon[33500]: pgmap v617: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 807 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 7184/39420 objects degraded (18.224%) 2024-03-26T05:14:10.959 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:10 smithi132 ceph-mon[33500]: 4.6 scrub starts 2024-03-26T05:14:10.990 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:11.042 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:11.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:11.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:10 smithi080 ceph-mon[29691]: 3.2 scrub starts 2024-03-26T05:14:11.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:10 smithi080 ceph-mon[29691]: 3.2 scrub ok 2024-03-26T05:14:11.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:10 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3193203701' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:11.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:10 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:11.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:10 smithi080 ceph-mon[29691]: pgmap v617: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 807 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 7184/39420 objects degraded (18.224%) 2024-03-26T05:14:11.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:10 smithi080 ceph-mon[29691]: 4.6 scrub starts 2024-03-26T05:14:11.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:10 smithi080 ceph-mon[40404]: 3.2 scrub starts 2024-03-26T05:14:11.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:10 smithi080 ceph-mon[40404]: 3.2 scrub ok 2024-03-26T05:14:11.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:10 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3193203701' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:11.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:10 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:11.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:10 smithi080 ceph-mon[40404]: pgmap v617: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 807 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 7184/39420 objects degraded (18.224%) 2024-03-26T05:14:11.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:10 smithi080 ceph-mon[40404]: 4.6 scrub starts 2024-03-26T05:14:11.144 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:11.174 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:11.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:11.276 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:11.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:11.325 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:11.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:11.426 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:11.463 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:11.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:11.565 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:11.592 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:11.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:11.694 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:11.754 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:11.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:11.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:11.847 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:11.856 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:11.872 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:11.916 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:11.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:11.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[29691]: 4.6 scrub ok 2024-03-26T05:14:11.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[29691]: 5.b scrub starts 2024-03-26T05:14:11.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[29691]: 5.b scrub ok 2024-03-26T05:14:11.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[29691]: 3.10 scrub starts 2024-03-26T05:14:11.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[29691]: 3.10 scrub ok 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[29691]: 6.19 scrub starts 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[29691]: osdmap e310: 8 total, 7 up, 8 in 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[29691]: 6.19 scrub ok 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 7184/39420 objects degraded (18.224%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[40404]: 4.6 scrub ok 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[40404]: 5.b scrub starts 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[40404]: 5.b scrub ok 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[40404]: 3.10 scrub starts 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[40404]: 3.10 scrub ok 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[40404]: 6.19 scrub starts 2024-03-26T05:14:11.991 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:14:11.992 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:14:11.992 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[40404]: osdmap e310: 8 total, 7 up, 8 in 2024-03-26T05:14:11.992 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[40404]: 6.19 scrub ok 2024-03-26T05:14:11.992 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:11 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 7184/39420 objects degraded (18.224%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:12.017 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:11 smithi132 ceph-mon[33500]: 4.6 scrub ok 2024-03-26T05:14:12.044 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:11 smithi132 ceph-mon[33500]: 5.b scrub starts 2024-03-26T05:14:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:11 smithi132 ceph-mon[33500]: 5.b scrub ok 2024-03-26T05:14:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:11 smithi132 ceph-mon[33500]: 3.10 scrub starts 2024-03-26T05:14:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:11 smithi132 ceph-mon[33500]: 3.10 scrub ok 2024-03-26T05:14:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:11 smithi132 ceph-mon[33500]: 6.19 scrub starts 2024-03-26T05:14:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:11 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:14:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:11 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:14:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:11 smithi132 ceph-mon[33500]: osdmap e310: 8 total, 7 up, 8 in 2024-03-26T05:14:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:11 smithi132 ceph-mon[33500]: 6.19 scrub ok 2024-03-26T05:14:12.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:11 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 7184/39420 objects degraded (18.224%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:12.047 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:12.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:12.149 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:12.200 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:12.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:12.302 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:12.344 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:12.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:12.369 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:14:12.369 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:14:12.446 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:12.474 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:12.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:12.576 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:12.615 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:12.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:12.716 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:12.751 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:12.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:12.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:12.853 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:12.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:12.883 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:12.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:12.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:13.015 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:13.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:13.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:12 smithi132 ceph-mon[33500]: 3.0 scrub starts 2024-03-26T05:14:13.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:12 smithi132 ceph-mon[33500]: 3.0 scrub ok 2024-03-26T05:14:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:12 smithi132 ceph-mon[33500]: pgmap v619: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 452 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 4169/22359 objects degraded (18.646%) 2024-03-26T05:14:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:12 smithi132 ceph-mon[33500]: 3.6 scrub starts 2024-03-26T05:14:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:12 smithi132 ceph-mon[33500]: 3.6 scrub ok 2024-03-26T05:14:13.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:12 smithi080 ceph-mon[40404]: 3.0 scrub starts 2024-03-26T05:14:13.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:12 smithi080 ceph-mon[40404]: 3.0 scrub ok 2024-03-26T05:14:13.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:12 smithi080 ceph-mon[40404]: pgmap v619: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 452 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 4169/22359 objects degraded (18.646%) 2024-03-26T05:14:13.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:12 smithi080 ceph-mon[40404]: 3.6 scrub starts 2024-03-26T05:14:13.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:12 smithi080 ceph-mon[40404]: 3.6 scrub ok 2024-03-26T05:14:13.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:12 smithi080 ceph-mon[29691]: 3.0 scrub starts 2024-03-26T05:14:13.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:12 smithi080 ceph-mon[29691]: 3.0 scrub ok 2024-03-26T05:14:13.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:12 smithi080 ceph-mon[29691]: pgmap v619: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 452 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 4169/22359 objects degraded (18.646%) 2024-03-26T05:14:13.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:12 smithi080 ceph-mon[29691]: 3.6 scrub starts 2024-03-26T05:14:13.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:12 smithi080 ceph-mon[29691]: 3.6 scrub ok 2024-03-26T05:14:13.117 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:13.151 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T05:14:13.158 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:13.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:13.262 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:13.290 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:13.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:13.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:13.392 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:13.428 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:13.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:13.498 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:14:13.530 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:13.569 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:13.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:13.670 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:14:13.670 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:13.706 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:13.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:13.809 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:13.839 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:13.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:13.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:13.920 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:13 smithi132 ceph-mon[33500]: 5.7 scrub starts 2024-03-26T05:14:13.920 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:13 smithi132 ceph-mon[33500]: 5.7 scrub ok 2024-03-26T05:14:13.941 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:13.983 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:13.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:13.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:14.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:13 smithi080 ceph-mon[40404]: 5.7 scrub starts 2024-03-26T05:14:14.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:13 smithi080 ceph-mon[40404]: 5.7 scrub ok 2024-03-26T05:14:14.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:13 smithi080 ceph-mon[29691]: 5.7 scrub starts 2024-03-26T05:14:14.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:13 smithi080 ceph-mon[29691]: 5.7 scrub ok 2024-03-26T05:14:14.086 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:14.112 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:14.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:14.214 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:14.257 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:14.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:14.360 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:14.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:14:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:14:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:14:14.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:14.400 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:14.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:14.501 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:14.528 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:14.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:14.629 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:14.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:14.677 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:14.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:14.778 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:14.814 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:14.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:14.915 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:14.945 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:14.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:14.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:14 smithi132 ceph-mon[33500]: 9.4 scrub starts 2024-03-26T05:14:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:14 smithi132 ceph-mon[33500]: pgmap v620: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 452 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.13k op/s; 4169/22359 objects degraded (18.646%) 2024-03-26T05:14:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:14 smithi132 ceph-mon[33500]: 5.5 scrub starts 2024-03-26T05:14:15.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:14 smithi132 ceph-mon[33500]: 9.4 scrub ok 2024-03-26T05:14:15.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:14 smithi132 ceph-mon[33500]: 5.5 scrub ok 2024-03-26T05:14:15.047 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:15.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:15.098 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:15.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:15.109 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:14 smithi080 ceph-mon[40404]: 9.4 scrub starts 2024-03-26T05:14:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:14 smithi080 ceph-mon[40404]: pgmap v620: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 452 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.13k op/s; 4169/22359 objects degraded (18.646%) 2024-03-26T05:14:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:14 smithi080 ceph-mon[40404]: 5.5 scrub starts 2024-03-26T05:14:15.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:14 smithi080 ceph-mon[40404]: 9.4 scrub ok 2024-03-26T05:14:15.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:14 smithi080 ceph-mon[40404]: 5.5 scrub ok 2024-03-26T05:14:15.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:14 smithi080 ceph-mon[29691]: 9.4 scrub starts 2024-03-26T05:14:15.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:14 smithi080 ceph-mon[29691]: pgmap v620: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 452 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.13k op/s; 4169/22359 objects degraded (18.646%) 2024-03-26T05:14:15.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:14 smithi080 ceph-mon[29691]: 5.5 scrub starts 2024-03-26T05:14:15.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:14 smithi080 ceph-mon[29691]: 9.4 scrub ok 2024-03-26T05:14:15.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:14 smithi080 ceph-mon[29691]: 5.5 scrub ok 2024-03-26T05:14:15.199 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:15.246 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:15.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:15.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:15.342 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:14:15.342 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:14:15.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:15.389 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:15.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:15.491 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:15.521 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:15.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:15.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:15.623 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:15.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:15.652 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:15.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:15.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:14:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:14:15.753 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:15.812 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:15.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:15.885 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:14:15.914 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:15.937 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:15.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:15.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:15.979 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:15 smithi132 ceph-mon[33500]: 6.1f deep-scrub starts 2024-03-26T05:14:15.979 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:15 smithi132 ceph-mon[33500]: 6.1f deep-scrub ok 2024-03-26T05:14:15.979 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:15 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:16.015 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T05:14:16.039 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:16.087 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:16.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:16.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:15 smithi080 ceph-mon[40404]: 6.1f deep-scrub starts 2024-03-26T05:14:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:15 smithi080 ceph-mon[40404]: 6.1f deep-scrub ok 2024-03-26T05:14:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:15 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:16.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:15 smithi080 ceph-mon[29691]: 6.1f deep-scrub starts 2024-03-26T05:14:16.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:15 smithi080 ceph-mon[29691]: 6.1f deep-scrub ok 2024-03-26T05:14:16.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:15 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:16.189 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:16.219 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:16.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:16.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:16.322 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:16.371 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Removed 121580 objects 2024-03-26T05:14:16.371 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Clean up completed and total clean up time :97.6176 2024-03-26T05:14:16.378 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:14:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:16.553 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:16.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:16.654 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:16.677 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:16.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:16.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:16.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:16.804 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:16.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:16.904 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:16.928 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:16.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:17.030 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:17.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:16 smithi132 ceph-mon[33500]: 3.5 scrub starts 2024-03-26T05:14:17.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:16 smithi132 ceph-mon[33500]: 3.5 scrub ok 2024-03-26T05:14:17.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:16 smithi132 ceph-mon[33500]: pgmap v621: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 246 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 2544/12480 objects degraded (20.385%) 2024-03-26T05:14:17.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:16 smithi132 ceph-mon[33500]: 6.4 scrub starts 2024-03-26T05:14:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:16 smithi132 ceph-mon[33500]: 6.4 scrub ok 2024-03-26T05:14:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:16 smithi132 ceph-mon[33500]: 4.b scrub starts 2024-03-26T05:14:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:16 smithi132 ceph-mon[33500]: 4.b scrub ok 2024-03-26T05:14:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:16 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 4169/22359 objects degraded (18.646%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:17.054 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:17.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[29691]: 3.5 scrub starts 2024-03-26T05:14:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[29691]: 3.5 scrub ok 2024-03-26T05:14:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[29691]: pgmap v621: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 246 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 2544/12480 objects degraded (20.385%) 2024-03-26T05:14:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[29691]: 6.4 scrub starts 2024-03-26T05:14:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[29691]: 6.4 scrub ok 2024-03-26T05:14:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[29691]: 4.b scrub starts 2024-03-26T05:14:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[29691]: 4.b scrub ok 2024-03-26T05:14:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 4169/22359 objects degraded (18.646%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:17.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[40404]: 3.5 scrub starts 2024-03-26T05:14:17.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[40404]: 3.5 scrub ok 2024-03-26T05:14:17.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[40404]: pgmap v621: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 246 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 2544/12480 objects degraded (20.385%) 2024-03-26T05:14:17.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[40404]: 6.4 scrub starts 2024-03-26T05:14:17.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[40404]: 6.4 scrub ok 2024-03-26T05:14:17.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[40404]: 4.b scrub starts 2024-03-26T05:14:17.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[40404]: 4.b scrub ok 2024-03-26T05:14:17.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:16 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 4169/22359 objects degraded (18.646%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:17.156 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:17.181 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:17.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:17.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:17.308 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:17.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:17.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:17.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:17.433 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:17.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:17.535 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:17.558 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:17.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:17.660 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:17.683 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:17.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:17.784 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:17.808 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:17.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:17.824 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-26T05:14:17.824 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-26T05:14:17.909 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:17.932 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:17.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:17.997 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:14:17.997 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:14:18.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:18.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:17 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:14:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:17 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:14:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:17 smithi132 ceph-mon[33500]: 3.1e scrub starts 2024-03-26T05:14:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:17 smithi132 ceph-mon[33500]: 3.1e scrub ok 2024-03-26T05:14:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:17 smithi132 ceph-mon[33500]: 3.1 scrub starts 2024-03-26T05:14:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:17 smithi132 ceph-mon[33500]: 3.1 scrub ok 2024-03-26T05:14:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:17 smithi132 ceph-mon[33500]: 9.0 scrub starts 2024-03-26T05:14:18.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:17 smithi132 ceph-mon[33500]: 9.0 scrub ok 2024-03-26T05:14:18.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:18.060 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:18.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:18.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:14:18.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:14:18.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[29691]: 3.1e scrub starts 2024-03-26T05:14:18.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[29691]: 3.1e scrub ok 2024-03-26T05:14:18.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[29691]: 3.1 scrub starts 2024-03-26T05:14:18.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[29691]: 3.1 scrub ok 2024-03-26T05:14:18.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[29691]: 9.0 scrub starts 2024-03-26T05:14:18.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[29691]: 9.0 scrub ok 2024-03-26T05:14:18.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:14:18.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:14:18.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[40404]: 3.1e scrub starts 2024-03-26T05:14:18.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[40404]: 3.1e scrub ok 2024-03-26T05:14:18.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[40404]: 3.1 scrub starts 2024-03-26T05:14:18.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[40404]: 3.1 scrub ok 2024-03-26T05:14:18.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[40404]: 9.0 scrub starts 2024-03-26T05:14:18.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:17 smithi080 ceph-mon[40404]: 9.0 scrub ok 2024-03-26T05:14:18.160 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:18.183 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:18.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:18.285 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:18.309 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:18.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:18.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:18.436 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:18.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:18.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:14:18.438 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:14:18.438 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-26T05:14:18.537 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:18.561 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:18.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:18.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:18.595 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T05:14:18.662 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:18.701 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:18.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:18.803 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:18.828 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:18.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:18.875 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:14:18.929 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:18.941 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:18 smithi080 ceph-mon[40404]: 3.1d scrub starts 2024-03-26T05:14:18.941 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:18 smithi080 ceph-mon[40404]: 3.1d scrub ok 2024-03-26T05:14:18.941 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:18 smithi080 ceph-mon[40404]: pgmap v622: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 51 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.36k op/s; 570/3105 objects degraded (18.357%) 2024-03-26T05:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:18 smithi080 ceph-mon[29691]: 3.1d scrub starts 2024-03-26T05:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:18 smithi080 ceph-mon[29691]: 3.1d scrub ok 2024-03-26T05:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:18 smithi080 ceph-mon[29691]: pgmap v622: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 51 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.36k op/s; 570/3105 objects degraded (18.357%) 2024-03-26T05:14:18.953 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:18.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:18.993 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:14:18.993 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:14:18.993 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:14:19.001 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:14:19.021 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:14:19.040 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:14:19.055 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:19.060 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:14:19.061 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:14:19.061 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:14:19.061 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:14:19.061 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:14:19.061 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:14:19.062 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:14:19.062 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:14:19.062 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:14:19.079 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:19.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:19.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:19.175 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:18 smithi132 ceph-mon[33500]: 3.1d scrub starts 2024-03-26T05:14:19.175 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:18 smithi132 ceph-mon[33500]: 3.1d scrub ok 2024-03-26T05:14:19.175 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:18 smithi132 ceph-mon[33500]: pgmap v622: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 51 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.36k op/s; 570/3105 objects degraded (18.357%) 2024-03-26T05:14:19.181 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:19.206 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:19.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:19.308 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:19.331 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:19.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:19.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:19.433 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:19.456 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:19.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:19.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:19.558 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:19.581 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:19.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:19.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:19.683 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:19.707 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:19.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:19.809 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:19.832 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:19.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:19.934 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:19.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:19.958 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:19.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:20.020 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:19 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2823914279' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:20.020 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:19 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2823914279' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:20.059 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:20.082 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:20.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:20.184 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:20.208 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:20.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:20.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:20.230 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:19 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2823914279' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:20.303 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:14:20.303 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:14:20.310 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:20.333 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:20.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:20.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:20.435 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:20.458 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:20.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:20.471 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:14:20.471 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:14:20.471 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2024-03-26T05:14:20.471 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:14:20.560 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:20.584 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:20.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:20.686 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:20.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:20.811 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:20.836 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:20.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:20.849 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:20.877 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:20.938 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:20.963 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:20.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:21.065 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:21.089 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:21.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:21.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:21.119 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T05:14:21.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:20 smithi132 ceph-mon[33500]: 6.f scrub starts 2024-03-26T05:14:21.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:20 smithi132 ceph-mon[33500]: 6.f scrub ok 2024-03-26T05:14:21.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:20 smithi132 ceph-mon[33500]: 3.18 scrub starts 2024-03-26T05:14:21.144 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:20 smithi132 ceph-mon[33500]: 3.18 scrub ok 2024-03-26T05:14:21.144 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:20 smithi132 ceph-mon[33500]: pgmap v623: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 44 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 453/2754 objects degraded (16.449%) 2024-03-26T05:14:21.144 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:20 smithi132 ceph-mon[33500]: 3.1f scrub starts 2024-03-26T05:14:21.144 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:20 smithi132 ceph-mon[33500]: 3.1f scrub ok 2024-03-26T05:14:21.144 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:14:21.144 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:14:21.190 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:21.214 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:21.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:21.215 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[40404]: 6.f scrub starts 2024-03-26T05:14:21.215 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[40404]: 6.f scrub ok 2024-03-26T05:14:21.215 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[40404]: 3.18 scrub starts 2024-03-26T05:14:21.215 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[40404]: 3.18 scrub ok 2024-03-26T05:14:21.215 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[40404]: pgmap v623: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 44 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 453/2754 objects degraded (16.449%) 2024-03-26T05:14:21.215 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[40404]: 3.1f scrub starts 2024-03-26T05:14:21.215 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[40404]: 3.1f scrub ok 2024-03-26T05:14:21.216 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:14:21.216 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:14:21.216 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[29691]: 6.f scrub starts 2024-03-26T05:14:21.216 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[29691]: 6.f scrub ok 2024-03-26T05:14:21.216 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[29691]: 3.18 scrub starts 2024-03-26T05:14:21.216 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[29691]: 3.18 scrub ok 2024-03-26T05:14:21.216 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[29691]: pgmap v623: 148 pgs: 87 active+clean, 33 active+undersized, 28 active+undersized+degraded; 44 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 453/2754 objects degraded (16.449%) 2024-03-26T05:14:21.216 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[29691]: 3.1f scrub starts 2024-03-26T05:14:21.216 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[29691]: 3.1f scrub ok 2024-03-26T05:14:21.216 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:14:21.216 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:14:21.316 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:21.340 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:21.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:21.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:21.361 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:14:21.442 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:21.466 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:21.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:21.567 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:21.593 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:21.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:21.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:21.695 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:21.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:21.720 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:21.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:21.821 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:21.844 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:21.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:21.887 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:14:21.946 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:21.971 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:21.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:22.070 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:21 smithi080 ceph-mon[29691]: 5.15 scrub starts 2024-03-26T05:14:22.070 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:21 smithi080 ceph-mon[29691]: 5.15 scrub ok 2024-03-26T05:14:22.070 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:21 smithi080 ceph-mon[29691]: 5.c scrub starts 2024-03-26T05:14:22.070 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:21 smithi080 ceph-mon[29691]: 5.c scrub ok 2024-03-26T05:14:22.070 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:21 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2335952473' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:14:22.070 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:21 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 453/2754 objects degraded (16.449%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:22.070 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:21 smithi080 ceph-mon[40404]: 5.15 scrub starts 2024-03-26T05:14:22.070 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:21 smithi080 ceph-mon[40404]: 5.15 scrub ok 2024-03-26T05:14:22.071 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:21 smithi080 ceph-mon[40404]: 5.c scrub starts 2024-03-26T05:14:22.071 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:21 smithi080 ceph-mon[40404]: 5.c scrub ok 2024-03-26T05:14:22.071 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:21 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2335952473' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:14:22.071 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:21 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 453/2754 objects degraded (16.449%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:22.072 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:22.095 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:22.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:22.190 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:21 smithi132 ceph-mon[33500]: 5.15 scrub starts 2024-03-26T05:14:22.190 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:21 smithi132 ceph-mon[33500]: 5.15 scrub ok 2024-03-26T05:14:22.191 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:21 smithi132 ceph-mon[33500]: 5.c scrub starts 2024-03-26T05:14:22.191 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:21 smithi132 ceph-mon[33500]: 5.c scrub ok 2024-03-26T05:14:22.191 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:21 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2335952473' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:14:22.191 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:21 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 453/2754 objects degraded (16.449%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:22.197 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:22.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:22.222 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:22.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:22.304 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:14:22.322 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:22.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:22.375 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:22.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:22.475 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:22.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:22.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:22.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:22.600 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:22.624 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:22.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:22.726 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:22.749 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:22.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:22.851 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:22.874 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:22.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:22.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:22.975 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:22.984 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:22 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:14:22.984 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:22 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2335952473' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:14:22.984 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:22 smithi080 ceph-mon[40404]: osdmap e311: 8 total, 7 up, 8 in 2024-03-26T05:14:22.984 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:22 smithi080 ceph-mon[40404]: pgmap v625: 148 pgs: 87 active+clean, 36 active+undersized, 25 active+undersized+degraded; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 722 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:22.984 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:22 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:14:22.984 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:22 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2335952473' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:14:22.984 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:22 smithi080 ceph-mon[29691]: osdmap e311: 8 total, 7 up, 8 in 2024-03-26T05:14:22.984 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:22 smithi080 ceph-mon[29691]: pgmap v625: 148 pgs: 87 active+clean, 36 active+undersized, 25 active+undersized+degraded; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 722 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:22.999 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:23.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:23.101 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:23.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:23.124 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:23.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:23.170 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:22 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:14:23.170 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:22 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2335952473' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:14:23.170 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:22 smithi132 ceph-mon[33500]: osdmap e311: 8 total, 7 up, 8 in 2024-03-26T05:14:23.171 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:22 smithi132 ceph-mon[33500]: pgmap v625: 148 pgs: 87 active+clean, 36 active+undersized, 25 active+undersized+degraded; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 722 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:23.218 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:14:23.218 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:14:23.218 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2024-03-26T05:14:23.218 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2024-03-26T05:14:23.218 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:14:23.225 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:23.248 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:23.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:23.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:23.349 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:23.375 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:23.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:23.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:23.477 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:23.478 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:14:23.478 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:14:23.478 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:14:23.478 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:14:23.479 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:14:23.479 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:14:23.479 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:14:23.479 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:14:23.479 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:14:23.479 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:14:23.479 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:14:23.479 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:14:23.503 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:23.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:23.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:23.604 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:23.628 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:23.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:23.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:23.730 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:23.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:23.755 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:23.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:23.856 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:23.880 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:23.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:23.893 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T05:14:23.982 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:24.009 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:24.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:24.101 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:14:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:14:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:14:24.111 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:24.136 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:24.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:24.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:14:24.238 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:24.261 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:24.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:24.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:24.363 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:24.388 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:24.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:24.489 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:24.514 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:24.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:24.615 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:24.625 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:24.634 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:14:24.634 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:14:24.641 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:24.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:24.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:24.743 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:24.766 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:24.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:24.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:24.867 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:24.892 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:24.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:24.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:24.993 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:25.017 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:25.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:25.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:25.119 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:25.143 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:25.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:25.244 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:25.268 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:25.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:25.299 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:25 smithi080 ceph-mon[40404]: pgmap v626: 148 pgs: 87 active+clean, 36 active+undersized, 25 active+undersized+degraded; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 722 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:25.300 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:25 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:14:25.300 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:25 smithi080 ceph-mon[29691]: pgmap v626: 148 pgs: 87 active+clean, 36 active+undersized, 25 active+undersized+degraded; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 722 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:25.300 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:25 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:14:25.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:25 smithi132 ceph-mon[33500]: pgmap v626: 148 pgs: 87 active+clean, 36 active+undersized, 25 active+undersized+degraded; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 722 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:25.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:25 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:14:25.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:25.394 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:25.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:25.496 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:25.519 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:25.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:25.621 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:25.644 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:25.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:25.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:25.645 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:14:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:14:25.746 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:25.769 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:25.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:25.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:25.870 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:25.895 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:25.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:25.995 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:26.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:26.021 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:26.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:26.063 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:14:26.063 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:14:26.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198054 8.9 8:99f5ccfa:::benchmark_data_smithi080_109213_object83537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:53.895008+0000", 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 272.17643407399999, 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20899105000000001, 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198054 2024-03-26T05:14:26.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895008+0000", 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895008+0000", 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895010+0000", 2024-03-26T05:14:26.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.72e-06 2024-03-26T05:14:26.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895014+0000", 2024-03-26T05:14:26.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.69e-06 2024-03-26T05:14:26.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895028+0000", 2024-03-26T05:14:26.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3808000000000001e-05 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895036+0000", 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6459999999999993e-06 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.981756+0000", 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.086720452000000003 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.981876+0000", 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119364 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.982000+0000", 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012400099999999999 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992267+0000", 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010267642 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992404+0000", 2024-03-26T05:14:26.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000136713 2024-03-26T05:14:26.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.103815+0000", 2024-03-26T05:14:26.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.111411407 2024-03-26T05:14:26.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.103923+0000", 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000107909 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.103999+0000", 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5698000000000006e-05 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198055 8.9 8:9376f681:::benchmark_data_smithi080_109213_object83538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.070 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:53.895393+0000", 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 272.176048831, 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.208813532, 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198055 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895393+0000", 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895393+0000", 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895394+0000", 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6400000000000002e-07 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895397+0000", 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5499999999999999e-06 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895410+0000", 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2989e-05 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895417+0000", 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4330000000000001e-06 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.984852+0000", 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.089435597000000006 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.984975+0000", 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000122328 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.985100+0000", 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000125513 2024-03-26T05:14:26.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992272+0000", 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0071720559999999996 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002218+0000", 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.009945934 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104058+0000", 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.101839334 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104151+0000", 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3468999999999996e-05 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104207+0000", 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5665000000000003e-05 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198065 8.9 8:9d5183f8:::benchmark_data_smithi080_109213_object83548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.076 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:53.906108+0000", 2024-03-26T05:14:26.077 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 272.16533396599999, 2024-03-26T05:14:26.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.19828168400000001, 2024-03-26T05:14:26.077 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.077 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.077 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.077 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.077 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.077 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198065 2024-03-26T05:14:26.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.077 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.077 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.078 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.906108+0000", 2024-03-26T05:14:26.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.906108+0000", 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.906109+0000", 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2099999999999995e-07 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.906112+0000", 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0510000000000001e-06 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.906122+0000", 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7829999999999993e-06 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.906129+0000", 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2080000000000003e-06 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.988438+0000", 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.082309388999999997 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.988554+0000", 2024-03-26T05:14:26.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011518099999999999 2024-03-26T05:14:26.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.988686+0000", 2024-03-26T05:14:26.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132564 2024-03-26T05:14:26.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.998891+0000", 2024-03-26T05:14:26.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010205182 2024-03-26T05:14:26.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002261+0000", 2024-03-26T05:14:26.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0033697000000000002 2024-03-26T05:14:26.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104254+0000", 2024-03-26T05:14:26.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.101992572 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104333+0000", 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9555999999999995e-05 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104390+0000", 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6576999999999997e-05 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.083 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198081 8.9 8:950f9248:::benchmark_data_smithi080_109213_object83564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:53.917093+0000", 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 272.15434857999998, 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18745711200000001, 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198081 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.917093+0000", 2024-03-26T05:14:26.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.917093+0000", 2024-03-26T05:14:26.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.917094+0000", 2024-03-26T05:14:26.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.61e-07 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.917098+0000", 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7610000000000001e-06 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.917111+0000", 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2676e-05 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.917117+0000", 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9639999999999997e-06 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992434+0000", 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.075316607999999993 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992546+0000", 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000112419 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992657+0000", 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000110328 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002229+0000", 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0095720879999999994 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002294+0000", 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4789000000000006e-05 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104438+0000", 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.102143832 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104498+0000", 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0931999999999999e-05 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104551+0000", 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2054e-05 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198086 8.9 8:9e1cc76a:::benchmark_data_smithi080_109213_object83569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:53.922111+0000", 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 272.14933106699999, 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18262619699999999, 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198086 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.922111+0000", 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.922111+0000", 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.922112+0000", 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.184e-06 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.922117+0000", 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2549999999999997e-06 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.922133+0000", 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5877999999999999e-05 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.922142+0000", 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0559999999999994e-06 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.995725+0000", 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.073583027999999995 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.995859+0000", 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000134108 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.995989+0000", 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000129417 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002303+0000", 2024-03-26T05:14:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0063138350000000003 2024-03-26T05:14:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002694+0000", 2024-03-26T05:14:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00039094999999999999 2024-03-26T05:14:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104638+0000", 2024-03-26T05:14:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10194389099999999 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104685+0000", 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7749e-05 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104737+0000", 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1845999999999998e-05 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198508 8.9 8:986ea8e3:::benchmark_data_smithi080_109213_object83991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.353239+0000", 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 271.71820319900002, 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20569538500000001, 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198508 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353239+0000", 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353239+0000", 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353240+0000", 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4100000000000001e-06 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353245+0000", 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2000000000000002e-06 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353258+0000", 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2106000000000001e-05 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353272+0000", 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4893e-05 2024-03-26T05:14:26.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.406729+0000", 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.053456469 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.406825+0000", 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6075000000000004e-05 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.406903+0000", 2024-03-26T05:14:26.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8450999999999998e-05 2024-03-26T05:14:26.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.409021+0000", 2024-03-26T05:14:26.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002117819 2024-03-26T05:14:26.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.410391+0000", 2024-03-26T05:14:26.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001370008 2024-03-26T05:14:26.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.558821+0000", 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.148429591 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.558877+0000", 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6178000000000003e-05 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.558934+0000", 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7185000000000002e-05 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198509 8.9 8:914b5413:::benchmark_data_smithi080_109213_object83992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.353322+0000", 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 271.71812010600001, 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.205746507, 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.102 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198509 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353322+0000", 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353322+0000", 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353322+0000", 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9500000000000003e-07 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353325+0000", 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.909e-06 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353333+0000", 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3699999999999997e-06 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353340+0000", 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1969999999999999e-06 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.410429+0000", 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.057089411999999999 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.410567+0000", 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000137405 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.410721+0000", 2024-03-26T05:14:26.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015416099999999999 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416309+0000", 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0055884469999999999 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416351+0000", 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2101999999999998e-05 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.558984+0000", 2024-03-26T05:14:26.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14263231300000001 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.559015+0000", 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.1498000000000002e-05 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.559068+0000", 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3198000000000001e-05 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.107 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198524 8.9 8:97161286:::benchmark_data_smithi080_109213_object84007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.366616+0000", 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 271.70482568400001, 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20856360500000001, 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198524 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366616+0000", 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366616+0000", 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366618+0000", 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4479999999999999e-06 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366623+0000", 2024-03-26T05:14:26.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3310000000000003e-06 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366639+0000", 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.59e-05 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366648+0000", 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5639999999999997e-06 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.412563+0000", 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.045915688000000003 2024-03-26T05:14:26.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.412676+0000", 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000113008 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.412809+0000", 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013275799999999999 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416343+0000", 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0035344500000000002 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416354+0000", 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0794e-05 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575077+0000", 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.15872312899999999 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575136+0000", 2024-03-26T05:14:26.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8783000000000003e-05 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575180+0000", 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3751999999999997e-05 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198530 8.9 8:9a064d51:::benchmark_data_smithi080_109213_object84013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.370993+0000", 2024-03-26T05:14:26.113 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 271.70044863999999, 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20430893899999999, 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198530 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.370993+0000", 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.370993+0000", 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.370995+0000", 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3349999999999999e-06 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.371001+0000", 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2369999999999997e-06 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.371012+0000", 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0692e-05 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.371018+0000", 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9390000000000002e-06 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.414470+0000", 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043452706000000001 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.414555+0000", 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4563999999999999e-05 2024-03-26T05:14:26.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.414637+0000", 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2443999999999999e-05 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416408+0000", 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0017704999999999999 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416541+0000", 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000133337 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575224+0000", 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.15868285500000001 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575262+0000", 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7988999999999998e-05 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575302+0000", 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0340999999999999e-05 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.118 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:199631 8.9 8:951c68a4:::benchmark_data_smithi080_109213_object85114:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:55.624693+0000", 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 270.44674902899999, 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20792925800000001, 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 199631 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.119 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624693+0000", 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624693+0000", 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624695+0000", 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5770000000000001e-06 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624699+0000", 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6999999999999999e-06 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624710+0000", 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0747000000000001e-05 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624717+0000", 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1489999999999997e-06 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.655682+0000", 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.030964476000000001 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.655762+0000", 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0160999999999994e-05 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.655849+0000", 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7125000000000004e-05 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.123 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.660996+0000", 2024-03-26T05:14:26.123 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0051472660000000002 2024-03-26T05:14:26.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.123 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.663215+0000", 2024-03-26T05:14:26.123 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0022190790000000001 2024-03-26T05:14:26.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.123 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832537+0000", 2024-03-26T05:14:26.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.169322062 2024-03-26T05:14:26.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832580+0000", 2024-03-26T05:14:26.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3037999999999997e-05 2024-03-26T05:14:26.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832622+0000", 2024-03-26T05:14:26.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1878000000000001e-05 2024-03-26T05:14:26.124 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:199636 8.9 8:959205e3:::benchmark_data_smithi080_109213_object85119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:55.645400+0000", 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 270.42604201500001, 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18738101200000001, 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 199636 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.125 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.645400+0000", 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.645400+0000", 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.645401+0000", 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5149999999999999e-06 2024-03-26T05:14:26.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.645408+0000", 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9890000000000003e-06 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.645420+0000", 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1250000000000001e-05 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.645428+0000", 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9149999999999996e-06 2024-03-26T05:14:26.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.657862+0000", 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.012434007 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.657948+0000", 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6234000000000001e-05 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.658071+0000", 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000123225 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.661254+0000", 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031827270000000002 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.663395+0000", 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021413059999999999 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832675+0000", 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.169279755 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.129 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832725+0000", 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0367000000000001e-05 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832781+0000", 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5721999999999999e-05 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.130 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:199926 8.9 8:9bc5c9ee:::benchmark_data_smithi080_109213_object85409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.131 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:26.132 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:55.992851+0000", 2024-03-26T05:14:26.132 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 270.07859110999999, 2024-03-26T05:14:26.132 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.201735579, 2024-03-26T05:14:26.132 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.132 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.132 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.132 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.132 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.132 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 199926 2024-03-26T05:14:26.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.132 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992851+0000", 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992851+0000", 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992852+0000", 2024-03-26T05:14:26.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5999999999999991e-07 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992855+0000", 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2789999999999999e-06 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992865+0000", 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3999999999999998e-06 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992870+0000", 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4389999999999999e-06 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.992911+0000", 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1449000000000002e-05 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.993006+0000", 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4477999999999999e-05 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.135 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.993085+0000", 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9538999999999999e-05 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.996438+0000", 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00335213 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.996976+0000", 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053849499999999995 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.194301+0000", 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.197324468 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.194517+0000", 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021599999999999999 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.194586+0000", 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9942000000000002e-05 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.137 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:200274 8.9 8:906226e0:::benchmark_data_smithi080_109213_object85757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:56.335381+0000", 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 269.736060506, 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.17162375199999999, 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 200274 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.335381+0000", 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.335381+0000", 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.335383+0000", 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4160000000000001e-06 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.335387+0000", 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1479999999999999e-06 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.335397+0000", 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6760000000000002e-06 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.335404+0000", 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0570000000000004e-06 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.380146+0000", 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044741784 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.380281+0000", 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000135666 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.380483+0000", 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00020144600000000001 2024-03-26T05:14:26.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.390206+0000", 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0097238089999999999 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.390355+0000", 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014864300000000001 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.506924+0000", 2024-03-26T05:14:26.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.116569326 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.506961+0000", 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6928000000000003e-05 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:56.507005+0000", 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3853000000000002e-05 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:201000 8.9 8:97ede170:::benchmark_data_smithi080_109213_object86483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:57.050509+0000", 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 269.020932641, 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.188192953, 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 201000 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.144 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.050509+0000", 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.050509+0000", 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.050511+0000", 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.703e-06 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.050516+0000", 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9790000000000001e-06 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.050532+0000", 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.615e-05 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.050540+0000", 2024-03-26T05:14:26.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9389999999999997e-06 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.119649+0000", 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.069108881999999996 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.119782+0000", 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000133156 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.119904+0000", 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000121557 2024-03-26T05:14:26.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.132767+0000", 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.012863495000000001 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.136898+0000", 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0041304360000000004 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.238609+0000", 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.101711413 2024-03-26T05:14:26.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.238659+0000", 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9845999999999997e-05 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:57.238702+0000", 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3396999999999998e-05 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.149 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:202429 8.9 8:943a69cc:::benchmark_data_smithi080_109213_object87912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:58.502083+0000", 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 267.56935877299998, 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.193700766, 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 202429 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.502083+0000", 2024-03-26T05:14:26.150 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.502083+0000", 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.502084+0000", 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0049999999999999e-06 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.502091+0000", 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5300000000000002e-06 2024-03-26T05:14:26.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.502104+0000", 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2792999999999999e-05 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.502111+0000", 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5129999999999999e-06 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.517204+0000", 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.015093367 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.517335+0000", 2024-03-26T05:14:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013098799999999999 2024-03-26T05:14:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.517478+0000", 2024-03-26T05:14:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014280599999999999 2024-03-26T05:14:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.520055+0000", 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0025765559999999998 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.520070+0000", 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4970999999999999e-05 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.695655+0000", 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.17558531699999999 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.154 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.695714+0000", 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8881000000000001e-05 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.695784+0000", 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0038999999999998e-05 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:202824 8.9 8:90537ff0:::benchmark_data_smithi080_109213_object88307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:58.982640+0000", 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 267.08880214599998, 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18913682500000001, 2024-03-26T05:14:26.155 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 202824 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.982640+0000", 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.156 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.982640+0000", 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.982641+0000", 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.42e-06 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.982646+0000", 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0250000000000002e-06 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.982658+0000", 2024-03-26T05:14:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2099000000000001e-05 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:58.982666+0000", 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8080000000000005e-06 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.028333+0000", 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.045667253999999997 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.028418+0000", 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4122999999999994e-05 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.028501+0000", 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3112e-05 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.030935+0000", 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002434322 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.031142+0000", 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00020722500000000001 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.171647+0000", 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14050507800000001 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.171705+0000", 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7630999999999998e-05 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.171777+0000", 2024-03-26T05:14:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1728000000000002e-05 2024-03-26T05:14:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:203255 8.9 8:9507306b:::benchmark_data_smithi080_109213_object88738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:14:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:59.393074+0000", 2024-03-26T05:14:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 266.67836820399998, 2024-03-26T05:14:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.21242334900000001, 2024-03-26T05:14:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 203255 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393074+0000", 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393074+0000", 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393075+0000", 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0079999999999999e-06 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393079+0000", 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8909999999999997e-06 2024-03-26T05:14:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393092+0000", 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2927e-05 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393097+0000", 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7459999999999997e-06 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.164 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.475807+0000", 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.082709258999999993 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.475886+0000", 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9282000000000004e-05 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.476011+0000", 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000125425 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.482040+0000", 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0060289230000000003 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.486423+0000", 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043832439999999997 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.605408+0000", 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11898417 2024-03-26T05:14:26.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.605444+0000", 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6127999999999997e-05 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.605497+0000", 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3346000000000001e-05 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.167 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24356.0:3175 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94789128796928 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:12:26.361163+0000", 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 119.71027929500001, 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.43763786900000001, 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24356", 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 3175 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.168 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361163+0000", 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361163+0000", 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361165+0000", 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.9269999999999999e-06 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.169 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361170+0000", 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7490000000000004e-06 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361187+0000", 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6344000000000001e-05 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361198+0000", 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0868e-05 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361267+0000", 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9015000000000002e-05 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361278+0000", 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1096e-05 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.796862+0000", 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.43558413099999999 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.171 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.797030+0000", 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016853500000000001 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.798801+0000", 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001770204 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24356.0:3176 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94789134008320 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:12:26.361223+0000", 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 119.710218985, 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.43790922500000001, 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24356", 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 3176 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361223+0000", 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361223+0000", 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361224+0000", 2024-03-26T05:14:26.174 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2e-07 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361226+0000", 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.7360000000000001e-06 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361234+0000", 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8050000000000007e-06 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361241+0000", 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9949999999999999e-06 2024-03-26T05:14:26.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361297+0000", 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6232e-05 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361306+0000", 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3659999999999992e-06 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.797188+0000", 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.43588267200000003 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.797316+0000", 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012806199999999999 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.799132+0000", 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001815837 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.177 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24356.0:3177 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94789128803840 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:14:26.178 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:12:26.361253+0000", 2024-03-26T05:14:26.178 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 119.71018918999999, 2024-03-26T05:14:26.178 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.44081309600000002, 2024-03-26T05:14:26.178 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:14:26.178 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:14:26.178 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:14:26.178 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24356", 2024-03-26T05:14:26.178 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:14:26.178 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 3177 2024-03-26T05:14:26.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.178 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:14:26.178 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361253+0000", 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361253+0000", 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361253+0000", 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8000000000000001e-07 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361256+0000", 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.3819999999999998e-06 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361263+0000", 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.379e-06 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361268+0000", 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.31e-06 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.180 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361324+0000", 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6106000000000001e-05 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361335+0000", 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0271e-05 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.799570+0000", 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.43823521100000001 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.181 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.799717+0000", 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014688200000000001 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.802066+0000", 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0023491749999999998 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:14:26.182 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:14:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:26.207 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:26.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:26.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:26 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:26.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:26 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:14:26.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:26 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:14:26.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:26 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:14:26.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:26 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1737078208' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:14:26.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:26 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:14:26.308 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:26.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:26.334 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:26.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:26.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:26 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:26.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:26 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:14:26.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:26 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:14:26.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:26 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:14:26.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:26 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1737078208' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:14:26.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:26 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:14:26.336 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:26 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:26.337 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:26 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:14:26.337 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:26 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:14:26.337 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:26 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:14:26.337 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:26 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1737078208' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:14:26.337 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:26 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:14:26.435 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:26.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:26.460 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:26.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:26.561 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:26.571 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'unique_pool_2' removed 2024-03-26T05:14:26.585 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:26.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:26.687 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:26.711 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:26.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:26.812 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:26.838 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:26.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:26.940 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:26.965 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:26.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:26.973 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T05:14:27.059 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:14:27.066 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:27.089 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:27.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:27.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:27.191 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:27.214 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:27.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:27.316 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:27.340 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:27.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:27.441 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:27.464 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:27.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:27.467 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:27.482 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:14:27.505 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:14:27.525 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:14:27.544 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:14:27.564 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:14:27.566 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:27.587 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:14:27.592 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:27.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:27.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:27.608 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:14:27.608 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:27 smithi080 ceph-mon[29691]: pgmap v627: 148 pgs: 87 active+clean, 40 active+undersized, 21 active+undersized+degraded; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 393 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:27.608 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:27 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:14:27.608 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:27 smithi080 ceph-mon[29691]: osdmap e312: 8 total, 7 up, 8 in 2024-03-26T05:14:27.608 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:27 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 91/690 objects degraded (13.188%), 21 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:27.608 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:27 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4037026754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:27.608 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:27 smithi080 ceph-mon[40404]: pgmap v627: 148 pgs: 87 active+clean, 40 active+undersized, 21 active+undersized+degraded; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 393 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:27.609 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:27 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:14:27.609 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:27 smithi080 ceph-mon[40404]: osdmap e312: 8 total, 7 up, 8 in 2024-03-26T05:14:27.609 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:27 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 91/690 objects degraded (13.188%), 21 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:27.609 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:27 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4037026754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:27.609 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:14:27.609 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:14:27.609 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:14:27.609 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:14:27.609 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:14:27.609 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:14:27.609 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:14:27.609 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:14:27.694 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:27.718 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:27.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:27.820 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:27.843 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:27.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:27.945 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:27.966 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T05:14:27.967 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T05:14:27.967 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-26T05:14:27.968 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-26T05:14:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:27.970 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:27.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:28.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:27 smithi132 ceph-mon[33500]: pgmap v627: 148 pgs: 87 active+clean, 40 active+undersized, 21 active+undersized+degraded; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 393 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:28.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:27 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:14:28.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:27 smithi132 ceph-mon[33500]: osdmap e312: 8 total, 7 up, 8 in 2024-03-26T05:14:28.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:27 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 91/690 objects degraded (13.188%), 21 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:28.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:27 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4037026754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:28.071 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:28.095 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:28.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:28.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:28.196 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:28.220 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:28.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:28.322 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:28.345 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:28.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:28.446 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:28.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:28.471 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:28.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:28.571 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:14:28.582 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:14:28.595 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:28.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:28.697 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:28.721 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:28.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:28.822 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:28.827 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:28 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:14:28.827 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:28 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:14:28.827 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:28 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:14:28.828 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:28 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:14:28.828 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:28 smithi080 ceph-mon[40404]: osdmap e313: 8 total, 7 up, 8 in 2024-03-26T05:14:28.828 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:28 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:14:28.828 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:28 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:14:28.828 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:28 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:14:28.828 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:28 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:14:28.828 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:28 smithi080 ceph-mon[29691]: osdmap e313: 8 total, 7 up, 8 in 2024-03-26T05:14:28.850 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:28.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:28.952 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:28.977 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:28.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:28.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:29.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:28 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:14:29.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:28 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:14:29.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:28 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:14:29.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:28 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:14:29.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:28 smithi132 ceph-mon[33500]: osdmap e313: 8 total, 7 up, 8 in 2024-03-26T05:14:29.078 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:29.106 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:29.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:29.208 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:29.232 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:29.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:29.258 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:29.296 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:29.333 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:29.357 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:29.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:29.458 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:29.482 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:29.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:29.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:29.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:29.611 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:29.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:29.635 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-03-26T05:14:29.635 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:14:29.635 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-26T05:14:29.635 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T05:14:29.635 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T05:14:29.636 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-26T05:14:29.659 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:29 smithi080 ceph-mon[29691]: pgmap v630: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 669 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:29.659 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:29 smithi080 ceph-mon[40404]: pgmap v630: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 669 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:29.712 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:29.735 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:29.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:29.837 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:29.860 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:29.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:29.962 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:29.986 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:29.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:30.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:29 smithi132 ceph-mon[33500]: pgmap v630: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 669 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:30.087 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:30.115 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:30.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:30.217 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:30.244 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:30.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:30.345 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:30.370 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:30.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:30.471 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:30.496 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:30.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:30.583 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:14:30.597 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:30.625 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:30.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:30.727 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:30.752 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:30.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:30.853 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:30.881 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:30.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:30.982 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:31.007 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:31.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:31.108 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:31.136 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:31.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:31.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:31.238 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:31.262 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:31.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:31.363 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:31.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:31.388 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:31.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:31.489 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:31.512 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:31.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:31.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:31.614 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:31.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:31.640 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:31.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:31.741 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:31.764 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:31.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:31.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:31.792 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:31 smithi080 ceph-mon[40404]: pgmap v631: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:31.792 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:31 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:14:31.792 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:31 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:14:31.792 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:31 smithi080 ceph-mon[40404]: osdmap e314: 8 total, 7 up, 8 in 2024-03-26T05:14:31.792 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:31 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 91/690 objects degraded (13.188%), 19 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:31.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:31 smithi080 ceph-mon[29691]: pgmap v631: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:31.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:31 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:14:31.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:31 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:14:31.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:31 smithi080 ceph-mon[29691]: osdmap e314: 8 total, 7 up, 8 in 2024-03-26T05:14:31.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:31 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 91/690 objects degraded (13.188%), 19 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:31.866 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:31.891 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:31.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:31.922 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:14:31.922 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:14:31.937 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:31.992 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:32.020 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:32.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:32.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:32.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:31 smithi132 ceph-mon[33500]: pgmap v631: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:32.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:31 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:14:32.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:31 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:14:32.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:31 smithi132 ceph-mon[33500]: osdmap e314: 8 total, 7 up, 8 in 2024-03-26T05:14:32.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:31 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 91/690 objects degraded (13.188%), 19 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:32.121 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:32.159 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:32.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:32.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:32.262 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:32.286 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:32.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:32.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:32.388 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:32.415 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:32.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:32.517 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:32.541 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:32.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:32.643 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:32.667 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:32.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:32.769 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:32.792 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:32.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:32.894 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:32.918 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:32.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:33.019 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:33.044 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:33.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:33.098 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T05:14:33.145 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:33.167 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:33.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:33.269 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:33.294 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:33.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:33.396 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:33.424 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:33.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:33.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:33.526 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:33.549 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:33.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:33.610 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:14:33.651 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:33.675 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:33.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:33.776 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:33.804 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:33.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:33.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:33 smithi080 ceph-mon[29691]: pgmap v633: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:33.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:33 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2454096025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:33.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:33 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:33.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:33 smithi080 ceph-mon[40404]: pgmap v633: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:33.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:33 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2454096025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:33.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:33 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:33.906 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:33.932 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:33.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:34.034 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:34.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:33 smithi132 ceph-mon[33500]: pgmap v633: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:34.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:33 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2454096025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:34.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:33 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:34.058 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:34.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:34.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:34.183 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:34.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:34.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:34.284 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:34.307 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:34.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:34.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:34.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:14:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:14:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:14:34.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:34.433 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:34.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:34.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:34.562 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:34.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:34.664 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:34.689 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:34.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:34.791 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:34.815 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:34.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:34.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:34 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:14:34.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:34 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:14:34.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:34 smithi080 ceph-mon[29691]: osdmap e315: 8 total, 7 up, 8 in 2024-03-26T05:14:34.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:34 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:14:34.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:34 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:14:34.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:34 smithi080 ceph-mon[40404]: osdmap e315: 8 total, 7 up, 8 in 2024-03-26T05:14:34.917 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:34.942 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:34.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:35.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:34 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:14:35.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:34 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:14:35.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:34 smithi132 ceph-mon[33500]: osdmap e315: 8 total, 7 up, 8 in 2024-03-26T05:14:35.045 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:35.070 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:35.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:35.137 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:35.171 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:35.196 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:35.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:35.298 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:35.323 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:35.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:35.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:35.425 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:35.449 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:35.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:35.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:35.577 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:35.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:35.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:35.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:35.702 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:35.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:35.804 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:35.828 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:35.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:35.829 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:35 smithi080 ceph-mon[40404]: pgmap v635: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 794 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:35.829 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:14:35.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:14:35.830 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:35 smithi080 ceph-mon[29691]: pgmap v635: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 794 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:35.830 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:14:35.830 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:14:35.929 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:35.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:35 smithi132 ceph-mon[33500]: pgmap v635: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 794 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:35.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:14:35.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:14:35.943 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:14:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:14:35.953 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:35.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:35.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:36.055 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:36.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:36.079 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:36.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:36.180 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:36.203 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:36.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:36.305 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:36.328 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:36.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:36.430 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:36.454 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:36.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:36.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:36.556 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:36.579 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:36.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:36.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:36.680 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:36.703 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:36.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:36.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:36.805 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:36.828 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:36.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:36.899 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:36.899 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:36.930 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:36.954 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:36.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:36.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:37.056 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:37.080 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:37.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:37.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:37.140 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:14:37.141 INFO:teuthology.orchestra.run.smithi080.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T05:14:37.141 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:14:37.182 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:37.206 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:37.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:37.307 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:37.331 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:37.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:37.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:37.433 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:37.458 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:37.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:37.480 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:14:37.559 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:37.584 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:37.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:37.685 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:37.708 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:37.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:37.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:37.798 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:37 smithi080 ceph-mon[29691]: pgmap v636: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:37.798 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:37 smithi080 ceph-mon[40404]: pgmap v636: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:37.810 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:37.833 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:37.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:37.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:37.846 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:14:37.865 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:14:37.884 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:14:37.904 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:14:37.924 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:14:37.935 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:37.943 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:14:37.959 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:37.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:37.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:37.964 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:14:37.965 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:14:37.965 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:14:37.965 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:14:37.965 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:14:37.965 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:14:37.965 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:14:37.965 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:14:37.965 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:14:38.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:37 smithi132 ceph-mon[33500]: pgmap v636: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:38.061 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:38.086 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:38.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:38.186 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:38.210 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:38.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:38.312 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:38.335 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:38.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:38.436 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:38.460 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:38.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:38.562 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:38.586 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:38.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:38.688 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:38.711 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:38.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:38.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:38.798 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:38 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/333309273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:38.799 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:38 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/333309273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:38.813 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:14:38.823 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:14:38.841 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:38.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:38.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:38.860 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:38.943 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:38.971 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:38.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:38.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:39.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:38 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/333309273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:39.073 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:39.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:39.102 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:39.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:39.202 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:39.229 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:39.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:39.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:39.331 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:39.358 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:39.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:39.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:39.459 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:39.486 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:39.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:39.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:39.588 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:39.615 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:39.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:39.717 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:39.743 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:39.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:39.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:39.845 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:39.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:39.874 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:39.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:39.962 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:39 smithi080 ceph-mon[40404]: pgmap v637: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:39.962 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:39 smithi080 ceph-mon[29691]: pgmap v637: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:39.974 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:40.004 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:40.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:40.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:39 smithi132 ceph-mon[33500]: pgmap v637: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:40.105 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:40.132 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:40.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:40.233 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:40.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:40.258 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:40.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:40.359 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:40.378 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:40.385 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:40.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:40.486 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:40.511 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:40.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:40.614 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:40.638 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:40.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:40.650 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'unique_pool_3' created 2024-03-26T05:14:40.740 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:40 smithi080 ceph-mon[40404]: pgmap v638: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:40.740 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:40 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/824759106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T05:14:40.740 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:40 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T05:14:40.740 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:40 smithi080 ceph-mon[29691]: pgmap v638: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:40.740 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:40 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/824759106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T05:14:40.740 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:40 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T05:14:40.741 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:40.764 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:40.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:40.824 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:14:40.866 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:40.889 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:40.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:40.991 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:41.016 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:41.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:41.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:41.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:40 smithi132 ceph-mon[33500]: pgmap v638: 132 pgs: 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:41.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:40 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/824759106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T05:14:41.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:40 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T05:14:41.117 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:41.140 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:41.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:41.242 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:41.270 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:41.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:41.372 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:41.396 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:41.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:41.498 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:41.523 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:41.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:41.547 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-26T05:14:41.625 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:41.659 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:41.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:41.760 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:41.787 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:41.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:41.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:41.888 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:41.914 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:41.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:41.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:41.937 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:41 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-26T05:14:41.937 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:41 smithi080 ceph-mon[40404]: osdmap e316: 8 total, 7 up, 8 in 2024-03-26T05:14:41.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:41 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-26T05:14:41.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:41 smithi080 ceph-mon[29691]: osdmap e316: 8 total, 7 up, 8 in 2024-03-26T05:14:42.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:42.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:42.041 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:42.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:42.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:41 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-26T05:14:42.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:41 smithi132 ceph-mon[33500]: osdmap e316: 8 total, 7 up, 8 in 2024-03-26T05:14:42.143 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:42.166 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:42.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:42.268 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:42.296 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:42.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:42.398 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:42.423 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:42.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:42.524 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:42.548 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:42.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:42.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:42.649 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:42.678 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:42.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:42.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:42.780 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:42.807 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:42.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:42.905 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:42 smithi080 ceph-mon[29691]: osdmap e317: 8 total, 7 up, 8 in 2024-03-26T05:14:42.906 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:42 smithi080 ceph-mon[29691]: pgmap v641: 148 pgs: 3 creating+peering, 13 unknown, 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:42.906 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:42 smithi080 ceph-mon[40404]: osdmap e317: 8 total, 7 up, 8 in 2024-03-26T05:14:42.906 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:42 smithi080 ceph-mon[40404]: pgmap v641: 148 pgs: 3 creating+peering, 13 unknown, 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:42.908 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:42.934 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:42.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:43.036 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:43.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:42 smithi132 ceph-mon[33500]: osdmap e317: 8 total, 7 up, 8 in 2024-03-26T05:14:43.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:42 smithi132 ceph-mon[33500]: pgmap v641: 148 pgs: 3 creating+peering, 13 unknown, 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:43.062 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:43.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:43.163 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:43.187 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:14:43.188 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:14:43.188 INFO:teuthology.orchestra.run.smithi080.stdout: "complaint_time": 30, 2024-03-26T05:14:43.188 INFO:teuthology.orchestra.run.smithi080.stdout: "num_blocked_ops": 0 2024-03-26T05:14:43.188 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:14:43.190 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:43.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:43.220 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-03-26T05:14:43.221 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:14:43.291 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:43.318 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:43.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:43.420 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:43.444 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:43.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:43.546 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:43.573 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:43.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:43.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:43.674 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:43.700 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:43.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:43.802 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:43.825 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:43.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:43.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:43.847 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:43.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:43 smithi080 ceph-mon[40404]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:14:43.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:43 smithi080 ceph-mon[40404]: osdmap e318: 8 total, 7 up, 8 in 2024-03-26T05:14:43.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:43 smithi080 ceph-mon[29691]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:14:43.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:43 smithi080 ceph-mon[29691]: osdmap e318: 8 total, 7 up, 8 in 2024-03-26T05:14:43.928 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:43.953 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:43.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:44.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:43 smithi132 ceph-mon[33500]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:14:44.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:43 smithi132 ceph-mon[33500]: osdmap e318: 8 total, 7 up, 8 in 2024-03-26T05:14:44.054 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:44.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:44.079 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:44.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:44.180 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:44.204 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:44.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:44.306 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:44.318 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:14:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:14:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:14:44.334 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:44.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:44.344 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T05:14:44.435 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:44.463 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:44.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:44.565 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:44.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:44.594 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:44.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:44.695 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:44.721 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:44.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:44.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:44.823 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:44.851 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:44.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:44.853 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:44.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:44 smithi080 ceph-mon[40404]: pgmap v643: 148 pgs: 3 creating+peering, 13 unknown, 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:44.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:44 smithi080 ceph-mon[29691]: pgmap v643: 148 pgs: 3 creating+peering, 13 unknown, 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:44.952 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:44.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:44.977 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:44.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:45.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:44 smithi132 ceph-mon[33500]: pgmap v643: 148 pgs: 3 creating+peering, 13 unknown, 80 active+clean, 33 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:45.078 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:45.106 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:45.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:45.208 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:45.233 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:45.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:45.335 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:45.359 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:45.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:45.460 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:45.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:45.484 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:45.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:45.516 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:45.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:45.610 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:45.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:45.676 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:14:45.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:45.739 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:45.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:45.841 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:45.866 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:45.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:45.966 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:45.995 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:45.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:46.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:45 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2345453186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:14:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:45 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:46.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:14:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:14:46.082 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:45 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2345453186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:14:46.082 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:45 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:46.082 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:45 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2345453186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:14:46.082 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:45 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:46.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:46.122 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:46.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:46.223 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:46.246 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:46.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:46.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:46.372 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:46.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:46.473 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:46.497 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:46.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:46.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:46.599 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:46.626 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:46.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:46.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:46.728 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:46.756 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:46.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:46.857 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:46.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:14:46.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2345453186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:14:46.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[40404]: osdmap e319: 8 total, 7 up, 8 in 2024-03-26T05:14:46.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[40404]: 4.4 scrub starts 2024-03-26T05:14:46.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[40404]: 4.4 scrub ok 2024-03-26T05:14:46.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[40404]: 6.10 scrub starts 2024-03-26T05:14:46.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[40404]: 6.10 scrub ok 2024-03-26T05:14:46.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[40404]: pgmap v645: 148 pgs: 2 creating+peering, 91 active+clean, 36 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:46.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:14:46.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2345453186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:14:46.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[29691]: osdmap e319: 8 total, 7 up, 8 in 2024-03-26T05:14:46.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[29691]: 4.4 scrub starts 2024-03-26T05:14:46.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[29691]: 4.4 scrub ok 2024-03-26T05:14:46.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[29691]: 6.10 scrub starts 2024-03-26T05:14:46.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[29691]: 6.10 scrub ok 2024-03-26T05:14:46.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:46 smithi080 ceph-mon[29691]: pgmap v645: 148 pgs: 2 creating+peering, 91 active+clean, 36 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:46.881 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:46.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:46.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:46.983 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:46 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:14:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:46 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2345453186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:14:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:46 smithi132 ceph-mon[33500]: osdmap e319: 8 total, 7 up, 8 in 2024-03-26T05:14:47.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:46 smithi132 ceph-mon[33500]: 4.4 scrub starts 2024-03-26T05:14:47.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:46 smithi132 ceph-mon[33500]: 4.4 scrub ok 2024-03-26T05:14:47.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:46 smithi132 ceph-mon[33500]: 6.10 scrub starts 2024-03-26T05:14:47.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:46 smithi132 ceph-mon[33500]: 6.10 scrub ok 2024-03-26T05:14:47.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:46 smithi132 ceph-mon[33500]: pgmap v645: 148 pgs: 2 creating+peering, 91 active+clean, 36 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:47.047 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:47.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:47.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:47.149 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:47.177 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:47.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:47.278 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:47.302 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:47.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:47.404 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:47.431 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:47.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:47.533 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:47.561 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:47.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:47.663 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:47.689 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:47.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:47.790 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:47.816 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:47.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:47.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:47.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:47 smithi080 ceph-mon[29691]: 3.4 scrub starts 2024-03-26T05:14:47.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:47 smithi080 ceph-mon[29691]: 3.4 scrub ok 2024-03-26T05:14:47.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:47 smithi080 ceph-mon[40404]: 3.4 scrub starts 2024-03-26T05:14:47.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:47 smithi080 ceph-mon[40404]: 3.4 scrub ok 2024-03-26T05:14:47.918 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:47.944 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:47.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:48.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:47 smithi132 ceph-mon[33500]: 3.4 scrub starts 2024-03-26T05:14:48.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:47 smithi132 ceph-mon[33500]: 3.4 scrub ok 2024-03-26T05:14:48.045 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:48.070 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:48.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:48.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:48.172 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:48.196 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:48.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:48.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:48.322 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:48.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:48.424 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:48.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:48.448 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:48.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:48.549 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:48.573 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:48.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:48.675 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:48.700 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:48.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:48.802 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:48.826 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:48.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:48.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:48.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:48 smithi080 ceph-mon[29691]: 2.2 scrub starts 2024-03-26T05:14:48.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:48 smithi080 ceph-mon[29691]: 2.2 scrub ok 2024-03-26T05:14:48.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:48 smithi080 ceph-mon[29691]: pgmap v646: 148 pgs: 92 active+clean, 37 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 803 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:48.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:48 smithi080 ceph-mon[40404]: 2.2 scrub starts 2024-03-26T05:14:48.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:48 smithi080 ceph-mon[40404]: 2.2 scrub ok 2024-03-26T05:14:48.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:48 smithi080 ceph-mon[40404]: pgmap v646: 148 pgs: 92 active+clean, 37 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 803 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:48.928 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:48.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:48.986 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:14:49.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:48 smithi132 ceph-mon[33500]: 2.2 scrub starts 2024-03-26T05:14:49.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:48 smithi132 ceph-mon[33500]: 2.2 scrub ok 2024-03-26T05:14:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:48 smithi132 ceph-mon[33500]: pgmap v646: 148 pgs: 92 active+clean, 37 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 803 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:49.054 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:49.078 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:49.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:49.179 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:49.204 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:49.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:49.219 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:14:49.219 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:14:49.219 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:14:49.219 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:14:49.219 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.219 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24356.0:1902 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94789128791168 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T05:14:49.219 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:05:35.871899+0000", 2024-03-26T05:14:49.219 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 553.35479029299995, 2024-03-26T05:14:49.219 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.55856863400000001, 2024-03-26T05:14:49.219 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.219 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:14:49.219 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24356", 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 1902 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.871899+0000", 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.871899+0000", 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.871901+0000", 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4529999999999999e-06 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.871904+0000", 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2329999999999998e-06 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.871914+0000", 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0441e-05 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.221 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.871921+0000", 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5139999999999998e-06 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872073+0000", 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015249 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op must wait for map", 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.872079+0000", 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9750000000000004e-06 2024-03-26T05:14:49.222 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.882805+0000", 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010726051 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:35.882814+0000", 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6090000000000001e-06 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.423086+0000", 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.54027256800000001 2024-03-26T05:14:49.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for degraded object", 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.423362+0000", 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00027567500000000002 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.428581+0000", 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0052186919999999996 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.428689+0000", 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010869 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.224 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:05:36.430468+0000", 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0017782430000000001 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:138926 8.8 8:10c99b2e:::benchmark_data_smithi080_109213_object24409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:50.225267+0000", 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 359.00142234200001, 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.147903376, 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.225 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 138926 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225267+0000", 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225267+0000", 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225269+0000", 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.762e-06 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225275+0000", 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1070000000000001e-06 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225288+0000", 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3018e-05 2024-03-26T05:14:49.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225297+0000", 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8890000000000008e-06 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225467+0000", 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000170371 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225602+0000", 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013448600000000001 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.228 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225739+0000", 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000137145 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.232793+0000", 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0070541029999999999 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.239818+0000", 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0070250080000000001 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.229 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.239829+0000", 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0876000000000001e-05 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.372463+0000", 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13263451000000001 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.372880+0000", 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000416921 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.231 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.231 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.373170+0000", 2024-03-26T05:14:49.231 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00029018000000000002 2024-03-26T05:14:49.231 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.231 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.231 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.231 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.231 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:138927 8.8 8:18e36ad7:::benchmark_data_smithi080_109213_object24410:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:14:49.231 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:50.225318+0000", 2024-03-26T05:14:49.231 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 359.00137098499999, 2024-03-26T05:14:49.232 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.151279733, 2024-03-26T05:14:49.232 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.232 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.232 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.232 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:14:49.232 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:49.232 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 138927 2024-03-26T05:14:49.232 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.232 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225318+0000", 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225318+0000", 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225319+0000", 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2100000000000002e-07 2024-03-26T05:14:49.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225321+0000", 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6029999999999999e-06 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225328+0000", 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4949999999999996e-06 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225333+0000", 2024-03-26T05:14:49.234 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4410000000000001e-06 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.229159+0000", 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0038261189999999998 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.229294+0000", 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013497800000000001 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.229441+0000", 2024-03-26T05:14:49.235 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000146479 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.232830+0000", 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.003389211 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.239893+0000", 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0070629619999999999 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.239901+0000", 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8909999999999995e-06 2024-03-26T05:14:49.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.376390+0000", 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.136489262 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.376480+0000", 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0116000000000003e-05 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.376598+0000", 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117755 2024-03-26T05:14:49.237 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:138931 8.8 8:13f499e4:::benchmark_data_smithi080_109213_object24414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:50.225340+0000", 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 359.00134919700002, 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.15177058800000001, 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:49.238 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 138931 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225340+0000", 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225340+0000", 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225340+0000", 2024-03-26T05:14:49.239 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.9299999999999999e-07 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225343+0000", 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5809999999999999e-06 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225349+0000", 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.755e-06 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.240 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.225354+0000", 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2549999999999997e-06 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.233157+0000", 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0078024949999999996 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.233287+0000", 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000130856 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.233413+0000", 2024-03-26T05:14:49.241 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012534699999999999 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.239860+0000", 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0064472699999999997 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.239905+0000", 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4978000000000002e-05 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.242 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.239927+0000", 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1838999999999999e-05 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.376637+0000", 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13671028900000001 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.376996+0000", 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000359348 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.377111+0000", 2024-03-26T05:14:49.243 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000114282 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:139288 8.c 8:37374267:::benchmark_data_smithi080_109213_object24771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:50.694422+0000", 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 358.53226713200002, 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14953480399999999, 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 139288 2024-03-26T05:14:49.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.694422+0000", 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.694422+0000", 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.694424+0000", 2024-03-26T05:14:49.245 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6109999999999999e-06 2024-03-26T05:14:49.246 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.246 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.246 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.246 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.694430+0000", 2024-03-26T05:14:49.246 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7069999999999997e-06 2024-03-26T05:14:49.246 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.246 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.246 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.246 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.694443+0000", 2024-03-26T05:14:49.246 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3506000000000001e-05 2024-03-26T05:14:49.246 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.246 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.694452+0000", 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8939999999999999e-06 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.700509+0000", 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.006056751 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.700612+0000", 2024-03-26T05:14:49.247 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103397 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.700719+0000", 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000106637 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.843783+0000", 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.143064104 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.248 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.843820+0000", 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7523e-05 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.843866+0000", 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6026999999999998e-05 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.843912+0000", 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5398999999999997e-05 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.249 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.843957+0000", 2024-03-26T05:14:49.250 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5247999999999998e-05 2024-03-26T05:14:49.250 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.250 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.250 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.250 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.250 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.250 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:139290 8.c 8:3b446d1e:::benchmark_data_smithi080_109213_object24773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:14:49.250 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:50.694988+0000", 2024-03-26T05:14:49.250 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 358.53170109899997, 2024-03-26T05:14:49.250 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.161270513, 2024-03-26T05:14:49.250 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.250 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 139290 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.694988+0000", 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.694988+0000", 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.251 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.694989+0000", 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.043e-06 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.694994+0000", 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5669999999999996e-06 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.695005+0000", 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1409e-05 2024-03-26T05:14:49.252 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.695014+0000", 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8510000000000005e-06 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.704501+0000", 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0094869670000000007 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.836718+0000", 2024-03-26T05:14:49.253 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13221691899999999 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.836864+0000", 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000146422 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.845561+0000", 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0086966030000000007 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.254 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.845620+0000", 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8953000000000003e-05 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.856165+0000", 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010544974 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.856219+0000", 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3606000000000001e-05 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.856259+0000", 2024-03-26T05:14:49.255 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0198999999999999e-05 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:139292 8.c 8:3895b146:::benchmark_data_smithi080_109213_object24775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:50.696336+0000", 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 358.53035339500002, 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.16005011799999999, 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.256 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 139292 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.696336+0000", 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.696336+0000", 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.257 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.696337+0000", 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0899999999999999e-06 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.696341+0000", 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9210000000000002e-06 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.258 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.696352+0000", 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1062e-05 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.696359+0000", 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3939999999999998e-06 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.839811+0000", 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14345170900000001 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.259 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.840851+0000", 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010394289999999999 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,7", 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.840963+0000", 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000112273 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.845609+0000", 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0046462939999999996 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.260 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.845625+0000", 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5588000000000001e-05 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.856305+0000", 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010679892999999999 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.856351+0000", 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6585000000000002e-05 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.261 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.856386+0000", 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4879999999999998e-05 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:139297 8.8 8:1b7a481e:::benchmark_data_smithi080_109213_object24780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:50.697472+0000", 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 358.52921773600002, 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14697400799999999, 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.262 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 139297 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.697472+0000", 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.263 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.697472+0000", 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.697472+0000", 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.44e-07 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.697476+0000", 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3019999999999999e-06 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.264 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.697484+0000", 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5909999999999996e-06 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.697490+0000", 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.297e-06 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.700328+0000", 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0028371350000000002 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.265 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.700447+0000", 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118977 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.700578+0000", 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013121500000000001 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.839885+0000", 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13930693599999999 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.266 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.839905+0000", 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0540999999999999e-05 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.840532+0000", 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00062703300000000004 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.844267+0000", 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00373428 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.267 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.844330+0000", 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3480999999999997e-05 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:50.844446+0000", 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011557600000000001 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:162664 8.e 8:70bb5d55:::benchmark_data_smithi080_109213_object48147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:09:18.429689+0000", 2024-03-26T05:14:49.268 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 330.797000481, 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.15164423599999999, 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 162664 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.429689+0000", 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.269 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.429689+0000", 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.429690+0000", 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3799999999999996e-07 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.429693+0000", 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7790000000000002e-06 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.270 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.271 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.429702+0000", 2024-03-26T05:14:49.271 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7649999999999999e-06 2024-03-26T05:14:49.271 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.271 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.271 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.271 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.429709+0000", 2024-03-26T05:14:49.271 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.781e-06 2024-03-26T05:14:49.271 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.271 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.271 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.271 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.430222+0000", 2024-03-26T05:14:49.272 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000512759 2024-03-26T05:14:49.272 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.272 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.430335+0000", 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000112824 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,4", 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.430448+0000", 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000113521 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.435485+0000", 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0050366789999999996 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.273 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.567201+0000", 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.131716693 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.581145+0000", 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.013944066999999999 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.581290+0000", 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014451000000000001 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.274 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.581333+0000", 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3120000000000001e-05 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:162665 8.e 8:748d884b:::benchmark_data_smithi080_109213_object48148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:09:18.429723+0000", 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 330.79696621900001, 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.15474381100000001, 2024-03-26T05:14:49.275 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 162665 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.429723+0000", 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.276 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.429723+0000", 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.429724+0000", 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9499999999999998e-07 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.429726+0000", 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1519999999999999e-06 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.277 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.429732+0000", 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0279999999999997e-06 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.429736+0000", 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7330000000000003e-06 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.433190+0000", 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0034530899999999998 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.278 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.433283+0000", 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.3528000000000003e-05 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,4", 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.433355+0000", 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1707999999999999e-05 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.435620+0000", 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022651120000000001 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.279 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.567236+0000", 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.131615865 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.584247+0000", 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.017011745000000002 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.584423+0000", 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017589299999999999 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:09:18.584467+0000", 2024-03-26T05:14:49.280 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3562000000000001e-05 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:110002 9.9 9:9e933090:::benchmark_data_smithi080_136801_object110001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.832023+0000", 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 143.39466654700001, 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98611885300000002, 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 110002 2024-03-26T05:14:49.281 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.832023+0000", 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.832023+0000", 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.282 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.832025+0000", 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1830000000000001e-06 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.832053+0000", 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.845e-05 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.832076+0000", 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2770000000000001e-05 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.283 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.832086+0000", 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9660000000000008e-06 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.832149+0000", 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2835000000000004e-05 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.832157+0000", 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0099999999999995e-06 2024-03-26T05:14:49.284 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.812197+0000", 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98004036500000002 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.812354+0000", 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015643499999999999 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5", 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.812541+0000", 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000187646 2024-03-26T05:14:49.285 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817914+0000", 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0053730369999999998 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817967+0000", 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2089999999999998e-05 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.818039+0000", 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2319999999999999e-05 2024-03-26T05:14:49.286 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.818142+0000", 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000102746 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:110014 9.5 9:a0066748:::benchmark_data_smithi080_136801_object110013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.884246+0000", 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 143.342443175, 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93625166500000001, 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.287 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 110014 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.884246+0000", 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.884246+0000", 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.288 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.884248+0000", 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.37e-06 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.884270+0000", 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2484999999999999e-05 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.884284+0000", 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3592e-05 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.884294+0000", 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9320000000000008e-06 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.884357+0000", 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3256000000000006e-05 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.884366+0000", 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1919999999999994e-06 2024-03-26T05:14:49.290 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.813160+0000", 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.92879444200000005 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.814758+0000", 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001597125 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5", 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.816090+0000", 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001332012 2024-03-26T05:14:49.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820132+0000", 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0040425360000000002 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820168+0000", 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6208999999999998e-05 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820369+0000", 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020104399999999999 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.292 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820498+0000", 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012846999999999999 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:219043 9.5 9:a5171b29:::benchmark_data_smithi080_136801_object97460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:13:56.859479+0000", 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 52.367210757999999, 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.163237677, 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.293 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 219043 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.859479+0000", 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.859479+0000", 2024-03-26T05:14:49.294 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.859480+0000", 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.44e-06 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.859485+0000", 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9189999999999998e-06 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.859497+0000", 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2276000000000001e-05 2024-03-26T05:14:49.295 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.859505+0000", 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2160000000000006e-06 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.861752+0000", 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002246362 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.861872+0000", 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012008200000000001 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.296 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5", 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.861994+0000", 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012219600000000001 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.866037+0000", 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0040432580000000001 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.022564+0000", 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.15652623199999999 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.297 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.022620+0000", 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6647000000000002e-05 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.022716+0000", 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6049000000000001e-05 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.298 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.25268.0:219046 9.8 e308/265 9:1c8c3937:::benchmark_data_smithi080_136801_object97463:head v 308'13420, mlcod=308'13418)", 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:13:56.864362+0000", 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 52.362327759999999, 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.165359652, 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.864362+0000", 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.864362+0000", 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.299 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.864363+0000", 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0529999999999999e-06 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.864371+0000", 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6659999999999997e-06 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.864386+0000", 2024-03-26T05:14:49.300 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4415e-05 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.864394+0000", 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0679999999999997e-06 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.866115+0000", 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0017214929999999999 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.866136+0000", 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0676999999999999e-05 2024-03-26T05:14:49.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.029685+0000", 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.163548994 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.029721+0000", 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6285999999999998e-05 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.302 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:219056 9.b 9:d29ef351:::benchmark_data_smithi080_136801_object97473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:13:56.868177+0000", 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 52.358512269999999, 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.165842451, 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 219056 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868177+0000", 2024-03-26T05:14:49.303 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868177+0000", 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868180+0000", 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4830000000000002e-06 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868186+0000", 2024-03-26T05:14:49.304 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3799999999999999e-06 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868202+0000", 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5692000000000002e-05 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868211+0000", 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9090000000000006e-06 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868278+0000", 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7206999999999995e-05 2024-03-26T05:14:49.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868400+0000", 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000121857 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.022656+0000", 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.154256541 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.029251+0000", 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0065944610000000002 2024-03-26T05:14:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.029444+0000", 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019356899999999999 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.033722+0000", 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0042778030000000002 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.033909+0000", 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000186548 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.034020+0000", 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000111001 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.25268.0:219055 9.8 e308/265 9:1098c907:::benchmark_data_smithi080_136801_object97472:head v 308'13421, mlcod=308'13418)", 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:13:56.868433+0000", 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 52.358256163999997, 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.16132057799999999, 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868433+0000", 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868433+0000", 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868434+0000", 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2600000000000001e-07 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868440+0000", 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4579999999999998e-06 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868449+0000", 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8200000000000003e-06 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.868456+0000", 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5849999999999996e-06 2024-03-26T05:14:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.022702+0000", 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.154246156 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.022727+0000", 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5435999999999998e-05 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.029726+0000", 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0069981039999999998 2024-03-26T05:14:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.029754+0000", 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.8192999999999999e-05 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:219061 9.9 9:99165bd0:::benchmark_data_smithi080_136801_object97478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:13:56.873969+0000", 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 52.352720437000002, 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.162125935, 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 219061 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.873969+0000", 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.873969+0000", 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.873971+0000", 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1909999999999999e-06 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:56.873979+0000", 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7160000000000003e-06 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.022551+0000", 2024-03-26T05:14:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.148572069 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.022567+0000", 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6066000000000002e-05 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.022626+0000", 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8724999999999998e-05 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.022743+0000", 2024-03-26T05:14:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117081 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5", 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.022861+0000", 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011859800000000001 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.032411+0000", 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0095498379999999997 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.035774+0000", 2024-03-26T05:14:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.003362922 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.035969+0000", 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019502500000000001 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:13:57.036095+0000", 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000125704 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.317 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.24356.0:3412 5.3 5:c609908c:::notify.5:head [watch ping cookie 94789128791168 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e311)", 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:14:23.580022+0000", 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 25.646667048000001, 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00045203600000000001, 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.24356", 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 3412 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:23.580022+0000", 2024-03-26T05:14:49.318 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:23.580022+0000", 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:23.580025+0000", 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3949999999999999e-06 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:23.580036+0000", 2024-03-26T05:14:49.319 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1744e-05 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:23.580063+0000", 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6786000000000001e-05 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:23.580079+0000", 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5290000000000001e-05 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:23.580213+0000", 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013479200000000001 2024-03-26T05:14:49.320 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:23.580302+0000", 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8733999999999994e-05 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:23.580474+0000", 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017229499999999999 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "rep_scrubmap(6.10 e213 from shard 1)", 2024-03-26T05:14:49.321 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:14:46.009338+0000", 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.2173510630000002, 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014964758, 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.009338+0000", 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.009338+0000", 2024-03-26T05:14:49.322 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.009339+0000", 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6300000000000004e-07 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.009343+0000", 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0269999999999997e-06 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.009353+0000", 2024-03-26T05:14:49.323 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.004e-05 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.009361+0000", 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0809999999999994e-06 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.024285+0000", 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014923557 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.024293+0000", 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0490000000000004e-06 2024-03-26T05:14:49.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.024303+0000", 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0241000000000001e-05 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "rep_scrubmap(6.10 e213 from shard 5)", 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:14:46.009499+0000", 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 3.2171908380000001, 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014864469999999999, 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:14:49.325 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.009499+0000", 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.009499+0000", 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.009499+0000", 2024-03-26T05:14:49.326 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0999999999999998e-07 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.009503+0000", 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.861e-06 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.009514+0000", 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0858e-05 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.009520+0000", 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4139999999999999e-06 2024-03-26T05:14:49.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.024331+0000", 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014810996999999999 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.024338+0000", 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8970000000000002e-06 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:14:46.024363+0000", 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4732999999999999e-05 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.328 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.329 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:14:49.329 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:14:49.329 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:14:49.329 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:49.357 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:49.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:49.459 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:49.483 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:49.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:49.585 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:49.611 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:49.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:49.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:49.713 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:49.739 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:49.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:49.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:49.787 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:49 smithi080 ceph-mon[29691]: 5.3 scrub starts 2024-03-26T05:14:49.787 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:49 smithi080 ceph-mon[29691]: 5.3 scrub ok 2024-03-26T05:14:49.787 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:49 smithi080 ceph-mon[40404]: 5.3 scrub starts 2024-03-26T05:14:49.787 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:49 smithi080 ceph-mon[40404]: 5.3 scrub ok 2024-03-26T05:14:49.841 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:49.864 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:49.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:49.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:49.966 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:49.989 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:49.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:49.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:50.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:49 smithi132 ceph-mon[33500]: 5.3 scrub starts 2024-03-26T05:14:50.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:49 smithi132 ceph-mon[33500]: 5.3 scrub ok 2024-03-26T05:14:50.090 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:50.119 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:50.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:50.220 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:50.249 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:50.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:50.350 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:50.375 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:50.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:50.422 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T05:14:50.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:50.505 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:50.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:50.607 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:50.634 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:50.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:50.695 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-26T05:14:50.736 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:50.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:50.761 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:50.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:50.862 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:50.887 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:50.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:50.988 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:51.015 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:51.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:50 smithi132 ceph-mon[33500]: 3.19 scrub starts 2024-03-26T05:14:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:50 smithi132 ceph-mon[33500]: 3.19 scrub ok 2024-03-26T05:14:51.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:50 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3602313521' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:14:51.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:50 smithi132 ceph-mon[33500]: pgmap v647: 148 pgs: 92 active+clean, 37 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:51.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:14:51.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:14:51.094 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:50 smithi080 ceph-mon[40404]: 3.19 scrub starts 2024-03-26T05:14:51.094 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:50 smithi080 ceph-mon[40404]: 3.19 scrub ok 2024-03-26T05:14:51.094 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:50 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3602313521' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:14:51.095 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:50 smithi080 ceph-mon[40404]: pgmap v647: 148 pgs: 92 active+clean, 37 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:51.095 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:14:51.095 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:14:51.095 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:50 smithi080 ceph-mon[29691]: 3.19 scrub starts 2024-03-26T05:14:51.095 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:50 smithi080 ceph-mon[29691]: 3.19 scrub ok 2024-03-26T05:14:51.095 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:50 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3602313521' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:14:51.095 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:50 smithi080 ceph-mon[29691]: pgmap v647: 148 pgs: 92 active+clean, 37 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:51.095 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:14:51.095 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:14:51.117 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:51.143 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:51.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:51.245 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:51.271 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:51.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:51.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:51.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:51.399 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:51.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:51.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:51.501 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:51.527 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:51.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:51.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:51.629 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:51.653 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:51.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:51.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:14:51.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:14:51.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3602313521' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:14:51.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[29691]: osdmap e320: 8 total, 7 up, 8 in 2024-03-26T05:14:51.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[29691]: 6.b scrub starts 2024-03-26T05:14:51.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[29691]: 4.d scrub starts 2024-03-26T05:14:51.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[29691]: 4.d scrub ok 2024-03-26T05:14:51.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[29691]: 6.b scrub ok 2024-03-26T05:14:51.755 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:51.782 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:51.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:51.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:51.884 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:51.911 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:51.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:51.930 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:51.992 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:14:51.992 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:14:51.993 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:14:51.993 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3602313521' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:14:51.993 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[40404]: osdmap e320: 8 total, 7 up, 8 in 2024-03-26T05:14:51.993 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[40404]: 6.b scrub starts 2024-03-26T05:14:51.993 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[40404]: 4.d scrub starts 2024-03-26T05:14:51.993 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[40404]: 4.d scrub ok 2024-03-26T05:14:51.993 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:51 smithi080 ceph-mon[40404]: 6.b scrub ok 2024-03-26T05:14:52.011 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:14:52.013 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:52.031 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:14:52.038 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:52.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:52.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:51 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:14:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:51 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:14:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:51 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3602313521' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:14:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:51 smithi132 ceph-mon[33500]: osdmap e320: 8 total, 7 up, 8 in 2024-03-26T05:14:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:51 smithi132 ceph-mon[33500]: 6.b scrub starts 2024-03-26T05:14:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:51 smithi132 ceph-mon[33500]: 4.d scrub starts 2024-03-26T05:14:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:51 smithi132 ceph-mon[33500]: 4.d scrub ok 2024-03-26T05:14:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:51 smithi132 ceph-mon[33500]: 6.b scrub ok 2024-03-26T05:14:52.050 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:14:52.071 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:14:52.090 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:14:52.111 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:14:52.112 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:14:52.112 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:14:52.112 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:14:52.112 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:14:52.112 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:14:52.112 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:14:52.112 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:14:52.112 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:14:52.141 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:52.165 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:52.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:52.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:52.266 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:52.290 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:52.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:52.392 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:52.415 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:52.420 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:52.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:52.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:52.476 DEBUG:teuthology.orchestra.run.smithi080:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-26T05:14:52.477 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-26T05:14:52.477 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:14:52.479 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T05:14:52.479 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T05:14:52.522 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:52.548 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:52.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:52.588 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:hints = 1 2024-03-26T05:14:52.641 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T05:14:52.641 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Object prefix: benchmark_data_smithi080_161990 2024-03-26T05:14:52.642 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:14:52.642 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 0 0 0 0 0 0 - 0 2024-03-26T05:14:52.650 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:52.692 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:52.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:52.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:52.794 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:52.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:52 smithi080 ceph-mon[29691]: 6.13 scrub starts 2024-03-26T05:14:52.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:52 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1085462721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:52.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:52 smithi080 ceph-mon[29691]: 6.13 scrub ok 2024-03-26T05:14:52.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:52 smithi080 ceph-mon[29691]: pgmap v649: 148 pgs: 92 active+clean, 37 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:52.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:52 smithi080 ceph-mon[40404]: 6.13 scrub starts 2024-03-26T05:14:52.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:52 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1085462721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:52.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:52 smithi080 ceph-mon[40404]: 6.13 scrub ok 2024-03-26T05:14:52.806 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:52 smithi080 ceph-mon[40404]: pgmap v649: 148 pgs: 92 active+clean, 37 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:52.828 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:52.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:52.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:52.930 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:52.967 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:52.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:53.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:52 smithi132 ceph-mon[33500]: 6.13 scrub starts 2024-03-26T05:14:53.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:52 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1085462721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:14:53.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:52 smithi132 ceph-mon[33500]: 6.13 scrub ok 2024-03-26T05:14:53.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:52 smithi132 ceph-mon[33500]: pgmap v649: 148 pgs: 92 active+clean, 37 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:53.069 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:53.109 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:53.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:53.210 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:53.247 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:53.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:53.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:53.349 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:53.389 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:53.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:53.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:53.491 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:53.531 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:53.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:53.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:53.591 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 1 16 3870 3854 240.357 240.875 0.00858617 0.00413768 2024-03-26T05:14:53.633 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:53.672 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:53.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:53.773 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:53.810 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:53.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:53.912 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:53.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:53.950 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:53.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:53.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[40404]: 5.a scrub starts 2024-03-26T05:14:53.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[40404]: 5.a scrub ok 2024-03-26T05:14:53.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[40404]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:14:53.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[40404]: 4.12 scrub starts 2024-03-26T05:14:53.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[40404]: 4.12 scrub ok 2024-03-26T05:14:53.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[40404]: 6.1a scrub starts 2024-03-26T05:14:53.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[40404]: 6.1a scrub ok 2024-03-26T05:14:53.997 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[29691]: 5.a scrub starts 2024-03-26T05:14:53.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[29691]: 5.a scrub ok 2024-03-26T05:14:53.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[29691]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:14:53.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[29691]: 4.12 scrub starts 2024-03-26T05:14:53.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[29691]: 4.12 scrub ok 2024-03-26T05:14:53.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[29691]: 6.1a scrub starts 2024-03-26T05:14:53.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:53 smithi080 ceph-mon[29691]: 6.1a scrub ok 2024-03-26T05:14:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:53 smithi132 ceph-mon[33500]: 5.a scrub starts 2024-03-26T05:14:54.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:53 smithi132 ceph-mon[33500]: 5.a scrub ok 2024-03-26T05:14:54.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:53 smithi132 ceph-mon[33500]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:14:54.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:53 smithi132 ceph-mon[33500]: 4.12 scrub starts 2024-03-26T05:14:54.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:53 smithi132 ceph-mon[33500]: 4.12 scrub ok 2024-03-26T05:14:54.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:53 smithi132 ceph-mon[33500]: 6.1a scrub starts 2024-03-26T05:14:54.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:53 smithi132 ceph-mon[33500]: 6.1a scrub ok 2024-03-26T05:14:54.050 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:54.089 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:54.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:54.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:54.190 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:54.226 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:54.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:54.254 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:14:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:14:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:14:54.326 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:54.366 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:54.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:54.469 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:54.502 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:54.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:54.592 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 2 16 7546 7530 235.049 229.75 0.00605712 0.00424158 2024-03-26T05:14:54.603 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:54.636 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:54.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:54.737 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:54.775 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:54.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:54.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:54 smithi080 ceph-mon[29691]: 4.1c scrub starts 2024-03-26T05:14:54.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:54 smithi080 ceph-mon[29691]: 4.1c scrub ok 2024-03-26T05:14:54.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:54 smithi080 ceph-mon[29691]: 6.e scrub starts 2024-03-26T05:14:54.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:54 smithi080 ceph-mon[29691]: pgmap v650: 148 pgs: 92 active+clean, 37 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:54.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:54 smithi080 ceph-mon[29691]: 6.e scrub ok 2024-03-26T05:14:54.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:54 smithi080 ceph-mon[40404]: 4.1c scrub starts 2024-03-26T05:14:54.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:54 smithi080 ceph-mon[40404]: 4.1c scrub ok 2024-03-26T05:14:54.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:54 smithi080 ceph-mon[40404]: 6.e scrub starts 2024-03-26T05:14:54.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:54 smithi080 ceph-mon[40404]: pgmap v650: 148 pgs: 92 active+clean, 37 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:54.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:54 smithi080 ceph-mon[40404]: 6.e scrub ok 2024-03-26T05:14:54.876 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:54.915 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:54.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:55.017 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:55.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:54 smithi132 ceph-mon[33500]: 4.1c scrub starts 2024-03-26T05:14:55.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:54 smithi132 ceph-mon[33500]: 4.1c scrub ok 2024-03-26T05:14:55.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:54 smithi132 ceph-mon[33500]: 6.e scrub starts 2024-03-26T05:14:55.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:54 smithi132 ceph-mon[33500]: pgmap v650: 148 pgs: 92 active+clean, 37 active+undersized, 19 active+undersized+degraded; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/690 objects degraded (13.188%) 2024-03-26T05:14:55.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:54 smithi132 ceph-mon[33500]: 6.e scrub ok 2024-03-26T05:14:55.058 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:55.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:55.123 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:14:55.124 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:14:55.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:55.201 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:55.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:55.303 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:55.338 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:55.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:55.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:55.440 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:55.465 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:55.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:55.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:55.567 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:55.591 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 3 16 10794 10778 224.368 203 0.00444464 0.00445083 2024-03-26T05:14:55.602 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:55.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:55.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:55.704 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:55.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:55.742 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:55.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:55.842 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:55.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:55.884 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:55.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:55.985 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:55.998 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:55 smithi080 ceph-mon[40404]: 6.a scrub starts 2024-03-26T05:14:55.998 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:55 smithi080 ceph-mon[40404]: 6.a scrub ok 2024-03-26T05:14:55.998 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:55 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:55.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:55 smithi080 ceph-mon[29691]: 6.a scrub starts 2024-03-26T05:14:55.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:55 smithi080 ceph-mon[29691]: 6.a scrub ok 2024-03-26T05:14:55.999 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:55 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:56.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:56.019 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:56.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:56.032 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:14:56.032 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:14:56.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:55 smithi132 ceph-mon[33500]: 6.a scrub starts 2024-03-26T05:14:56.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:55 smithi132 ceph-mon[33500]: 6.a scrub ok 2024-03-26T05:14:56.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:55 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:14:56.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:14:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:14:56.120 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:56.162 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:56.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:56.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:56.265 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:56.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:56.306 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:56.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:56.407 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:56.443 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:56.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:56.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:56.545 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:56.578 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:56.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:56.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:56.591 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 4 15 13724 13709 214.074 183.188 0.00986472 0.00466279 2024-03-26T05:14:56.680 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:56.714 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:56.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:56.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:56.764 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:56 smithi080 ceph-mon[40404]: 4.7 scrub starts 2024-03-26T05:14:56.764 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:56 smithi080 ceph-mon[40404]: 4.7 scrub ok 2024-03-26T05:14:56.764 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:56 smithi080 ceph-mon[40404]: 4.9 scrub starts 2024-03-26T05:14:56.764 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:56 smithi080 ceph-mon[40404]: 4.9 scrub ok 2024-03-26T05:14:56.765 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:56 smithi080 ceph-mon[40404]: pgmap v651: 148 pgs: 92 active+clean, 34 active+undersized, 22 active+undersized+degraded; 511 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 51 MiB/s wr, 817 op/s; 1783/25206 objects degraded (7.074%) 2024-03-26T05:14:56.765 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:56 smithi080 ceph-mon[29691]: 4.7 scrub starts 2024-03-26T05:14:56.765 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:56 smithi080 ceph-mon[29691]: 4.7 scrub ok 2024-03-26T05:14:56.765 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:56 smithi080 ceph-mon[29691]: 4.9 scrub starts 2024-03-26T05:14:56.765 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:56 smithi080 ceph-mon[29691]: 4.9 scrub ok 2024-03-26T05:14:56.765 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:56 smithi080 ceph-mon[29691]: pgmap v651: 148 pgs: 92 active+clean, 34 active+undersized, 22 active+undersized+degraded; 511 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 51 MiB/s wr, 817 op/s; 1783/25206 objects degraded (7.074%) 2024-03-26T05:14:56.816 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:56.857 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:56.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:56.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:56.959 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:57.000 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:57.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:57.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:56 smithi132 ceph-mon[33500]: 4.7 scrub starts 2024-03-26T05:14:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:56 smithi132 ceph-mon[33500]: 4.7 scrub ok 2024-03-26T05:14:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:56 smithi132 ceph-mon[33500]: 4.9 scrub starts 2024-03-26T05:14:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:56 smithi132 ceph-mon[33500]: 4.9 scrub ok 2024-03-26T05:14:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:56 smithi132 ceph-mon[33500]: pgmap v651: 148 pgs: 92 active+clean, 34 active+undersized, 22 active+undersized+degraded; 511 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 51 MiB/s wr, 817 op/s; 1783/25206 objects degraded (7.074%) 2024-03-26T05:14:57.102 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:14:57.105 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:14:57.110 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T05:14:57.124 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:14:57.133 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:57.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:14:57.236 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:57.285 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:57.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:57.389 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:57.429 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:57.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:57.531 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:57.569 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:57.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:57.593 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 5 15 16379 16364 204.445 165.938 0.00391827 0.00488588 2024-03-26T05:14:57.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:57.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:57.709 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:57.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:57.748 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:14:57.810 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:57.848 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:57.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:57.950 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:57.988 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:57.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:58.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:57 smithi132 ceph-mon[33500]: 5.f scrub starts 2024-03-26T05:14:58.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:57 smithi132 ceph-mon[33500]: 5.f scrub ok 2024-03-26T05:14:58.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:57 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 1783/25206 objects degraded (7.074%), 22 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:58.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:57 smithi132 ceph-mon[33500]: 4.c scrub starts 2024-03-26T05:14:58.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:57 smithi132 ceph-mon[33500]: 4.c scrub ok 2024-03-26T05:14:58.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:57 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/754921244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:58.089 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:58.100 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:57 smithi080 ceph-mon[29691]: 5.f scrub starts 2024-03-26T05:14:58.100 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:57 smithi080 ceph-mon[29691]: 5.f scrub ok 2024-03-26T05:14:58.100 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:57 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 1783/25206 objects degraded (7.074%), 22 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:58.100 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:57 smithi080 ceph-mon[29691]: 4.c scrub starts 2024-03-26T05:14:58.100 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:57 smithi080 ceph-mon[29691]: 4.c scrub ok 2024-03-26T05:14:58.100 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:57 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/754921244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:58.100 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:57 smithi080 ceph-mon[40404]: 5.f scrub starts 2024-03-26T05:14:58.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:57 smithi080 ceph-mon[40404]: 5.f scrub ok 2024-03-26T05:14:58.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:57 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 1783/25206 objects degraded (7.074%), 22 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:14:58.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:57 smithi080 ceph-mon[40404]: 4.c scrub starts 2024-03-26T05:14:58.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:57 smithi080 ceph-mon[40404]: 4.c scrub ok 2024-03-26T05:14:58.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:57 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/754921244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:14:58.130 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:58.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:58.232 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:58.269 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:58.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:58.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:58.413 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:58.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:58.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:58.515 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:58.554 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:58.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:58.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:58.592 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 6 16 19249 19233 200.255 179.312 0.00297059 0.0049873 2024-03-26T05:14:58.657 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:58.694 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:58.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:58.796 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:14:58.830 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:58.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:14:58.932 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:14:58.956 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:58.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:14:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:58 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:14:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/754921244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:14:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:58 smithi132 ceph-mon[33500]: osdmap e321: 8 total, 7 up, 8 in 2024-03-26T05:14:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:58 smithi132 ceph-mon[33500]: pgmap v653: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 603 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 2520/29586 objects degraded (8.518%) 2024-03-26T05:14:59.058 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:59.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:59.092 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:59.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:59.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:58 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:14:59.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/754921244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:14:59.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:58 smithi080 ceph-mon[29691]: osdmap e321: 8 total, 7 up, 8 in 2024-03-26T05:14:59.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:58 smithi080 ceph-mon[29691]: pgmap v653: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 603 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 2520/29586 objects degraded (8.518%) 2024-03-26T05:14:59.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:58 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:14:59.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/754921244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:14:59.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:58 smithi080 ceph-mon[40404]: osdmap e321: 8 total, 7 up, 8 in 2024-03-26T05:14:59.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:58 smithi080 ceph-mon[40404]: pgmap v653: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 603 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 2520/29586 objects degraded (8.518%) 2024-03-26T05:14:59.193 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:59.233 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:59.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:59.335 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:14:59.369 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:14:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:59.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:14:59.471 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:14:59.508 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:59.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:14:59.592 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 7 16 21743 21727 193.915 155.875 0.00594273 0.00515284 2024-03-26T05:14:59.610 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:14:59.649 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:59.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:59.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:14:59.751 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:59.790 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:59.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:14:59.892 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:14:59.914 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:59 smithi080 ceph-mon[40404]: 4.10 scrub starts 2024-03-26T05:14:59.914 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:14:59 smithi080 ceph-mon[40404]: 4.10 scrub ok 2024-03-26T05:14:59.914 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:59 smithi080 ceph-mon[29691]: 4.10 scrub starts 2024-03-26T05:14:59.914 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:14:59 smithi080 ceph-mon[29691]: 4.10 scrub ok 2024-03-26T05:14:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:14:59.945 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:14:59.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:00.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:59 smithi132 ceph-mon[33500]: 4.10 scrub starts 2024-03-26T05:15:00.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:14:59 smithi132 ceph-mon[33500]: 4.10 scrub ok 2024-03-26T05:15:00.046 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:00.085 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:00.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:00.186 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:00.221 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:00.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:00.322 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:00.363 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:00.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:00.465 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:00.478 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:00.517 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:00.518 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:00.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:00.592 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 8 16 24008 23992 187.371 141.562 0.00421878 0.00533084 2024-03-26T05:15:00.619 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:00.669 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:00.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:00.771 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:00.804 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:00.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:00.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[29691]: 6.c scrub starts 2024-03-26T05:15:00.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[29691]: 6.c scrub ok 2024-03-26T05:15:00.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[29691]: 4.1 scrub starts 2024-03-26T05:15:00.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[29691]: 4.1 scrub ok 2024-03-26T05:15:00.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[29691]: 6.1d scrub starts 2024-03-26T05:15:00.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[29691]: pgmap v654: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 1.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 117 MiB/s wr, 1.88k op/s; 4333/53280 objects degraded (8.133%) 2024-03-26T05:15:00.876 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[29691]: 6.1d scrub ok 2024-03-26T05:15:00.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[40404]: 6.c scrub starts 2024-03-26T05:15:00.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[40404]: 6.c scrub ok 2024-03-26T05:15:00.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[40404]: 4.1 scrub starts 2024-03-26T05:15:00.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[40404]: 4.1 scrub ok 2024-03-26T05:15:00.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[40404]: 6.1d scrub starts 2024-03-26T05:15:00.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[40404]: pgmap v654: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 1.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 117 MiB/s wr, 1.88k op/s; 4333/53280 objects degraded (8.133%) 2024-03-26T05:15:00.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:00 smithi080 ceph-mon[40404]: 6.1d scrub ok 2024-03-26T05:15:00.906 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:00.936 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:15:00.940 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:15:00.950 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:00.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:00.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:01.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:00 smithi132 ceph-mon[33500]: 6.c scrub starts 2024-03-26T05:15:01.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:00 smithi132 ceph-mon[33500]: 6.c scrub ok 2024-03-26T05:15:01.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:00 smithi132 ceph-mon[33500]: 4.1 scrub starts 2024-03-26T05:15:01.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:00 smithi132 ceph-mon[33500]: 4.1 scrub ok 2024-03-26T05:15:01.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:00 smithi132 ceph-mon[33500]: 6.1d scrub starts 2024-03-26T05:15:01.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:00 smithi132 ceph-mon[33500]: pgmap v654: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 1.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 117 MiB/s wr, 1.88k op/s; 4333/53280 objects degraded (8.133%) 2024-03-26T05:15:01.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:00 smithi132 ceph-mon[33500]: 6.1d scrub ok 2024-03-26T05:15:01.051 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:01.091 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:01.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:01.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:01.193 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:01.216 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:01.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:01.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:01.318 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:01.351 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:01.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:01.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:01.453 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:01.491 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:01.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:01.592 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 9 16 26054 26038 180.76 127.875 0.00489465 0.00552046 2024-03-26T05:15:01.592 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:01.620 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:01.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:01.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:01.697 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:15:01.722 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:01.784 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:01.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:01.885 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:01.918 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:01.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:02.020 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:02.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:01 smithi132 ceph-mon[33500]: 2.1 scrub starts 2024-03-26T05:15:02.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:01 smithi132 ceph-mon[33500]: 2.1 scrub ok 2024-03-26T05:15:02.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:01 smithi132 ceph-mon[33500]: 3.1c deep-scrub starts 2024-03-26T05:15:02.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:01 smithi132 ceph-mon[33500]: 3.1c deep-scrub ok 2024-03-26T05:15:02.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:01 smithi132 ceph-mon[33500]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:15:02.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:01 smithi132 ceph-mon[33500]: 4.1b deep-scrub starts 2024-03-26T05:15:02.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:01 smithi132 ceph-mon[33500]: 4.1b deep-scrub ok 2024-03-26T05:15:02.061 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:02.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:02.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[29691]: 2.1 scrub starts 2024-03-26T05:15:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[29691]: 2.1 scrub ok 2024-03-26T05:15:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[29691]: 3.1c deep-scrub starts 2024-03-26T05:15:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[29691]: 3.1c deep-scrub ok 2024-03-26T05:15:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[29691]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:15:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[29691]: 4.1b deep-scrub starts 2024-03-26T05:15:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[29691]: 4.1b deep-scrub ok 2024-03-26T05:15:02.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[40404]: 2.1 scrub starts 2024-03-26T05:15:02.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[40404]: 2.1 scrub ok 2024-03-26T05:15:02.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[40404]: 3.1c deep-scrub starts 2024-03-26T05:15:02.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[40404]: 3.1c deep-scrub ok 2024-03-26T05:15:02.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[40404]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:15:02.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[40404]: 4.1b deep-scrub starts 2024-03-26T05:15:02.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:01 smithi080 ceph-mon[40404]: 4.1b deep-scrub ok 2024-03-26T05:15:02.163 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:02.201 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:02.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:02.303 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:02.339 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:02.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:02.440 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:02.478 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:02.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:02.580 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:02.592 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 10 16 28270 28254 176.534 138.5 0.0141418 0.00565863 2024-03-26T05:15:02.616 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:02.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:02.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:02.718 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:02.758 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:02.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:02.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:02.860 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:02.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:02.898 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:02.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:02.999 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:03.032 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:03.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:03.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:02 smithi132 ceph-mon[33500]: pgmap v655: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.32k op/s; 5979/70293 objects degraded (8.506%) 2024-03-26T05:15:03.097 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:02 smithi080 ceph-mon[29691]: pgmap v655: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.32k op/s; 5979/70293 objects degraded (8.506%) 2024-03-26T05:15:03.097 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:02 smithi080 ceph-mon[40404]: pgmap v655: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.32k op/s; 5979/70293 objects degraded (8.506%) 2024-03-26T05:15:03.135 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:03.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:03.177 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:03.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:03.278 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:03.319 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:03.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:03.421 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:03.457 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:03.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:03.559 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:03.583 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:03.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:03.592 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 11 16 30471 30455 172.991 137.562 0.0022507 0.00574567 2024-03-26T05:15:03.684 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:03.725 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:03.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:03.827 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:03.863 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:03.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:03.965 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:04.001 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:04.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:04.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:03 smithi080 ceph-mon[40404]: 6.d scrub starts 2024-03-26T05:15:04.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:03 smithi080 ceph-mon[40404]: 6.d scrub ok 2024-03-26T05:15:04.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:03 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 5979/70293 objects degraded (8.506%), 23 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:04.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:03 smithi080 ceph-mon[40404]: 3.1a scrub starts 2024-03-26T05:15:04.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:03 smithi080 ceph-mon[40404]: 3.1a scrub ok 2024-03-26T05:15:04.036 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:03 smithi080 ceph-mon[29691]: 6.d scrub starts 2024-03-26T05:15:04.036 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:03 smithi080 ceph-mon[29691]: 6.d scrub ok 2024-03-26T05:15:04.037 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:03 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 5979/70293 objects degraded (8.506%), 23 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:04.037 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:03 smithi080 ceph-mon[29691]: 3.1a scrub starts 2024-03-26T05:15:04.037 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:03 smithi080 ceph-mon[29691]: 3.1a scrub ok 2024-03-26T05:15:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:03 smithi132 ceph-mon[33500]: 6.d scrub starts 2024-03-26T05:15:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:03 smithi132 ceph-mon[33500]: 6.d scrub ok 2024-03-26T05:15:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:03 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 5979/70293 objects degraded (8.506%), 23 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:03 smithi132 ceph-mon[33500]: 3.1a scrub starts 2024-03-26T05:15:04.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:03 smithi132 ceph-mon[33500]: 3.1a scrub ok 2024-03-26T05:15:04.102 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:04.136 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:04.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:04.238 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:04.276 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:04.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:04.312 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:15:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:15:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:15:04.378 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:04.417 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:04.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:04.518 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:04.551 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:04.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:04.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:04.592 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 12 16 32843 32827 170.928 148.25 0.00605203 0.00584615 2024-03-26T05:15:04.653 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:04.692 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:04.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:04.794 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:04.824 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:04.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:04.925 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:04.964 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:04.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:05.039 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:04 smithi080 ceph-mon[29691]: 4.18 scrub starts 2024-03-26T05:15:05.039 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:04 smithi080 ceph-mon[29691]: 4.18 scrub ok 2024-03-26T05:15:05.039 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:04 smithi080 ceph-mon[29691]: pgmap v656: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.32k op/s; 5979/70293 objects degraded (8.506%) 2024-03-26T05:15:05.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:04 smithi132 ceph-mon[33500]: 4.18 scrub starts 2024-03-26T05:15:05.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:04 smithi132 ceph-mon[33500]: 4.18 scrub ok 2024-03-26T05:15:05.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:04 smithi132 ceph-mon[33500]: pgmap v656: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.32k op/s; 5979/70293 objects degraded (8.506%) 2024-03-26T05:15:05.043 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:04 smithi080 ceph-mon[40404]: 4.18 scrub starts 2024-03-26T05:15:05.043 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:04 smithi080 ceph-mon[40404]: 4.18 scrub ok 2024-03-26T05:15:05.043 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:04 smithi080 ceph-mon[40404]: pgmap v656: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.32k op/s; 5979/70293 objects degraded (8.506%) 2024-03-26T05:15:05.066 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:05.100 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":654,"stamp":"2024-03-26T05:15:00.032208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1149442398,"num_objects":17760,"num_object_clones":0,"num_object_copies":53280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17760,"num_whiteouts":0,"num_read":1849,"num_read_kb":1665,"num_write":18623,"num_write_kb":1124381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":1774172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18706,"ondisk_log_size":18706,"up":388,"acting":388,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":449,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9953020,"kb_used_data":6385596,"kb_used_omap":12,"kb_used_meta":3567347,"kb_avail":740008196,"statfs":{"total":767960285184,"available":757768392704,"internally_reserved":0,"allocated":6538850304,"data_stored":6536197613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3652963888},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1148846080,"num_objects":17530,"num_object_clones":0,"num_object_copies":52590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17530,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":17530,"num_write_kb":1121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.329890"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":570,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.753144+0000","last_change":"2024-03-26T05:14:53.055275+0000","last_active":"2024-03-26T05:14:57.753144+0000","last_peered":"2024-03-26T05:14:57.753144+0000","last_clean":"2024-03-26T05:14:57.753144+0000","last_became_active":"2024-03-26T05:07:59.996436+0000","last_became_peered":"2024-03-26T05:07:59.996436+0000","last_unstale":"2024-03-26T05:14:57.753144+0000","last_undegraded":"2024-03-26T05:14:57.753144+0000","last_fullsized":"2024-03-26T05:14:57.753144+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:53.055153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:53.055153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:21.428475+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":565,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.938651+0000","last_change":"2024-03-26T05:12:26.805941+0000","last_active":"2024-03-26T05:14:50.938651+0000","last_peered":"2024-03-26T05:14:50.938651+0000","last_clean":"2024-03-26T05:12:19.745617+0000","last_became_active":"2024-03-26T05:12:26.805941+0000","last_became_peered":"2024-03-26T05:12:26.805941+0000","last_unstale":"2024-03-26T05:14:50.938651+0000","last_undegraded":"2024-03-26T05:14:50.938651+0000","last_fullsized":"2024-03-26T05:12:26.799894+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:30.527764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:30.527764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:50.967501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":589,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.705756+0000","last_change":"2024-03-26T05:14:20.034981+0000","last_active":"2024-03-26T05:14:50.705756+0000","last_peered":"2024-03-26T05:14:50.705756+0000","last_clean":"2024-03-26T05:14:50.705756+0000","last_became_active":"2024-03-26T05:07:59.994431+0000","last_became_peered":"2024-03-26T05:07:59.994431+0000","last_unstale":"2024-03-26T05:14:50.705756+0000","last_undegraded":"2024-03-26T05:14:50.705756+0000","last_fullsized":"2024-03-26T05:14:50.705756+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:20.034875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:20.034875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:42.643604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"254'21","reported_seq":596,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.753012+0000","last_change":"2024-03-26T05:12:54.218563+0000","last_active":"2024-03-26T05:14:57.753012+0000","last_peered":"2024-03-26T05:14:57.753012+0000","last_clean":"2024-03-26T05:14:57.753012+0000","last_became_active":"2024-03-26T05:00:24.943263+0000","last_became_peered":"2024-03-26T05:00:24.943263+0000","last_unstale":"2024-03-26T05:14:57.753012+0000","last_undegraded":"2024-03-26T05:14:57.753012+0000","last_fullsized":"2024-03-26T05:14:57.753012+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:12:54.218453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:12:54.218453+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":533,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.926934+0000","last_change":"2024-03-26T05:12:26.807652+0000","last_active":"2024-03-26T05:14:50.926934+0000","last_peered":"2024-03-26T05:14:50.926934+0000","last_clean":"2024-03-26T05:12:19.741866+0000","last_became_active":"2024-03-26T05:12:26.807652+0000","last_became_peered":"2024-03-26T05:12:26.807652+0000","last_unstale":"2024-03-26T05:14:50.926934+0000","last_undegraded":"2024-03-26T05:14:50.926934+0000","last_fullsized":"2024-03-26T05:12:26.800890+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:09:59.528596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:09:59.528596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:55.926093+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":563,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.752868+0000","last_change":"2024-03-26T05:13:59.872345+0000","last_active":"2024-03-26T05:14:57.752868+0000","last_peered":"2024-03-26T05:14:57.752868+0000","last_clean":"2024-03-26T05:14:57.752868+0000","last_became_active":"2024-03-26T05:00:26.955070+0000","last_became_peered":"2024-03-26T05:00:26.955070+0000","last_unstale":"2024-03-26T05:14:57.752868+0000","last_undegraded":"2024-03-26T05:14:57.752868+0000","last_fullsized":"2024-03-26T05:14:57.752868+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:59.872240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:13:59.872240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:28.647632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":568,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.927580+0000","last_change":"2024-03-26T05:14:16.816742+0000","last_active":"2024-03-26T05:14:50.927580+0000","last_peered":"2024-03-26T05:14:50.927580+0000","last_clean":"2024-03-26T05:14:50.927580+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:14:50.927580+0000","last_undegraded":"2024-03-26T05:14:50.927580+0000","last_fullsized":"2024-03-26T05:14:50.927580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:16.816655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.816655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:29.107106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"254'51","reported_seq":645,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.927851+0000","last_change":"2024-03-26T05:13:59.679698+0000","last_active":"2024-03-26T05:14:50.927851+0000","last_peered":"2024-03-26T05:14:50.927851+0000","last_clean":"2024-03-26T05:14:50.927851+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:14:50.927851+0000","last_undegraded":"2024-03-26T05:14:50.927851+0000","last_fullsized":"2024-03-26T05:14:50.927851+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'51","last_scrub_stamp":"2024-03-26T05:13:59.679432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:59.679432+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:12.748354+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":559,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.705482+0000","last_change":"2024-03-26T05:12:26.806082+0000","last_active":"2024-03-26T05:14:50.705482+0000","last_peered":"2024-03-26T05:14:50.705482+0000","last_clean":"2024-03-26T05:12:19.739769+0000","last_became_active":"2024-03-26T05:12:26.806082+0000","last_became_peered":"2024-03-26T05:12:26.806082+0000","last_unstale":"2024-03-26T05:14:50.705482+0000","last_undegraded":"2024-03-26T05:14:50.705482+0000","last_fullsized":"2024-03-26T05:12:26.795916+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:04.037472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:04.037472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:50.990104+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":243,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:51.573417+0000","last_change":"2024-03-26T05:12:26.802256+0000","last_active":"2024-03-26T05:14:51.573417+0000","last_peered":"2024-03-26T05:14:51.573417+0000","last_clean":"2024-03-26T05:12:19.741381+0000","last_became_active":"2024-03-26T05:12:26.802256+0000","last_became_peered":"2024-03-26T05:12:26.802256+0000","last_unstale":"2024-03-26T05:14:51.573417+0000","last_undegraded":"2024-03-26T05:14:51.573417+0000","last_fullsized":"2024-03-26T05:12:26.791108+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:04.643034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:04.643034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:52.522622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":248,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.574481+0000","last_change":"2024-03-26T05:14:17.792824+0000","last_active":"2024-03-26T05:14:51.574481+0000","last_peered":"2024-03-26T05:14:51.574481+0000","last_clean":"2024-03-26T05:14:51.574481+0000","last_became_active":"2024-03-26T05:07:59.947557+0000","last_became_peered":"2024-03-26T05:07:59.947557+0000","last_unstale":"2024-03-26T05:14:51.574481+0000","last_undegraded":"2024-03-26T05:14:51.574481+0000","last_fullsized":"2024-03-26T05:14:51.574481+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:17.792745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:17.792745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:21.140892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"254'27","reported_seq":615,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.753323+0000","last_change":"2024-03-26T05:13:22.026515+0000","last_active":"2024-03-26T05:14:57.753323+0000","last_peered":"2024-03-26T05:14:57.753323+0000","last_clean":"2024-03-26T05:14:57.753323+0000","last_became_active":"2024-03-26T05:00:24.940209+0000","last_became_peered":"2024-03-26T05:00:24.940209+0000","last_unstale":"2024-03-26T05:14:57.753323+0000","last_undegraded":"2024-03-26T05:14:57.753323+0000","last_fullsized":"2024-03-26T05:14:57.753323+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:13:22.026354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:22.026354+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":248,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.573642+0000","last_change":"2024-03-26T05:14:10.840006+0000","last_active":"2024-03-26T05:14:51.573642+0000","last_peered":"2024-03-26T05:14:51.573642+0000","last_clean":"2024-03-26T05:14:51.573642+0000","last_became_active":"2024-03-26T05:07:59.998777+0000","last_became_peered":"2024-03-26T05:07:59.998777+0000","last_unstale":"2024-03-26T05:14:51.573642+0000","last_undegraded":"2024-03-26T05:14:51.573642+0000","last_fullsized":"2024-03-26T05:14:51.573642+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:10.839912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:10.839912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:38.402615+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":64,"reported_epoch":321,"state":"active+undersized","last_fresh":"2024-03-26T05:14:57.753818+0000","last_change":"2024-03-26T05:12:26.791031+0000","last_active":"2024-03-26T05:14:57.753818+0000","last_peered":"2024-03-26T05:14:57.753818+0000","last_clean":"2024-03-26T05:00:25.933152+0000","last_became_active":"2024-03-26T05:12:26.791031+0000","last_became_peered":"2024-03-26T05:12:26.791031+0000","last_unstale":"2024-03-26T05:14:57.753818+0000","last_undegraded":"2024-03-26T05:14:57.753818+0000","last_fullsized":"2024-03-26T05:12:26.787449+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_clean_scrub_stamp":"2024-03-26T05:12:18.959687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:00.957609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":240,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.574312+0000","last_change":"2024-03-26T05:13:11.849759+0000","last_active":"2024-03-26T05:14:51.574312+0000","last_peered":"2024-03-26T05:14:51.574312+0000","last_clean":"2024-03-26T05:14:51.574312+0000","last_became_active":"2024-03-26T05:08:00.011262+0000","last_became_peered":"2024-03-26T05:08:00.011262+0000","last_unstale":"2024-03-26T05:14:51.574312+0000","last_undegraded":"2024-03-26T05:14:51.574312+0000","last_fullsized":"2024-03-26T05:14:51.574312+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:11.849682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:13:11.849682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"254'13","reported_seq":583,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.753067+0000","last_change":"2024-03-26T05:12:55.257984+0000","last_active":"2024-03-26T05:14:57.753067+0000","last_peered":"2024-03-26T05:14:57.753067+0000","last_clean":"2024-03-26T05:14:57.753067+0000","last_became_active":"2024-03-26T05:00:24.942861+0000","last_became_peered":"2024-03-26T05:00:24.942861+0000","last_unstale":"2024-03-26T05:14:57.753067+0000","last_undegraded":"2024-03-26T05:14:57.753067+0000","last_fullsized":"2024-03-26T05:14:57.753067+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'13","last_scrub_stamp":"2024-03-26T05:12:55.257845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:12:55.257845+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":580,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.754434+0000","last_change":"2024-03-26T05:14:05.787837+0000","last_active":"2024-03-26T05:14:57.754434+0000","last_peered":"2024-03-26T05:14:57.754434+0000","last_clean":"2024-03-26T05:14:57.754434+0000","last_became_active":"2024-03-26T05:07:59.942829+0000","last_became_peered":"2024-03-26T05:07:59.942829+0000","last_unstale":"2024-03-26T05:14:57.754434+0000","last_undegraded":"2024-03-26T05:14:57.754434+0000","last_fullsized":"2024-03-26T05:14:57.754434+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:05.787712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:14:05.787712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:22.343352+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":558,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.939452+0000","last_change":"2024-03-26T05:13:54.862260+0000","last_active":"2024-03-26T05:14:50.939452+0000","last_peered":"2024-03-26T05:14:50.939452+0000","last_clean":"2024-03-26T05:14:50.939452+0000","last_became_active":"2024-03-26T05:00:26.960266+0000","last_became_peered":"2024-03-26T05:00:26.960266+0000","last_unstale":"2024-03-26T05:14:50.939452+0000","last_undegraded":"2024-03-26T05:14:50.939452+0000","last_fullsized":"2024-03-26T05:14:50.939452+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:54.862047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:13:54.862047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:09.312242+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":571,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.705142+0000","last_change":"2024-03-26T05:12:26.799393+0000","last_active":"2024-03-26T05:14:50.705142+0000","last_peered":"2024-03-26T05:14:50.705142+0000","last_clean":"2024-03-26T05:12:19.738964+0000","last_became_active":"2024-03-26T05:12:26.799393+0000","last_became_peered":"2024-03-26T05:12:26.799393+0000","last_unstale":"2024-03-26T05:14:50.705142+0000","last_undegraded":"2024-03-26T05:14:50.705142+0000","last_fullsized":"2024-03-26T05:12:26.795552+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:04.980730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:12:04.980730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:50.990117+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"254'27","reported_seq":621,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:53.363227+0000","last_change":"2024-03-26T05:14:53.363298+0000","last_active":"2024-03-26T05:14:53.363227+0000","last_peered":"2024-03-26T05:14:53.363227+0000","last_clean":"2024-03-26T05:14:53.363227+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:14:53.363227+0000","last_undegraded":"2024-03-26T05:14:53.363227+0000","last_fullsized":"2024-03-26T05:14:53.363227+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:14:53.363147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:53.363147+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:54.464449+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":568,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.928300+0000","last_change":"2024-03-26T05:14:15.038846+0000","last_active":"2024-03-26T05:14:50.928300+0000","last_peered":"2024-03-26T05:14:50.928300+0000","last_clean":"2024-03-26T05:14:50.928300+0000","last_became_active":"2024-03-26T05:07:59.959705+0000","last_became_peered":"2024-03-26T05:07:59.959705+0000","last_unstale":"2024-03-26T05:14:50.928300+0000","last_undegraded":"2024-03-26T05:14:50.928300+0000","last_fullsized":"2024-03-26T05:14:50.928300+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:15.038608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:15.038608+0000","last_clean_scrub_stamp":"2024-03-26T05:14:15.038608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:18.866042+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":562,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.754284+0000","last_change":"2024-03-26T05:14:04.831113+0000","last_active":"2024-03-26T05:14:57.754284+0000","last_peered":"2024-03-26T05:14:57.754284+0000","last_clean":"2024-03-26T05:14:57.754284+0000","last_became_active":"2024-03-26T05:00:26.961946+0000","last_became_peered":"2024-03-26T05:00:26.961946+0000","last_unstale":"2024-03-26T05:14:57.754284+0000","last_undegraded":"2024-03-26T05:14:57.754284+0000","last_fullsized":"2024-03-26T05:14:57.754284+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:04.830989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:04.830989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:13.020717+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":578,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.753494+0000","last_change":"2024-03-26T05:13:01.143981+0000","last_active":"2024-03-26T05:14:57.753494+0000","last_peered":"2024-03-26T05:14:57.753494+0000","last_clean":"2024-03-26T05:14:57.753494+0000","last_became_active":"2024-03-26T05:00:23.346944+0000","last_became_peered":"2024-03-26T05:00:23.346944+0000","last_unstale":"2024-03-26T05:14:57.753494+0000","last_undegraded":"2024-03-26T05:14:57.753494+0000","last_fullsized":"2024-03-26T05:14:57.753494+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:01.143856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:13:01.143856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"254'26","reported_seq":600,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.927727+0000","last_change":"2024-03-26T05:13:55.718708+0000","last_active":"2024-03-26T05:14:50.927727+0000","last_peered":"2024-03-26T05:14:50.927727+0000","last_clean":"2024-03-26T05:14:50.927727+0000","last_became_active":"2024-03-26T05:00:24.936723+0000","last_became_peered":"2024-03-26T05:00:24.936723+0000","last_unstale":"2024-03-26T05:14:50.927727+0000","last_undegraded":"2024-03-26T05:14:50.927727+0000","last_fullsized":"2024-03-26T05:14:50.927727+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:13:55.718533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:55.718533+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:09.735999+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":385,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:51.574029+0000","last_change":"2024-03-26T05:12:26.803403+0000","last_active":"2024-03-26T05:14:51.574029+0000","last_peered":"2024-03-26T05:14:51.574029+0000","last_clean":"2024-03-26T05:07:55.694823+0000","last_became_active":"2024-03-26T05:12:26.803403+0000","last_became_peered":"2024-03-26T05:12:26.803403+0000","last_unstale":"2024-03-26T05:14:51.574029+0000","last_undegraded":"2024-03-26T05:12:26.791176+0000","last_fullsized":"2024-03-26T05:12:26.791060+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:12:13.995144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:13.995144+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:52.522651+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"294'9","reported_seq":759,"reported_epoch":321,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:57.751948+0000","last_change":"2024-03-26T05:12:26.795772+0000","last_active":"2024-03-26T05:14:57.751948+0000","last_peered":"2024-03-26T05:14:57.751948+0000","last_clean":"2024-03-26T05:12:23.556404+0000","last_became_active":"2024-03-26T05:12:26.795772+0000","last_became_peered":"2024-03-26T05:12:26.795772+0000","last_unstale":"2024-03-26T05:14:57.751948+0000","last_undegraded":"2024-03-26T05:12:26.790732+0000","last_fullsized":"2024-03-26T05:12:26.790617+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:11:19.698504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:19.698504+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":575,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.938693+0000","last_change":"2024-03-26T05:14:49.948907+0000","last_active":"2024-03-26T05:14:50.938693+0000","last_peered":"2024-03-26T05:14:50.938693+0000","last_clean":"2024-03-26T05:14:50.938693+0000","last_became_active":"2024-03-26T05:00:22.773093+0000","last_became_peered":"2024-03-26T05:00:22.773093+0000","last_unstale":"2024-03-26T05:14:50.938693+0000","last_undegraded":"2024-03-26T05:14:50.938693+0000","last_fullsized":"2024-03-26T05:14:50.938693+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:49.948818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:49.948818+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:14.067138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'32","reported_seq":617,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:50.705500+0000","last_change":"2024-03-26T05:12:26.804500+0000","last_active":"2024-03-26T05:14:50.705500+0000","last_peered":"2024-03-26T05:14:50.705500+0000","last_clean":"2024-03-26T05:12:19.739941+0000","last_became_active":"2024-03-26T05:12:26.804500+0000","last_became_peered":"2024-03-26T05:12:26.804500+0000","last_unstale":"2024-03-26T05:14:50.705500+0000","last_undegraded":"2024-03-26T05:12:26.796883+0000","last_fullsized":"2024-03-26T05:12:26.796767+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'32","last_scrub_stamp":"2024-03-26T05:12:15.970313+0000","last_deep_scrub":"290'32","last_deep_scrub_stamp":"2024-03-26T05:12:15.970313+0000","last_clean_scrub_stamp":"2024-03-26T05:12:15.970313+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:50.990174+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":566,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.939519+0000","last_change":"2024-03-26T05:13:15.959501+0000","last_active":"2024-03-26T05:14:50.939519+0000","last_peered":"2024-03-26T05:14:50.939519+0000","last_clean":"2024-03-26T05:14:50.939519+0000","last_became_active":"2024-03-26T05:07:59.995261+0000","last_became_peered":"2024-03-26T05:07:59.995261+0000","last_unstale":"2024-03-26T05:14:50.939519+0000","last_undegraded":"2024-03-26T05:14:50.939519+0000","last_fullsized":"2024-03-26T05:14:50.939519+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:15.959229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:13:15.959229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":557,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.705789+0000","last_change":"2024-03-26T05:12:26.798694+0000","last_active":"2024-03-26T05:14:50.705789+0000","last_peered":"2024-03-26T05:14:50.705789+0000","last_clean":"2024-03-26T05:12:19.740077+0000","last_became_active":"2024-03-26T05:12:26.798694+0000","last_became_peered":"2024-03-26T05:12:26.798694+0000","last_unstale":"2024-03-26T05:14:50.705789+0000","last_undegraded":"2024-03-26T05:14:50.705789+0000","last_fullsized":"2024-03-26T05:12:26.794841+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:27.080959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:27.080959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:50.990073+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":566,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.927188+0000","last_change":"2024-03-26T05:14:19.823236+0000","last_active":"2024-03-26T05:14:50.927188+0000","last_peered":"2024-03-26T05:14:50.927188+0000","last_clean":"2024-03-26T05:14:50.927188+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:14:50.927188+0000","last_undegraded":"2024-03-26T05:14:50.927188+0000","last_fullsized":"2024-03-26T05:14:50.927188+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:19.823158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:19.823158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:43.081047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"254'25","reported_seq":691,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.751520+0000","last_change":"2024-03-26T05:14:50.429468+0000","last_active":"2024-03-26T05:14:57.751520+0000","last_peered":"2024-03-26T05:14:57.751520+0000","last_clean":"2024-03-26T05:14:57.751520+0000","last_became_active":"2024-03-26T05:07:59.962007+0000","last_became_peered":"2024-03-26T05:07:59.962007+0000","last_unstale":"2024-03-26T05:14:57.751520+0000","last_undegraded":"2024-03-26T05:14:57.751520+0000","last_fullsized":"2024-03-26T05:14:57.751520+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:14:50.429340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:50.429340+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:00.465354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":63,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.941021+0000","last_change":"2024-03-26T05:12:26.794893+0000","last_active":"2024-03-26T05:14:50.941021+0000","last_peered":"2024-03-26T05:14:50.941021+0000","last_clean":"2024-03-26T05:00:27.943471+0000","last_became_active":"2024-03-26T05:12:26.794893+0000","last_became_peered":"2024-03-26T05:12:26.794893+0000","last_unstale":"2024-03-26T05:14:50.941021+0000","last_undegraded":"2024-03-26T05:14:50.941021+0000","last_fullsized":"2024-03-26T05:12:26.793302+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:10.020572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:10.020572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":563,"reported_epoch":321,"state":"active+undersized","last_fresh":"2024-03-26T05:14:57.749929+0000","last_change":"2024-03-26T05:12:26.799213+0000","last_active":"2024-03-26T05:14:57.749929+0000","last_peered":"2024-03-26T05:14:57.749929+0000","last_clean":"2024-03-26T05:12:19.743473+0000","last_became_active":"2024-03-26T05:12:26.799213+0000","last_became_peered":"2024-03-26T05:12:26.799213+0000","last_unstale":"2024-03-26T05:14:57.749929+0000","last_undegraded":"2024-03-26T05:14:57.749929+0000","last_fullsized":"2024-03-26T05:12:26.791611+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.693592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.693592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":589,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.705589+0000","last_change":"2024-03-26T05:13:55.940656+0000","last_active":"2024-03-26T05:14:50.705589+0000","last_peered":"2024-03-26T05:14:50.705589+0000","last_clean":"2024-03-26T05:14:50.705589+0000","last_became_active":"2024-03-26T05:07:59.993996+0000","last_became_peered":"2024-03-26T05:07:59.993996+0000","last_unstale":"2024-03-26T05:14:50.705589+0000","last_undegraded":"2024-03-26T05:14:50.705589+0000","last_fullsized":"2024-03-26T05:14:50.705589+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:55.940541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:13:55.940541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:10.604546+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"254'10","reported_seq":569,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.927280+0000","last_change":"2024-03-26T05:13:05.819992+0000","last_active":"2024-03-26T05:14:50.927280+0000","last_peered":"2024-03-26T05:14:50.927280+0000","last_clean":"2024-03-26T05:14:50.927280+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:14:50.927280+0000","last_undegraded":"2024-03-26T05:14:50.927280+0000","last_fullsized":"2024-03-26T05:14:50.927280+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'10","last_scrub_stamp":"2024-03-26T05:13:05.819859+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:13:05.819859+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":555,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:52.001464+0000","last_change":"2024-03-26T05:14:52.001489+0000","last_active":"2024-03-26T05:14:52.001464+0000","last_peered":"2024-03-26T05:14:52.001464+0000","last_clean":"2024-03-26T05:14:52.001464+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:14:52.001464+0000","last_undegraded":"2024-03-26T05:14:52.001464+0000","last_fullsized":"2024-03-26T05:14:52.001464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:52.001365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_clean_scrub_stamp":"2024-03-26T05:14:52.001365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:53.555937+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":561,"reported_epoch":321,"state":"active+undersized","last_fresh":"2024-03-26T05:14:57.752880+0000","last_change":"2024-03-26T05:12:26.805355+0000","last_active":"2024-03-26T05:14:57.752880+0000","last_peered":"2024-03-26T05:14:57.752880+0000","last_clean":"2024-03-26T05:12:13.651420+0000","last_became_active":"2024-03-26T05:12:26.805355+0000","last_became_peered":"2024-03-26T05:12:26.805355+0000","last_unstale":"2024-03-26T05:14:57.752880+0000","last_undegraded":"2024-03-26T05:14:57.752880+0000","last_fullsized":"2024-03-26T05:12:26.786977+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:23.957542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:11:23.957542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:00.957530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":235,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:51.574509+0000","last_change":"2024-03-26T05:12:26.806655+0000","last_active":"2024-03-26T05:14:51.574509+0000","last_peered":"2024-03-26T05:14:51.574509+0000","last_clean":"2024-03-26T05:12:19.743537+0000","last_became_active":"2024-03-26T05:12:26.806655+0000","last_became_peered":"2024-03-26T05:12:26.806655+0000","last_unstale":"2024-03-26T05:14:51.574509+0000","last_undegraded":"2024-03-26T05:14:51.574509+0000","last_fullsized":"2024-03-26T05:12:26.791797+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:19.513035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:11:19.513035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:52.522586+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"254'25","reported_seq":593,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:50.926917+0000","last_change":"2024-03-26T05:12:26.820515+0000","last_active":"2024-03-26T05:14:50.926917+0000","last_peered":"2024-03-26T05:14:50.926917+0000","last_clean":"2024-03-26T05:12:19.741292+0000","last_became_active":"2024-03-26T05:12:26.820515+0000","last_became_peered":"2024-03-26T05:12:26.820515+0000","last_unstale":"2024-03-26T05:14:50.926917+0000","last_undegraded":"2024-03-26T05:12:26.798120+0000","last_fullsized":"2024-03-26T05:12:26.797798+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:11:18.408463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:11:18.408463+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:55.926182+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":585,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.704242+0000","last_change":"2024-03-26T05:14:46.024625+0000","last_active":"2024-03-26T05:14:50.704242+0000","last_peered":"2024-03-26T05:14:50.704242+0000","last_clean":"2024-03-26T05:14:50.704242+0000","last_became_active":"2024-03-26T05:07:59.994210+0000","last_became_peered":"2024-03-26T05:07:59.994210+0000","last_unstale":"2024-03-26T05:14:50.704242+0000","last_undegraded":"2024-03-26T05:14:50.704242+0000","last_fullsized":"2024-03-26T05:14:50.704242+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:46.024526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:46.024526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:12.539256+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":551,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.927722+0000","last_change":"2024-03-26T05:14:05.728212+0000","last_active":"2024-03-26T05:14:50.927722+0000","last_peered":"2024-03-26T05:14:50.927722+0000","last_clean":"2024-03-26T05:14:50.927722+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:14:50.927722+0000","last_undegraded":"2024-03-26T05:14:50.927722+0000","last_fullsized":"2024-03-26T05:14:50.927722+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:05.728106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:14:05.728106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:07.919876+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":62,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.927320+0000","last_change":"2024-03-26T05:12:26.808345+0000","last_active":"2024-03-26T05:14:50.927320+0000","last_peered":"2024-03-26T05:14:50.927320+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:12:26.808345+0000","last_became_peered":"2024-03-26T05:12:26.808345+0000","last_unstale":"2024-03-26T05:14:50.927320+0000","last_undegraded":"2024-03-26T05:14:50.927320+0000","last_fullsized":"2024-03-26T05:12:26.791305+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:33.016072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:11:33.016072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:55.926195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"254'15","reported_seq":596,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:52.954314+0000","last_change":"2024-03-26T05:14:52.954459+0000","last_active":"2024-03-26T05:14:52.954314+0000","last_peered":"2024-03-26T05:14:52.954314+0000","last_clean":"2024-03-26T05:14:52.954314+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:14:52.954314+0000","last_undegraded":"2024-03-26T05:14:52.954314+0000","last_fullsized":"2024-03-26T05:14:52.954314+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'15","last_scrub_stamp":"2024-03-26T05:14:52.954230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:52.954230+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:16.600536+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":559,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.927559+0000","last_change":"2024-03-26T05:14:00.723869+0000","last_active":"2024-03-26T05:14:50.927559+0000","last_peered":"2024-03-26T05:14:50.927559+0000","last_clean":"2024-03-26T05:14:50.927559+0000","last_became_active":"2024-03-26T05:07:59.999703+0000","last_became_peered":"2024-03-26T05:07:59.999703+0000","last_unstale":"2024-03-26T05:14:50.927559+0000","last_undegraded":"2024-03-26T05:14:50.927559+0000","last_fullsized":"2024-03-26T05:14:50.927559+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:00.723630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:14:00.723630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:27.514443+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":579,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.939163+0000","last_change":"2024-03-26T05:14:00.861515+0000","last_active":"2024-03-26T05:14:50.939163+0000","last_peered":"2024-03-26T05:14:50.939163+0000","last_clean":"2024-03-26T05:14:50.939163+0000","last_became_active":"2024-03-26T05:08:00.001601+0000","last_became_peered":"2024-03-26T05:08:00.001601+0000","last_unstale":"2024-03-26T05:14:50.939163+0000","last_undegraded":"2024-03-26T05:14:50.939163+0000","last_fullsized":"2024-03-26T05:14:50.939163+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:00.861398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:14:00.861398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:08.452829+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":558,"reported_epoch":321,"state":"active+undersized","last_fresh":"2024-03-26T05:14:57.753604+0000","last_change":"2024-03-26T05:12:26.796125+0000","last_active":"2024-03-26T05:14:57.753604+0000","last_peered":"2024-03-26T05:14:57.753604+0000","last_clean":"2024-03-26T05:12:19.733579+0000","last_became_active":"2024-03-26T05:12:26.796125+0000","last_became_peered":"2024-03-26T05:12:26.796125+0000","last_unstale":"2024-03-26T05:14:57.753604+0000","last_undegraded":"2024-03-26T05:14:57.753604+0000","last_fullsized":"2024-03-26T05:12:26.789044+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:21.515294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:21.515294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:00.957544+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"254'24","reported_seq":603,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.753440+0000","last_change":"2024-03-26T05:13:56.875745+0000","last_active":"2024-03-26T05:14:57.753440+0000","last_peered":"2024-03-26T05:14:57.753440+0000","last_clean":"2024-03-26T05:14:57.753440+0000","last_became_active":"2024-03-26T05:00:24.942278+0000","last_became_peered":"2024-03-26T05:00:24.942278+0000","last_unstale":"2024-03-26T05:14:57.753440+0000","last_undegraded":"2024-03-26T05:14:57.753440+0000","last_fullsized":"2024-03-26T05:14:57.753440+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'24","last_scrub_stamp":"2024-03-26T05:13:56.875610+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:13:56.875610+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:09.653876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":559,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.705196+0000","last_change":"2024-03-26T05:12:26.804377+0000","last_active":"2024-03-26T05:14:50.705196+0000","last_peered":"2024-03-26T05:14:50.705196+0000","last_clean":"2024-03-26T05:12:19.739652+0000","last_became_active":"2024-03-26T05:12:26.804377+0000","last_became_peered":"2024-03-26T05:12:26.804377+0000","last_unstale":"2024-03-26T05:14:50.705196+0000","last_undegraded":"2024-03-26T05:14:50.705196+0000","last_fullsized":"2024-03-26T05:12:26.796240+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:17.975697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_clean_scrub_stamp":"2024-03-26T05:12:17.975697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:50.990144+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"216'10","reported_seq":335,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.574492+0000","last_change":"2024-03-26T05:14:20.704552+0000","last_active":"2024-03-26T05:14:51.574492+0000","last_peered":"2024-03-26T05:14:51.574492+0000","last_clean":"2024-03-26T05:14:51.574492+0000","last_became_active":"2024-03-26T05:08:00.005648+0000","last_became_peered":"2024-03-26T05:08:00.005648+0000","last_unstale":"2024-03-26T05:14:51.574492+0000","last_undegraded":"2024-03-26T05:14:51.574492+0000","last_fullsized":"2024-03-26T05:14:51.574492+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:14:20.704431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:20.704431+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:50.597417+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":64,"reported_epoch":321,"state":"active+undersized","last_fresh":"2024-03-26T05:14:57.753256+0000","last_change":"2024-03-26T05:12:26.792500+0000","last_active":"2024-03-26T05:14:57.753256+0000","last_peered":"2024-03-26T05:14:57.753256+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:12:26.792500+0000","last_became_peered":"2024-03-26T05:12:26.792500+0000","last_unstale":"2024-03-26T05:14:57.753256+0000","last_undegraded":"2024-03-26T05:14:57.753256+0000","last_fullsized":"2024-03-26T05:12:26.787718+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:01.962048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:01.962048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:58.000330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"254'32","reported_seq":612,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:50.928138+0000","last_change":"2024-03-26T05:12:26.823453+0000","last_active":"2024-03-26T05:14:50.928138+0000","last_peered":"2024-03-26T05:14:50.928138+0000","last_clean":"2024-03-26T05:12:19.742067+0000","last_became_active":"2024-03-26T05:12:26.823453+0000","last_became_peered":"2024-03-26T05:12:26.823453+0000","last_unstale":"2024-03-26T05:14:50.928138+0000","last_undegraded":"2024-03-26T05:12:26.801999+0000","last_fullsized":"2024-03-26T05:12:26.801890+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'32","last_scrub_stamp":"2024-03-26T05:12:07.585374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.585374+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:55.926211+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":563,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.754320+0000","last_change":"2024-03-26T05:13:17.125999+0000","last_active":"2024-03-26T05:14:57.754320+0000","last_peered":"2024-03-26T05:14:57.754320+0000","last_clean":"2024-03-26T05:14:57.754320+0000","last_became_active":"2024-03-26T05:07:59.942512+0000","last_became_peered":"2024-03-26T05:07:59.942512+0000","last_unstale":"2024-03-26T05:14:57.754320+0000","last_undegraded":"2024-03-26T05:14:57.754320+0000","last_fullsized":"2024-03-26T05:14:57.754320+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:17.125868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:13:17.125868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"294'10","reported_seq":734,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:50.928421+0000","last_change":"2024-03-26T05:12:26.813690+0000","last_active":"2024-03-26T05:14:50.928421+0000","last_peered":"2024-03-26T05:14:50.928421+0000","last_clean":"2024-03-26T05:12:23.559293+0000","last_became_active":"2024-03-26T05:12:26.813690+0000","last_became_peered":"2024-03-26T05:12:26.813690+0000","last_unstale":"2024-03-26T05:14:50.928421+0000","last_undegraded":"2024-03-26T05:12:26.798220+0000","last_fullsized":"2024-03-26T05:12:26.798102+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'9","last_scrub_stamp":"2024-03-26T05:10:30.592948+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:10:30.592948+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:55.926112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":572,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.704454+0000","last_change":"2024-03-26T05:12:26.804440+0000","last_active":"2024-03-26T05:14:50.704454+0000","last_peered":"2024-03-26T05:14:50.704454+0000","last_clean":"2024-03-26T05:12:19.738482+0000","last_became_active":"2024-03-26T05:12:26.804440+0000","last_became_peered":"2024-03-26T05:12:26.804440+0000","last_unstale":"2024-03-26T05:14:50.704454+0000","last_undegraded":"2024-03-26T05:14:50.704454+0000","last_fullsized":"2024-03-26T05:12:26.794341+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:59.989341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:59.989341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:50.990131+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"254'26","reported_seq":264,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:51.574214+0000","last_change":"2024-03-26T05:12:26.811984+0000","last_active":"2024-03-26T05:14:51.574214+0000","last_peered":"2024-03-26T05:14:51.574214+0000","last_clean":"2024-03-26T05:12:19.744509+0000","last_became_active":"2024-03-26T05:12:26.811984+0000","last_became_peered":"2024-03-26T05:12:26.811984+0000","last_unstale":"2024-03-26T05:14:51.574214+0000","last_undegraded":"2024-03-26T05:12:26.792151+0000","last_fullsized":"2024-03-26T05:12:26.792040+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:12:07.599746+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.599746+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:52.522612+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":566,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.941081+0000","last_change":"2024-03-26T05:12:26.794017+0000","last_active":"2024-03-26T05:14:50.941081+0000","last_peered":"2024-03-26T05:14:50.941081+0000","last_clean":"2024-03-26T05:12:19.754702+0000","last_became_active":"2024-03-26T05:12:26.794017+0000","last_became_peered":"2024-03-26T05:12:26.794017+0000","last_unstale":"2024-03-26T05:14:50.941081+0000","last_undegraded":"2024-03-26T05:14:50.941081+0000","last_fullsized":"2024-03-26T05:12:26.790840+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:13.142714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:13.142714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":545,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.928272+0000","last_change":"2024-03-26T05:12:26.825408+0000","last_active":"2024-03-26T05:14:50.928272+0000","last_peered":"2024-03-26T05:14:50.928272+0000","last_clean":"2024-03-26T05:12:19.766203+0000","last_became_active":"2024-03-26T05:12:26.825408+0000","last_became_peered":"2024-03-26T05:12:26.825408+0000","last_unstale":"2024-03-26T05:14:50.928272+0000","last_undegraded":"2024-03-26T05:14:50.928272+0000","last_fullsized":"2024-03-26T05:12:26.820690+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:19.364263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:11:19.364263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:55.926167+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":64,"reported_epoch":321,"state":"active+undersized","last_fresh":"2024-03-26T05:14:57.753727+0000","last_change":"2024-03-26T05:12:26.795370+0000","last_active":"2024-03-26T05:14:57.753727+0000","last_peered":"2024-03-26T05:14:57.753727+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:12:26.795370+0000","last_became_peered":"2024-03-26T05:12:26.795370+0000","last_unstale":"2024-03-26T05:14:57.753727+0000","last_undegraded":"2024-03-26T05:14:57.753727+0000","last_fullsized":"2024-03-26T05:12:26.788030+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.939576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.939576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:00.957585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"254'27","reported_seq":617,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:50.705513+0000","last_change":"2024-03-26T05:12:26.804729+0000","last_active":"2024-03-26T05:14:50.705513+0000","last_peered":"2024-03-26T05:14:50.705513+0000","last_clean":"2024-03-26T05:12:19.739995+0000","last_became_active":"2024-03-26T05:12:26.804729+0000","last_became_peered":"2024-03-26T05:12:26.804729+0000","last_unstale":"2024-03-26T05:14:50.705513+0000","last_undegraded":"2024-03-26T05:12:26.798275+0000","last_fullsized":"2024-03-26T05:12:26.798159+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:12:11.885908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:12:11.885908+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:50.990160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":65,"reported_epoch":321,"state":"active+undersized","last_fresh":"2024-03-26T05:14:57.754570+0000","last_change":"2024-03-26T05:12:26.795814+0000","last_active":"2024-03-26T05:14:57.754570+0000","last_peered":"2024-03-26T05:14:57.754570+0000","last_clean":"2024-03-26T05:00:27.943471+0000","last_became_active":"2024-03-26T05:12:26.795814+0000","last_became_peered":"2024-03-26T05:12:26.795814+0000","last_unstale":"2024-03-26T05:14:57.754570+0000","last_undegraded":"2024-03-26T05:14:57.754570+0000","last_fullsized":"2024-03-26T05:12:26.790166+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:07.006139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.006139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":248,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.574367+0000","last_change":"2024-03-26T05:14:04.756574+0000","last_active":"2024-03-26T05:14:51.574367+0000","last_peered":"2024-03-26T05:14:51.574367+0000","last_clean":"2024-03-26T05:14:51.574367+0000","last_became_active":"2024-03-26T05:07:59.997434+0000","last_became_peered":"2024-03-26T05:07:59.997434+0000","last_unstale":"2024-03-26T05:14:51.574367+0000","last_undegraded":"2024-03-26T05:14:51.574367+0000","last_fullsized":"2024-03-26T05:14:51.574367+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:04.756452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:04.756452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:24.605450+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":611,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.749360+0000","last_change":"2024-03-26T05:14:10.457609+0000","last_active":"2024-03-26T05:14:57.749360+0000","last_peered":"2024-03-26T05:14:57.749360+0000","last_clean":"2024-03-26T05:14:57.749360+0000","last_became_active":"2024-03-26T05:07:59.992217+0000","last_became_peered":"2024-03-26T05:07:59.992217+0000","last_unstale":"2024-03-26T05:14:57.749360+0000","last_undegraded":"2024-03-26T05:14:57.749360+0000","last_fullsized":"2024-03-26T05:14:57.749360+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:14:10.457383+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_clean_scrub_stamp":"2024-03-26T05:14:10.457383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:37.215967+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"254'18","reported_seq":580,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:50.928485+0000","last_change":"2024-03-26T05:12:26.826573+0000","last_active":"2024-03-26T05:14:50.928485+0000","last_peered":"2024-03-26T05:14:50.928485+0000","last_clean":"2024-03-26T05:12:19.766379+0000","last_became_active":"2024-03-26T05:12:26.826573+0000","last_became_peered":"2024-03-26T05:12:26.826573+0000","last_unstale":"2024-03-26T05:14:50.928485+0000","last_undegraded":"2024-03-26T05:12:26.820828+0000","last_fullsized":"2024-03-26T05:12:26.820722+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'18","last_scrub_stamp":"2024-03-26T05:10:36.560689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:10:36.560689+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"320'556","reported_seq":572,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:55.189174+0000","last_change":"2024-03-26T05:14:41.669766+0000","last_active":"2024-03-26T05:14:55.189174+0000","last_peered":"2024-03-26T05:14:55.189174+0000","last_clean":"2024-03-26T05:14:55.189174+0000","last_became_active":"2024-03-26T05:14:41.669464+0000","last_became_peered":"2024-03-26T05:14:41.669464+0000","last_unstale":"2024-03-26T05:14:55.189174+0000","last_undegraded":"2024-03-26T05:14:55.189174+0000","last_fullsized":"2024-03-26T05:14:55.189174+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:59.944387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36438016,"num_objects":556,"num_object_clones":0,"num_object_copies":1668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":556,"num_write_kb":35584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":253,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:54.651193+0000","last_change":"2024-03-26T05:14:54.651241+0000","last_active":"2024-03-26T05:14:54.651193+0000","last_peered":"2024-03-26T05:14:54.651193+0000","last_clean":"2024-03-26T05:14:54.651193+0000","last_became_active":"2024-03-26T05:08:00.000166+0000","last_became_peered":"2024-03-26T05:08:00.000166+0000","last_unstale":"2024-03-26T05:14:54.651193+0000","last_undegraded":"2024-03-26T05:14:54.651193+0000","last_fullsized":"2024-03-26T05:14:54.651193+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:14:54.651113+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:14:54.651113+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:08.393835+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":282,"reported_epoch":321,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:57.754015+0000","last_change":"2024-03-26T05:12:26.797516+0000","last_active":"2024-03-26T05:14:57.754015+0000","last_peered":"2024-03-26T05:14:57.754015+0000","last_clean":"2024-03-26T05:05:06.821667+0000","last_became_active":"2024-03-26T05:12:26.797516+0000","last_became_peered":"2024-03-26T05:12:26.797516+0000","last_unstale":"2024-03-26T05:14:57.754015+0000","last_undegraded":"2024-03-26T05:12:26.789206+0000","last_fullsized":"2024-03-26T05:12:26.789088+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.959120+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:00.957559+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"254'48","reported_seq":291,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:51.573407+0000","last_change":"2024-03-26T05:12:26.801013+0000","last_active":"2024-03-26T05:14:51.573407+0000","last_peered":"2024-03-26T05:14:51.573407+0000","last_clean":"2024-03-26T05:12:19.981236+0000","last_became_active":"2024-03-26T05:12:26.801013+0000","last_became_peered":"2024-03-26T05:12:26.801013+0000","last_unstale":"2024-03-26T05:14:51.573407+0000","last_undegraded":"2024-03-26T05:12:26.790279+0000","last_fullsized":"2024-03-26T05:12:26.790127+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'48","last_scrub_stamp":"2024-03-26T05:12:01.634109+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:12:01.634109+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:52.522639+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"294'9","reported_seq":121,"reported_epoch":321,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:57.749733+0000","last_change":"2024-03-26T05:12:26.798755+0000","last_active":"2024-03-26T05:14:57.749733+0000","last_peered":"2024-03-26T05:14:57.749733+0000","last_clean":"2024-03-26T05:00:30.162327+0000","last_became_active":"2024-03-26T05:12:26.798755+0000","last_became_peered":"2024-03-26T05:12:26.798755+0000","last_unstale":"2024-03-26T05:14:57.749733+0000","last_undegraded":"2024-03-26T05:12:26.793286+0000","last_fullsized":"2024-03-26T05:12:26.793174+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:12:03.961391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:03.961391+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"321'1469","reported_seq":1489,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:59.989477+0000","last_change":"2024-03-26T05:14:41.671297+0000","last_active":"2024-03-26T05:14:59.989477+0000","last_peered":"2024-03-26T05:14:59.989477+0000","last_clean":"2024-03-26T05:14:59.989477+0000","last_became_active":"2024-03-26T05:14:41.670908+0000","last_became_peered":"2024-03-26T05:14:41.670908+0000","last_unstale":"2024-03-26T05:14:59.989477+0000","last_undegraded":"2024-03-26T05:14:59.989477+0000","last_fullsized":"2024-03-26T05:14:59.989477+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":1469,"log_dups_size":0,"ondisk_log_size":1469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:52.422128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96272384,"num_objects":1469,"num_object_clones":0,"num_object_copies":4407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1469,"num_write_kb":94016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":561,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.031059+0000","last_change":"2024-03-26T05:14:51.031081+0000","last_active":"2024-03-26T05:14:51.031059+0000","last_peered":"2024-03-26T05:14:51.031059+0000","last_clean":"2024-03-26T05:14:51.031059+0000","last_became_active":"2024-03-26T05:08:55.556543+0000","last_became_peered":"2024-03-26T05:08:55.556543+0000","last_unstale":"2024-03-26T05:14:51.031059+0000","last_undegraded":"2024-03-26T05:14:51.031059+0000","last_fullsized":"2024-03-26T05:14:51.031059+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:51.030982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:51.030982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:16.872827+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":64,"reported_epoch":321,"state":"active+undersized","last_fresh":"2024-03-26T05:14:57.754141+0000","last_change":"2024-03-26T05:12:26.805999+0000","last_active":"2024-03-26T05:14:57.754141+0000","last_peered":"2024-03-26T05:14:57.754141+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:12:26.805999+0000","last_became_peered":"2024-03-26T05:12:26.805999+0000","last_unstale":"2024-03-26T05:14:57.754141+0000","last_undegraded":"2024-03-26T05:14:57.754141+0000","last_fullsized":"2024-03-26T05:12:26.788979+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:20.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:20.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:00.957597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"254'37","reported_seq":630,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.754072+0000","last_change":"2024-03-26T05:14:55.963922+0000","last_active":"2024-03-26T05:14:57.754072+0000","last_peered":"2024-03-26T05:14:57.754072+0000","last_clean":"2024-03-26T05:14:57.754072+0000","last_became_active":"2024-03-26T05:00:24.943639+0000","last_became_peered":"2024-03-26T05:00:24.943639+0000","last_unstale":"2024-03-26T05:14:57.754072+0000","last_undegraded":"2024-03-26T05:14:57.754072+0000","last_fullsized":"2024-03-26T05:14:57.754072+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-26T05:14:55.963773+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_clean_scrub_stamp":"2024-03-26T05:14:55.963773+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:02.526824+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":567,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.941426+0000","last_change":"2024-03-26T05:14:01.337189+0000","last_active":"2024-03-26T05:14:50.941426+0000","last_peered":"2024-03-26T05:14:50.941426+0000","last_clean":"2024-03-26T05:14:50.941426+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:14:50.941426+0000","last_undegraded":"2024-03-26T05:14:50.941426+0000","last_fullsized":"2024-03-26T05:14:50.941426+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:01.336959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.432732+0000","last_clean_scrub_stamp":"2024-03-26T05:14:01.336959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:12.641112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"320'723","reported_seq":741,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:56.055724+0000","last_change":"2024-03-26T05:14:41.671445+0000","last_active":"2024-03-26T05:14:56.055724+0000","last_peered":"2024-03-26T05:14:56.055724+0000","last_clean":"2024-03-26T05:14:56.055724+0000","last_became_active":"2024-03-26T05:14:41.671159+0000","last_became_peered":"2024-03-26T05:14:41.671159+0000","last_unstale":"2024-03-26T05:14:56.055724+0000","last_undegraded":"2024-03-26T05:14:56.055724+0000","last_fullsized":"2024-03-26T05:14:56.055724+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":723,"log_dups_size":0,"ondisk_log_size":723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:00.267592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47382528,"num_objects":723,"num_object_clones":0,"num_object_copies":2169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":723,"num_write_kb":46272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":63,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.941482+0000","last_change":"2024-03-26T05:12:26.804715+0000","last_active":"2024-03-26T05:14:50.941482+0000","last_peered":"2024-03-26T05:14:50.941482+0000","last_clean":"2024-03-26T05:00:27.943471+0000","last_became_active":"2024-03-26T05:12:26.804715+0000","last_became_peered":"2024-03-26T05:12:26.804715+0000","last_unstale":"2024-03-26T05:14:50.941482+0000","last_undegraded":"2024-03-26T05:14:50.941482+0000","last_fullsized":"2024-03-26T05:12:26.795708+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:05.986499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_clean_scrub_stamp":"2024-03-26T05:12:05.986499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":587,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.941620+0000","last_change":"2024-03-26T05:14:10.201428+0000","last_active":"2024-03-26T05:14:50.941620+0000","last_peered":"2024-03-26T05:14:50.941620+0000","last_clean":"2024-03-26T05:14:50.941620+0000","last_became_active":"2024-03-26T05:08:00.000009+0000","last_became_peered":"2024-03-26T05:08:00.000009+0000","last_unstale":"2024-03-26T05:14:50.941620+0000","last_undegraded":"2024-03-26T05:14:50.941620+0000","last_fullsized":"2024-03-26T05:14:50.941620+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:10.201320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:10.201320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:16.798697+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":385,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:51.573445+0000","last_change":"2024-03-26T05:12:26.802470+0000","last_active":"2024-03-26T05:14:51.573445+0000","last_peered":"2024-03-26T05:14:51.573445+0000","last_clean":"2024-03-26T05:07:55.690606+0000","last_became_active":"2024-03-26T05:12:26.802470+0000","last_became_peered":"2024-03-26T05:12:26.802470+0000","last_unstale":"2024-03-26T05:14:51.573445+0000","last_undegraded":"2024-03-26T05:14:51.573445+0000","last_fullsized":"2024-03-26T05:12:26.791511+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:27.963495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:27.963495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:52.522599+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"254'41","reported_seq":634,"reported_epoch":321,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:57.751050+0000","last_change":"2024-03-26T05:12:26.816012+0000","last_active":"2024-03-26T05:14:57.751050+0000","last_peered":"2024-03-26T05:14:57.751050+0000","last_clean":"2024-03-26T05:12:19.743443+0000","last_became_active":"2024-03-26T05:12:26.816012+0000","last_became_peered":"2024-03-26T05:12:26.816012+0000","last_unstale":"2024-03-26T05:14:57.751050+0000","last_undegraded":"2024-03-26T05:12:26.791532+0000","last_fullsized":"2024-03-26T05:12:26.791418+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'41","last_scrub_stamp":"2024-03-26T05:11:24.658818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:11:24.658818+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":10}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"320'636","reported_seq":654,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:55.405795+0000","last_change":"2024-03-26T05:14:41.674537+0000","last_active":"2024-03-26T05:14:55.405795+0000","last_peered":"2024-03-26T05:14:55.405795+0000","last_clean":"2024-03-26T05:14:55.405795+0000","last_became_active":"2024-03-26T05:14:41.674204+0000","last_became_peered":"2024-03-26T05:14:41.674204+0000","last_unstale":"2024-03-26T05:14:55.405795+0000","last_undegraded":"2024-03-26T05:14:55.405795+0000","last_fullsized":"2024-03-26T05:14:55.405795+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:40.990314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41680896,"num_objects":636,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":636,"num_write_kb":40704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":552,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.704518+0000","last_change":"2024-03-26T05:12:26.799187+0000","last_active":"2024-03-26T05:14:50.704518+0000","last_peered":"2024-03-26T05:14:50.704518+0000","last_clean":"2024-03-26T05:12:19.739391+0000","last_became_active":"2024-03-26T05:12:26.799187+0000","last_became_peered":"2024-03-26T05:12:26.799187+0000","last_unstale":"2024-03-26T05:14:50.704518+0000","last_undegraded":"2024-03-26T05:14:50.704518+0000","last_fullsized":"2024-03-26T05:12:26.794340+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:30.164093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:30.164093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:50.990056+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":575,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:52.371871+0000","last_change":"2024-03-26T05:14:52.371890+0000","last_active":"2024-03-26T05:14:52.371871+0000","last_peered":"2024-03-26T05:14:52.371871+0000","last_clean":"2024-03-26T05:14:52.371871+0000","last_became_active":"2024-03-26T05:00:26.955917+0000","last_became_peered":"2024-03-26T05:00:26.955917+0000","last_unstale":"2024-03-26T05:14:52.371871+0000","last_undegraded":"2024-03-26T05:14:52.371871+0000","last_fullsized":"2024-03-26T05:14:52.371871+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:52.371799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:52.371799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:16.058063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":249,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.574173+0000","last_change":"2024-03-26T05:14:00.685481+0000","last_active":"2024-03-26T05:14:51.574173+0000","last_peered":"2024-03-26T05:14:51.574173+0000","last_clean":"2024-03-26T05:14:51.574173+0000","last_became_active":"2024-03-26T05:07:59.959978+0000","last_became_peered":"2024-03-26T05:07:59.959978+0000","last_unstale":"2024-03-26T05:14:51.574173+0000","last_undegraded":"2024-03-26T05:14:51.574173+0000","last_fullsized":"2024-03-26T05:14:51.574173+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:00.685364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:00.685364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:17.678469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"254'21","reported_seq":610,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.705903+0000","last_change":"2024-03-26T05:14:16.094396+0000","last_active":"2024-03-26T05:14:50.705903+0000","last_peered":"2024-03-26T05:14:50.705903+0000","last_clean":"2024-03-26T05:14:50.705903+0000","last_became_active":"2024-03-26T05:00:24.935733+0000","last_became_peered":"2024-03-26T05:00:24.935733+0000","last_unstale":"2024-03-26T05:14:50.705903+0000","last_undegraded":"2024-03-26T05:14:50.705903+0000","last_fullsized":"2024-03-26T05:14:50.705903+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:14:16.094249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.094249+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:44.605892+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"321'1465","reported_seq":1485,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:59.989748+0000","last_change":"2024-03-26T05:14:41.671380+0000","last_active":"2024-03-26T05:14:59.989748+0000","last_peered":"2024-03-26T05:14:59.989748+0000","last_clean":"2024-03-26T05:14:59.989748+0000","last_became_active":"2024-03-26T05:14:41.671031+0000","last_became_peered":"2024-03-26T05:14:41.671031+0000","last_unstale":"2024-03-26T05:14:59.989748+0000","last_undegraded":"2024-03-26T05:14:59.989748+0000","last_fullsized":"2024-03-26T05:14:59.989748+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:07.027772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96010240,"num_objects":1465,"num_object_clones":0,"num_object_copies":4395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1465,"num_write_kb":93760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":542,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.927087+0000","last_change":"2024-03-26T05:12:26.806592+0000","last_active":"2024-03-26T05:14:50.927087+0000","last_peered":"2024-03-26T05:14:50.927087+0000","last_clean":"2024-03-26T05:12:19.741894+0000","last_became_active":"2024-03-26T05:12:26.806592+0000","last_became_peered":"2024-03-26T05:12:26.806592+0000","last_unstale":"2024-03-26T05:14:50.927087+0000","last_undegraded":"2024-03-26T05:14:50.927087+0000","last_fullsized":"2024-03-26T05:12:26.791171+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:31.518660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:31.518660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:55.926155+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":253,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.574192+0000","last_change":"2024-03-26T05:14:47.525396+0000","last_active":"2024-03-26T05:14:51.574192+0000","last_peered":"2024-03-26T05:14:51.574192+0000","last_clean":"2024-03-26T05:14:51.574192+0000","last_became_active":"2024-03-26T05:07:59.998511+0000","last_became_peered":"2024-03-26T05:07:59.998511+0000","last_unstale":"2024-03-26T05:14:51.574192+0000","last_undegraded":"2024-03-26T05:14:51.574192+0000","last_fullsized":"2024-03-26T05:14:51.574192+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:14:47.525284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:14:47.525284+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:53.176809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":564,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.753209+0000","last_change":"2024-03-26T05:14:02.881924+0000","last_active":"2024-03-26T05:14:57.753209+0000","last_peered":"2024-03-26T05:14:57.753209+0000","last_clean":"2024-03-26T05:14:57.753209+0000","last_became_active":"2024-03-26T05:00:22.770487+0000","last_became_peered":"2024-03-26T05:00:22.770487+0000","last_unstale":"2024-03-26T05:14:57.753209+0000","last_undegraded":"2024-03-26T05:14:57.753209+0000","last_fullsized":"2024-03-26T05:14:57.753209+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:02.881800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:02.881800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:05.327198+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"254'46","reported_seq":827,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.939140+0000","last_change":"2024-03-26T05:14:45.973151+0000","last_active":"2024-03-26T05:14:50.939140+0000","last_peered":"2024-03-26T05:14:50.939140+0000","last_clean":"2024-03-26T05:14:50.939140+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:14:50.939140+0000","last_undegraded":"2024-03-26T05:14:50.939140+0000","last_fullsized":"2024-03-26T05:14:50.939140+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'46","last_scrub_stamp":"2024-03-26T05:14:45.973027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:45.973027+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:47.493198+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":569,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.705924+0000","last_change":"2024-03-26T05:14:14.131676+0000","last_active":"2024-03-26T05:14:50.705924+0000","last_peered":"2024-03-26T05:14:50.705924+0000","last_clean":"2024-03-26T05:14:50.705924+0000","last_became_active":"2024-03-26T05:00:26.964475+0000","last_became_peered":"2024-03-26T05:00:26.964475+0000","last_unstale":"2024-03-26T05:14:50.705924+0000","last_undegraded":"2024-03-26T05:14:50.705924+0000","last_fullsized":"2024-03-26T05:14:50.705924+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:14.131427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:14.131427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:29.840324+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"320'593","reported_seq":608,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:55.183981+0000","last_change":"2024-03-26T05:14:52.609100+0000","last_active":"2024-03-26T05:14:55.183981+0000","last_peered":"2024-03-26T05:14:55.183981+0000","last_clean":"2024-03-26T05:14:40.645004+0000","last_became_active":"2024-03-26T05:14:41.666096+0000","last_became_peered":"2024-03-26T05:14:41.666096+0000","last_unstale":"2024-03-26T05:14:55.183981+0000","last_undegraded":"2024-03-26T05:14:52.604612+0000","last_fullsized":"2024-03-26T05:14:41.653791+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:47.776924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38862848,"num_objects":593,"num_object_clones":0,"num_object_copies":1779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":593,"num_write_kb":37952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":593}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":65,"reported_epoch":321,"state":"active+undersized","last_fresh":"2024-03-26T05:14:57.754376+0000","last_change":"2024-03-26T05:12:26.800058+0000","last_active":"2024-03-26T05:14:57.754376+0000","last_peered":"2024-03-26T05:14:57.754376+0000","last_clean":"2024-03-26T05:00:27.943471+0000","last_became_active":"2024-03-26T05:12:26.800058+0000","last_became_peered":"2024-03-26T05:12:26.800058+0000","last_unstale":"2024-03-26T05:14:57.754376+0000","last_undegraded":"2024-03-26T05:14:57.754376+0000","last_fullsized":"2024-03-26T05:12:26.791101+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:20.054579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:20.054579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":568,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.939091+0000","last_change":"2024-03-26T05:14:11.975264+0000","last_active":"2024-03-26T05:14:50.939091+0000","last_peered":"2024-03-26T05:14:50.939091+0000","last_clean":"2024-03-26T05:14:50.939091+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:14:50.939091+0000","last_undegraded":"2024-03-26T05:14:50.939091+0000","last_fullsized":"2024-03-26T05:14:50.939091+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:11.975035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:11.975035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:35.551105+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":584,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.941385+0000","last_change":"2024-03-26T05:13:57.369670+0000","last_active":"2024-03-26T05:14:50.941385+0000","last_peered":"2024-03-26T05:14:50.941385+0000","last_clean":"2024-03-26T05:14:50.941385+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:14:50.941385+0000","last_undegraded":"2024-03-26T05:14:50.941385+0000","last_fullsized":"2024-03-26T05:14:50.941385+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:57.369522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:13:57.369522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:57.970765+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"254'23","reported_seq":390,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:55.315382+0000","last_change":"2024-03-26T05:14:55.315443+0000","last_active":"2024-03-26T05:14:55.315382+0000","last_peered":"2024-03-26T05:14:55.315382+0000","last_clean":"2024-03-26T05:14:55.315382+0000","last_became_active":"2024-03-26T05:08:00.001398+0000","last_became_peered":"2024-03-26T05:08:00.001398+0000","last_unstale":"2024-03-26T05:14:55.315382+0000","last_undegraded":"2024-03-26T05:14:55.315382+0000","last_fullsized":"2024-03-26T05:14:55.315382+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'23","last_scrub_stamp":"2024-03-26T05:14:55.315296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:55.315296+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:56.284400+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":582,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.942148+0000","last_change":"2024-03-26T05:12:26.801678+0000","last_active":"2024-03-26T05:14:50.942148+0000","last_peered":"2024-03-26T05:14:50.942148+0000","last_clean":"2024-03-26T05:12:19.754479+0000","last_became_active":"2024-03-26T05:12:26.801678+0000","last_became_peered":"2024-03-26T05:12:26.801678+0000","last_unstale":"2024-03-26T05:14:50.942148+0000","last_undegraded":"2024-03-26T05:14:50.942148+0000","last_fullsized":"2024-03-26T05:12:26.792427+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:07.171121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.171121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"321'1446","reported_seq":1466,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:59.980972+0000","last_change":"2024-03-26T05:14:41.675109+0000","last_active":"2024-03-26T05:14:59.980972+0000","last_peered":"2024-03-26T05:14:59.980972+0000","last_clean":"2024-03-26T05:14:59.980972+0000","last_became_active":"2024-03-26T05:14:41.674748+0000","last_became_peered":"2024-03-26T05:14:41.674748+0000","last_unstale":"2024-03-26T05:14:59.980972+0000","last_undegraded":"2024-03-26T05:14:59.980972+0000","last_fullsized":"2024-03-26T05:14:59.980972+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":1446,"log_dups_size":0,"ondisk_log_size":1446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:44.710212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94765056,"num_objects":1446,"num_object_clones":0,"num_object_copies":4338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1446,"num_write_kb":92544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":249,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.574576+0000","last_change":"2024-03-26T05:13:57.791760+0000","last_active":"2024-03-26T05:14:51.574576+0000","last_peered":"2024-03-26T05:14:51.574576+0000","last_clean":"2024-03-26T05:14:51.574576+0000","last_became_active":"2024-03-26T05:07:59.959853+0000","last_became_peered":"2024-03-26T05:07:59.959853+0000","last_unstale":"2024-03-26T05:14:51.574576+0000","last_undegraded":"2024-03-26T05:14:51.574576+0000","last_fullsized":"2024-03-26T05:14:51.574576+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:57.791637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:03.848208+0000","last_clean_scrub_stamp":"2024-03-26T05:13:57.791637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:12.672955+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":365,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:50.704804+0000","last_change":"2024-03-26T05:12:26.799253+0000","last_active":"2024-03-26T05:14:50.704804+0000","last_peered":"2024-03-26T05:14:50.704804+0000","last_clean":"2024-03-26T05:01:05.801809+0000","last_became_active":"2024-03-26T05:12:26.799253+0000","last_became_peered":"2024-03-26T05:12:26.799253+0000","last_unstale":"2024-03-26T05:14:50.704804+0000","last_undegraded":"2024-03-26T05:12:26.795468+0000","last_fullsized":"2024-03-26T05:12:26.795368+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:50.990190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":579,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.927266+0000","last_change":"2024-03-26T05:14:09.841525+0000","last_active":"2024-03-26T05:14:50.927266+0000","last_peered":"2024-03-26T05:14:50.927266+0000","last_clean":"2024-03-26T05:14:50.927266+0000","last_became_active":"2024-03-26T05:07:59.960061+0000","last_became_peered":"2024-03-26T05:07:59.960061+0000","last_unstale":"2024-03-26T05:14:50.927266+0000","last_undegraded":"2024-03-26T05:14:50.927266+0000","last_fullsized":"2024-03-26T05:14:50.927266+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:09.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:09.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:27.701986+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"294'37","reported_seq":800,"reported_epoch":321,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:57.754277+0000","last_change":"2024-03-26T05:12:26.796936+0000","last_active":"2024-03-26T05:14:57.754277+0000","last_peered":"2024-03-26T05:14:57.754277+0000","last_clean":"2024-03-26T05:12:23.556509+0000","last_became_active":"2024-03-26T05:12:26.796936+0000","last_became_peered":"2024-03-26T05:12:26.796936+0000","last_unstale":"2024-03-26T05:14:57.754277+0000","last_undegraded":"2024-03-26T05:12:26.790649+0000","last_fullsized":"2024-03-26T05:12:26.790539+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'36","last_scrub_stamp":"2024-03-26T05:11:20.689640+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:11:20.689640+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":393,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.941933+0000","last_change":"2024-03-26T05:12:26.809399+0000","last_active":"2024-03-26T05:14:50.941933+0000","last_peered":"2024-03-26T05:14:50.941933+0000","last_clean":"2024-03-26T05:07:55.694924+0000","last_became_active":"2024-03-26T05:12:26.809399+0000","last_became_peered":"2024-03-26T05:12:26.809399+0000","last_unstale":"2024-03-26T05:14:50.941933+0000","last_undegraded":"2024-03-26T05:14:50.941933+0000","last_fullsized":"2024-03-26T05:12:26.791110+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:14.948209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:14.948209+0000","last_clean_scrub_stamp":"2024-03-26T05:12:14.948209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"320'737","reported_seq":751,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:56.045266+0000","last_change":"2024-03-26T05:14:52.615711+0000","last_active":"2024-03-26T05:14:56.045266+0000","last_peered":"2024-03-26T05:14:56.045266+0000","last_clean":"2024-03-26T05:14:40.645004+0000","last_became_active":"2024-03-26T05:14:41.673577+0000","last_became_peered":"2024-03-26T05:14:41.673577+0000","last_unstale":"2024-03-26T05:14:56.045266+0000","last_undegraded":"2024-03-26T05:14:52.605102+0000","last_fullsized":"2024-03-26T05:14:41.663663+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:05.664145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48300032,"num_objects":737,"num_object_clones":0,"num_object_copies":2211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":737,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":737,"num_write_kb":47168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":737}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":575,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.938863+0000","last_change":"2024-03-26T05:14:16.082365+0000","last_active":"2024-03-26T05:14:50.938863+0000","last_peered":"2024-03-26T05:14:50.938863+0000","last_clean":"2024-03-26T05:14:50.938863+0000","last_became_active":"2024-03-26T05:08:00.001657+0000","last_became_peered":"2024-03-26T05:08:00.001657+0000","last_unstale":"2024-03-26T05:14:50.938863+0000","last_undegraded":"2024-03-26T05:14:50.938863+0000","last_fullsized":"2024-03-26T05:14:50.938863+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:16.082247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.082247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:29.758135+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":467,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.753938+0000","last_change":"2024-03-26T05:14:16.863466+0000","last_active":"2024-03-26T05:14:57.753938+0000","last_peered":"2024-03-26T05:14:57.753938+0000","last_clean":"2024-03-26T05:14:57.753938+0000","last_became_active":"2024-03-26T05:07:59.992489+0000","last_became_peered":"2024-03-26T05:07:59.992489+0000","last_unstale":"2024-03-26T05:14:57.753938+0000","last_undegraded":"2024-03-26T05:14:57.753938+0000","last_fullsized":"2024-03-26T05:14:57.753938+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:16.863389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.863389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:29.694667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":355,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:54.262789+0000","last_change":"2024-03-26T05:12:26.810246+0000","last_active":"2024-03-26T05:14:54.262789+0000","last_peered":"2024-03-26T05:14:54.262789+0000","last_clean":"2024-03-26T05:01:36.779364+0000","last_became_active":"2024-03-26T05:12:26.810246+0000","last_became_peered":"2024-03-26T05:12:26.810246+0000","last_unstale":"2024-03-26T05:14:54.262789+0000","last_undegraded":"2024-03-26T05:12:26.796272+0000","last_fullsized":"2024-03-26T05:12:26.796137+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:12:02.961845+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:12:02.961845+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:57.947232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":214,"num_read_kb":209,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"254'30","reported_seq":655,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.705401+0000","last_change":"2024-03-26T05:14:10.147920+0000","last_active":"2024-03-26T05:14:50.705401+0000","last_peered":"2024-03-26T05:14:50.705401+0000","last_clean":"2024-03-26T05:14:50.705401+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:14:50.705401+0000","last_undegraded":"2024-03-26T05:14:50.705401+0000","last_fullsized":"2024-03-26T05:14:50.705401+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'30","last_scrub_stamp":"2024-03-26T05:14:10.147766+0000","last_deep_scrub":"187'18","last_deep_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_clean_scrub_stamp":"2024-03-26T05:14:10.147766+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:19.886647+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":249,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.574492+0000","last_change":"2024-03-26T05:14:12.823709+0000","last_active":"2024-03-26T05:14:51.574492+0000","last_peered":"2024-03-26T05:14:51.574492+0000","last_clean":"2024-03-26T05:14:51.574492+0000","last_became_active":"2024-03-26T05:07:59.999990+0000","last_became_peered":"2024-03-26T05:07:59.999990+0000","last_unstale":"2024-03-26T05:14:51.574492+0000","last_undegraded":"2024-03-26T05:14:51.574492+0000","last_fullsized":"2024-03-26T05:14:51.574492+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:12.823589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:12.823589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:37.266929+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"321'1439","reported_seq":1459,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:59.981173+0000","last_change":"2024-03-26T05:14:41.674398+0000","last_active":"2024-03-26T05:14:59.981173+0000","last_peered":"2024-03-26T05:14:59.981173+0000","last_clean":"2024-03-26T05:14:59.981173+0000","last_became_active":"2024-03-26T05:14:41.673934+0000","last_became_peered":"2024-03-26T05:14:41.673934+0000","last_unstale":"2024-03-26T05:14:59.981173+0000","last_undegraded":"2024-03-26T05:14:59.981173+0000","last_fullsized":"2024-03-26T05:14:59.981173+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":1439,"log_dups_size":0,"ondisk_log_size":1439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:41.638704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94306304,"num_objects":1439,"num_object_clones":0,"num_object_copies":4317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1439,"num_write_kb":92096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":555,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.938573+0000","last_change":"2024-03-26T05:14:07.008841+0000","last_active":"2024-03-26T05:14:50.938573+0000","last_peered":"2024-03-26T05:14:50.938573+0000","last_clean":"2024-03-26T05:14:50.938573+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:14:50.938573+0000","last_undegraded":"2024-03-26T05:14:50.938573+0000","last_fullsized":"2024-03-26T05:14:50.938573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:07.008713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:14:07.008713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:36.536700+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":587,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.939088+0000","last_change":"2024-03-26T05:14:46.997307+0000","last_active":"2024-03-26T05:14:50.939088+0000","last_peered":"2024-03-26T05:14:50.939088+0000","last_clean":"2024-03-26T05:14:50.939088+0000","last_became_active":"2024-03-26T05:08:00.003099+0000","last_became_peered":"2024-03-26T05:08:00.003099+0000","last_unstale":"2024-03-26T05:14:50.939088+0000","last_undegraded":"2024-03-26T05:14:50.939088+0000","last_fullsized":"2024-03-26T05:14:50.939088+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:46.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:11:29.514915+0000","last_clean_scrub_stamp":"2024-03-26T05:14:46.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:02.601058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"254'44","reported_seq":692,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:50.704758+0000","last_change":"2024-03-26T05:12:26.806879+0000","last_active":"2024-03-26T05:14:50.704758+0000","last_peered":"2024-03-26T05:14:50.704758+0000","last_clean":"2024-03-26T05:12:19.738884+0000","last_became_active":"2024-03-26T05:12:26.806879+0000","last_became_peered":"2024-03-26T05:12:26.806879+0000","last_unstale":"2024-03-26T05:14:50.704758+0000","last_undegraded":"2024-03-26T05:12:26.800582+0000","last_fullsized":"2024-03-26T05:12:26.800471+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'44","last_scrub_stamp":"2024-03-26T05:11:29.126183+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.126183+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:50.990090+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":588,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.751067+0000","last_change":"2024-03-26T05:13:58.537222+0000","last_active":"2024-03-26T05:14:57.751067+0000","last_peered":"2024-03-26T05:14:57.751067+0000","last_clean":"2024-03-26T05:14:57.751067+0000","last_became_active":"2024-03-26T05:07:59.991484+0000","last_became_peered":"2024-03-26T05:07:59.991484+0000","last_unstale":"2024-03-26T05:14:57.751067+0000","last_undegraded":"2024-03-26T05:14:57.751067+0000","last_fullsized":"2024-03-26T05:14:57.751067+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:58.537133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:13:58.537133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:06.112195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"321'1463","reported_seq":1480,"reported_epoch":321,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:59.942233+0000","last_change":"2024-03-26T05:14:52.611572+0000","last_active":"2024-03-26T05:14:59.942233+0000","last_peered":"2024-03-26T05:14:59.942233+0000","last_clean":"2024-03-26T05:14:40.645004+0000","last_became_active":"2024-03-26T05:14:41.667075+0000","last_became_peered":"2024-03-26T05:14:41.667075+0000","last_unstale":"2024-03-26T05:14:59.942233+0000","last_undegraded":"2024-03-26T05:14:52.604992+0000","last_fullsized":"2024-03-26T05:14:41.661842+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":1463,"log_dups_size":0,"ondisk_log_size":1463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:04.634087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95879168,"num_objects":1463,"num_object_clones":0,"num_object_copies":4389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1463,"num_write_kb":93632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1463}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":566,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.704708+0000","last_change":"2024-03-26T05:14:07.165543+0000","last_active":"2024-03-26T05:14:50.704708+0000","last_peered":"2024-03-26T05:14:50.704708+0000","last_clean":"2024-03-26T05:14:50.704708+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:14:50.704708+0000","last_undegraded":"2024-03-26T05:14:50.704708+0000","last_fullsized":"2024-03-26T05:14:50.704708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:07.165430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:07.165430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:24.631985+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":249,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.574255+0000","last_change":"2024-03-26T05:14:15.770146+0000","last_active":"2024-03-26T05:14:51.574255+0000","last_peered":"2024-03-26T05:14:51.574255+0000","last_clean":"2024-03-26T05:14:51.574255+0000","last_became_active":"2024-03-26T05:07:59.960768+0000","last_became_peered":"2024-03-26T05:07:59.960768+0000","last_unstale":"2024-03-26T05:14:51.574255+0000","last_undegraded":"2024-03-26T05:14:51.574255+0000","last_fullsized":"2024-03-26T05:14:51.574255+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:15.770027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:15.770027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:23.821205+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"254'26","reported_seq":624,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.941685+0000","last_change":"2024-03-26T05:14:05.219911+0000","last_active":"2024-03-26T05:14:50.941685+0000","last_peered":"2024-03-26T05:14:50.941685+0000","last_clean":"2024-03-26T05:14:50.941685+0000","last_became_active":"2024-03-26T05:07:59.960351+0000","last_became_peered":"2024-03-26T05:07:59.960351+0000","last_unstale":"2024-03-26T05:14:50.941685+0000","last_undegraded":"2024-03-26T05:14:50.941685+0000","last_fullsized":"2024-03-26T05:14:50.941685+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:14:05.219767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:05.219767+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:31.449224+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"216'11","reported_seq":782,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.706062+0000","last_change":"2024-03-26T05:14:48.976961+0000","last_active":"2024-03-26T05:14:50.706062+0000","last_peered":"2024-03-26T05:14:50.706062+0000","last_clean":"2024-03-26T05:14:50.706062+0000","last_became_active":"2024-03-26T05:07:59.994682+0000","last_became_peered":"2024-03-26T05:07:59.994682+0000","last_unstale":"2024-03-26T05:14:50.706062+0000","last_undegraded":"2024-03-26T05:14:50.706062+0000","last_fullsized":"2024-03-26T05:14:50.706062+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:14:48.976873+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:14:48.976873+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:04.257132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"321'1372","reported_seq":1392,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:59.989200+0000","last_change":"2024-03-26T05:14:41.668609+0000","last_active":"2024-03-26T05:14:59.989200+0000","last_peered":"2024-03-26T05:14:59.989200+0000","last_clean":"2024-03-26T05:14:59.989200+0000","last_became_active":"2024-03-26T05:14:41.668381+0000","last_became_peered":"2024-03-26T05:14:41.668381+0000","last_unstale":"2024-03-26T05:14:59.989200+0000","last_undegraded":"2024-03-26T05:14:59.989200+0000","last_fullsized":"2024-03-26T05:14:59.989200+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":1372,"log_dups_size":0,"ondisk_log_size":1372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:00.958605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89915392,"num_objects":1372,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1372,"num_write_kb":87808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":65,"reported_epoch":321,"state":"active+undersized","last_fresh":"2024-03-26T05:14:57.751987+0000","last_change":"2024-03-26T05:12:26.798078+0000","last_active":"2024-03-26T05:14:57.751987+0000","last_peered":"2024-03-26T05:14:57.751987+0000","last_clean":"2024-03-26T05:00:27.943471+0000","last_became_active":"2024-03-26T05:12:26.798078+0000","last_became_peered":"2024-03-26T05:12:26.798078+0000","last_unstale":"2024-03-26T05:14:57.751987+0000","last_undegraded":"2024-03-26T05:14:57.751987+0000","last_fullsized":"2024-03-26T05:12:26.794803+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:22.910006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:22.910006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":586,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.705409+0000","last_change":"2024-03-26T05:14:12.114121+0000","last_active":"2024-03-26T05:14:50.705409+0000","last_peered":"2024-03-26T05:14:50.705409+0000","last_clean":"2024-03-26T05:14:50.705409+0000","last_became_active":"2024-03-26T05:00:22.773594+0000","last_became_peered":"2024-03-26T05:00:22.773594+0000","last_unstale":"2024-03-26T05:14:50.705409+0000","last_undegraded":"2024-03-26T05:14:50.705409+0000","last_fullsized":"2024-03-26T05:14:50.705409+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:12.113993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:12.113993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:32.590913+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"254'34","reported_seq":661,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.753628+0000","last_change":"2024-03-26T05:12:19.310247+0000","last_active":"2024-03-26T05:14:57.753628+0000","last_peered":"2024-03-26T05:14:57.753628+0000","last_clean":"2024-03-26T05:14:57.753628+0000","last_became_active":"2024-03-26T05:07:59.942510+0000","last_became_peered":"2024-03-26T05:07:59.942510+0000","last_unstale":"2024-03-26T05:14:57.753628+0000","last_undegraded":"2024-03-26T05:14:57.753628+0000","last_fullsized":"2024-03-26T05:14:57.753628+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'34","last_scrub_stamp":"2024-03-26T05:12:19.309684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:12:19.309684+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":735,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.927651+0000","last_change":"2024-03-26T05:13:56.721364+0000","last_active":"2024-03-26T05:14:50.927651+0000","last_peered":"2024-03-26T05:14:50.927651+0000","last_clean":"2024-03-26T05:14:50.927651+0000","last_became_active":"2024-03-26T05:00:26.957486+0000","last_became_peered":"2024-03-26T05:00:26.957486+0000","last_unstale":"2024-03-26T05:14:50.927651+0000","last_undegraded":"2024-03-26T05:14:50.927651+0000","last_fullsized":"2024-03-26T05:14:50.927651+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:13:56.721250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:13:56.721250+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:25.936253+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"321'1389","reported_seq":1409,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:59.920388+0000","last_change":"2024-03-26T05:14:41.672323+0000","last_active":"2024-03-26T05:14:59.920388+0000","last_peered":"2024-03-26T05:14:59.920388+0000","last_clean":"2024-03-26T05:14:59.920388+0000","last_became_active":"2024-03-26T05:14:41.671997+0000","last_became_peered":"2024-03-26T05:14:41.671997+0000","last_unstale":"2024-03-26T05:14:59.920388+0000","last_undegraded":"2024-03-26T05:14:59.920388+0000","last_fullsized":"2024-03-26T05:14:59.920388+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":1389,"log_dups_size":0,"ondisk_log_size":1389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:54.840276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91029504,"num_objects":1389,"num_object_clones":0,"num_object_copies":4167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1389,"num_write_kb":88896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":559,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.753234+0000","last_change":"2024-03-26T05:13:54.923843+0000","last_active":"2024-03-26T05:14:57.753234+0000","last_peered":"2024-03-26T05:14:57.753234+0000","last_clean":"2024-03-26T05:14:57.753234+0000","last_became_active":"2024-03-26T05:00:28.980498+0000","last_became_peered":"2024-03-26T05:00:28.980498+0000","last_unstale":"2024-03-26T05:14:57.753234+0000","last_undegraded":"2024-03-26T05:14:57.753234+0000","last_fullsized":"2024-03-26T05:14:57.753234+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:54.923744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:13:54.923744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":554,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.927103+0000","last_change":"2024-03-26T05:12:26.821873+0000","last_active":"2024-03-26T05:14:50.927103+0000","last_peered":"2024-03-26T05:14:50.927103+0000","last_clean":"2024-03-26T05:12:19.767293+0000","last_became_active":"2024-03-26T05:12:26.821873+0000","last_became_peered":"2024-03-26T05:12:26.821873+0000","last_unstale":"2024-03-26T05:14:50.927103+0000","last_undegraded":"2024-03-26T05:14:50.927103+0000","last_fullsized":"2024-03-26T05:12:26.808949+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:17.750535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:17.750535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:55.926239+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"254'52","reported_seq":643,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:50.926902+0000","last_change":"2024-03-26T05:12:26.811802+0000","last_active":"2024-03-26T05:14:50.926902+0000","last_peered":"2024-03-26T05:14:50.926902+0000","last_clean":"2024-03-26T05:12:19.741444+0000","last_became_active":"2024-03-26T05:12:26.811802+0000","last_became_peered":"2024-03-26T05:12:26.811802+0000","last_unstale":"2024-03-26T05:14:50.926902+0000","last_undegraded":"2024-03-26T05:12:26.792367+0000","last_fullsized":"2024-03-26T05:12:26.792245+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'52","last_scrub_stamp":"2024-03-26T05:12:06.588253+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:12:06.588253+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:55.926226+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+undersized","last_fresh":"2024-03-26T05:14:57.753379+0000","last_change":"2024-03-26T05:12:26.805397+0000","last_active":"2024-03-26T05:14:57.753379+0000","last_peered":"2024-03-26T05:14:57.753379+0000","last_clean":"2024-03-26T05:12:19.732211+0000","last_became_active":"2024-03-26T05:12:26.805397+0000","last_became_peered":"2024-03-26T05:12:26.805397+0000","last_unstale":"2024-03-26T05:14:57.753379+0000","last_undegraded":"2024-03-26T05:14:57.753379+0000","last_fullsized":"2024-03-26T05:12:26.787441+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.411139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.411139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:00.957512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"321'1449","reported_seq":1465,"reported_epoch":321,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:59.930751+0000","last_change":"2024-03-26T05:14:52.606206+0000","last_active":"2024-03-26T05:14:59.930751+0000","last_peered":"2024-03-26T05:14:59.930751+0000","last_clean":"2024-03-26T05:14:40.645004+0000","last_became_active":"2024-03-26T05:14:41.670810+0000","last_became_peered":"2024-03-26T05:14:41.670810+0000","last_unstale":"2024-03-26T05:14:59.930751+0000","last_undegraded":"2024-03-26T05:14:52.604388+0000","last_fullsized":"2024-03-26T05:14:41.661822+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":1449,"log_dups_size":0,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:52.313528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94961664,"num_objects":1449,"num_object_clones":0,"num_object_copies":4347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1449,"num_write_kb":92736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1449}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":241,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.574098+0000","last_change":"2024-03-26T05:13:19.069302+0000","last_active":"2024-03-26T05:14:51.574098+0000","last_peered":"2024-03-26T05:14:51.574098+0000","last_clean":"2024-03-26T05:14:51.574098+0000","last_became_active":"2024-03-26T05:08:00.005676+0000","last_became_peered":"2024-03-26T05:08:00.005676+0000","last_unstale":"2024-03-26T05:14:51.574098+0000","last_undegraded":"2024-03-26T05:14:51.574098+0000","last_fullsized":"2024-03-26T05:14:51.574098+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:19.069174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_clean_scrub_stamp":"2024-03-26T05:13:19.069174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"216'10","reported_seq":768,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.754592+0000","last_change":"2024-03-26T05:14:03.822542+0000","last_active":"2024-03-26T05:14:57.754592+0000","last_peered":"2024-03-26T05:14:57.754592+0000","last_clean":"2024-03-26T05:14:57.754592+0000","last_became_active":"2024-03-26T05:07:59.997429+0000","last_became_peered":"2024-03-26T05:07:59.997429+0000","last_unstale":"2024-03-26T05:14:57.754592+0000","last_undegraded":"2024-03-26T05:14:57.754592+0000","last_fullsized":"2024-03-26T05:14:57.754592+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:14:03.822423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:03.822423+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:14.383814+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":554,"reported_epoch":320,"state":"active+undersized","last_fresh":"2024-03-26T05:14:50.927376+0000","last_change":"2024-03-26T05:12:26.817034+0000","last_active":"2024-03-26T05:14:50.927376+0000","last_peered":"2024-03-26T05:14:50.927376+0000","last_clean":"2024-03-26T05:12:19.741099+0000","last_became_active":"2024-03-26T05:12:26.817034+0000","last_became_peered":"2024-03-26T05:12:26.817034+0000","last_unstale":"2024-03-26T05:14:50.927376+0000","last_undegraded":"2024-03-26T05:14:50.927376+0000","last_fullsized":"2024-03-26T05:12:26.791928+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:40.541340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:10:40.541340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:55.926140+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"292'34","reported_seq":621,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.939414+0000","last_change":"2024-03-26T05:14:02.845403+0000","last_active":"2024-03-26T05:14:50.939414+0000","last_peered":"2024-03-26T05:14:50.939414+0000","last_clean":"2024-03-26T05:14:50.939414+0000","last_became_active":"2024-03-26T05:00:24.935688+0000","last_became_peered":"2024-03-26T05:00:24.935688+0000","last_unstale":"2024-03-26T05:14:50.939414+0000","last_undegraded":"2024-03-26T05:14:50.939414+0000","last_fullsized":"2024-03-26T05:14:50.939414+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'34","last_scrub_stamp":"2024-03-26T05:14:02.845258+0000","last_deep_scrub":"292'34","last_deep_scrub_stamp":"2024-03-26T05:12:23.778630+0000","last_clean_scrub_stamp":"2024-03-26T05:14:02.845258+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:32.576587+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"321'1409","reported_seq":1429,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:59.981988+0000","last_change":"2024-03-26T05:14:41.675033+0000","last_active":"2024-03-26T05:14:59.981988+0000","last_peered":"2024-03-26T05:14:59.981988+0000","last_clean":"2024-03-26T05:14:59.981988+0000","last_became_active":"2024-03-26T05:14:41.674614+0000","last_became_peered":"2024-03-26T05:14:41.674614+0000","last_unstale":"2024-03-26T05:14:59.981988+0000","last_undegraded":"2024-03-26T05:14:59.981988+0000","last_fullsized":"2024-03-26T05:14:59.981988+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":1409,"log_dups_size":0,"ondisk_log_size":1409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:02.963701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92340224,"num_objects":1409,"num_object_clones":0,"num_object_copies":4227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1409,"num_write_kb":90176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":559,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.927689+0000","last_change":"2024-03-26T05:13:21.796632+0000","last_active":"2024-03-26T05:14:50.927689+0000","last_peered":"2024-03-26T05:14:50.927689+0000","last_clean":"2024-03-26T05:14:50.927689+0000","last_became_active":"2024-03-26T05:07:59.957995+0000","last_became_peered":"2024-03-26T05:07:59.957995+0000","last_unstale":"2024-03-26T05:14:50.927689+0000","last_undegraded":"2024-03-26T05:14:50.927689+0000","last_fullsized":"2024-03-26T05:14:50.927689+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:21.796526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:13:21.796526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":241,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:51.574470+0000","last_change":"2024-03-26T05:13:07.883159+0000","last_active":"2024-03-26T05:14:51.574470+0000","last_peered":"2024-03-26T05:14:51.574470+0000","last_clean":"2024-03-26T05:14:51.574470+0000","last_became_active":"2024-03-26T05:07:59.947747+0000","last_became_peered":"2024-03-26T05:07:59.947747+0000","last_unstale":"2024-03-26T05:14:51.574470+0000","last_undegraded":"2024-03-26T05:14:51.574470+0000","last_fullsized":"2024-03-26T05:14:51.574470+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:07.883082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:13:07.883082+0000","last_clean_scrub_stamp":"2024-03-26T05:13:07.883082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":566,"reported_epoch":321,"state":"active+undersized","last_fresh":"2024-03-26T05:14:57.753388+0000","last_change":"2024-03-26T05:12:26.794081+0000","last_active":"2024-03-26T05:14:57.753388+0000","last_peered":"2024-03-26T05:14:57.753388+0000","last_clean":"2024-03-26T05:12:19.732296+0000","last_became_active":"2024-03-26T05:12:26.794081+0000","last_became_peered":"2024-03-26T05:12:26.794081+0000","last_unstale":"2024-03-26T05:14:57.753388+0000","last_undegraded":"2024-03-26T05:14:57.753388+0000","last_fullsized":"2024-03-26T05:12:26.787723+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:29.465975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.465975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:00.957573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"254'25","reported_seq":610,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.754343+0000","last_change":"2024-03-26T05:13:04.195129+0000","last_active":"2024-03-26T05:14:57.754343+0000","last_peered":"2024-03-26T05:14:57.754343+0000","last_clean":"2024-03-26T05:14:57.754343+0000","last_became_active":"2024-03-26T05:00:24.944093+0000","last_became_peered":"2024-03-26T05:00:24.944093+0000","last_unstale":"2024-03-26T05:14:57.754343+0000","last_undegraded":"2024-03-26T05:14:57.754343+0000","last_fullsized":"2024-03-26T05:14:57.754343+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:13:04.195003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:04.195003+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"320'678","reported_seq":696,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:55.398009+0000","last_change":"2024-03-26T05:14:41.667178+0000","last_active":"2024-03-26T05:14:55.398009+0000","last_peered":"2024-03-26T05:14:55.398009+0000","last_clean":"2024-03-26T05:14:55.398009+0000","last_became_active":"2024-03-26T05:14:41.666892+0000","last_became_peered":"2024-03-26T05:14:41.666892+0000","last_unstale":"2024-03-26T05:14:55.398009+0000","last_undegraded":"2024-03-26T05:14:55.398009+0000","last_fullsized":"2024-03-26T05:14:55.398009+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:54.363535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44433408,"num_objects":678,"num_object_clones":0,"num_object_copies":2034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":678,"num_write_kb":43392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":563,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.942112+0000","last_change":"2024-03-26T05:14:19.269141+0000","last_active":"2024-03-26T05:14:50.942112+0000","last_peered":"2024-03-26T05:14:50.942112+0000","last_clean":"2024-03-26T05:14:50.942112+0000","last_became_active":"2024-03-26T05:00:28.981666+0000","last_became_peered":"2024-03-26T05:00:28.981666+0000","last_unstale":"2024-03-26T05:14:50.942112+0000","last_undegraded":"2024-03-26T05:14:50.942112+0000","last_fullsized":"2024-03-26T05:14:50.942112+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:19.269035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:19.269035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:40.592091+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":567,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.939208+0000","last_change":"2024-03-26T05:14:21.059979+0000","last_active":"2024-03-26T05:14:50.939208+0000","last_peered":"2024-03-26T05:14:50.939208+0000","last_clean":"2024-03-26T05:14:50.939208+0000","last_became_active":"2024-03-26T05:00:26.959648+0000","last_became_peered":"2024-03-26T05:00:26.959648+0000","last_unstale":"2024-03-26T05:14:50.939208+0000","last_undegraded":"2024-03-26T05:14:50.939208+0000","last_fullsized":"2024-03-26T05:14:50.939208+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:21.059907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:14:21.059907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:42.802204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":598,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.750727+0000","last_change":"2024-03-26T05:14:16.382616+0000","last_active":"2024-03-26T05:14:57.750727+0000","last_peered":"2024-03-26T05:14:57.750727+0000","last_clean":"2024-03-26T05:14:57.750727+0000","last_became_active":"2024-03-26T05:06:06.521150+0000","last_became_peered":"2024-03-26T05:06:06.521150+0000","last_unstale":"2024-03-26T05:14:57.750727+0000","last_undegraded":"2024-03-26T05:14:57.750727+0000","last_fullsized":"2024-03-26T05:14:57.750727+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:16.382538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.382538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:40.411732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"254'31","reported_seq":616,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.754029+0000","last_change":"2024-03-26T05:14:50.944638+0000","last_active":"2024-03-26T05:14:57.754029+0000","last_peered":"2024-03-26T05:14:57.754029+0000","last_clean":"2024-03-26T05:14:57.754029+0000","last_became_active":"2024-03-26T05:00:24.942414+0000","last_became_peered":"2024-03-26T05:00:24.942414+0000","last_unstale":"2024-03-26T05:14:57.754029+0000","last_undegraded":"2024-03-26T05:14:57.754029+0000","last_fullsized":"2024-03-26T05:14:57.754029+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'31","last_scrub_stamp":"2024-03-26T05:14:50.944513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:50.944513+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:01.388428+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"320'706","reported_seq":723,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:55.873073+0000","last_change":"2024-03-26T05:14:41.674905+0000","last_active":"2024-03-26T05:14:55.873073+0000","last_peered":"2024-03-26T05:14:55.873073+0000","last_clean":"2024-03-26T05:14:55.873073+0000","last_became_active":"2024-03-26T05:14:41.674554+0000","last_became_peered":"2024-03-26T05:14:41.674554+0000","last_unstale":"2024-03-26T05:14:55.873073+0000","last_undegraded":"2024-03-26T05:14:55.873073+0000","last_fullsized":"2024-03-26T05:14:55.873073+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:10.604204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46268416,"num_objects":706,"num_object_clones":0,"num_object_copies":2118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":706,"num_write_kb":45184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":559,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.754641+0000","last_change":"2024-03-26T05:14:54.067625+0000","last_active":"2024-03-26T05:14:57.754641+0000","last_peered":"2024-03-26T05:14:57.754641+0000","last_clean":"2024-03-26T05:14:57.754641+0000","last_became_active":"2024-03-26T05:00:28.978452+0000","last_became_peered":"2024-03-26T05:00:28.978452+0000","last_unstale":"2024-03-26T05:14:57.754641+0000","last_undegraded":"2024-03-26T05:14:57.754641+0000","last_fullsized":"2024-03-26T05:14:57.754641+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:54.067506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:54.067506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:15.072812+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"294'12","reported_seq":768,"reported_epoch":320,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:14:50.941903+0000","last_change":"2024-03-26T05:12:26.801682+0000","last_active":"2024-03-26T05:14:50.941903+0000","last_peered":"2024-03-26T05:14:50.941903+0000","last_clean":"2024-03-26T05:12:23.556995+0000","last_became_active":"2024-03-26T05:12:26.801682+0000","last_became_peered":"2024-03-26T05:12:26.801682+0000","last_unstale":"2024-03-26T05:14:50.941903+0000","last_undegraded":"2024-03-26T05:12:26.790226+0000","last_fullsized":"2024-03-26T05:12:26.790079+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:12:08.228910+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:12:08.228910+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:14:57.369566+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":573,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-26T05:14:50.927610+0000","last_change":"2024-03-26T05:14:08.827220+0000","last_active":"2024-03-26T05:14:50.927610+0000","last_peered":"2024-03-26T05:14:50.927610+0000","last_clean":"2024-03-26T05:14:50.927610+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:14:50.927610+0000","last_undegraded":"2024-03-26T05:14:50.927610+0000","last_fullsized":"2024-03-26T05:14:50.927610+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:08.827109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:08.827109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:15:12.020129+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"254'20","reported_seq":597,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-26T05:14:57.754179+0000","last_change":"2024-03-26T05:14:56.990059+0000","last_active":"2024-03-26T05:14:57.754179+0000","last_peered":"2024-03-26T05:14:57.754179+0000","last_clean":"2024-03-26T05:14:57.754179+0000","last_became_active":"2024-03-26T05:00:24.940222+0000","last_became_peered":"2024-03-26T05:00:24.940222+0000","last_unstale":"2024-03-26T05:14:57.754179+0000","last_undegraded":"2024-03-26T05:14:57.754179+0000","last_fullsized":"2024-03-26T05:14:57.754179+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'20","last_scrub_stamp":"2024-03-26T05:14:56.989928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:56.989928+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:24.652122+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1148846080,"num_objects":17530,"num_object_clones":0,"num_object_copies":52590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17530,"num_write_kb":1121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2727411712,"data_stored":2727411712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17530,"ondisk_log_size":17530,"up":44,"acting":44,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":214,"num_read_kb":209,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":80,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1413,"num_read_kb":1138,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":1870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":948,"ondisk_log_size":948,"up":85,"acting":85,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":82,"acting":82,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":84,"acting":84,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005205,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4287492,"kb_used_data":3941180,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":89457660,"statfs":{"total":95995035648,"available":91604643840,"internally_reserved":0,"allocated":4035768320,"data_stored":4035435774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":354613706},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332281,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940920,"kb_used_data":594608,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92804232,"statfs":{"total":95995035648,"available":95031533568,"internally_reserved":0,"allocated":608878592,"data_stored":608559792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":354613702},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575058,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":626256,"kb_used_data":107784,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":93118896,"statfs":{"total":95995035648,"available":95353749504,"internally_reserved":0,"allocated":110370816,"data_stored":110019571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738583,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917512,"kb_used_data":480960,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92827640,"statfs":{"total":95995035648,"available":95055503360,"internally_reserved":0,"allocated":492503040,"data_stored":492161266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411533,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":615916,"kb_used_data":237668,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93129236,"statfs":{"total":95995035648,"available":95364337664,"internally_reserved":0,"allocated":243372032,"data_stored":243056861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117191,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798084,"kb_used_data":229628,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92947068,"statfs":{"total":95995035648,"available":95177797632,"internally_reserved":0,"allocated":235139072,"data_stored":234799661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":582089159},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920964,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":986956,"kb_used_data":640644,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92758196,"statfs":{"total":95995035648,"available":94984392704,"internally_reserved":0,"allocated":656019456,"data_stored":655680701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659325,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":779884,"kb_used_data":153124,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92965268,"statfs":{"total":95995035648,"available":95196434432,"internally_reserved":0,"allocated":156798976,"data_stored":156483987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":493420544,"data_stored":493420544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143589376,"data_stored":143589376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":251265024,"data_stored":251265024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250413056,"data_stored":250413056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":737148928,"data_stored":737148928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":196280320,"data_stored":196280320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":655294464,"data_stored":655294464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:15:05.107 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-26T05:15:05.107 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-26T05:15:05.112 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:05.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:05.214 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:05.247 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:05.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:05.349 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:05.382 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:05.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:05.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:05.484 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:05.528 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:05.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:05.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:05.592 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 13 16 34919 34903 167.761 129.75 0.00825355 0.0059562 2024-03-26T05:15:05.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:05.659 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:05.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:05.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:05.760 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:05.799 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:05.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:05.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:05.901 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:05.936 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:05.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:05.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:15:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:15:06.037 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:06.061 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:06.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:06.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:06.162 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:06.207 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:06.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:06.309 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:06.343 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:06.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:06.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:06.432 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:06.432 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:15:06.432 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:06 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:06.432 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:06.432 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:15:06.432 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:06 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:06.445 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:06.486 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:06.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:06.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:06.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:15:06.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:06 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:06.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:06.592 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 14 16 36915 36899 164.688 124.75 0.00536729 0.00606558 2024-03-26T05:15:06.619 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:06.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:06.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:06.722 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:06.779 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:06.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:06.880 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:06.920 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:06.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:06.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:07.022 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:07.067 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:07.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:07.167 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:07.201 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:07.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:07.302 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:07.343 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:07.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:07.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:07.423 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:15:07.445 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:07.471 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:15:07.478 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:07.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:07.518 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:15:07.556 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:15:07.580 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:07.594 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 15 16 38899 38883 161.966 124 0.011229 0.0061684 2024-03-26T05:15:07.594 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:15:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:07.624 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:07.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:07.633 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:15:07.672 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:15:07.673 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:15:07.673 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:15:07.673 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:15:07.673 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:15:07.673 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:15:07.673 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:15:07.673 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:15:07.673 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:15:07.725 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:07.761 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:07.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:07.844 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:07 smithi080 ceph-mon[29691]: 6.17 scrub starts 2024-03-26T05:15:07.844 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:07 smithi080 ceph-mon[29691]: pgmap v657: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 154 MiB/s wr, 2.47k op/s; 8022/99216 objects degraded (8.085%) 2024-03-26T05:15:07.844 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:07 smithi080 ceph-mon[29691]: 6.17 scrub ok 2024-03-26T05:15:07.844 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:07 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3466059853' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:07.844 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:07 smithi080 ceph-mon[40404]: 6.17 scrub starts 2024-03-26T05:15:07.844 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:07 smithi080 ceph-mon[40404]: pgmap v657: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 154 MiB/s wr, 2.47k op/s; 8022/99216 objects degraded (8.085%) 2024-03-26T05:15:07.844 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:07 smithi080 ceph-mon[40404]: 6.17 scrub ok 2024-03-26T05:15:07.845 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:07 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3466059853' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:07.863 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:07.900 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:07.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:08.001 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:08.038 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:08.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:07 smithi132 ceph-mon[33500]: 6.17 scrub starts 2024-03-26T05:15:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:07 smithi132 ceph-mon[33500]: pgmap v657: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 154 MiB/s wr, 2.47k op/s; 8022/99216 objects degraded (8.085%) 2024-03-26T05:15:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:07 smithi132 ceph-mon[33500]: 6.17 scrub ok 2024-03-26T05:15:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:07 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3466059853' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:08.064 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:08.068 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:08.139 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:08.185 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:08.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:08.287 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:08.328 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:08.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:08.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:08.430 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:08.471 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:08.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:08.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:08.573 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:08.593 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 16 16 40911 40895 159.702 125.75 0.0115131 0.00625532 2024-03-26T05:15:08.603 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:08.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:08.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:15:08.635 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:15:08.705 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:08.744 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:08.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:08.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:08.846 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:08.850 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:08 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 8690/103281 objects degraded (8.414%), 23 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:08.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:08 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 8690/103281 objects degraded (8.414%), 23 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:08.870 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:08.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:08.972 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:09.018 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:09.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:09.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:08 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 8690/103281 objects degraded (8.414%), 23 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:09.120 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:09.152 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:09.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:09.253 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:09.293 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:09.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:09.395 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:09.434 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:09.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:09.536 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:09.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:09.577 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:09.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:09.593 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 17 16 42753 42737 157.08 115.125 0.0048661 0.00636081 2024-03-26T05:15:09.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:09.717 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:09.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:09.732 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:09 smithi080 ceph-mon[29691]: pgmap v658: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 995 B/s rd, 149 MiB/s wr, 2.39k op/s; 8690/103281 objects degraded (8.414%) 2024-03-26T05:15:09.732 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:09 smithi080 ceph-mon[40404]: pgmap v658: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 995 B/s rd, 149 MiB/s wr, 2.39k op/s; 8690/103281 objects degraded (8.414%) 2024-03-26T05:15:09.819 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:09.861 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:09.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:09.964 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:10.001 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:10.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:10.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:10.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:09 smithi132 ceph-mon[33500]: pgmap v658: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 995 B/s rd, 149 MiB/s wr, 2.39k op/s; 8690/103281 objects degraded (8.414%) 2024-03-26T05:15:10.103 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:10.148 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:10.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:10.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:10.299 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:10.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:10.401 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:10.441 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:10.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:10.542 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:10.574 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:10.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:10.593 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 18 16 44717 44701 155.173 122.75 0.00350716 0.00643957 2024-03-26T05:15:10.636 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:15:10.676 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:10.726 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:10.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:10.829 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:10.879 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:10.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:10.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:10.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:10 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:15:10.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:10 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:15:10.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:10 smithi080 ceph-mon[29691]: osdmap e322: 8 total, 7 up, 8 in 2024-03-26T05:15:10.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:10 smithi080 ceph-mon[29691]: 5.13 scrub starts 2024-03-26T05:15:10.924 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:10 smithi080 ceph-mon[29691]: 5.13 scrub ok 2024-03-26T05:15:10.924 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:10 smithi080 ceph-mon[29691]: pgmap v660: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s; 9954/120090 objects degraded (8.289%) 2024-03-26T05:15:10.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:10 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:15:10.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:10 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:15:10.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:10 smithi080 ceph-mon[40404]: osdmap e322: 8 total, 7 up, 8 in 2024-03-26T05:15:10.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:10 smithi080 ceph-mon[40404]: 5.13 scrub starts 2024-03-26T05:15:10.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:10 smithi080 ceph-mon[40404]: 5.13 scrub ok 2024-03-26T05:15:10.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:10 smithi080 ceph-mon[40404]: pgmap v660: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s; 9954/120090 objects degraded (8.289%) 2024-03-26T05:15:10.982 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:11.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:11.029 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:11.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:11.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:10 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:15:11.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:10 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:15:11.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:10 smithi132 ceph-mon[33500]: osdmap e322: 8 total, 7 up, 8 in 2024-03-26T05:15:11.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:10 smithi132 ceph-mon[33500]: 5.13 scrub starts 2024-03-26T05:15:11.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:10 smithi132 ceph-mon[33500]: 5.13 scrub ok 2024-03-26T05:15:11.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:10 smithi132 ceph-mon[33500]: pgmap v660: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s; 9954/120090 objects degraded (8.289%) 2024-03-26T05:15:11.130 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:11.177 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:11.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:11.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:11.271 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:15:11.271 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:15:11.271 INFO:teuthology.orchestra.run.smithi080.stdout: "complaint_time": 30, 2024-03-26T05:15:11.271 INFO:teuthology.orchestra.run.smithi080.stdout: "num_blocked_ops": 0 2024-03-26T05:15:11.272 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:15:11.279 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:11.322 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:11.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:11.423 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:11.459 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:11.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:11.561 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:11.594 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 19 15 46452 46437 152.716 108.5 0.00833042 0.0065435 2024-03-26T05:15:11.600 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:11.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:11.702 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:11.737 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:11.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:11.839 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:11.871 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:11.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:11.954 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:11 smithi080 ceph-mon[40404]: 5.12 scrub starts 2024-03-26T05:15:11.955 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:11 smithi080 ceph-mon[40404]: 5.12 scrub ok 2024-03-26T05:15:11.955 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:11 smithi080 ceph-mon[40404]: 4.1d deep-scrub starts 2024-03-26T05:15:11.955 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:11 smithi080 ceph-mon[40404]: 4.1d deep-scrub ok 2024-03-26T05:15:11.955 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:11 smithi080 ceph-mon[40404]: 3.17 deep-scrub starts 2024-03-26T05:15:11.955 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:11 smithi080 ceph-mon[40404]: 3.17 deep-scrub ok 2024-03-26T05:15:11.955 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:11 smithi080 ceph-mon[29691]: 5.12 scrub starts 2024-03-26T05:15:11.955 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:11 smithi080 ceph-mon[29691]: 5.12 scrub ok 2024-03-26T05:15:11.955 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:11 smithi080 ceph-mon[29691]: 4.1d deep-scrub starts 2024-03-26T05:15:11.955 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:11 smithi080 ceph-mon[29691]: 4.1d deep-scrub ok 2024-03-26T05:15:11.955 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:11 smithi080 ceph-mon[29691]: 3.17 deep-scrub starts 2024-03-26T05:15:11.955 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:11 smithi080 ceph-mon[29691]: 3.17 deep-scrub ok 2024-03-26T05:15:11.973 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:12.021 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:12.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:11 smithi132 ceph-mon[33500]: 5.12 scrub starts 2024-03-26T05:15:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:11 smithi132 ceph-mon[33500]: 5.12 scrub ok 2024-03-26T05:15:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:11 smithi132 ceph-mon[33500]: 4.1d deep-scrub starts 2024-03-26T05:15:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:11 smithi132 ceph-mon[33500]: 4.1d deep-scrub ok 2024-03-26T05:15:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:11 smithi132 ceph-mon[33500]: 3.17 deep-scrub starts 2024-03-26T05:15:12.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:11 smithi132 ceph-mon[33500]: 3.17 deep-scrub ok 2024-03-26T05:15:12.122 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:12.158 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:12.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:12.260 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:12.293 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:12.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:12.395 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:12.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:12.431 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:12.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:12.532 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:12.568 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:12.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:12.594 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:15:12.604672+0000 min lat: 0.00134127 max lat: 0.108701 avg lat: 0.00661214 2024-03-26T05:15:12.594 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:15:12.594 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 20 16 48387 48371 151.124 120.875 0.00711504 0.00661214 2024-03-26T05:15:12.669 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:12.704 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:12.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:12.806 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:12.836 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:12.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:12.862 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T05:15:12.938 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:12.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:12 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:15:12.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:12 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:15:12.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:12 smithi080 ceph-mon[29691]: 5.1d scrub starts 2024-03-26T05:15:12.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:12 smithi080 ceph-mon[29691]: 5.1d scrub ok 2024-03-26T05:15:12.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:12 smithi080 ceph-mon[29691]: pgmap v661: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s; 11168/132774 objects degraded (8.411%) 2024-03-26T05:15:12.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:12 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:15:12.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:12 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:15:12.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:12 smithi080 ceph-mon[40404]: 5.1d scrub starts 2024-03-26T05:15:12.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:12 smithi080 ceph-mon[40404]: 5.1d scrub ok 2024-03-26T05:15:12.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:12 smithi080 ceph-mon[40404]: pgmap v661: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s; 11168/132774 objects degraded (8.411%) 2024-03-26T05:15:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:12.990 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:12.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:13.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:12 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:15:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:12 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:15:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:12 smithi132 ceph-mon[33500]: 5.1d scrub starts 2024-03-26T05:15:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:12 smithi132 ceph-mon[33500]: 5.1d scrub ok 2024-03-26T05:15:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:12 smithi132 ceph-mon[33500]: pgmap v661: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s; 11168/132774 objects degraded (8.411%) 2024-03-26T05:15:13.090 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:13.122 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:13.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:13.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:13.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:13.258 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:13.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:13.360 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:13.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:13.407 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:13.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:13.508 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:13.565 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:13.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:13.594 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 21 16 50047 50031 148.863 103.75 0.0113767 0.00671324 2024-03-26T05:15:13.665 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:13.714 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:13.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:13.778 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:13 smithi080 ceph-mon[40404]: 3.b scrub starts 2024-03-26T05:15:13.778 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:13 smithi080 ceph-mon[40404]: 3.b scrub ok 2024-03-26T05:15:13.778 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:13 smithi080 ceph-mon[29691]: 3.b scrub starts 2024-03-26T05:15:13.778 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:13 smithi080 ceph-mon[29691]: 3.b scrub ok 2024-03-26T05:15:13.816 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:13.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:13.876 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:13.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:13.904 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:13.919 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:13.978 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:14.026 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:14.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:14.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:14.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:13 smithi132 ceph-mon[33500]: 3.b scrub starts 2024-03-26T05:15:14.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:13 smithi132 ceph-mon[33500]: 3.b scrub ok 2024-03-26T05:15:14.057 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:15:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:15:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:15:14.128 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:14.178 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:14.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:14.279 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:14.314 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:14.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:14.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:14.417 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:14.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:14.455 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:14.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:14.556 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:14.595 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 22 16 51578 51562 146.446 95.6875 0.0101992 0.00682357 2024-03-26T05:15:14.598 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:14.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:14.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:14.700 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:14.701 INFO:teuthology.orchestra.run.smithi080.stderr:set pool 1 pg_num to 11 2024-03-26T05:15:14.712 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:15:14.726 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:14.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:14.828 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:14.861 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:14.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:14.963 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:15.002 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:15.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:14 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/911740024' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:15:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:14 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:15:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:14 smithi132 ceph-mon[33500]: pgmap v662: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s; 11168/132774 objects degraded (8.411%) 2024-03-26T05:15:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:14 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1766220990' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-26T05:15:15.104 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:14 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/911740024' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:15:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:14 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:15:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:14 smithi080 ceph-mon[29691]: pgmap v662: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s; 11168/132774 objects degraded (8.411%) 2024-03-26T05:15:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:14 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1766220990' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-26T05:15:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:14 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/911740024' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:15:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:14 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:15:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:14 smithi080 ceph-mon[40404]: pgmap v662: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s; 11168/132774 objects degraded (8.411%) 2024-03-26T05:15:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:14 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1766220990' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-26T05:15:15.143 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:15.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:15.245 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:15.284 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:15.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:15.387 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:15.423 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:15.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:15.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:15.525 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:15.562 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:15.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:15.594 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 23 16 53322 53306 144.818 109 0.00365146 0.00690028 2024-03-26T05:15:15.664 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:15.694 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:15.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:15.796 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:15.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:15.831 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:15.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:15.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[40404]: 6.7 deep-scrub starts 2024-03-26T05:15:15.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[40404]: 6.7 deep-scrub ok 2024-03-26T05:15:15.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[40404]: 4.19 scrub starts 2024-03-26T05:15:15.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[40404]: 4.19 scrub ok 2024-03-26T05:15:15.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:15:15.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:15:15.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1766220990' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-26T05:15:15.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[40404]: osdmap e323: 8 total, 7 up, 8 in 2024-03-26T05:15:15.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:15.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[29691]: 6.7 deep-scrub starts 2024-03-26T05:15:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[29691]: 6.7 deep-scrub ok 2024-03-26T05:15:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[29691]: 4.19 scrub starts 2024-03-26T05:15:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[29691]: 4.19 scrub ok 2024-03-26T05:15:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:15:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:15:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1766220990' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-26T05:15:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[29691]: osdmap e323: 8 total, 7 up, 8 in 2024-03-26T05:15:15.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:15 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:15.931 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:15.970 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:15.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:15.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:16.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:15:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:15:16.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:15 smithi132 ceph-mon[33500]: 6.7 deep-scrub starts 2024-03-26T05:15:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:15 smithi132 ceph-mon[33500]: 6.7 deep-scrub ok 2024-03-26T05:15:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:15 smithi132 ceph-mon[33500]: 4.19 scrub starts 2024-03-26T05:15:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:15 smithi132 ceph-mon[33500]: 4.19 scrub ok 2024-03-26T05:15:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:15 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:15:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:15 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:15:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:15 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1766220990' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-26T05:15:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:15 smithi132 ceph-mon[33500]: osdmap e323: 8 total, 7 up, 8 in 2024-03-26T05:15:16.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:15 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:16.072 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:16.110 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:16.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:16.211 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:16.244 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:16.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:16.345 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:16.384 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:16.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:16.485 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:16.525 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:16.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:16.595 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 24 15 55091 55076 143.393 110.625 0.0119834 0.00696858 2024-03-26T05:15:16.628 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:16.659 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:16.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:16.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:16.761 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:16.807 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:16.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:16.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:16 smithi080 ceph-mon[40404]: pgmap v664: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.17k op/s; 12815/155478 objects degraded (8.242%) 2024-03-26T05:15:16.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:16 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-26T05:15:16.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:16 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:15:16.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:16 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:15:16.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:16 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 11168/132774 objects degraded (8.411%), 23 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:16.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:16 smithi080 ceph-mon[29691]: pgmap v664: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.17k op/s; 12815/155478 objects degraded (8.242%) 2024-03-26T05:15:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:16 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-26T05:15:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:16 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:15:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:16 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:15:16.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:16 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 11168/132774 objects degraded (8.411%), 23 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:16.908 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:16.931 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:16.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:17.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:17.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:16 smithi132 ceph-mon[33500]: pgmap v664: 148 pgs: 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.17k op/s; 12815/155478 objects degraded (8.242%) 2024-03-26T05:15:17.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:16 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-26T05:15:17.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:16 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:15:17.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:16 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:15:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:16 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 11168/132774 objects degraded (8.411%), 23 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:17.070 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:17.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:17.172 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:17.210 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:17.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:17.313 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:17.352 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:17.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:17.454 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:17.492 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:17.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:17.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:17.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:17.595 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 25 16 56601 56585 141.43 94.3125 0.00521164 0.00706624 2024-03-26T05:15:17.604 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:15:17.629 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:17.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:17.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:17.731 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:17.781 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:17 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-26T05:15:17.782 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:17 smithi080 ceph-mon[40404]: osdmap e324: 8 total, 7 up, 8 in 2024-03-26T05:15:17.782 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:17 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-26T05:15:17.782 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:17 smithi080 ceph-mon[29691]: osdmap e324: 8 total, 7 up, 8 in 2024-03-26T05:15:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:17.790 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:17.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:17.891 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:17.929 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:17.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:17.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:18.030 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:18.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:17 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-26T05:15:18.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:17 smithi132 ceph-mon[33500]: osdmap e324: 8 total, 7 up, 8 in 2024-03-26T05:15:18.064 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:18.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:18.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:18.166 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:18.207 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:18.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:18.309 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:18.349 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:18.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:18.451 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:18.493 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:18.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:18.595 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:18.596 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 26 16 58083 58067 139.552 92.625 0.0192214 0.00716111 2024-03-26T05:15:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:18.655 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:18.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:18.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:18.784 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:18.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:18.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:18.857 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:15:18.886 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:18.903 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:15:18.938 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:18.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:18.951 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:15:19.000 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:15:19.000 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:18 smithi080 ceph-mon[40404]: osdmap e325: 8 total, 7 up, 8 in 2024-03-26T05:15:19.000 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:18 smithi080 ceph-mon[40404]: pgmap v667: 158 pgs: 10 unknown, 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.44k op/s; 13380/158730 objects degraded (8.429%) 2024-03-26T05:15:19.000 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:18 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-26T05:15:19.000 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:18 smithi080 ceph-mon[29691]: osdmap e325: 8 total, 7 up, 8 in 2024-03-26T05:15:19.001 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:18 smithi080 ceph-mon[29691]: pgmap v667: 158 pgs: 10 unknown, 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.44k op/s; 13380/158730 objects degraded (8.429%) 2024-03-26T05:15:19.001 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:18 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-26T05:15:19.040 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:19.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:18 smithi132 ceph-mon[33500]: osdmap e325: 8 total, 7 up, 8 in 2024-03-26T05:15:19.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:18 smithi132 ceph-mon[33500]: pgmap v667: 158 pgs: 10 unknown, 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.44k op/s; 13380/158730 objects degraded (8.429%) 2024-03-26T05:15:19.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:18 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-26T05:15:19.053 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:15:19.066 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:19.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:19.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:19.130 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:15:19.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:19.172 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:15:19.177 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:15:19.177 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:15:19.177 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:15:19.177 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:15:19.177 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:15:19.177 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:15:19.177 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:15:19.177 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T05:15:19.210 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:19.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:19.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:19.312 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:19.350 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:19.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:19.365 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:15:19.365 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:15:19.452 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:19.477 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:19.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:19.579 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:19.595 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 27 15 59477 59462 137.613 87.1875 0.00839222 0.0072621 2024-03-26T05:15:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:19.618 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:19.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:19.719 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:19.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:19.761 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:19.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:19.862 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:19.891 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:19.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:19.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:19.993 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:20.034 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:20.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:20.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:20.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:19 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-26T05:15:20.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:19 smithi132 ceph-mon[33500]: osdmap e326: 8 total, 7 up, 8 in 2024-03-26T05:15:20.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:19 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2066151518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:20.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:19 smithi132 ceph-mon[33500]: osdmap e327: 8 total, 7 up, 8 in 2024-03-26T05:15:20.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:19 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-26T05:15:20.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:19 smithi080 ceph-mon[40404]: osdmap e326: 8 total, 7 up, 8 in 2024-03-26T05:15:20.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:19 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2066151518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:20.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:19 smithi080 ceph-mon[40404]: osdmap e327: 8 total, 7 up, 8 in 2024-03-26T05:15:20.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:19 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-26T05:15:20.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:19 smithi080 ceph-mon[29691]: osdmap e326: 8 total, 7 up, 8 in 2024-03-26T05:15:20.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:19 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2066151518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:20.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:19 smithi080 ceph-mon[29691]: osdmap e327: 8 total, 7 up, 8 in 2024-03-26T05:15:20.136 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:20.171 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:20.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:20.274 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:20.298 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:20.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:20.400 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:20.442 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:20.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:20.544 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:20.581 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:20.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:20.595 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 28 16 60950 60934 135.984 92 0.00687597 0.00734955 2024-03-26T05:15:20.684 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:20.719 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:20.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:20.821 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:20.858 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:20.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:20.960 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:20.995 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:20.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:20.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:21.097 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:21.134 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:21.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:21.235 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:21.274 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:21.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:21.376 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:21.412 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:21.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:21.514 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:21.553 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:21.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:21.595 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 29 16 62347 62331 134.306 87.3125 0.00255983 0.00742973 2024-03-26T05:15:21.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:21 smithi080 ceph-mon[40404]: pgmap v670: 158 pgs: 2 remapped+peering, 8 unknown, 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 13 KiB/s rd, 86 MiB/s wr, 1.39k op/s; 14363/171915 objects degraded (8.355%); 79 KiB/s, 0 objects/s recovering 2024-03-26T05:15:21.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:21.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:15:21.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:21 smithi080 ceph-mon[40404]: osdmap e328: 8 total, 7 up, 8 in 2024-03-26T05:15:21.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:21 smithi080 ceph-mon[40404]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-26T05:15:21.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:21 smithi080 ceph-mon[29691]: pgmap v670: 158 pgs: 2 remapped+peering, 8 unknown, 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 13 KiB/s rd, 86 MiB/s wr, 1.39k op/s; 14363/171915 objects degraded (8.355%); 79 KiB/s, 0 objects/s recovering 2024-03-26T05:15:21.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:21.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:15:21.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:21 smithi080 ceph-mon[29691]: osdmap e328: 8 total, 7 up, 8 in 2024-03-26T05:15:21.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:21 smithi080 ceph-mon[29691]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-26T05:15:21.654 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:21.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:21.678 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:21.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:21.779 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:21 smithi132 ceph-mon[33500]: pgmap v670: 158 pgs: 2 remapped+peering, 8 unknown, 92 active+clean, 33 active+undersized, 23 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 13 KiB/s rd, 86 MiB/s wr, 1.39k op/s; 14363/171915 objects degraded (8.355%); 79 KiB/s, 0 objects/s recovering 2024-03-26T05:15:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:15:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:21 smithi132 ceph-mon[33500]: osdmap e328: 8 total, 7 up, 8 in 2024-03-26T05:15:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:21 smithi132 ceph-mon[33500]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-26T05:15:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:21.819 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:21.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:21.920 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:21.955 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:21.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:22.056 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:22.099 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:22.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:22.196 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:15:22.196 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:15:22.201 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:22.238 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:22.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:22.292 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-03-26T05:15:22.292 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:15:22.292 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-03-26T05:15:22.292 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-03-26T05:15:22.293 DEBUG:teuthology.orchestra.run.smithi132:osd.7> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T05:15:22.295 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl start ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.7 2024-03-26T05:15:22.341 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:22.418 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:22.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:22.441 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:22.519 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:22.562 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:22.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:22.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:22.595 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 30 16 63737 63721 132.725 86.875 0.0102534 0.00752939 2024-03-26T05:15:22.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:22 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 14363/171915 objects degraded (8.355%), 23 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:22.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:22 smithi080 ceph-mon[40404]: osdmap e329: 8 total, 7 up, 8 in 2024-03-26T05:15:22.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:22 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 14363/171915 objects degraded (8.355%), 23 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:22.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:22 smithi080 ceph-mon[29691]: osdmap e329: 8 total, 7 up, 8 in 2024-03-26T05:15:22.664 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:22.684 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:22 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 14363/171915 objects degraded (8.355%), 23 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:22.684 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:22 smithi132 ceph-mon[33500]: osdmap e329: 8 total, 7 up, 8 in 2024-03-26T05:15:22.684 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:22 smithi132 systemd[1]: Starting Ceph osd.7 for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:15:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:22.689 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:22.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:22.790 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:22.828 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:22.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:22.929 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:22.972 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:22.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:23.001 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:22 smithi132 podman[122775]: 2024-03-26 05:15:22.60271506 +0000 UTC m=+0.037041217 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:15:23.001 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 podman[122775]: 2024-03-26 05:15:23.011221867 +0000 UTC m=+0.445548047 container create df85ac369a10a52db7180bce2328426a3cb54a0807d9a88d2fa9f519da4997fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-26T05:15:23.075 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:23.110 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:23.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:23.120 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T05:15:23.212 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:23.245 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:23.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:23.346 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:23.461 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:23.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:23.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:23.541 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 podman[122775]: 2024-03-26 05:15:23.289625834 +0000 UTC m=+0.723951991 container init df85ac369a10a52db7180bce2328426a3cb54a0807d9a88d2fa9f519da4997fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True) 2024-03-26T05:15:23.541 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 podman[122775]: 2024-03-26 05:15:23.293713084 +0000 UTC m=+0.728039248 container start df85ac369a10a52db7180bce2328426a3cb54a0807d9a88d2fa9f519da4997fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T05:15:23.542 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 podman[122775]: 2024-03-26 05:15:23.383024876 +0000 UTC m=+0.817351044 container attach df85ac369a10a52db7180bce2328426a3cb54a0807d9a88d2fa9f519da4997fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124) 2024-03-26T05:15:23.562 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:23.596 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 31 16 65148 65132 131.287 88.1875 0.00692568 0.00761205 2024-03-26T05:15:23.602 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:23.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:23.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:23.705 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:23.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:23.744 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:23.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:23.845 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:23.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:23 smithi080 ceph-mon[29691]: pgmap v673: 158 pgs: 2 remapped+peering, 92 active+clean, 34 active+undersized, 8 active+undersized+remapped, 22 active+undersized+degraded; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 MiB/s wr, 1.93k op/s; 15313/181932 objects degraded (8.417%); 1/181932 objects misplaced (0.001%); 144 KiB/s, 0 objects/s recovering 2024-03-26T05:15:23.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:23 smithi080 ceph-mon[29691]: osdmap e330: 8 total, 7 up, 8 in 2024-03-26T05:15:23.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:23 smithi080 ceph-mon[40404]: pgmap v673: 158 pgs: 2 remapped+peering, 92 active+clean, 34 active+undersized, 8 active+undersized+remapped, 22 active+undersized+degraded; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 MiB/s wr, 1.93k op/s; 15313/181932 objects degraded (8.417%); 1/181932 objects misplaced (0.001%); 144 KiB/s, 0 objects/s recovering 2024-03-26T05:15:23.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:23 smithi080 ceph-mon[40404]: osdmap e330: 8 total, 7 up, 8 in 2024-03-26T05:15:23.888 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:23.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:23.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:23.972 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:23 smithi132 ceph-mon[33500]: pgmap v673: 158 pgs: 2 remapped+peering, 92 active+clean, 34 active+undersized, 8 active+undersized+remapped, 22 active+undersized+degraded; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 MiB/s wr, 1.93k op/s; 15313/181932 objects degraded (8.417%); 1/181932 objects misplaced (0.001%); 144 KiB/s, 0 objects/s recovering 2024-03-26T05:15:23.973 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:23 smithi132 ceph-mon[33500]: osdmap e330: 8 total, 7 up, 8 in 2024-03-26T05:15:23.990 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:24.027 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:24.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:24.128 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:24.165 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:24.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:24.197 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:15:24.266 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate[122823]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate[122823]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate[122823]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate[122823]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 bash[122775]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 bash[122775]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 bash[122775]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 bash[122775]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 bash[122775]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 bash[122775]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 bash[122775]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate[122823]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate[122823]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T05:15:24.271 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:23 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate[122823]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-26T05:15:24.272 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:24 smithi132 podman[122775]: 2024-03-26 05:15:24.036266854 +0000 UTC m=+1.470593017 container died df85ac369a10a52db7180bce2328426a3cb54a0807d9a88d2fa9f519da4997fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T05:15:24.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:24.317 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:24.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:24.367 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:15:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:15:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:15:24.418 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:24.454 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:24.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:24.556 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:24.596 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:24.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:24.598 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 32 16 66472 66456 129.763 82.75 0.00999052 0.0077012 2024-03-26T05:15:24.697 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:24.733 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:24.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:24.835 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:24.875 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:24.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:24.977 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:24.991 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:24 smithi132 podman[122993]: 2024-03-26 05:15:24.686795551 +0000 UTC m=+0.639844415 container remove df85ac369a10a52db7180bce2328426a3cb54a0807d9a88d2fa9f519da4997fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T05:15:25.009 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:25.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:25.111 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:25.148 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:25.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:25.249 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:25.267 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:25 smithi132 podman[123077]: 2024-03-26 05:15:24.908166367 +0000 UTC m=+0.036479801 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:15:25.267 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:25 smithi132 podman[123077]: 2024-03-26 05:15:25.029940494 +0000 UTC m=+0.158253928 container create ae9ffa1b045527d180f2c6dd5ce3b13115027dad875931631dae6ad18f4ea3a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T05:15:25.288 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:25.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:25.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:25.390 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:25.391 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T05:15:25.423 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:25.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:25.525 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:25.541 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:25 smithi132 podman[123077]: 2024-03-26 05:15:25.28293127 +0000 UTC m=+0.411244701 container init ae9ffa1b045527d180f2c6dd5ce3b13115027dad875931631dae6ad18f4ea3a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-26T05:15:25.542 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:25 smithi132 podman[123077]: 2024-03-26 05:15:25.290087231 +0000 UTC m=+0.418400665 container start ae9ffa1b045527d180f2c6dd5ce3b13115027dad875931631dae6ad18f4ea3a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-26T05:15:25.542 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:25 smithi132 ceph-osd[123131]: -- 172.21.15.132:0/889546123 <== mon.2 v2:172.21.15.80:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5607687cd600 con 0x560768755000 2024-03-26T05:15:25.542 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:25 smithi132 ceph-osd[123131]: -- 172.21.15.132:0/889546123 <== mon.2 v2:172.21.15.80:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5607687831e0 con 0x560768755000 2024-03-26T05:15:25.542 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:25 smithi132 bash[123077]: ae9ffa1b045527d180f2c6dd5ce3b13115027dad875931631dae6ad18f4ea3a5 2024-03-26T05:15:25.542 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:25 smithi132 systemd[1]: Started Ceph osd.7 for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:15:25.560 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:25.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:25.597 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 33 16 67885 67869 128.507 88.3125 0.00969248 0.00777701 2024-03-26T05:15:25.662 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:25.698 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:25.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:25.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:25.800 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:25.815 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:15:25.838 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:25.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:25.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:25 smithi080 ceph-mon[29691]: pgmap v675: 158 pgs: 2 remapped+peering, 92 active+clean, 34 active+undersized, 8 active+undersized+remapped, 22 active+undersized+degraded; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.80k op/s; 15313/181932 objects degraded (8.417%); 1/181932 objects misplaced (0.001%); 134 KiB/s, 0 objects/s recovering 2024-03-26T05:15:25.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:25 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:15:25.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:25 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:15:25.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:25 smithi080 ceph-mon[29691]: osdmap e331: 8 total, 7 up, 8 in 2024-03-26T05:15:25.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:25 smithi080 ceph-mon[40404]: pgmap v675: 158 pgs: 2 remapped+peering, 92 active+clean, 34 active+undersized, 8 active+undersized+remapped, 22 active+undersized+degraded; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.80k op/s; 15313/181932 objects degraded (8.417%); 1/181932 objects misplaced (0.001%); 134 KiB/s, 0 objects/s recovering 2024-03-26T05:15:25.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:25 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:15:25.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:25 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:15:25.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:25 smithi080 ceph-mon[40404]: osdmap e331: 8 total, 7 up, 8 in 2024-03-26T05:15:25.939 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:25.950 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:25 smithi132 ceph-mon[33500]: pgmap v675: 158 pgs: 2 remapped+peering, 92 active+clean, 34 active+undersized, 8 active+undersized+remapped, 22 active+undersized+degraded; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.80k op/s; 15313/181932 objects degraded (8.417%); 1/181932 objects misplaced (0.001%); 134 KiB/s, 0 objects/s recovering 2024-03-26T05:15:25.950 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:25 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:15:25.950 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:25 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:15:25.950 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:25 smithi132 ceph-mon[33500]: osdmap e331: 8 total, 7 up, 8 in 2024-03-26T05:15:25.950 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:15:25.596+0000 7fb15b1517c0 -1 Falling back to public interface 2024-03-26T05:15:25.951 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:15:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:15:25.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:25.989 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:25.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:26.090 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:26.127 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:26.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:26.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:26.229 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:26.275 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:26.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:26.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:26.377 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:26.422 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:26.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:26.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:26.523 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:26.561 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:26.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:26.598 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 34 16 69241 69225 127.219 84.75 0.0111502 0.00785577 2024-03-26T05:15:26.663 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:26.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:26.726 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:26.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:26.827 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:26.858 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:26 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:26.858 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:26 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:15:26.858 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:26 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 15313/181932 objects degraded (8.417%), 22 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:26.861 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:26.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:26.862 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:26 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:26.862 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:26 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:15:26.862 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:26 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 15313/181932 objects degraded (8.417%), 22 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:26.963 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:26.996 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:26.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:27.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:26 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:27.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:26 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:15:27.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:26 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 15313/181932 objects degraded (8.417%), 22 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:27.098 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:27.134 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:27.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:27.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:27.236 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:27.259 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:27.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:27.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:27.361 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:27.405 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:27.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:27.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:27.477 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:27.477 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:27.507 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:27.557 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:27.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:27.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:27.598 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 35 16 70421 70405 125.691 73.75 0.0167188 0.00795111 2024-03-26T05:15:27.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:27 smithi080 ceph-mon[29691]: pgmap v677: 158 pgs: 1 active+clean+remapped, 98 active+clean, 34 active+undersized, 3 active+undersized+remapped, 22 active+undersized+degraded; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s; 16624/200241 objects degraded (8.302%); 1/200241 objects misplaced (0.000%); 50 KiB/s, 0 objects/s recovering 2024-03-26T05:15:27.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:27 smithi080 ceph-mon[29691]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-26T05:15:27.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:27 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:15:27.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:27 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:15:27.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:27 smithi080 ceph-mon[40404]: pgmap v677: 158 pgs: 1 active+clean+remapped, 98 active+clean, 34 active+undersized, 3 active+undersized+remapped, 22 active+undersized+degraded; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s; 16624/200241 objects degraded (8.302%); 1/200241 objects misplaced (0.000%); 50 KiB/s, 0 objects/s recovering 2024-03-26T05:15:27.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:27 smithi080 ceph-mon[40404]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-26T05:15:27.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:27 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:15:27.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:27 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:15:27.659 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:27.690 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:27.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:27.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:27.722 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:27 smithi132 ceph-mon[33500]: pgmap v677: 158 pgs: 1 active+clean+remapped, 98 active+clean, 34 active+undersized, 3 active+undersized+remapped, 22 active+undersized+degraded; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s; 16624/200241 objects degraded (8.302%); 1/200241 objects misplaced (0.000%); 50 KiB/s, 0 objects/s recovering 2024-03-26T05:15:27.722 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:27 smithi132 ceph-mon[33500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-26T05:15:27.722 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:27 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:15:27.723 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:27 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:15:27.792 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:27.796 INFO:teuthology.orchestra.run.smithi132.stderr:no valid command found; 10 closest matches: 2024-03-26T05:15:27.796 INFO:teuthology.orchestra.run.smithi132.stderr:0 2024-03-26T05:15:27.796 INFO:teuthology.orchestra.run.smithi132.stderr:1 2024-03-26T05:15:27.796 INFO:teuthology.orchestra.run.smithi132.stderr:2 2024-03-26T05:15:27.796 INFO:teuthology.orchestra.run.smithi132.stderr:abort 2024-03-26T05:15:27.796 INFO:teuthology.orchestra.run.smithi132.stderr:assert 2024-03-26T05:15:27.796 INFO:teuthology.orchestra.run.smithi132.stderr:config diff 2024-03-26T05:15:27.796 INFO:teuthology.orchestra.run.smithi132.stderr:config diff get 2024-03-26T05:15:27.796 INFO:teuthology.orchestra.run.smithi132.stderr:config get 2024-03-26T05:15:27.796 INFO:teuthology.orchestra.run.smithi132.stderr:config help [] 2024-03-26T05:15:27.796 INFO:teuthology.orchestra.run.smithi132.stderr:config set ... 2024-03-26T05:15:27.797 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: invalid command 2024-03-26T05:15:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:27.844 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:27.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:27.945 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:27.997 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:27.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:28.099 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:28.137 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:28.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:28.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:28.239 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:28.279 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:28.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:28.380 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:28.405 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:28.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:28.506 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:28.552 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:28.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:28.598 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 36 16 71667 71651 124.362 77.875 0.00534902 0.00803103 2024-03-26T05:15:28.646 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:15:28.653 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:28.690 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:28.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:28.791 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:28.828 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:28.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:28.930 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:28.970 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:28.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:28.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:29.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:28 smithi080 ceph-mon[29691]: osdmap e332: 8 total, 7 up, 8 in 2024-03-26T05:15:29.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:28 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2332364973' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:15:29.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:28 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:15:29.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:28 smithi080 ceph-mon[40404]: osdmap e332: 8 total, 7 up, 8 in 2024-03-26T05:15:29.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:28 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2332364973' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:15:29.027 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:28 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:15:29.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:28 smithi132 ceph-mon[33500]: osdmap e332: 8 total, 7 up, 8 in 2024-03-26T05:15:29.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:28 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2332364973' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:15:29.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:28 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:15:29.072 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:29.111 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:29.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:29.213 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:29.247 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:29.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:29.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:29.349 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:29.385 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:29.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:29.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:29.487 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:29.530 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:29.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:29.598 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 37 16 73026 73010 123.297 84.9375 0.00664505 0.00810581 2024-03-26T05:15:29.632 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:29.697 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:29.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:29.799 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:29.838 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:29.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:29.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:29 smithi080 ceph-mon[29691]: pgmap v679: 158 pgs: 4 active+clean+remapped, 98 active+clean, 34 active+undersized, 22 active+undersized+degraded; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.16k op/s; 17055/202860 objects degraded (8.407%); 1/202860 objects misplaced (0.000%) 2024-03-26T05:15:29.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:29 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:15:29.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:29 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:15:29.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:29 smithi080 ceph-mon[29691]: osdmap e333: 8 total, 7 up, 8 in 2024-03-26T05:15:29.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:29 smithi080 ceph-mon[40404]: pgmap v679: 158 pgs: 4 active+clean+remapped, 98 active+clean, 34 active+undersized, 22 active+undersized+degraded; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.16k op/s; 17055/202860 objects degraded (8.407%); 1/202860 objects misplaced (0.000%) 2024-03-26T05:15:29.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:29 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:15:29.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:29 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:15:29.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:29 smithi080 ceph-mon[40404]: osdmap e333: 8 total, 7 up, 8 in 2024-03-26T05:15:29.940 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:29.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:29.980 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:29.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:30.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:29 smithi132 ceph-mon[33500]: pgmap v679: 158 pgs: 4 active+clean+remapped, 98 active+clean, 34 active+undersized, 22 active+undersized+degraded; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.16k op/s; 17055/202860 objects degraded (8.407%); 1/202860 objects misplaced (0.000%) 2024-03-26T05:15:30.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:29 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:15:30.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:29 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:15:30.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:29 smithi132 ceph-mon[33500]: osdmap e333: 8 total, 7 up, 8 in 2024-03-26T05:15:30.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:29 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:15:29.839+0000 7fb15b1517c0 -1 osd.7 292 log_to_monitors true 2024-03-26T05:15:30.081 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:30.119 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:30.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:15:30.136 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-26T05:15:30.220 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:30.257 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:30.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:30.359 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:30.429 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:30.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:30.531 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:30.571 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:30.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:30.599 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 38 15 74356 74341 122.24 83.1875 0.00685313 0.00817613 2024-03-26T05:15:30.673 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:30.706 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:30.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:30.808 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:30.835 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:30 smithi080 ceph-mon[29691]: from='osd.7 [v2:172.21.15.132:6824/1165391760,v1:172.21.15.132:6825/1165391760]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:15:30.835 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:30 smithi080 ceph-mon[29691]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:15:30.835 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:30 smithi080 ceph-mon[40404]: from='osd.7 [v2:172.21.15.132:6824/1165391760,v1:172.21.15.132:6825/1165391760]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:15:30.835 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:30 smithi080 ceph-mon[40404]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:15:30.836 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:30.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:30.937 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:30.961 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:30.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:31.006 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:30 smithi132 ceph-mon[33500]: from='osd.7 [v2:172.21.15.132:6824/1165391760,v1:172.21.15.132:6825/1165391760]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:15:31.007 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:30 smithi132 ceph-mon[33500]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:15:31.063 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:31.098 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:31.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:31.200 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:31.237 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:31.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:31.339 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:31.379 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:31.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:31.481 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:31.520 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:31.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:31.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:31.541 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:15:31 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:15:31.275+0000 7fb1485a1700 -1 osd.7 334 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T05:15:31.599 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 39 16 75260 75244 120.552 56.4375 0.0175018 0.00828979 2024-03-26T05:15:31.623 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:31.659 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:31.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:31.676 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:15:31.676 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:15:31.762 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:31.793 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:31.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:31.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[40404]: pgmap v681: 158 pgs: 4 active+clean+remapped, 98 active+clean, 34 active+undersized, 22 active+undersized+degraded; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.76k op/s; 17899/213693 objects degraded (8.376%); 1/213693 objects misplaced (0.000%) 2024-03-26T05:15:31.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[40404]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T05:15:31.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[40404]: from='osd.7 [v2:172.21.15.132:6824/1165391760,v1:172.21.15.132:6825/1165391760]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:15:31.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[40404]: osdmap e334: 8 total, 7 up, 8 in 2024-03-26T05:15:31.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[40404]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:15:31.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[40404]: from='osd.7 ' entity='osd.7' 2024-03-26T05:15:31.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 17899/213693 objects degraded (8.376%), 22 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:31.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[40404]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T05:15:31.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[40404]: osd.7 [v2:172.21.15.132:6824/1165391760,v1:172.21.15.132:6825/1165391760] boot 2024-03-26T05:15:31.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[40404]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T05:15:31.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:15:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[29691]: pgmap v681: 158 pgs: 4 active+clean+remapped, 98 active+clean, 34 active+undersized, 22 active+undersized+degraded; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.76k op/s; 17899/213693 objects degraded (8.376%); 1/213693 objects misplaced (0.000%) 2024-03-26T05:15:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[29691]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T05:15:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[29691]: from='osd.7 [v2:172.21.15.132:6824/1165391760,v1:172.21.15.132:6825/1165391760]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:15:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[29691]: osdmap e334: 8 total, 7 up, 8 in 2024-03-26T05:15:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[29691]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:15:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[29691]: from='osd.7 ' entity='osd.7' 2024-03-26T05:15:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 17899/213693 objects degraded (8.376%), 22 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[29691]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T05:15:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[29691]: osd.7 [v2:172.21.15.132:6824/1165391760,v1:172.21.15.132:6825/1165391760] boot 2024-03-26T05:15:31.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[29691]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T05:15:31.868 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:31 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:15:31.894 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:31.925 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:31.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:31.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:32.023 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:15:32.027 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:32.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:31 smithi132 ceph-mon[33500]: pgmap v681: 158 pgs: 4 active+clean+remapped, 98 active+clean, 34 active+undersized, 22 active+undersized+degraded; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.76k op/s; 17899/213693 objects degraded (8.376%); 1/213693 objects misplaced (0.000%) 2024-03-26T05:15:32.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:31 smithi132 ceph-mon[33500]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T05:15:32.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:31 smithi132 ceph-mon[33500]: from='osd.7 [v2:172.21.15.132:6824/1165391760,v1:172.21.15.132:6825/1165391760]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:15:32.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:31 smithi132 ceph-mon[33500]: osdmap e334: 8 total, 7 up, 8 in 2024-03-26T05:15:32.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:31 smithi132 ceph-mon[33500]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:15:32.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:31 smithi132 ceph-mon[33500]: from='osd.7 ' entity='osd.7' 2024-03-26T05:15:32.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:31 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 17899/213693 objects degraded (8.376%), 22 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:32.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:31 smithi132 ceph-mon[33500]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T05:15:32.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:31 smithi132 ceph-mon[33500]: osd.7 [v2:172.21.15.132:6824/1165391760,v1:172.21.15.132:6825/1165391760] boot 2024-03-26T05:15:32.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:31 smithi132 ceph-mon[33500]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T05:15:32.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:31 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:15:32.042 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:15:32.051 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:32.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:32.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:32.061 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:15:32.081 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:15:32.101 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:15:32.121 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:15:32.141 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:15:32.153 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:32.161 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:15:32.162 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:15:32.162 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:15:32.162 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:15:32.163 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:15:32.163 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:15:32.163 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:15:32.163 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:15:32.163 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:15:32.178 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:32.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:32.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:32.280 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:32.303 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:32.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:32.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:32.405 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:32.431 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:32.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:32.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:32.533 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:32.556 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:32.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:32.600 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:15:32.610642+0000 min lat: 0.00134127 max lat: 0.204432 avg lat: 0.00829499 2024-03-26T05:15:32.600 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:15:32.600 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 40 16 75305 75289 117.608 2.8125 0.00360797 0.00829499 2024-03-26T05:15:32.659 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:32.681 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:32.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:32.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:32.783 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:32.799 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:32 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:15:32.799 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:32 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:15:32.799 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:32 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2560365437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:32.799 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:32 smithi080 ceph-mon[40404]: pgmap v684: 158 pgs: 4 active+clean+remapped, 98 active+clean, 34 active+undersized, 22 active+undersized+degraded; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 92 MiB/s wr, 1.47k op/s; 18735/222273 objects degraded (8.429%); 1/222273 objects misplaced (0.000%) 2024-03-26T05:15:32.799 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:32 smithi080 ceph-mon[40404]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T05:15:32.799 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:32 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:15:32.799 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:32 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:15:32.800 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:32 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2560365437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:32.800 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:32 smithi080 ceph-mon[29691]: pgmap v684: 158 pgs: 4 active+clean+remapped, 98 active+clean, 34 active+undersized, 22 active+undersized+degraded; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 92 MiB/s wr, 1.47k op/s; 18735/222273 objects degraded (8.429%); 1/222273 objects misplaced (0.000%) 2024-03-26T05:15:32.800 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:32 smithi080 ceph-mon[29691]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T05:15:32.809 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:32.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:32.824 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T05:15:32.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:32.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:33.039 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:33.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:32 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:15:33.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:32 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:15:33.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:32 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2560365437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:33.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:32 smithi132 ceph-mon[33500]: pgmap v684: 158 pgs: 4 active+clean+remapped, 98 active+clean, 34 active+undersized, 22 active+undersized+degraded; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 92 MiB/s wr, 1.47k op/s; 18735/222273 objects degraded (8.429%); 1/222273 objects misplaced (0.000%) 2024-03-26T05:15:33.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:32 smithi132 ceph-mon[33500]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T05:15:33.066 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:33.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:33.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:33.192 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:33.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:33.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:33.294 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:33.317 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:33.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:33.397 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:15:33.419 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:33.463 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:33.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:33.565 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:33.589 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:33.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:33.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:33.600 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 41 16 75305 75289 114.739 0 - 0.00829499 2024-03-26T05:15:33.691 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:33.720 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:33.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:33.822 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:33.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:33.861 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:33.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:33.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:33 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:15:33.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:33 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:15:33.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:33 smithi080 ceph-mon[40404]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T05:15:33.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:33 smithi080 ceph-mon[40404]: 10.1 starting backfill to osd.7 from (0'0,0'0] MAX to 334'4701 2024-03-26T05:15:33.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:33 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:15:33.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:33 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:15:33.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:33 smithi080 ceph-mon[29691]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T05:15:33.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:33 smithi080 ceph-mon[29691]: 10.1 starting backfill to osd.7 from (0'0,0'0] MAX to 334'4701 2024-03-26T05:15:33.920 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:33 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:15:33.920 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:33 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:15:33.920 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:33 smithi132 ceph-mon[33500]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T05:15:33.920 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:33 smithi132 ceph-mon[33500]: 10.1 starting backfill to osd.7 from (0'0,0'0] MAX to 334'4701 2024-03-26T05:15:33.962 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:33.995 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:33.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:33.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:34.096 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:34.136 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:34.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:34.182 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:34.237 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:34.291 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:34.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:34.324 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:15:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:15:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:15:34.392 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:34.430 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:34.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:34.497 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:34.531 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:34.575 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:34.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:34.600 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 42 16 76187 76171 113.32 27.5625 0.0749966 0.00881764 2024-03-26T05:15:34.676 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:34.720 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:34.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:34.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:34 smithi132 ceph-mon[33500]: 10.c starting backfill to osd.7 from (0'0,0'0] MAX to 334'4813 2024-03-26T05:15:34.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:34 smithi132 ceph-mon[33500]: pgmap v687: 158 pgs: 4 active+clean+remapped, 98 active+clean, 34 active+undersized, 22 active+undersized+degraded; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 715 op/s; 18735/222273 objects degraded (8.429%); 1/222273 objects misplaced (0.000%) 2024-03-26T05:15:34.822 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:34.865 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:34.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:34.966 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:35.001 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:35.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:35.102 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:35.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:34 smithi080 ceph-mon[29691]: 10.c starting backfill to osd.7 from (0'0,0'0] MAX to 334'4813 2024-03-26T05:15:35.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:34 smithi080 ceph-mon[29691]: pgmap v687: 158 pgs: 4 active+clean+remapped, 98 active+clean, 34 active+undersized, 22 active+undersized+degraded; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 715 op/s; 18735/222273 objects degraded (8.429%); 1/222273 objects misplaced (0.000%) 2024-03-26T05:15:35.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:34 smithi080 ceph-mon[40404]: 10.c starting backfill to osd.7 from (0'0,0'0] MAX to 334'4813 2024-03-26T05:15:35.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:34 smithi080 ceph-mon[40404]: pgmap v687: 158 pgs: 4 active+clean+remapped, 98 active+clean, 34 active+undersized, 22 active+undersized+degraded; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 715 op/s; 18735/222273 objects degraded (8.429%); 1/222273 objects misplaced (0.000%) 2024-03-26T05:15:35.136 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T05:15:35.147 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:35.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:35.249 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:35.283 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:35.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:35.384 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:35.432 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:35.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:35.505 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:15:35.533 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:35.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:35.582 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:35.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:35.600 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 43 16 77112 77096 112.03 57.8125 0.0110023 0.00891864 2024-03-26T05:15:35.684 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:35.722 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:35.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:35.823 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:35.865 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:35.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:35.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:36.008 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:36.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:36.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:15:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:15:36.108 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:36.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:36.146 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:36.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:36.247 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:36.286 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:36.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:36.387 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:36.409 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:15:36.427 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:36.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:36.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:36.529 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:36.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:36.576 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:36.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:36.600 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 44 16 78024 78008 110.779 57 0.0157848 0.00902093 2024-03-26T05:15:36.608 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:36 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:36.608 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:36 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:15:36.609 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:36.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:36 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:36.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:36 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:15:36.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:36.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:36 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:36.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:36 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:15:36.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:36.677 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:36.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:36.722 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:36.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:36.823 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:36.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:36.874 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:36.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:36.975 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:37.012 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:37.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:37.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:37.148 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:37.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:37.249 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:37.285 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:37.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:37.387 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:37.430 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:37.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:37.501 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:15:37.501 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:15:37.502 INFO:teuthology.orchestra.run.smithi080.stdout: "complaint_time": 30, 2024-03-26T05:15:37.502 INFO:teuthology.orchestra.run.smithi080.stdout: "num_blocked_ops": 0 2024-03-26T05:15:37.502 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:15:37.532 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:37.572 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:37.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:37.600 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 45 16 78951 78935 109.605 57.9375 0.00612741 0.00911735 2024-03-26T05:15:37.603 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:37 smithi080 ceph-mon[40404]: pgmap v688: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+remapped, 134 active+clean, 11 active+undersized, 1 active+recovering+undersized+degraded+remapped, 10 active+undersized+degraded; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 42 MiB/s wr, 673 op/s; 18868/229521 objects degraded (8.221%); 1.3 MiB/s, 21 objects/s recovering 2024-03-26T05:15:37.603 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:37 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:37.603 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:37 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:37.603 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:37 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 18868/229521 objects degraded (8.221%), 12 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:37.603 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:37 smithi080 ceph-mon[29691]: pgmap v688: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+remapped, 134 active+clean, 11 active+undersized, 1 active+recovering+undersized+degraded+remapped, 10 active+undersized+degraded; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 42 MiB/s wr, 673 op/s; 18868/229521 objects degraded (8.221%); 1.3 MiB/s, 21 objects/s recovering 2024-03-26T05:15:37.604 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:37 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:37.604 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:37 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:37.604 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:37 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 18868/229521 objects degraded (8.221%), 12 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:37.614 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:37 smithi132 ceph-mon[33500]: pgmap v688: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+remapped, 134 active+clean, 11 active+undersized, 1 active+recovering+undersized+degraded+remapped, 10 active+undersized+degraded; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 42 MiB/s wr, 673 op/s; 18868/229521 objects degraded (8.221%); 1.3 MiB/s, 21 objects/s recovering 2024-03-26T05:15:37.615 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:37 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:37.615 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:37 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:37.615 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:37 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 18868/229521 objects degraded (8.221%), 12 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-26T05:15:37.673 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:37.712 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:37.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:37.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:37.815 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:37.855 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:37.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:37.956 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:37.995 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:37.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:38.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:38.129 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:38.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:38.231 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:38.269 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:38.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:38.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:38.370 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:38.371 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:15:38.404 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:38.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(10.9 338/336 [PushOp(10:9cca2eb7:::benchmark_data_smithi080_161990_object1323:head, version: 320'88, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:9cca2eb7:::benchmark_data_smithi080_161990_object1323:head@320'88, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:15:38.426470+0000", 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0021072199999999999, 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021609839999999999, 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:15:38.419 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:15:38.426470+0000", 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:15:38.426470+0000", 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:15:38.426471+0000", 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2399999999999998e-07 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:15:38.420 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:15:38.426501+0000", 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0162999999999999e-05 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:15:38.426522+0000", 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.0852999999999998e-05 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:15:38.426539+0000", 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.7204e-05 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:15:38.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:15:38.427036+0000", 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00049726700000000002 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:15:38.427056+0000", 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.9633e-05 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 1 2024-03-26T05:15:38.422 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:15:38.484 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T05:15:38.505 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:38.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:38 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2451822242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:15:38.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:38 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:15:38.573 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:38.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:38.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:38.600 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 46 16 79917 79901 108.535 60.375 0.00400146 0.00920555 2024-03-26T05:15:38.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:38 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2451822242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:15:38.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:38 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:15:38.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:38 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2451822242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:15:38.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:38 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:15:38.675 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:38.714 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:38.717 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:38.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:38.818 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:38.854 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:38.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:38.956 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:38.997 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:38.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:39.099 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:39.136 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:39.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:39.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:39.237 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:39.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:39.277 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:39.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:39.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:39.417 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:39.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:39.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:39.519 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:39.555 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:39.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:39.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:39.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[40404]: pgmap v689: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 29 MiB/s wr, 469 op/s; 19059/231351 objects degraded (8.238%); 1.7 MiB/s, 27 objects/s recovering 2024-03-26T05:15:39.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[40404]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:15:39.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:15:39.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:15:39.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[40404]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T05:15:39.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[40404]: 5.1f scrub starts 2024-03-26T05:15:39.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[40404]: 5.1f scrub ok 2024-03-26T05:15:39.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[29691]: pgmap v689: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 29 MiB/s wr, 469 op/s; 19059/231351 objects degraded (8.238%); 1.7 MiB/s, 27 objects/s recovering 2024-03-26T05:15:39.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[29691]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:15:39.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:15:39.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:15:39.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[29691]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T05:15:39.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[29691]: 5.1f scrub starts 2024-03-26T05:15:39.585 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:39 smithi080 ceph-mon[29691]: 5.1f scrub ok 2024-03-26T05:15:39.601 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 47 16 80879 80863 107.503 60.125 0.0135741 0.00929646 2024-03-26T05:15:39.656 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:39.691 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:39.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:39.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:39 smithi132 ceph-mon[33500]: pgmap v689: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 29 MiB/s wr, 469 op/s; 19059/231351 objects degraded (8.238%); 1.7 MiB/s, 27 objects/s recovering 2024-03-26T05:15:39.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:39 smithi132 ceph-mon[33500]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:15:39.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:39 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:15:39.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:39 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:15:39.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:39 smithi132 ceph-mon[33500]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T05:15:39.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:39 smithi132 ceph-mon[33500]: 5.1f scrub starts 2024-03-26T05:15:39.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:39 smithi132 ceph-mon[33500]: 5.1f scrub ok 2024-03-26T05:15:39.793 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:39.830 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:39.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:39.931 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:39.974 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:39.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:39.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:40.076 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:40.115 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:40.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:40.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:40.217 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:40.260 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:40.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:40.361 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:40.399 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:40.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:40.501 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:40.536 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:40.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:40.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:40.601 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 48 16 81887 81871 106.576 63 0.00760856 0.00937721 2024-03-26T05:15:40.638 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:40.678 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:40.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:40.718 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:40 smithi132 ceph-mon[33500]: 5.19 scrub starts 2024-03-26T05:15:40.718 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:40 smithi132 ceph-mon[33500]: 5.19 scrub ok 2024-03-26T05:15:40.718 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:40 smithi132 ceph-mon[33500]: 1.6 scrub starts 2024-03-26T05:15:40.719 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:40 smithi132 ceph-mon[33500]: 1.6 scrub ok 2024-03-26T05:15:40.780 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:40.819 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:40.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:40.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:40 smithi080 ceph-mon[29691]: 5.19 scrub starts 2024-03-26T05:15:40.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:40 smithi080 ceph-mon[29691]: 5.19 scrub ok 2024-03-26T05:15:40.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:40 smithi080 ceph-mon[29691]: 1.6 scrub starts 2024-03-26T05:15:40.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:40 smithi080 ceph-mon[29691]: 1.6 scrub ok 2024-03-26T05:15:40.838 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:40 smithi080 ceph-mon[40404]: 5.19 scrub starts 2024-03-26T05:15:40.838 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:40 smithi080 ceph-mon[40404]: 5.19 scrub ok 2024-03-26T05:15:40.838 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:40 smithi080 ceph-mon[40404]: 1.6 scrub starts 2024-03-26T05:15:40.838 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:40 smithi080 ceph-mon[40404]: 1.6 scrub ok 2024-03-26T05:15:40.921 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:40.955 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:40.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:41.057 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:41.115 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:41.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:41.217 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:41.227 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:15:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:41.253 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:41.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:41.354 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:41.391 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:41.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:41.426 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:41.493 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:41.521 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:41 smithi080 ceph-mon[29691]: 1.4 scrub starts 2024-03-26T05:15:41.521 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:41 smithi080 ceph-mon[29691]: 1.4 scrub ok 2024-03-26T05:15:41.521 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:41 smithi080 ceph-mon[29691]: pgmap v691: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 45 MiB/s wr, 717 op/s; 18831/238344 objects degraded (7.901%); 5.7 MiB/s, 91 objects/s recovering 2024-03-26T05:15:41.521 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:41 smithi080 ceph-mon[29691]: 1.7 scrub starts 2024-03-26T05:15:41.521 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:41 smithi080 ceph-mon[29691]: 1.7 scrub ok 2024-03-26T05:15:41.522 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:41 smithi080 ceph-mon[40404]: 1.4 scrub starts 2024-03-26T05:15:41.522 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:41 smithi080 ceph-mon[40404]: 1.4 scrub ok 2024-03-26T05:15:41.522 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:41 smithi080 ceph-mon[40404]: pgmap v691: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 45 MiB/s wr, 717 op/s; 18831/238344 objects degraded (7.901%); 5.7 MiB/s, 91 objects/s recovering 2024-03-26T05:15:41.522 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:41 smithi080 ceph-mon[40404]: 1.7 scrub starts 2024-03-26T05:15:41.522 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:41 smithi080 ceph-mon[40404]: 1.7 scrub ok 2024-03-26T05:15:41.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:41.542 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:41.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:41.603 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 49 16 82863 82847 105.642 61 0.0123524 0.00945984 2024-03-26T05:15:41.644 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:41.693 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:41.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:41.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:41 smithi132 ceph-mon[33500]: 1.4 scrub starts 2024-03-26T05:15:41.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:41 smithi132 ceph-mon[33500]: 1.4 scrub ok 2024-03-26T05:15:41.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:41 smithi132 ceph-mon[33500]: pgmap v691: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 45 MiB/s wr, 717 op/s; 18831/238344 objects degraded (7.901%); 5.7 MiB/s, 91 objects/s recovering 2024-03-26T05:15:41.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:41 smithi132 ceph-mon[33500]: 1.7 scrub starts 2024-03-26T05:15:41.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:41 smithi132 ceph-mon[33500]: 1.7 scrub ok 2024-03-26T05:15:41.795 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:41.843 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:41.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:41.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:41.986 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:41.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:41.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:42.088 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:42.122 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:42.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:42.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:42.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:42.261 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:42.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:42.363 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:42.406 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:42.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:42.508 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:42.544 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:42.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:42.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:42.603 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 50 16 83857 83841 104.771 62.125 0.00404029 0.00953754 2024-03-26T05:15:42.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[29691]: 6.1b scrub starts 2024-03-26T05:15:42.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[29691]: 6.1b scrub ok 2024-03-26T05:15:42.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 18831/238344 objects degraded (7.901%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:15:42.618 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[29691]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T05:15:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:15:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:15:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1220015187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:42.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[40404]: 6.1b scrub starts 2024-03-26T05:15:42.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[40404]: 6.1b scrub ok 2024-03-26T05:15:42.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 18831/238344 objects degraded (7.901%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:15:42.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[40404]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T05:15:42.619 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:15:42.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:15:42.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:42 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1220015187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:42.646 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:42.685 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:42.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:42.786 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:42.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:42 smithi132 ceph-mon[33500]: 6.1b scrub starts 2024-03-26T05:15:42.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:42 smithi132 ceph-mon[33500]: 6.1b scrub ok 2024-03-26T05:15:42.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:42 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 18831/238344 objects degraded (7.901%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:15:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:42 smithi132 ceph-mon[33500]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T05:15:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:42 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:15:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:42 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:15:42.794 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:42 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1220015187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:42.828 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:42.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:42.876 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:15:42.876 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:15:42.876 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:15:42.876 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:15:42.876 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4: {} 2024-03-26T05:15:42.876 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:15:42.876 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:15:42.876 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:15:42.877 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:15:42.877 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:15:42.877 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:15:42.877 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:15:42.877 INFO:teuthology.orchestra.run.smithi080.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T05:15:42.877 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:15:42.877 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:15:42.877 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:15:42.930 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:42.965 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:42.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:43.067 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:43.113 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:43.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:43.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:43.215 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:43.271 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:43.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:43.373 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:43.413 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:43.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:43.514 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:43.559 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:43.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:43.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:43.604 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 51 16 84849 84833 103.933 62 0.00881904 0.00961642 2024-03-26T05:15:43.662 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:43.702 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:43.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:43.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:43 smithi132 ceph-mon[33500]: 5.14 scrub starts 2024-03-26T05:15:43.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:43 smithi132 ceph-mon[33500]: 5.14 scrub ok 2024-03-26T05:15:43.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:43 smithi132 ceph-mon[33500]: pgmap v693: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 61 MiB/s wr, 967 op/s; 18902/245613 objects degraded (7.696%); 7.3 MiB/s, 116 objects/s recovering 2024-03-26T05:15:43.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:43 smithi132 ceph-mon[33500]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T05:15:43.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:43 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:43.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:43 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:43.804 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:43.838 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:43.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:43.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:43 smithi080 ceph-mon[29691]: 5.14 scrub starts 2024-03-26T05:15:43.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:43 smithi080 ceph-mon[29691]: 5.14 scrub ok 2024-03-26T05:15:43.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:43 smithi080 ceph-mon[29691]: pgmap v693: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 61 MiB/s wr, 967 op/s; 18902/245613 objects degraded (7.696%); 7.3 MiB/s, 116 objects/s recovering 2024-03-26T05:15:43.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:43 smithi080 ceph-mon[29691]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T05:15:43.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:43 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:43.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:43 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:43.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:43 smithi080 ceph-mon[40404]: 5.14 scrub starts 2024-03-26T05:15:43.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:43 smithi080 ceph-mon[40404]: 5.14 scrub ok 2024-03-26T05:15:43.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:43 smithi080 ceph-mon[40404]: pgmap v693: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 61 MiB/s wr, 967 op/s; 18902/245613 objects degraded (7.696%); 7.3 MiB/s, 116 objects/s recovering 2024-03-26T05:15:43.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:43 smithi080 ceph-mon[40404]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T05:15:43.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:43 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:43.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:43 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:43.939 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:43.974 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:43.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:44.052 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:44.076 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:44.106 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:44.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:44.208 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:44.257 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:44.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:44.265 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:15:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:15:44.358 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:44.404 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:44.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:44.505 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:44.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:44.545 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:44.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:44.603 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 52 16 85872 85856 103.164 63.9375 0.0067612 0.00968853 2024-03-26T05:15:44.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[40404]: 3.8 scrub starts 2024-03-26T05:15:44.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[40404]: 3.8 scrub ok 2024-03-26T05:15:44.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[40404]: 4.f scrub starts 2024-03-26T05:15:44.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[40404]: 4.f scrub ok 2024-03-26T05:15:44.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:15:44.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:15:44.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:44.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[29691]: 3.8 scrub starts 2024-03-26T05:15:44.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[29691]: 3.8 scrub ok 2024-03-26T05:15:44.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[29691]: 4.f scrub starts 2024-03-26T05:15:44.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[29691]: 4.f scrub ok 2024-03-26T05:15:44.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:15:44.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:15:44.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:44 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:44.646 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:44.684 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:44.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:44.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:44.786 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:44.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:44 smithi132 ceph-mon[33500]: 3.8 scrub starts 2024-03-26T05:15:44.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:44 smithi132 ceph-mon[33500]: 3.8 scrub ok 2024-03-26T05:15:44.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:44 smithi132 ceph-mon[33500]: 4.f scrub starts 2024-03-26T05:15:44.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:44 smithi132 ceph-mon[33500]: 4.f scrub ok 2024-03-26T05:15:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:44 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:15:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:44 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:15:44.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:44 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:44.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:44.825 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:44.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:44.926 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:44.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:44.966 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:44.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:45.068 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:45.111 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:45.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:45.212 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:45.248 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:45.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:45.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:45.350 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4 2024-03-26T05:15:45.386 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:45.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.4' 2024-03-26T05:15:45.489 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:45.533 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:45.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:45.604 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 53 16 86915 86899 102.447 65.1875 0.00611279 0.00975584 2024-03-26T05:15:45.635 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:45.674 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:45.675 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:45 smithi080 ceph-mon[40404]: 1.3 scrub starts 2024-03-26T05:15:45.675 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:45 smithi080 ceph-mon[40404]: 1.3 scrub ok 2024-03-26T05:15:45.675 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:45 smithi080 ceph-mon[40404]: pgmap v695: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s; 18902/245613 objects degraded (7.696%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T05:15:45.675 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:45 smithi080 ceph-mon[29691]: 1.3 scrub starts 2024-03-26T05:15:45.675 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:45 smithi080 ceph-mon[29691]: 1.3 scrub ok 2024-03-26T05:15:45.675 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:45 smithi080 ceph-mon[29691]: pgmap v695: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s; 18902/245613 objects degraded (7.696%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T05:15:45.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:45.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:45.699 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:45 smithi132 ceph-mon[33500]: 1.3 scrub starts 2024-03-26T05:15:45.699 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:45 smithi132 ceph-mon[33500]: 1.3 scrub ok 2024-03-26T05:15:45.699 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:45 smithi132 ceph-mon[33500]: pgmap v695: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s; 18902/245613 objects degraded (7.696%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T05:15:45.699 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:15:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:15:45.777 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:45.813 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:45.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:45.915 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:45.950 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:45.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:46.027 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:15:46.051 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:46.095 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:46.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:46.105 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:15:46.105 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:15:46.105 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T05:15:46.105 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T05:15:46.105 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.4 2024-03-26T05:15:46.197 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:46.237 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:46.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:46.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:46.339 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:46.381 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:46.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:46.483 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:46.541 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:46 smithi132 systemd[1]: Stopping Ceph osd.4 for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:15:46.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:46 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:46.549 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:46.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:46.604 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 54 16 87872 87856 101.657 59.8125 0.0228878 0.00983022 2024-03-26T05:15:46.650 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:46.696 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:46.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:46.759 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:46 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:46.759 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:46 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:46.798 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:46.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:46.836 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:46.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:46.937 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:46.971 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:46.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:46.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:47.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:46 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[38419]: 2024-03-26T05:15:46.668+0000 7f25a013d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:15:47.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:46 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[38419]: 2024-03-26T05:15:46.668+0000 7f25a013d700 -1 osd.4 341 *** Got signal Terminated *** 2024-03-26T05:15:47.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:46 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[38419]: 2024-03-26T05:15:46.668+0000 7f25a013d700 -1 osd.4 341 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:15:47.073 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:47.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:47.117 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:47.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:47.218 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:47.255 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:47.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:47.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:47.357 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:47.397 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:47.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:47.499 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:47.537 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:47.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:47.604 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 55 16 88856 88840 100.927 61.5 0.0134485 0.00990321 2024-03-26T05:15:47.610 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:15:47.610 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:15:47.638 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:47.675 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:47.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:47.703 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:47 smithi080 ceph-mon[40404]: pgmap v696: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 18308/257958 objects degraded (7.097%); 13 MiB/s, 208 objects/s recovering 2024-03-26T05:15:47.703 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:47 smithi080 ceph-mon[40404]: 4.5 scrub starts 2024-03-26T05:15:47.703 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:47 smithi080 ceph-mon[40404]: 4.5 scrub ok 2024-03-26T05:15:47.703 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:47 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 18308/257958 objects degraded (7.097%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:15:47.704 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:47 smithi080 ceph-mon[40404]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T05:15:47.704 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:47 smithi080 ceph-mon[40404]: osd.4 marked itself down and dead 2024-03-26T05:15:47.704 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:47 smithi080 ceph-mon[29691]: pgmap v696: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 18308/257958 objects degraded (7.097%); 13 MiB/s, 208 objects/s recovering 2024-03-26T05:15:47.704 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:47 smithi080 ceph-mon[29691]: 4.5 scrub starts 2024-03-26T05:15:47.704 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:47 smithi080 ceph-mon[29691]: 4.5 scrub ok 2024-03-26T05:15:47.704 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:47 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 18308/257958 objects degraded (7.097%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:15:47.704 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:47 smithi080 ceph-mon[29691]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T05:15:47.704 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:47 smithi080 ceph-mon[29691]: osd.4 marked itself down and dead 2024-03-26T05:15:47.777 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:47.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:47 smithi132 ceph-mon[33500]: pgmap v696: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 18308/257958 objects degraded (7.097%); 13 MiB/s, 208 objects/s recovering 2024-03-26T05:15:47.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:47 smithi132 ceph-mon[33500]: 4.5 scrub starts 2024-03-26T05:15:47.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:47 smithi132 ceph-mon[33500]: 4.5 scrub ok 2024-03-26T05:15:47.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:47 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 18308/257958 objects degraded (7.097%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:15:47.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:47 smithi132 ceph-mon[33500]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T05:15:47.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:47 smithi132 ceph-mon[33500]: osd.4 marked itself down and dead 2024-03-26T05:15:47.802 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:47.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:47.901 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:47.903 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:47.929 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:47.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:48.031 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:48.057 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:48.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:48.158 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:48.181 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:48.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:48.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:48.283 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:48.308 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:48.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:48.409 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:48.435 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:48.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:48.502 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:48 smithi132 podman[125556]: 2024-03-26 05:15:48.229435061 +0000 UTC m=+1.733018243 container died 70d4c78bb644aef2ef8a99b85b7d1fc299fd7f8e8d19e7560633c885bebf63a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-26T05:15:48.503 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:48 smithi132 ceph-mon[33500]: 4.17 scrub starts 2024-03-26T05:15:48.503 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:48 smithi132 ceph-mon[33500]: 4.17 scrub ok 2024-03-26T05:15:48.503 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:48 smithi132 ceph-mon[33500]: 6.4 scrub starts 2024-03-26T05:15:48.503 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:48 smithi132 ceph-mon[33500]: 6.4 scrub ok 2024-03-26T05:15:48.503 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:48 smithi132 ceph-mon[33500]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T05:15:48.503 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:48 smithi132 ceph-mon[33500]: osdmap e342: 8 total, 7 up, 8 in 2024-03-26T05:15:48.503 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:48 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1414138578' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:15:48.503 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:48 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:15:48.537 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:48.561 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:48.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:48.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:48.604 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 56 16 88885 88869 99.1573 1.8125 0.00845505 0.0099081 2024-03-26T05:15:48.635 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:15:48.663 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:48.704 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:48.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:48.791 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:48 smithi132 podman[125556]: 2024-03-26 05:15:48.603677962 +0000 UTC m=+2.107261145 container cleanup 70d4c78bb644aef2ef8a99b85b7d1fc299fd7f8e8d19e7560633c885bebf63a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-26T05:15:48.791 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:48 smithi132 bash[125556]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4 2024-03-26T05:15:48.806 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:48.833 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:48.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:48.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[40404]: 4.17 scrub starts 2024-03-26T05:15:48.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[40404]: 4.17 scrub ok 2024-03-26T05:15:48.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[40404]: 6.4 scrub starts 2024-03-26T05:15:48.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[40404]: 6.4 scrub ok 2024-03-26T05:15:48.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[40404]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T05:15:48.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[40404]: osdmap e342: 8 total, 7 up, 8 in 2024-03-26T05:15:48.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1414138578' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:15:48.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:15:48.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[29691]: 4.17 scrub starts 2024-03-26T05:15:48.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[29691]: 4.17 scrub ok 2024-03-26T05:15:48.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[29691]: 6.4 scrub starts 2024-03-26T05:15:48.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[29691]: 6.4 scrub ok 2024-03-26T05:15:48.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[29691]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T05:15:48.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[29691]: osdmap e342: 8 total, 7 up, 8 in 2024-03-26T05:15:48.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1414138578' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:15:48.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:48 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:15:48.935 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:48.962 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:48.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:48.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:49.064 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:49.090 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:49.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:49.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:49.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:49.219 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:49.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:49.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:49.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:48 smithi132 podman[125637]: 2024-03-26 05:15:48.880542994 +0000 UTC m=+0.648458710 container remove 70d4c78bb644aef2ef8a99b85b7d1fc299fd7f8e8d19e7560633c885bebf63a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-26T05:15:49.321 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:49.346 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:49.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:49.448 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:49.473 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:49.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:49.541 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:49 smithi132 podman[125778]: 2024-03-26 05:15:49.15563605 +0000 UTC m=+0.020762383 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:15:49.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:49 smithi132 ceph-mon[33500]: 3.0 scrub starts 2024-03-26T05:15:49.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:49 smithi132 ceph-mon[33500]: 3.0 scrub ok 2024-03-26T05:15:49.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:49 smithi132 ceph-mon[33500]: pgmap v699: 158 pgs: 1 stale+active+undersized+degraded+remapped+backfilling, 25 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_toofull, 129 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 52 MiB/s wr, 835 op/s; 18496/260751 objects degraded (7.093%); 10 MiB/s, 160 objects/s recovering 2024-03-26T05:15:49.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:49 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:15:49.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:49 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:15:49.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:49 smithi132 ceph-mon[33500]: osdmap e343: 8 total, 7 up, 8 in 2024-03-26T05:15:49.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:49 smithi132 ceph-mon[33500]: 10.1 continuing backfill to osd.7 from (320'1000,341'5569] 10:8529dc16:::benchmark_data_smithi080_161990_object88224:head to 341'5569 2024-03-26T05:15:49.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:49 smithi132 ceph-mon[33500]: 10.c continuing backfill to osd.7 from (320'600,341'5683] MIN to 341'5683 2024-03-26T05:15:49.575 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:49.601 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:49.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:49.604 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 57 16 89019 89003 97.5649 8.375 0.00378443 0.0100447 2024-03-26T05:15:49.685 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[29691]: 3.0 scrub starts 2024-03-26T05:15:49.685 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[29691]: 3.0 scrub ok 2024-03-26T05:15:49.685 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[29691]: pgmap v699: 158 pgs: 1 stale+active+undersized+degraded+remapped+backfilling, 25 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_toofull, 129 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 52 MiB/s wr, 835 op/s; 18496/260751 objects degraded (7.093%); 10 MiB/s, 160 objects/s recovering 2024-03-26T05:15:49.685 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:15:49.685 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:15:49.685 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[29691]: osdmap e343: 8 total, 7 up, 8 in 2024-03-26T05:15:49.685 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[29691]: 10.1 continuing backfill to osd.7 from (320'1000,341'5569] 10:8529dc16:::benchmark_data_smithi080_161990_object88224:head to 341'5569 2024-03-26T05:15:49.685 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[29691]: 10.c continuing backfill to osd.7 from (320'600,341'5683] MIN to 341'5683 2024-03-26T05:15:49.686 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[40404]: 3.0 scrub starts 2024-03-26T05:15:49.686 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[40404]: 3.0 scrub ok 2024-03-26T05:15:49.686 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[40404]: pgmap v699: 158 pgs: 1 stale+active+undersized+degraded+remapped+backfilling, 25 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_toofull, 129 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 52 MiB/s wr, 835 op/s; 18496/260751 objects degraded (7.093%); 10 MiB/s, 160 objects/s recovering 2024-03-26T05:15:49.686 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:15:49.686 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:15:49.686 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[40404]: osdmap e343: 8 total, 7 up, 8 in 2024-03-26T05:15:49.686 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[40404]: 10.1 continuing backfill to osd.7 from (320'1000,341'5569] 10:8529dc16:::benchmark_data_smithi080_161990_object88224:head to 341'5569 2024-03-26T05:15:49.686 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:49 smithi080 ceph-mon[40404]: 10.c continuing backfill to osd.7 from (320'600,341'5683] MIN to 341'5683 2024-03-26T05:15:49.687 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T05:15:49.703 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:49.751 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:49.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:49.842 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:49 smithi132 podman[125778]: 2024-03-26 05:15:49.592888958 +0000 UTC m=+0.458015276 container create 5ed20e72af7599bede2b96d893328c0ab9d18824d4f77b109fdccc0a41c3a234 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-26T05:15:49.842 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:49 smithi132 podman[125778]: 2024-03-26 05:15:49.85207494 +0000 UTC m=+0.717201267 container init 5ed20e72af7599bede2b96d893328c0ab9d18824d4f77b109fdccc0a41c3a234 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T05:15:49.852 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:49.879 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:49.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:49.981 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:50.010 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:50.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:50.112 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:50.140 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:50.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:50.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:50.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:49 smithi132 podman[125778]: 2024-03-26 05:15:49.856936249 +0000 UTC m=+0.722062570 container start 5ed20e72af7599bede2b96d893328c0ab9d18824d4f77b109fdccc0a41c3a234 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-26T05:15:50.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:49 smithi132 podman[125778]: 2024-03-26 05:15:49.951791262 +0000 UTC m=+0.816917588 container attach 5ed20e72af7599bede2b96d893328c0ab9d18824d4f77b109fdccc0a41c3a234 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-deactivate, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:15:50.242 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:50.269 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:50.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:50.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:50.399 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:50.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:50.500 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:50.527 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:50.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:50.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:50.541 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:15:50 smithi132 podman[125778]: 2024-03-26 05:15:50.256056822 +0000 UTC m=+1.121183132 container died 5ed20e72af7599bede2b96d893328c0ab9d18824d4f77b109fdccc0a41c3a234 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T05:15:50.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:50 smithi132 ceph-mon[33500]: 5.17 scrub starts 2024-03-26T05:15:50.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:50 smithi132 ceph-mon[33500]: 5.17 scrub ok 2024-03-26T05:15:50.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:50 smithi132 ceph-mon[33500]: 4.a deep-scrub starts 2024-03-26T05:15:50.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:50 smithi132 ceph-mon[33500]: 4.a deep-scrub ok 2024-03-26T05:15:50.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:50.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:15:50.604 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 58 16 89019 89003 95.883 0 - 0.0100447 2024-03-26T05:15:50.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:50 smithi080 ceph-mon[40404]: 5.17 scrub starts 2024-03-26T05:15:50.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:50 smithi080 ceph-mon[40404]: 5.17 scrub ok 2024-03-26T05:15:50.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:50 smithi080 ceph-mon[40404]: 4.a deep-scrub starts 2024-03-26T05:15:50.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:50 smithi080 ceph-mon[40404]: 4.a deep-scrub ok 2024-03-26T05:15:50.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:50.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:15:50.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:50 smithi080 ceph-mon[29691]: 5.17 scrub starts 2024-03-26T05:15:50.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:50 smithi080 ceph-mon[29691]: 5.17 scrub ok 2024-03-26T05:15:50.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:50 smithi080 ceph-mon[29691]: 4.a deep-scrub starts 2024-03-26T05:15:50.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:50 smithi080 ceph-mon[29691]: 4.a deep-scrub ok 2024-03-26T05:15:50.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:15:50.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:15:50.630 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:50.655 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:50.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:50.757 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:50.784 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:50.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:50.886 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:50.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:50.915 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:50.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:51.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:51.041 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:51.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:51.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:51.143 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:51.168 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:51.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:51.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:51.270 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:51.296 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:51.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:51.399 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:51.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:51.425 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:51.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:51.526 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:51.551 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:51.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:51.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:51.604 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 59 16 89019 89003 94.2582 0 - 0.0100447 2024-03-26T05:15:51.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:51 smithi080 ceph-mon[29691]: pgmap v701: 158 pgs: 1 stale+active+undersized+degraded+remapped+backfilling, 25 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_toofull, 129 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 52 MiB/s wr, 835 op/s; 18496/260751 objects degraded (7.093%); 10 MiB/s, 160 objects/s recovering 2024-03-26T05:15:51.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:51 smithi080 ceph-mon[40404]: pgmap v701: 158 pgs: 1 stale+active+undersized+degraded+remapped+backfilling, 25 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_toofull, 129 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 52 MiB/s wr, 835 op/s; 18496/260751 objects degraded (7.093%); 10 MiB/s, 160 objects/s recovering 2024-03-26T05:15:51.653 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:51.680 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:51.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:51.782 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:51.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:51 smithi132 ceph-mon[33500]: pgmap v701: 158 pgs: 1 stale+active+undersized+degraded+remapped+backfilling, 25 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_toofull, 129 active+clean, 2 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 52 MiB/s wr, 835 op/s; 18496/260751 objects degraded (7.093%); 10 MiB/s, 160 objects/s recovering 2024-03-26T05:15:51.808 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:51.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:51.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:51.938 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:51.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:52.039 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:52.067 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:52.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:52.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:52.169 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:52.194 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:52.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:52.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:52.296 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:52.324 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:52.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:52.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:52.426 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:52.453 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:52.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:52.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:52.555 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:52.578 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:52.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:52.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:52.604 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:15:52.615479+0000 min lat: 0.00134127 max lat: 2.16921 avg lat: 0.0100447 2024-03-26T05:15:52.605 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:15:52.605 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 60 16 89019 89003 92.6875 0 - 0.0100447 2024-03-26T05:15:52.679 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:52.701 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[40404]: 4.11 scrub starts 2024-03-26T05:15:52.701 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[40404]: 4.11 scrub ok 2024-03-26T05:15:52.701 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[40404]: 6.1 scrub starts 2024-03-26T05:15:52.701 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[40404]: 6.1 scrub ok 2024-03-26T05:15:52.701 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[40404]: 3.10 scrub starts 2024-03-26T05:15:52.701 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[40404]: 3.10 scrub ok 2024-03-26T05:15:52.701 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 18496/260751 objects degraded (7.093%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:15:52.701 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[29691]: 4.11 scrub starts 2024-03-26T05:15:52.701 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[29691]: 4.11 scrub ok 2024-03-26T05:15:52.701 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[29691]: 6.1 scrub starts 2024-03-26T05:15:52.702 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[29691]: 6.1 scrub ok 2024-03-26T05:15:52.702 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[29691]: 3.10 scrub starts 2024-03-26T05:15:52.702 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[29691]: 3.10 scrub ok 2024-03-26T05:15:52.702 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:52 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 18496/260751 objects degraded (7.093%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:15:52.706 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:52.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:52.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:52.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:52 smithi132 ceph-mon[33500]: 4.11 scrub starts 2024-03-26T05:15:52.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:52 smithi132 ceph-mon[33500]: 4.11 scrub ok 2024-03-26T05:15:52.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:52 smithi132 ceph-mon[33500]: 6.1 scrub starts 2024-03-26T05:15:52.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:52 smithi132 ceph-mon[33500]: 6.1 scrub ok 2024-03-26T05:15:52.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:52 smithi132 ceph-mon[33500]: 3.10 scrub starts 2024-03-26T05:15:52.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:52 smithi132 ceph-mon[33500]: 3.10 scrub ok 2024-03-26T05:15:52.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:52 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 18496/260751 objects degraded (7.093%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:15:52.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:52.835 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:52.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:52.937 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:52.960 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:52.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:53.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:53.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:53.091 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:53.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:53.191 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:53.217 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:53.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:53.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:53.319 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:53.346 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:53.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:53.448 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:53.474 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:53.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:53.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:53.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:53 smithi132 ceph-mon[33500]: pgmap v702: 158 pgs: 2 undersized+degraded+remapped+backfilling+peered, 37 active+undersized, 89 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 34 MiB/s wr, 541 op/s; 59472/267702 objects degraded (22.216%); 39 MiB/s, 625 objects/s recovering 2024-03-26T05:15:53.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:53 smithi132 ceph-mon[33500]: 4.4 scrub starts 2024-03-26T05:15:53.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:53 smithi132 ceph-mon[33500]: 4.4 scrub ok 2024-03-26T05:15:53.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:53 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:15:53.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:53 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:15:53.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:53 smithi132 ceph-mon[33500]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T05:15:53.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:53 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2878929663' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:53.551 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:15:53.551 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:15:53.552 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:15:53.576 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:53.599 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:53.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:53.605 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 61 16 89019 89003 91.1682 0 - 0.0100447 2024-03-26T05:15:53.701 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:53.728 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:53.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:53.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:53.804 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:15:53.830 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:53.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[40404]: pgmap v702: 158 pgs: 2 undersized+degraded+remapped+backfilling+peered, 37 active+undersized, 89 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 34 MiB/s wr, 541 op/s; 59472/267702 objects degraded (22.216%); 39 MiB/s, 625 objects/s recovering 2024-03-26T05:15:53.831 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[40404]: 4.4 scrub starts 2024-03-26T05:15:53.831 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[40404]: 4.4 scrub ok 2024-03-26T05:15:53.831 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:15:53.831 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:15:53.831 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[40404]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T05:15:53.831 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2878929663' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:53.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[29691]: pgmap v702: 158 pgs: 2 undersized+degraded+remapped+backfilling+peered, 37 active+undersized, 89 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 34 MiB/s wr, 541 op/s; 59472/267702 objects degraded (22.216%); 39 MiB/s, 625 objects/s recovering 2024-03-26T05:15:53.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[29691]: 4.4 scrub starts 2024-03-26T05:15:53.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[29691]: 4.4 scrub ok 2024-03-26T05:15:53.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:15:53.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:15:53.832 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[29691]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T05:15:53.832 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:53 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2878929663' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:15:53.855 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:53.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:53.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:53.958 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:53.985 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:53.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:53.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:54.086 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:54.112 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:54.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:54.115 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:15:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:15:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:15:54.214 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:54.241 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:54.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:54.343 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:54.369 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:54.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:54.471 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:54.495 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:54.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:54.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:54.597 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:54.604 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 62 16 89019 89003 89.698 0 - 0.0100447 2024-03-26T05:15:54.620 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:54.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:54.722 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:54.748 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:54.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:54.850 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:54.874 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:54.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:54.975 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:54.999 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:55.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:55.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:55.101 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:55.134 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:55.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:55.236 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:55.262 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:55.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:55.285 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:15:55.363 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:55.390 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:55.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:55.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:55.492 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:55.520 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:55.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:55.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:55.605 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 63 16 89019 89003 88.2744 0 - 0.0100447 2024-03-26T05:15:55.622 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:55.645 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:55.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:55.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:55.747 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:55.773 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:55.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:55.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:55.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:55 smithi132 ceph-mon[33500]: pgmap v703: 158 pgs: 2 undersized+degraded+remapped+backfilling+peered, 37 active+undersized, 89 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 19 MiB/s wr, 310 op/s; 59472/267702 objects degraded (22.216%); 30 MiB/s, 487 objects/s recovering 2024-03-26T05:15:55.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:15:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:15:55.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:55 smithi080 ceph-mon[29691]: pgmap v703: 158 pgs: 2 undersized+degraded+remapped+backfilling+peered, 37 active+undersized, 89 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 19 MiB/s wr, 310 op/s; 59472/267702 objects degraded (22.216%); 30 MiB/s, 487 objects/s recovering 2024-03-26T05:15:55.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:55 smithi080 ceph-mon[40404]: pgmap v703: 158 pgs: 2 undersized+degraded+remapped+backfilling+peered, 37 active+undersized, 89 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 19 MiB/s wr, 310 op/s; 59472/267702 objects degraded (22.216%); 30 MiB/s, 487 objects/s recovering 2024-03-26T05:15:55.875 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:55.901 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:55.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:56.003 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:56.030 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:56.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:56.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:56.132 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:56.160 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:56.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:56.261 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:56.287 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:56.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:56.389 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:56.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:56.414 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:56.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:56.515 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:15:56.542 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:56.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:15:56.605 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 64 16 89019 89003 86.8954 0 - 0.0100447 2024-03-26T05:15:56.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:56 smithi080 ceph-mon[40404]: 4.0 scrub starts 2024-03-26T05:15:56.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:56 smithi080 ceph-mon[40404]: 4.0 scrub ok 2024-03-26T05:15:56.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:56 smithi080 ceph-mon[40404]: 10.b scrub starts 2024-03-26T05:15:56.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:56 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:56.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:56 smithi080 ceph-mon[29691]: 4.0 scrub starts 2024-03-26T05:15:56.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:56 smithi080 ceph-mon[29691]: 4.0 scrub ok 2024-03-26T05:15:56.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:56 smithi080 ceph-mon[29691]: 10.b scrub starts 2024-03-26T05:15:56.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:56 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:56.619 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T05:15:56.644 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:56.669 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:15:56.669 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T05:15:56.669 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-03-26T05:15:56.670 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi132.front.sepia.ceph.com 2024-03-26T05:15:56.670 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -n osd.4 --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-03-26T05:15:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:56.699 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:56.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:56.729 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:56 smithi132 ceph-mon[33500]: 4.0 scrub starts 2024-03-26T05:15:56.729 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:56 smithi132 ceph-mon[33500]: 4.0 scrub ok 2024-03-26T05:15:56.729 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:56 smithi132 ceph-mon[33500]: 10.b scrub starts 2024-03-26T05:15:56.729 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:56 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:15:56.800 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:56.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:56.825 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:56.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:56.926 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:56.950 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:56.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:56.989 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/osd.4/config 2024-03-26T05:15:57.052 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:57.080 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:57.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:57.182 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:57.211 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:57.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:57.313 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:57.338 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:57.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:57.440 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:57.464 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:57.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:57.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:57.566 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:57.589 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:57.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:57.605 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 65 16 89201 89185 85.7336 1.42188 0.00386171 0.0108758 2024-03-26T05:15:57.691 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:57.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:57.718 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:57.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:57.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:57 smithi132 ceph-mon[33500]: pgmap v704: 158 pgs: 2 undersized+degraded+remapped+backfilling+peered, 37 active+undersized, 89 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 242 B/s rd, 17 MiB/s wr, 274 op/s; 59044/267702 objects degraded (22.056%); 30 MiB/s, 481 objects/s recovering 2024-03-26T05:15:57.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:57 smithi132 ceph-mon[33500]: 10.7 scrub starts 2024-03-26T05:15:57.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:57 smithi132 ceph-mon[33500]: osdmap e344: 8 total, 7 up, 8 in 2024-03-26T05:15:57.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:57 smithi132 ceph-mon[33500]: 10.b scrub ok 2024-03-26T05:15:57.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:57 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 59044/267702 objects degraded (22.056%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:15:57.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:57 smithi132 ceph-mon[33500]: osdmap e345: 8 total, 7 up, 8 in 2024-03-26T05:15:57.818 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:57.843 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:57.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:57.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:57 smithi080 ceph-mon[29691]: pgmap v704: 158 pgs: 2 undersized+degraded+remapped+backfilling+peered, 37 active+undersized, 89 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 242 B/s rd, 17 MiB/s wr, 274 op/s; 59044/267702 objects degraded (22.056%); 30 MiB/s, 481 objects/s recovering 2024-03-26T05:15:57.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:57 smithi080 ceph-mon[29691]: 10.7 scrub starts 2024-03-26T05:15:57.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:57 smithi080 ceph-mon[29691]: osdmap e344: 8 total, 7 up, 8 in 2024-03-26T05:15:57.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:57 smithi080 ceph-mon[29691]: 10.b scrub ok 2024-03-26T05:15:57.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:57 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 59044/267702 objects degraded (22.056%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:15:57.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:57 smithi080 ceph-mon[29691]: osdmap e345: 8 total, 7 up, 8 in 2024-03-26T05:15:57.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:57 smithi080 ceph-mon[40404]: pgmap v704: 158 pgs: 2 undersized+degraded+remapped+backfilling+peered, 37 active+undersized, 89 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 242 B/s rd, 17 MiB/s wr, 274 op/s; 59044/267702 objects degraded (22.056%); 30 MiB/s, 481 objects/s recovering 2024-03-26T05:15:57.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:57 smithi080 ceph-mon[40404]: 10.7 scrub starts 2024-03-26T05:15:57.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:57 smithi080 ceph-mon[40404]: osdmap e344: 8 total, 7 up, 8 in 2024-03-26T05:15:57.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:57 smithi080 ceph-mon[40404]: 10.b scrub ok 2024-03-26T05:15:57.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:57 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 59044/267702 objects degraded (22.056%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:15:57.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:57 smithi080 ceph-mon[40404]: osdmap e345: 8 total, 7 up, 8 in 2024-03-26T05:15:57.945 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:57.969 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:57.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:58.070 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:58.095 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:58.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:58.197 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:58.223 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:58.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:58.325 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:58.349 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:58.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:58.451 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:58.478 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:58.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:58.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:58.580 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:15:58.604 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:58.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:15:58.606 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 66 16 89201 89185 84.4348 0 - 0.0108758 2024-03-26T05:15:58.706 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:58.732 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:58.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:58.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:58.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:58 smithi132 ceph-mon[33500]: 4.14 deep-scrub starts 2024-03-26T05:15:58.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:58 smithi132 ceph-mon[33500]: 4.14 deep-scrub ok 2024-03-26T05:15:58.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:58 smithi132 ceph-mon[33500]: 10.7 scrub ok 2024-03-26T05:15:58.834 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:58.860 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:58.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:58.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:58 smithi080 ceph-mon[40404]: 4.14 deep-scrub starts 2024-03-26T05:15:58.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:58 smithi080 ceph-mon[40404]: 4.14 deep-scrub ok 2024-03-26T05:15:58.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:58 smithi080 ceph-mon[40404]: 10.7 scrub ok 2024-03-26T05:15:58.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:58 smithi080 ceph-mon[29691]: 4.14 deep-scrub starts 2024-03-26T05:15:58.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:58 smithi080 ceph-mon[29691]: 4.14 deep-scrub ok 2024-03-26T05:15:58.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:58 smithi080 ceph-mon[29691]: 10.7 scrub ok 2024-03-26T05:15:58.961 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:58.987 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:58.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:58.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:59.089 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:59.114 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:59.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:59.216 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:15:59.241 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:59.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:59.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:15:59.343 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:15:59.368 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:59.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:15:59.470 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:59.497 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:59.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:59.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:59.599 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:15:59.605 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 67 16 89201 89185 83.1748 0 - 0.0108758 2024-03-26T05:15:59.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:59 smithi080 ceph-mon[29691]: pgmap v707: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 undersized+degraded+remapped+peered, 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 255 B/s rd, 18 MiB/s wr, 289 op/s; 54253/267702 objects degraded (20.266%); 69 MiB/s, 1.11k objects/s recovering 2024-03-26T05:15:59.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:59 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:15:59.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:15:59 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:15:59.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:59 smithi080 ceph-mon[40404]: pgmap v707: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 undersized+degraded+remapped+peered, 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 255 B/s rd, 18 MiB/s wr, 289 op/s; 54253/267702 objects degraded (20.266%); 69 MiB/s, 1.11k objects/s recovering 2024-03-26T05:15:59.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:59 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:15:59.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:15:59 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:15:59.624 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:59.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:15:59.726 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:15:59.751 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:15:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:59.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:15:59.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:59 smithi132 ceph-mon[33500]: pgmap v707: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 undersized+degraded+remapped+peered, 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 255 B/s rd, 18 MiB/s wr, 289 op/s; 54253/267702 objects degraded (20.266%); 69 MiB/s, 1.11k objects/s recovering 2024-03-26T05:15:59.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:59 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:15:59.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:15:59 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:15:59.853 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:15:59.882 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:15:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:15:59.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:15:59.984 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:00.007 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:00.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:00.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:00.041 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:16:00.041 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:16:00.110 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:00.134 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:00.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:00.222 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:00.237 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:00.261 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:00.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:00.363 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:00.388 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:00.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:00.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:00.490 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:00.514 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:00.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:00.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:00.605 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 68 16 89201 89185 81.9518 0 - 0.0108758 2024-03-26T05:16:00.616 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:00.639 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:00.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:00.711 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:00 smithi132 ceph-mon[33500]: 3.7 scrub starts 2024-03-26T05:16:00.711 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:00 smithi132 ceph-mon[33500]: 3.7 scrub ok 2024-03-26T05:16:00.711 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:00 smithi132 ceph-mon[33500]: osdmap e346: 8 total, 7 up, 8 in 2024-03-26T05:16:00.741 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:00.765 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:00.766 INFO:teuthology.orchestra.run.smithi132.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T05:16:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:00.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:00.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:00 smithi080 ceph-mon[40404]: 3.7 scrub starts 2024-03-26T05:16:00.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:00 smithi080 ceph-mon[40404]: 3.7 scrub ok 2024-03-26T05:16:00.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:00 smithi080 ceph-mon[40404]: osdmap e346: 8 total, 7 up, 8 in 2024-03-26T05:16:00.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:00 smithi080 ceph-mon[29691]: 3.7 scrub starts 2024-03-26T05:16:00.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:00 smithi080 ceph-mon[29691]: 3.7 scrub ok 2024-03-26T05:16:00.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:00 smithi080 ceph-mon[29691]: osdmap e346: 8 total, 7 up, 8 in 2024-03-26T05:16:00.868 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:00.893 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:00.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:00.995 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:01.018 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:01.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:01.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:01.120 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:01.143 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:01.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:01.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:01.245 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:01.271 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:01.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:01.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:01.401 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:01.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:01.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:01.503 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:01.528 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:01.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:01.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:01.605 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 69 16 89273 89257 80.8295 1.125 0.0168493 0.0123666 2024-03-26T05:16:01.630 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:01.665 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:01.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:01.745 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:01 smithi132 ceph-mon[33500]: pgmap v709: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 undersized+degraded+remapped+peered, 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54253/267702 objects degraded (20.266%); 54 MiB/s, 869 objects/s recovering 2024-03-26T05:16:01.745 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:01 smithi132 ceph-mon[33500]: osdmap e347: 8 total, 7 up, 8 in 2024-03-26T05:16:01.767 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:01.803 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:01.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:01.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:01.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:01 smithi080 ceph-mon[29691]: pgmap v709: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 undersized+degraded+remapped+peered, 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54253/267702 objects degraded (20.266%); 54 MiB/s, 869 objects/s recovering 2024-03-26T05:16:01.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:01 smithi080 ceph-mon[29691]: osdmap e347: 8 total, 7 up, 8 in 2024-03-26T05:16:01.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:01 smithi080 ceph-mon[40404]: pgmap v709: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 undersized+degraded+remapped+peered, 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 28 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54253/267702 objects degraded (20.266%); 54 MiB/s, 869 objects/s recovering 2024-03-26T05:16:01.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:01 smithi080 ceph-mon[40404]: osdmap e347: 8 total, 7 up, 8 in 2024-03-26T05:16:01.905 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:01.937 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:01.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:01.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:02.039 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:02.041 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:16:02.086 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:02.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:02.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:02.191 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:02.228 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:02.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:02.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:02.330 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:02.359 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:02.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:02.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:02.460 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:02.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:02.501 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:02.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:02.602 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:02.605 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 70 16 90323 90307 80.6123 65.625 0.0250658 0.0123962 2024-03-26T05:16:02.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:02.631 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:02.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:02.731 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:02.765 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:02.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:02.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:02.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:02 smithi132 ceph-mon[33500]: osdmap e348: 8 total, 7 up, 8 in 2024-03-26T05:16:02.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:02 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 54253/267702 objects degraded (20.266%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:02.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:02 smithi132 ceph-mon[33500]: osdmap e349: 8 total, 7 up, 8 in 2024-03-26T05:16:02.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:02 smithi080 ceph-mon[40404]: osdmap e348: 8 total, 7 up, 8 in 2024-03-26T05:16:02.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:02 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 54253/267702 objects degraded (20.266%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:02.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:02 smithi080 ceph-mon[40404]: osdmap e349: 8 total, 7 up, 8 in 2024-03-26T05:16:02.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:02 smithi080 ceph-mon[29691]: osdmap e348: 8 total, 7 up, 8 in 2024-03-26T05:16:02.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:02 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 54253/267702 objects degraded (20.266%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:02.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:02 smithi080 ceph-mon[29691]: osdmap e349: 8 total, 7 up, 8 in 2024-03-26T05:16:02.867 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:02.903 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:02.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:03.005 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:03.039 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:03.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:03.141 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:03.179 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:03.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:03.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:03.281 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:03.314 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:03.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:03.415 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:03.452 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:03.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:03.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:03.554 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:03.591 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:03.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:03.605 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 71 16 91396 91380 80.4214 67.0625 0.0148095 0.0124294 2024-03-26T05:16:03.693 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:03.728 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:03.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:03.830 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:03.865 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:03.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:03 smithi080 ceph-mon[29691]: pgmap v713: 158 pgs: 1 peering, 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 29 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.8 MiB/s wr, 46 op/s; 46034/268248 objects degraded (17.161%); 41 MiB/s, 655 objects/s recovering 2024-03-26T05:16:03.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:03 smithi080 ceph-mon[29691]: 6.1f scrub starts 2024-03-26T05:16:03.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:03 smithi080 ceph-mon[29691]: 6.1f scrub ok 2024-03-26T05:16:03.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:03 smithi080 ceph-mon[40404]: pgmap v713: 158 pgs: 1 peering, 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 29 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.8 MiB/s wr, 46 op/s; 46034/268248 objects degraded (17.161%); 41 MiB/s, 655 objects/s recovering 2024-03-26T05:16:03.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:03 smithi080 ceph-mon[40404]: 6.1f scrub starts 2024-03-26T05:16:03.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:03 smithi080 ceph-mon[40404]: 6.1f scrub ok 2024-03-26T05:16:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:03.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:03.968 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:04.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:04.009 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:04.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:03 smithi132 ceph-mon[33500]: pgmap v713: 158 pgs: 1 peering, 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 29 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.8 MiB/s wr, 46 op/s; 46034/268248 objects degraded (17.161%); 41 MiB/s, 655 objects/s recovering 2024-03-26T05:16:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:03 smithi132 ceph-mon[33500]: 6.1f scrub starts 2024-03-26T05:16:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:03 smithi132 ceph-mon[33500]: 6.1f scrub ok 2024-03-26T05:16:04.110 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:04.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:04.143 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:04.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:04.244 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:04.283 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:04.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:04.318 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:16:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:16:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:16:04.384 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:04.421 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:04.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:04.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:04.523 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:04.561 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:04.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:04.606 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 72 16 92517 92501 80.2774 70.0625 0.00763825 0.0124521 2024-03-26T05:16:04.661 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:04.705 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:04.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:04.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:04.842 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:04.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:04.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:04 smithi080 ceph-mon[40404]: 6.11 scrub starts 2024-03-26T05:16:04.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:04 smithi080 ceph-mon[40404]: 6.11 scrub ok 2024-03-26T05:16:04.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:04 smithi080 ceph-mon[29691]: 6.11 scrub starts 2024-03-26T05:16:04.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:04 smithi080 ceph-mon[29691]: 6.11 scrub ok 2024-03-26T05:16:04.945 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:04.979 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:04.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:04.993 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:16:04.994 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:16:04.994 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 0 2024-03-26T05:16:04.994 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:16:05.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:04 smithi132 ceph-mon[33500]: 6.11 scrub starts 2024-03-26T05:16:05.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:04 smithi132 ceph-mon[33500]: 6.11 scrub ok 2024-03-26T05:16:05.081 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:05.118 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:05.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:05.219 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:05.232 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:05.268 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:05.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:05.369 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:05.410 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:05.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:05.511 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:05.546 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:05.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:05.606 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 73 16 93599 93583 80.1041 67.625 0.0205021 0.0124782 2024-03-26T05:16:05.648 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:05.683 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:05.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:05.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:05.785 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:05.823 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:05.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:05.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:05.909 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:05 smithi080 ceph-mon[29691]: pgmap v714: 158 pgs: 1 peering, 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 29 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.5 MiB/s wr, 40 op/s; 46034/268248 objects degraded (17.161%); 36 MiB/s, 572 objects/s recovering 2024-03-26T05:16:05.909 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:05.909 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:16:05.910 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:05 smithi080 ceph-mon[40404]: pgmap v714: 158 pgs: 1 peering, 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 29 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.5 MiB/s wr, 40 op/s; 46034/268248 objects degraded (17.161%); 36 MiB/s, 572 objects/s recovering 2024-03-26T05:16:05.910 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:05.910 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:16:05.925 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:05.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:05 smithi132 ceph-mon[33500]: pgmap v714: 158 pgs: 1 peering, 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 29 active+undersized+degraded; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.5 MiB/s wr, 40 op/s; 46034/268248 objects degraded (17.161%); 36 MiB/s, 572 objects/s recovering 2024-03-26T05:16:05.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:05.943 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:16:05.943 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:16:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:16:05.964 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:05.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:06.066 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:06.102 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:06.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:06.204 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:06.241 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:06.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:06.343 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:06.383 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:06.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:06.484 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:06.523 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:06.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:06.606 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 74 16 94752 94736 79.9953 72.0625 0.0169628 0.0124964 2024-03-26T05:16:06.624 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:06.660 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:06.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:06.762 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:06.803 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:06.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:06.817 INFO:teuthology.orchestra.run.smithi132.stdout:6.e 2024-03-26T05:16:06.817 INFO:teuthology.orchestra.run.smithi132.stdout:6.6 2024-03-26T05:16:06.817 INFO:teuthology.orchestra.run.smithi132.stdout:6.1e 2024-03-26T05:16:06.817 INFO:teuthology.orchestra.run.smithi132.stdout:6.12 2024-03-26T05:16:06.817 INFO:teuthology.orchestra.run.smithi132.stdout:5.f 2024-03-26T05:16:06.817 INFO:teuthology.orchestra.run.smithi132.stdout:5.c 2024-03-26T05:16:06.817 INFO:teuthology.orchestra.run.smithi132.stdout:5.a 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:5.9 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:5.5 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:6.f 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:6.2 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:5.1d 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:5.1a 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:5.1 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:5.11 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:4.f 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:4.d 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:6.7 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:1.3 2024-03-26T05:16:06.818 INFO:teuthology.orchestra.run.smithi132.stdout:5.10 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:4.c 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:10.d 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:3.13 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:4.1b 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:6.15 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:10.c 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:6.b 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:10.4 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:5.e 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:3.11 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:10.6 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:3.15 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:6.17 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:5.1f 2024-03-26T05:16:06.819 INFO:teuthology.orchestra.run.smithi132.stdout:4.5 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:6.1d 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:3.14 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:4.1d 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:10.0 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:1.5 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:5.18 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:10.1 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:4.e 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:3.1 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:5.1c 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:3.1a 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:6.1a 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:3.1b 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:3.6 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:3.1d 2024-03-26T05:16:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:5.0 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:3.3 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:3.e 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:3.f 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:4.13 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:6.14 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:10.e 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:1.2 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:4.18 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:3.19 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:10.2 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:4.1a 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:4.2 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:4.9 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:4.1 2024-03-26T05:16:06.821 INFO:teuthology.orchestra.run.smithi132.stdout:3.9 2024-03-26T05:16:06.822 INFO:teuthology.orchestra.run.smithi132.stdout:4.b 2024-03-26T05:16:06.836 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T05:16:06.884 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:06 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:06.884 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:06 smithi080 ceph-mon[29691]: 3.2 scrub starts 2024-03-26T05:16:06.884 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:06 smithi080 ceph-mon[29691]: 3.2 scrub ok 2024-03-26T05:16:06.884 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:06 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 46034/268248 objects degraded (17.161%), 31 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:06.884 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:06 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:06.884 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:06 smithi080 ceph-mon[40404]: 3.2 scrub starts 2024-03-26T05:16:06.884 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:06 smithi080 ceph-mon[40404]: 3.2 scrub ok 2024-03-26T05:16:06.884 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:06 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 46034/268248 objects degraded (17.161%), 31 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:06.904 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:06.945 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:06.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:06 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:06 smithi132 ceph-mon[33500]: 3.2 scrub starts 2024-03-26T05:16:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:06 smithi132 ceph-mon[33500]: 3.2 scrub ok 2024-03-26T05:16:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:06 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 46034/268248 objects degraded (17.161%), 31 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:07.046 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:07.081 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:07.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:07.182 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:07.220 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:07.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:07.320 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:07.359 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:07.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:07.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:07.460 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:07.496 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:07.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:07.598 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:07.606 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 75 16 95843 95827 79.8378 68.1875 0.00825039 0.0125204 2024-03-26T05:16:07.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:07.633 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:07.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:07.675 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:07 smithi080 ceph-mon[29691]: pgmap v715: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 37 MiB/s wr, 600 op/s; 53240/278490 objects degraded (19.117%); 30 MiB/s, 485 objects/s recovering 2024-03-26T05:16:07.675 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:07 smithi080 ceph-mon[29691]: 3.4 scrub starts 2024-03-26T05:16:07.675 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:07 smithi080 ceph-mon[29691]: 3.4 scrub ok 2024-03-26T05:16:07.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:07 smithi080 ceph-mon[40404]: pgmap v715: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 37 MiB/s wr, 600 op/s; 53240/278490 objects degraded (19.117%); 30 MiB/s, 485 objects/s recovering 2024-03-26T05:16:07.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:07 smithi080 ceph-mon[40404]: 3.4 scrub starts 2024-03-26T05:16:07.676 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:07 smithi080 ceph-mon[40404]: 3.4 scrub ok 2024-03-26T05:16:07.733 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:07.783 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:07.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:07.884 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:07.927 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:07.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:08.028 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:07 smithi132 ceph-mon[33500]: pgmap v715: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 37 MiB/s wr, 600 op/s; 53240/278490 objects degraded (19.117%); 30 MiB/s, 485 objects/s recovering 2024-03-26T05:16:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:07 smithi132 ceph-mon[33500]: 3.4 scrub starts 2024-03-26T05:16:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:07 smithi132 ceph-mon[33500]: 3.4 scrub ok 2024-03-26T05:16:08.073 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:08.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:08.135 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -n osd.4 --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 4.d --file /var/log/ceph/exp.4.d.4 2024-03-26T05:16:08.175 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:08.209 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:08.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:08.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:08.311 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:08.352 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:08.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:08.417 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/osd.4/config 2024-03-26T05:16:08.453 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:08.488 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:08.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:08.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:08.589 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:08.606 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 76 16 96864 96848 79.6269 63.8125 0.00932697 0.0125532 2024-03-26T05:16:08.619 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:08.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:08.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:08.642 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:16:08.721 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:08.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:08.757 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:08.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:08.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:08 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/774828329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:16:08.859 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:08.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:08 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/774828329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:16:08.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:08 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/774828329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:16:08.896 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:08.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:08.998 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:09.034 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:09.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:09.135 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:09.158 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:09.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:09.260 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:09.296 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:09.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:09.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:09.398 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:09.432 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:09.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:09.533 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:09.560 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:09.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:09.606 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 77 16 97917 97901 79.4475 65.8125 0.00825356 0.0125824 2024-03-26T05:16:09.662 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:09.716 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:09.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:09.817 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:09.843 INFO:teuthology.orchestra.run.smithi132.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T05:16:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:09.852 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:09.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:09.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:09 smithi080 ceph-mon[29691]: pgmap v716: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 34 MiB/s wr, 540 op/s; 53393/280365 objects degraded (19.044%); 22 MiB/s, 347 objects/s recovering 2024-03-26T05:16:09.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:09 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:16:09.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:09 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/774828329' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:16:09.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:09 smithi080 ceph-mon[29691]: osdmap e350: 8 total, 7 up, 8 in 2024-03-26T05:16:09.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:09 smithi080 ceph-mon[40404]: pgmap v716: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 34 MiB/s wr, 540 op/s; 53393/280365 objects degraded (19.044%); 22 MiB/s, 347 objects/s recovering 2024-03-26T05:16:09.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:09 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:16:09.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:09 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/774828329' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:16:09.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:09 smithi080 ceph-mon[40404]: osdmap e350: 8 total, 7 up, 8 in 2024-03-26T05:16:09.953 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:09.977 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:09 smithi132 ceph-mon[33500]: pgmap v716: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 34 MiB/s wr, 540 op/s; 53393/280365 objects degraded (19.044%); 22 MiB/s, 347 objects/s recovering 2024-03-26T05:16:09.977 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:09 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:16:09.978 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:09 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/774828329' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:16:09.978 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:09 smithi132 ceph-mon[33500]: osdmap e350: 8 total, 7 up, 8 in 2024-03-26T05:16:09.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:09.992 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:09.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:10.093 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:10.136 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:10.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:10.237 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:10.279 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:10.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:10.380 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:10.414 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:10.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:10.516 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:10.551 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:10.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:10.606 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 78 16 98989 98973 79.2879 67 0.00877935 0.0126068 2024-03-26T05:16:10.636 INFO:teuthology.orchestra.run.smithi132.stderr:Exporting 4.d info 4.d( v 254'31 (0'0,254'31] local-lis/les=69/70 n=8 ec=69/69 lis/c=69/69 les/c/f=70/70/0 sis=69) 2024-03-26T05:16:10.640 INFO:teuthology.orchestra.run.smithi132.stderr:Read #4:b019ea83:gc::gc.18:head# 2024-03-26T05:16:10.641 INFO:teuthology.orchestra.run.smithi132.stderr:Read #4:b110eaf6:::obj_delete_at_hint.0000000099:head# 2024-03-26T05:16:10.641 INFO:teuthology.orchestra.run.smithi132.stderr:Read #4:b1126f68:::obj_delete_at_hint.0000000056:head# 2024-03-26T05:16:10.641 INFO:teuthology.orchestra.run.smithi132.stderr:Read #4:b3052b9f:reshard::reshard.0000000013:head# 2024-03-26T05:16:10.641 INFO:teuthology.orchestra.run.smithi132.stderr:Read #4:b4f60f75:::obj_delete_at_hint.0000000072:head# 2024-03-26T05:16:10.641 INFO:teuthology.orchestra.run.smithi132.stderr:Read #4:b603df87:::obj_delete_at_hint.0000000062:head# 2024-03-26T05:16:10.641 INFO:teuthology.orchestra.run.smithi132.stderr:Read #4:b6c174be:::obj_delete_at_hint.0000000115:head# 2024-03-26T05:16:10.641 INFO:teuthology.orchestra.run.smithi132.stderr:Read #4:b72b1478:::obj_delete_at_hint.0000000033:head# 2024-03-26T05:16:10.641 INFO:teuthology.orchestra.run.smithi132.stderr:Export successful 2024-03-26T05:16:10.653 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:10.684 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:10.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:10.786 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:10.824 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:10.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:10.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:10.925 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:10.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:10.966 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:10.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:11.067 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:11.119 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:11.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:11.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:11.221 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:11.256 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:11.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:11.358 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:11.394 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:11.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:11.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:11.496 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:11.519 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:11.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:11.606 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 79 16 100014 99998 79.0951 64.0625 0.0107386 0.0126378 2024-03-26T05:16:11.621 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:11.653 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:11.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:11.675 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:11.710 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:16:11.749 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:16:11.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:11 smithi080 ceph-mon[29691]: pgmap v718: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 605 B/s rd, 48 MiB/s wr, 760 op/s; 54437/287529 objects degraded (18.933%); 3.3 MiB/s, 53 objects/s recovering 2024-03-26T05:16:11.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:11 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 54437/287529 objects degraded (18.933%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:11.749 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:11 smithi080 ceph-mon[29691]: osdmap e351: 8 total, 7 up, 8 in 2024-03-26T05:16:11.750 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:11 smithi080 ceph-mon[40404]: pgmap v718: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 605 B/s rd, 48 MiB/s wr, 760 op/s; 54437/287529 objects degraded (18.933%); 3.3 MiB/s, 53 objects/s recovering 2024-03-26T05:16:11.750 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:11 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 54437/287529 objects degraded (18.933%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:11.750 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:11 smithi080 ceph-mon[40404]: osdmap e351: 8 total, 7 up, 8 in 2024-03-26T05:16:11.755 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:11.787 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:16:11.788 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:11.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:11.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:11.826 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:16:11.871 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:16:11.890 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:11.917 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:16:11.932 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:11.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:11.961 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:16:11.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:16:11.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:16:11.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:16:11.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:16:11.962 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:16:11.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:16:11.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:16:11.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:16:12.035 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:11 smithi132 ceph-mon[33500]: pgmap v718: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 605 B/s rd, 48 MiB/s wr, 760 op/s; 54437/287529 objects degraded (18.933%); 3.3 MiB/s, 53 objects/s recovering 2024-03-26T05:16:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:11 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 54437/287529 objects degraded (18.933%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:11 smithi132 ceph-mon[33500]: osdmap e351: 8 total, 7 up, 8 in 2024-03-26T05:16:12.070 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:12.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:12.172 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:12.208 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:12.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:12.310 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:12.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:12.344 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:12.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:12.445 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:12.485 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:12.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:12.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:12.587 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:12.607 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:16:12.618315+0000 min lat: 0.00134127 max lat: 13.8808 avg lat: 0.0126541 2024-03-26T05:16:12.608 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:16:12.608 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 80 16 101152 101136 78.9944 71.125 0.0216517 0.0126541 2024-03-26T05:16:12.624 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:12.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:12.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:12.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:12.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:12.763 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:12.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:12.864 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -n osd.4 --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 4.d 2024-03-26T05:16:12.865 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:12.899 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:12.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:12.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:12.990 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:12 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3711057462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:12.991 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:12 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3711057462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:13.000 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:13.040 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:13.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:13.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:12 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3711057462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:13.116 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:16:13.142 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:13.149 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/osd.4/config 2024-03-26T05:16:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:16:13.196 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:16:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:13.217 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:13.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:13.318 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:13.359 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:13.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:13.460 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:13.501 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:13.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:13.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:13.603 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:13.608 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 81 15 102260 102245 78.8743 69.3125 0.0239221 0.0126733 2024-03-26T05:16:13.644 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:13.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:13.746 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:13.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:13.790 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:13.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:13.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:13 smithi080 ceph-mon[40404]: pgmap v720: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 55714/296631 objects degraded (18.782%); 6.6 MiB/s, 105 objects/s recovering 2024-03-26T05:16:13.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:13 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:16:13.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:13 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:16:13.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:13 smithi080 ceph-mon[40404]: osdmap e352: 8 total, 7 up, 8 in 2024-03-26T05:16:13.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:13 smithi080 ceph-mon[29691]: pgmap v720: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 55714/296631 objects degraded (18.782%); 6.6 MiB/s, 105 objects/s recovering 2024-03-26T05:16:13.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:13 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:16:13.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:13 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:16:13.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:13 smithi080 ceph-mon[29691]: osdmap e352: 8 total, 7 up, 8 in 2024-03-26T05:16:13.891 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:13.924 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:13.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:13.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:13.984 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:14.025 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:14.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:13 smithi132 ceph-mon[33500]: pgmap v720: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 55714/296631 objects degraded (18.782%); 6.6 MiB/s, 105 objects/s recovering 2024-03-26T05:16:14.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:13 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:16:14.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:13 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:16:14.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:13 smithi132 ceph-mon[33500]: osdmap e352: 8 total, 7 up, 8 in 2024-03-26T05:16:14.054 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:14.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:14.115 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:16:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:16:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:16:14.156 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:14.187 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:14.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:14.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:14.289 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:14.327 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:14.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:14.358 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:16:14.359 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:16:14.428 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:14.465 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:14.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:14.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:14.566 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:14.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:14.608 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:14.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:14.609 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 82 16 103305 103289 78.7081 65.25 0.00850342 0.0127006 2024-03-26T05:16:14.709 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:14.743 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:14.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:14.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:14.842 INFO:teuthology.orchestra.run.smithi132.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T05:16:14.845 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:14.886 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:14.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:14.974 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:14 smithi132 ceph-mon[33500]: pgmap v722: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 904 op/s; 55714/296631 objects degraded (18.782%); 4.1 MiB/s, 64 objects/s recovering 2024-03-26T05:16:14.988 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:15.032 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:15.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:15.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:15.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:14 smithi080 ceph-mon[40404]: pgmap v722: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 904 op/s; 55714/296631 objects degraded (18.782%); 4.1 MiB/s, 64 objects/s recovering 2024-03-26T05:16:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:14 smithi080 ceph-mon[29691]: pgmap v722: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 904 op/s; 55714/296631 objects degraded (18.782%); 4.1 MiB/s, 64 objects/s recovering 2024-03-26T05:16:15.134 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:15.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:15.173 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:15.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:15.197 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:16:15.274 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:15.307 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:15.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:15.398 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T05:16:15.409 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:15.441 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:15.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:15.542 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:15.589 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:15.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:15.609 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 83 16 104343 104327 78.5407 64.875 0.0136433 0.0127269 2024-03-26T05:16:15.662 INFO:teuthology.orchestra.run.smithi132.stdout: marking collection for removal 2024-03-26T05:16:15.663 INFO:teuthology.orchestra.run.smithi132.stdout:setting '_remove' omap key 2024-03-26T05:16:15.663 INFO:teuthology.orchestra.run.smithi132.stdout:finish_remove_pgs 4.d_head removing 4.d 2024-03-26T05:16:15.667 INFO:teuthology.orchestra.run.smithi132.stdout:Remove successful 2024-03-26T05:16:15.690 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:15.695 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:15 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:15.695 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:15 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:15.724 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:15.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:15.826 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:15.856 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:15.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:15.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:15.957 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:16.003 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:16.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:16.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:15 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:16.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:16:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:16:16.106 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:16.137 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:16.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:16.239 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:16.283 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:16.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:16.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:16.385 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:16.422 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:16.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:16.523 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:16.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:16.560 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:16.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:16.609 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 84 16 105459 105443 78.436 69.75 0.00953963 0.0127412 2024-03-26T05:16:16.662 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:16.696 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:16.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:16.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:16 smithi132 ceph-mon[33500]: pgmap v723: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 57768/310764 objects degraded (18.589%); 5.7 MiB/s, 91 objects/s recovering 2024-03-26T05:16:16.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:16 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 55714/296631 objects degraded (18.782%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:16.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:16 smithi132 ceph-mon[33500]: osdmap e353: 8 total, 7 up, 8 in 2024-03-26T05:16:16.798 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:16.822 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:16.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:16.923 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:16.957 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:16.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:16.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:17.003 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -n osd.4 --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.d.4 2024-03-26T05:16:17.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:16 smithi080 ceph-mon[40404]: pgmap v723: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 57768/310764 objects degraded (18.589%); 5.7 MiB/s, 91 objects/s recovering 2024-03-26T05:16:17.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:16 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 55714/296631 objects degraded (18.782%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:17.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:16 smithi080 ceph-mon[40404]: osdmap e353: 8 total, 7 up, 8 in 2024-03-26T05:16:17.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:16 smithi080 ceph-mon[29691]: pgmap v723: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 57768/310764 objects degraded (18.589%); 5.7 MiB/s, 91 objects/s recovering 2024-03-26T05:16:17.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:16 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 55714/296631 objects degraded (18.782%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:17.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:16 smithi080 ceph-mon[29691]: osdmap e353: 8 total, 7 up, 8 in 2024-03-26T05:16:17.059 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:17.136 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:17.139 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:17.142 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:17.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:17.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:17.243 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/osd.4/config 2024-03-26T05:16:17.284 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:17.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:17.386 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:17.430 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:17.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:17.533 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:17.566 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:17.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:17.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:17.609 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 85 16 106442 106426 78.2359 61.4375 0.00721948 0.0127769 2024-03-26T05:16:17.668 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:17.692 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:17.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:17.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:17.829 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:17.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:17.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:17.931 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:17.970 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:17.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:18.072 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:18.104 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:18.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:18.206 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:18.239 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:18.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:18.340 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:18.368 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:18.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:18.469 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:18.517 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:18.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:18.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:18.611 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 86 16 107476 107460 78.0776 64.625 0.014454 0.0128031 2024-03-26T05:16:18.620 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:18.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:18.656 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:18.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:18.757 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:18.791 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:18.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:18.893 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:18.929 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:18.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:18.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:19.031 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:19.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:19.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:19.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:19.134 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:16:19.170 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:19.207 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:19.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:19.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:19.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:19 smithi132 ceph-mon[33500]: pgmap v725: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 793 B/s rd, 52 MiB/s wr, 835 op/s; 57956/312789 objects degraded (18.529%); 4.2 MiB/s, 67 objects/s recovering 2024-03-26T05:16:19.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:19 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3958352152' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:19.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:19 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:19.309 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:19.347 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:19.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:19.449 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:19.475 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:19.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:19.576 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:19.609 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 87 16 108557 108541 77.9567 67.5625 0.0074107 0.0128229 2024-03-26T05:16:19.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:19 smithi080 ceph-mon[40404]: pgmap v725: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 793 B/s rd, 52 MiB/s wr, 835 op/s; 57956/312789 objects degraded (18.529%); 4.2 MiB/s, 67 objects/s recovering 2024-03-26T05:16:19.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:19 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3958352152' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:19.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:19 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:19.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:19 smithi080 ceph-mon[29691]: pgmap v725: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 793 B/s rd, 52 MiB/s wr, 835 op/s; 57956/312789 objects degraded (18.529%); 4.2 MiB/s, 67 objects/s recovering 2024-03-26T05:16:19.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:19 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3958352152' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:19.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:19 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:19.617 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:19.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:19.719 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:19.767 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:19.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:19.868 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:19.895 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:19.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:19.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:19.996 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:20.037 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:20.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:20.140 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:20.180 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:20.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:20.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:20.282 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:20.316 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:20.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:20.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:20 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:16:20.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:20 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:16:20.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:20 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:16:20.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:20 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:16:20.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:20 smithi080 ceph-mon[40404]: osdmap e354: 8 total, 7 up, 8 in 2024-03-26T05:16:20.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:20 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:16:20.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:20 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:16:20.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:20 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:16:20.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:20 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:16:20.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:20 smithi080 ceph-mon[29691]: osdmap e354: 8 total, 7 up, 8 in 2024-03-26T05:16:20.418 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:20.448 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:20.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:20.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:20.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:20 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:16:20.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:20 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:16:20.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:20 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:16:20.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:20 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:16:20.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:20 smithi132 ceph-mon[33500]: osdmap e354: 8 total, 7 up, 8 in 2024-03-26T05:16:20.550 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:20.594 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:20.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:20.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:20.609 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 88 16 109594 109578 77.8073 64.8125 0.0168009 0.0128468 2024-03-26T05:16:20.696 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:20.736 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:20.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:20.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:20.838 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:20.874 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:20.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:20.974 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:21.010 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:21.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:21.112 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:21.144 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:21.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:21.246 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:21.283 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:21.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:21.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:21.385 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:21.402 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:16:21.417 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:21.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:21.444 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:16:21.487 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:16:21.519 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:21.525 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:16:21.553 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:21.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:21.570 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:16:21.570 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:21 smithi080 ceph-mon[29691]: pgmap v727: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 691 B/s rd, 65 MiB/s wr, 1.03k op/s; 58917/319581 objects degraded (18.436%); 3.7 MiB/s, 58 objects/s recovering 2024-03-26T05:16:21.571 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:21.571 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:16:21.571 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:21 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3358024290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:21.571 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:21 smithi080 ceph-mon[40404]: pgmap v727: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 691 B/s rd, 65 MiB/s wr, 1.03k op/s; 58917/319581 objects degraded (18.436%); 3.7 MiB/s, 58 objects/s recovering 2024-03-26T05:16:21.571 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:21.571 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:16:21.571 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:21 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3358024290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:21.609 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:16:21.609 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 89 16 110689 110673 77.702 68.4375 0.020603 0.0128651 2024-03-26T05:16:21.655 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:21.658 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:16:21.658 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:16:21.658 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:16:21.658 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:16:21.658 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:16:21.658 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:16:21.658 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:16:21.658 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:16:21.659 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:16:21.704 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:21.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:21.754 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:21 smithi132 ceph-mon[33500]: pgmap v727: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 691 B/s rd, 65 MiB/s wr, 1.03k op/s; 58917/319581 objects degraded (18.436%); 3.7 MiB/s, 58 objects/s recovering 2024-03-26T05:16:21.754 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:21.754 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:16:21.754 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:21 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3358024290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:21.806 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:21.827 INFO:teuthology.orchestra.run.smithi132.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T05:16:21.843 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:21.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:21.944 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:21.980 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:21.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:21.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:22.081 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:22.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:22.126 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:22.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:22.194 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:16:22.194 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:16:22.194 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:16:22.194 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:16:22.194 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.194 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142549 8.0 8:0f39baff:::benchmark_data_smithi080_109213_object28032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:16:22.194 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.504592+0000", 2024-03-26T05:16:22.194 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 445.69728139199998, 2024-03-26T05:16:22.194 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98156162899999999, 2024-03-26T05:16:22.194 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.194 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142549 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504592+0000", 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504592+0000", 2024-03-26T05:16:22.195 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504593+0000", 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7100000000000002e-07 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504595+0000", 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4679999999999999e-06 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504602+0000", 2024-03-26T05:16:22.196 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5119999999999997e-06 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504607+0000", 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2379999999999997e-06 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504822+0000", 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021480699999999999 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.197 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.504829+0000", 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4590000000000001e-06 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.483043+0000", 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97821355600000004 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.483154+0000", 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000111266 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.198 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.483263+0000", 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000109616 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.485976+0000", 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027129250000000001 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.485989+0000", 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2797e-05 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.199 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.486051+0000", 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1488999999999994e-05 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.486108+0000", 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7441000000000001e-05 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.486154+0000", 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5584000000000003e-05 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.200 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142557 8.0 8:07e8ac51:::benchmark_data_smithi080_109213_object28040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.514964+0000", 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 445.68690963699999, 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97959498, 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142557 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.201 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.514964+0000", 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.514964+0000", 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.514965+0000", 2024-03-26T05:16:22.202 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5549999999999999e-06 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.514970+0000", 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8829999999999997e-06 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.514985+0000", 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4725999999999999e-05 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.203 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.514994+0000", 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8850000000000005e-06 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515051+0000", 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7119e-05 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515062+0000", 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1157e-05 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.204 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.486193+0000", 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97113100100000005 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.486297+0000", 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103472 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.486394+0000", 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7063000000000002e-05 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.205 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494287+0000", 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0078930370000000003 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494351+0000", 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4397e-05 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494448+0000", 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6775999999999996e-05 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.206 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494509+0000", 2024-03-26T05:16:22.207 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0908e-05 2024-03-26T05:16:22.207 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.207 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.207 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.207 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494559+0000", 2024-03-26T05:16:22.207 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0000999999999998e-05 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142560 8.0 8:03494147:::benchmark_data_smithi080_109213_object28043:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.515785+0000", 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 445.68608815300001, 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97893898300000004, 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.208 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142560 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515785+0000", 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515785+0000", 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.209 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515786+0000", 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1289999999999999e-06 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515791+0000", 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6670000000000004e-06 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515807+0000", 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5560999999999999e-05 2024-03-26T05:16:22.210 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515815+0000", 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5019999999999992e-06 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515877+0000", 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1439000000000006e-05 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.515885+0000", 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1200000000000002e-06 2024-03-26T05:16:22.211 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.489108+0000", 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97322330599999995 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.489231+0000", 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012295399999999999 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.489350+0000", 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118609 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.212 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494397+0000", 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0050477040000000001 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494406+0000", 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3999999999999992e-06 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494622+0000", 2024-03-26T05:16:22.213 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021623700000000001 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494681+0000", 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8885999999999998e-05 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494724+0000", 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3469e-05 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.214 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142575 8.0 8:0d7e4a70:::benchmark_data_smithi080_109213_object28058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.530049+0000", 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 445.67182475599998, 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97616341699999998, 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142575 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.215 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530049+0000", 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530049+0000", 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530050+0000", 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.42e-06 2024-03-26T05:16:22.216 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530054+0000", 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.641e-06 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530063+0000", 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6889999999999999e-06 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530070+0000", 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8210000000000004e-06 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.217 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530137+0000", 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7057000000000005e-05 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.530146+0000", 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1349999999999998e-06 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.491950+0000", 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96180365499999998 2024-03-26T05:16:22.218 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.492068+0000", 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117935 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.492192+0000", 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012364400000000001 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494734+0000", 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0025423289999999999 2024-03-26T05:16:22.219 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.500311+0000", 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055767949999999998 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506113+0000", 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0058020199999999997 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506164+0000", 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1505000000000002e-05 2024-03-26T05:16:22.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506212+0000", 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7771000000000001e-05 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142577 8.0 8:0fb818fc:::benchmark_data_smithi080_109213_object28060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.533454+0000", 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 445.668419148, 2024-03-26T05:16:22.221 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97287799699999999, 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142577 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533454+0000", 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.222 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533454+0000", 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533456+0000", 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.753e-06 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533461+0000", 2024-03-26T05:16:22.223 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1309999999999997e-06 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533476+0000", 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5241e-05 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533485+0000", 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2749999999999995e-06 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533534+0000", 2024-03-26T05:16:22.224 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9351999999999998e-05 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.533540+0000", 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8109999999999999e-06 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494773+0000", 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96123300899999997 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.494899+0000", 2024-03-26T05:16:22.225 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012616099999999999 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.495016+0000", 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117179 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.500360+0000", 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0053439079999999996 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.226 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.500372+0000", 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1649e-05 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506261+0000", 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0058893499999999998 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506301+0000", 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9548000000000002e-05 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.227 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506332+0000", 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.163e-05 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142588 8.0 8:0e33f004:::benchmark_data_smithi080_109213_object28071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.543025+0000", 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 445.65884882099999, 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96893715599999997, 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:22.228 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142588 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543025+0000", 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543025+0000", 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.229 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543026+0000", 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.144e-06 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543030+0000", 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9949999999999999e-06 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543041+0000", 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1202999999999999e-05 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543048+0000", 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4460000000000004e-06 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543303+0000", 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000254221 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.543312+0000", 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0389999999999993e-06 2024-03-26T05:16:22.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.497590+0000", 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95427875200000001 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.497727+0000", 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000136747 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.497849+0000", 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000121468 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.232 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506224+0000", 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0083755749999999997 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506364+0000", 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000140121 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.511871+0000", 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055066919999999997 2024-03-26T05:16:22.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.234 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.234 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.234 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.511923+0000", 2024-03-26T05:16:22.234 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:22.235 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1759999999999999e-05 2024-03-26T05:16:22.235 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.235 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.235 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.235 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.511962+0000", 2024-03-26T05:16:22.235 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8992999999999997e-05 2024-03-26T05:16:22.235 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.235 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142592 8.0 8:02c66466:::benchmark_data_smithi080_109213_object28075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:56.544864+0000", 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 445.65700934699998, 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96723239800000005, 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142592 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.236 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544864+0000", 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544864+0000", 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544865+0000", 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.234e-06 2024-03-26T05:16:22.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544871+0000", 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4009999999999996e-06 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544887+0000", 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5928000000000001e-05 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.238 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544895+0000", 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3590000000000008e-06 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544950+0000", 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4487000000000002e-05 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:56.544957+0000", 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8560000000000007e-06 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.239 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.500399+0000", 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95544189700000004 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.500508+0000", 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000108707 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.500609+0000", 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000100936 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.240 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506229+0000", 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0056203420000000004 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.506411+0000", 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018178500000000001 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.512004+0000", 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055931770000000004 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.241 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.512057+0000", 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2735000000000002e-05 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:57.512096+0000", 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9554000000000001e-05 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109994 9.c 9:34e37923:::benchmark_data_smithi080_136801_object109993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-26T05:16:22.242 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.792060+0000", 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 236.40981371000001, 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0406930759999999, 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109994 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.243 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792060+0000", 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792060+0000", 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792062+0000", 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7889999999999999e-06 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.244 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792085+0000", 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3445999999999999e-05 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792099+0000", 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4372000000000001e-05 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792110+0000", 2024-03-26T05:16:22.245 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0739e-05 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792816+0000", 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00070612300000000002 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792992+0000", 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017620600000000001 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,7", 2024-03-26T05:16:22.246 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.793149+0000", 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015638199999999999 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.809680+0000", 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016531033000000001 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.809693+0000", 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2781e-05 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.247 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.821562+0000", 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.01186924 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for degraded object", 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.822506+0000", 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00094436699999999999 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.832661+0000", 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.01015492 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.248 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.832753+0000", 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1677999999999999e-05 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109996 9.c 9:318dcee9:::benchmark_data_smithi080_136801_object109995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.817916+0000", 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 236.383957307, 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.016490052, 2024-03-26T05:16:22.249 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.250 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.250 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.250 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:16:22.250 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:16:22.250 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109996 2024-03-26T05:16:22.250 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.250 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.250 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.250 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.250 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817916+0000", 2024-03-26T05:16:22.250 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.250 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817916+0000", 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817917+0000", 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1600000000000001e-07 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.251 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817942+0000", 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5422000000000002e-05 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817950+0000", 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.977e-06 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817956+0000", 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.263e-06 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.252 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822354+0000", 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0043976070000000004 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822365+0000", 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.098e-05 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.822966+0000", 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0006012339999999 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.253 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.823106+0000", 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013994699999999999 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4", 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.823382+0000", 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00027621999999999998 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.832803+0000", 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0094210860000000004 2024-03-26T05:16:22.254 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.834257+0000", 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014535419999999999 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.834309+0000", 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2330999999999998e-05 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.834406+0000", 2024-03-26T05:16:22.255 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6927000000000002e-05 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109989 9.3 9:c1ef3764:::benchmark_data_smithi080_136801_object109988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.822224+0000", 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 236.37964932599999, 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.002023533, 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:16:22.256 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109989 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822224+0000", 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822224+0000", 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.257 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822227+0000", 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.8040000000000002e-06 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822261+0000", 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3615000000000003e-05 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822275+0000", 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4759e-05 2024-03-26T05:16:22.258 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822287+0000", 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1419e-05 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822922+0000", 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00063540799999999996 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822932+0000", 2024-03-26T05:16:22.259 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5510000000000005e-06 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.822466+0000", 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99953455999999996 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.824248+0000", 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001781417 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.260 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109993 9.3 9:c281200c:::benchmark_data_smithi080_136801_object109992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.822314+0000", 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 236.379559863, 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0022707790000001, 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109993 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.261 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822314+0000", 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822314+0000", 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822314+0000", 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6000000000000003e-07 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.262 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822356+0000", 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1310999999999998e-05 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822363+0000", 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9940000000000003e-06 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822368+0000", 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7200000000000003e-06 2024-03-26T05:16:22.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822959+0000", 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00059071500000000001 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822965+0000", 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0800000000000002e-06 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.824517+0000", 2024-03-26T05:16:22.264 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0015518800000001 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.824584+0000", 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7418999999999999e-05 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109998 9.3 9:cd2a8d52:::benchmark_data_smithi080_136801_object109997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.822378+0000", 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 236.37949546999999, 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0183832850000001, 2024-03-26T05:16:22.265 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109998 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822378+0000", 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.266 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822378+0000", 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822378+0000", 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6699999999999999e-07 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822401+0000", 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2480000000000002e-05 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.267 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822407+0000", 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5139999999999998e-06 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822413+0000", 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9549999999999997e-06 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822988+0000", 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00057432800000000003 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.268 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822994+0000", 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9800000000000003e-06 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.824610+0000", 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.001616391 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.826015+0000", 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014048019999999999 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.269 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1", 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.826423+0000", 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00040815099999999999 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840598+0000", 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014175045000000001 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840625+0000", 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6840000000000001e-05 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.270 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840672+0000", 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6598999999999999e-05 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840761+0000", 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9733000000000003e-05 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109999 9.3 9:c8cd8b0a:::benchmark_data_smithi080_136801_object109998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.829248+0000", 2024-03-26T05:16:22.271 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 236.37262525099999, 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0177361709999999, 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109999 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829248+0000", 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.272 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829248+0000", 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829251+0000", 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.706e-06 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829301+0000", 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0448000000000002e-05 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.273 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829327+0000", 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6041999999999999e-05 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829340+0000", 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2177999999999999e-05 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829560+0000", 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00022066499999999999 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.274 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829571+0000", 2024-03-26T05:16:22.275 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1049999999999999e-05 2024-03-26T05:16:22.275 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.830957+0000", 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.00138595 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.831076+0000", 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118507 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1", 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.831217+0000", 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014082499999999999 2024-03-26T05:16:22.276 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840769+0000", 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0095528499999999999 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.846840+0000", 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0060710260000000002 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.846896+0000", 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5721999999999999e-05 2024-03-26T05:16:22.277 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.846984+0000", 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8201999999999993e-05 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:110007 9.1 9:894e6b0c:::benchmark_data_smithi080_136801_object110006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.844428+0000", 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 236.35744549699999, 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97646292999999995, 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.278 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 110007 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844428+0000", 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844428+0000", 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.279 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844430+0000", 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.68e-06 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844464+0000", 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4585000000000002e-05 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844488+0000", 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3312e-05 2024-03-26T05:16:22.280 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844497+0000", 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4709999999999996e-06 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844569+0000", 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1805999999999997e-05 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844578+0000", 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5799999999999998e-06 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.281 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.811307+0000", 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96672887299999999 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.811467+0000", 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016000899999999999 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 6", 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.813451+0000", 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0019838109999999998 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.282 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817336+0000", 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0038850600000000001 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820757+0000", 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0034208289999999998 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820805+0000", 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7953e-05 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.283 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820891+0000", 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5960999999999999e-05 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:110010 9.3 9:cd2ac3f8:::benchmark_data_smithi080_136801_object110009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.848965+0000", 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 236.35290844599999, 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99831434500000005, 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.284 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 110010 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848965+0000", 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848965+0000", 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.285 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848966+0000", 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.381e-06 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848987+0000", 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0312000000000001e-05 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848999+0000", 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2170999999999999e-05 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.286 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.849009+0000", 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6700000000000006e-06 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.849085+0000", 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6884000000000005e-05 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.849096+0000", 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0179e-05 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.287 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.833721+0000", 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98462529899999995 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.833851+0000", 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012981200000000001 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1", 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.834023+0000", 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017234300000000001 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.288 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840776+0000", 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0067526269999999998 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.847155+0000", 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0063788459999999996 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.847201+0000", 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6702000000000001e-05 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.847279+0000", 2024-03-26T05:16:22.289 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8119000000000002e-05 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:110013 9.1 9:8edffc06:::benchmark_data_smithi080_136801_object110012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.857433+0000", 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 236.34444062700001, 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96361645399999996, 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:16:22.290 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 110013 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857433+0000", 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857433+0000", 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.291 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857434+0000", 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.178e-06 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857452+0000", 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7561e-05 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857463+0000", 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1099e-05 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857470+0000", 2024-03-26T05:16:22.292 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1999999999999997e-06 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857537+0000", 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6946999999999994e-05 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857547+0000", 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0669000000000001e-05 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817405+0000", 2024-03-26T05:16:22.293 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95985719199999997 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817539+0000", 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013459600000000001 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 6", 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817693+0000", 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015364900000000001 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820443+0000", 2024-03-26T05:16:22.294 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002750438 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820935+0000", 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00049171399999999997 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820979+0000", 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3495000000000003e-05 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.821049+0000", 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0716000000000005e-05 2024-03-26T05:16:22.295 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15657.0:81834 10.3 e338/316 10:cb3a1dfd:::benchmark_data_smithi080_161990_object81833:head v 338'5213, mlcod=338'5211)", 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:15:40.549406+0000", 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 41.652466963999998, 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.11973281600000001, 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.296 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.549406+0000", 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.549406+0000", 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.549407+0000", 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0400000000000005e-07 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.297 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.549429+0000", 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1549999999999999e-05 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.549445+0000", 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6089000000000001e-05 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.549453+0000", 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7360000000000001e-06 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.298 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.550098+0000", 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00064560799999999999 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.550144+0000", 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6097999999999998e-05 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.669093+0000", 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.11894822300000001 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.669139+0000", 2024-03-26T05:16:22.299 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6607999999999998e-05 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15657.0:81850 10.3 e338/316 10:cdaf78b7:::benchmark_data_smithi080_161990_object81849:head v 338'5214, mlcod=338'5211)", 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:15:40.577757+0000", 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 41.624116602000001, 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.091410963999999997, 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.300 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.577757+0000", 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.577757+0000", 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.577758+0000", 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3200000000000004e-07 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.577784+0000", 2024-03-26T05:16:22.301 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5867000000000002e-05 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.577794+0000", 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0862e-05 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.577801+0000", 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9299999999999997e-06 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.302 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.577881+0000", 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0086999999999994e-05 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.577900+0000", 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8873e-05 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.669144+0000", 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.091243590999999999 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.303 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.669168+0000", 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3921999999999999e-05 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15657.0:81883 10.3 e338/316 10:c5cf7a39:::benchmark_data_smithi080_161990_object81882:head v 338'5215, mlcod=338'5211)", 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:15:40.609179+0000", 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 41.592694131000002, 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.060266064000000001, 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.304 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.609179+0000", 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.609179+0000", 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.609181+0000", 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2449999999999999e-06 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.305 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.609205+0000", 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3975e-05 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.609220+0000", 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5692000000000002e-05 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.609229+0000", 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5529999999999993e-06 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.306 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.609406+0000", 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017714 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.609425+0000", 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9063999999999999e-05 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.669324+0000", 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.059898831999999999 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.307 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.669445+0000", 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000121563 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15657.0:81885 10.3 e338/316 10:c7f97687:::benchmark_data_smithi080_161990_object81884:head v 338'5216, mlcod=338'5211)", 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:15:40.612615+0000", 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 41.589258457, 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.056934054999999997, 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:16:22.308 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.612615+0000", 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.612615+0000", 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.612616+0000", 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1399999999999996e-07 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.612641+0000", 2024-03-26T05:16:22.309 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4706999999999999e-05 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.612656+0000", 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5034999999999999e-05 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.612664+0000", 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9780000000000005e-06 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.612734+0000", 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0030999999999994e-05 2024-03-26T05:16:22.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.612752+0000", 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8527000000000001e-05 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.669449+0000", 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.056696475000000003 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:40.669549+0000", 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000100488 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.311 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.312 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.312 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:16:22.312 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:16:22.312 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:16:22.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:22.313 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:22.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:22.414 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:22.453 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:22.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:22.555 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:22.593 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:22.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:22.609 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Total time run: 90.0083 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Total writes made: 111720 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Write size: 65536 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Object size: 65536 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Bandwidth (MB/sec): 77.5761 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev Bandwidth: 52.5606 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max bandwidth (MB/sec): 240.875 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min bandwidth (MB/sec): 0 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Average IOPS: 1241 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev IOPS: 840.982 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max IOPS: 3854 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min IOPS: 0 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Average Latency(s): 0.0128869 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev Latency(s): 0.136464 2024-03-26T05:16:22.610 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max latency(s): 13.8808 2024-03-26T05:16:22.611 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min latency(s): 0.00134127 2024-03-26T05:16:22.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:22 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 58917/319581 objects degraded (18.436%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:22.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:22 smithi080 ceph-mon[29691]: osdmap e355: 8 total, 7 up, 8 in 2024-03-26T05:16:22.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:22 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 58917/319581 objects degraded (18.436%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:22.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:22 smithi080 ceph-mon[40404]: osdmap e355: 8 total, 7 up, 8 in 2024-03-26T05:16:22.616 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T05:16:22.649 INFO:teuthology.orchestra.run.smithi132.stdout:get_pg_num_history pg_num_history pg_num_history(e341 pg_nums {1={20=1,324=11},2={66=3},3={67=32},4={69=32},5={71=32},6={73=32},9={265=16},10={316=16}} deleted_pools 312,9) 2024-03-26T05:16:22.649 INFO:teuthology.orchestra.run.smithi132.stdout:Importing pgid 4.d 2024-03-26T05:16:22.650 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:22 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 58917/319581 objects degraded (18.436%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:22.650 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:22 smithi132 ceph-mon[33500]: osdmap e355: 8 total, 7 up, 8 in 2024-03-26T05:16:22.650 INFO:teuthology.orchestra.run.smithi132.stdout:Write #4:b019ea83:gc::gc.18:head# 2024-03-26T05:16:22.650 INFO:teuthology.orchestra.run.smithi132.stdout:snapset 0=[]:{} 2024-03-26T05:16:22.651 INFO:teuthology.orchestra.run.smithi132.stdout:Write #4:b110eaf6:::obj_delete_at_hint.0000000099:head# 2024-03-26T05:16:22.651 INFO:teuthology.orchestra.run.smithi132.stdout:snapset 0=[]:{} 2024-03-26T05:16:22.651 INFO:teuthology.orchestra.run.smithi132.stdout:Write #4:b1126f68:::obj_delete_at_hint.0000000056:head# 2024-03-26T05:16:22.652 INFO:teuthology.orchestra.run.smithi132.stdout:snapset 0=[]:{} 2024-03-26T05:16:22.652 INFO:teuthology.orchestra.run.smithi132.stdout:Write #4:b3052b9f:reshard::reshard.0000000013:head# 2024-03-26T05:16:22.652 INFO:teuthology.orchestra.run.smithi132.stdout:snapset 0=[]:{} 2024-03-26T05:16:22.652 INFO:teuthology.orchestra.run.smithi132.stdout:Write #4:b4f60f75:::obj_delete_at_hint.0000000072:head# 2024-03-26T05:16:22.652 INFO:teuthology.orchestra.run.smithi132.stdout:snapset 0=[]:{} 2024-03-26T05:16:22.653 INFO:teuthology.orchestra.run.smithi132.stdout:Write #4:b603df87:::obj_delete_at_hint.0000000062:head# 2024-03-26T05:16:22.653 INFO:teuthology.orchestra.run.smithi132.stdout:snapset 0=[]:{} 2024-03-26T05:16:22.653 INFO:teuthology.orchestra.run.smithi132.stdout:Write #4:b6c174be:::obj_delete_at_hint.0000000115:head# 2024-03-26T05:16:22.653 INFO:teuthology.orchestra.run.smithi132.stdout:snapset 0=[]:{} 2024-03-26T05:16:22.653 INFO:teuthology.orchestra.run.smithi132.stdout:Write #4:b72b1478:::obj_delete_at_hint.0000000033:head# 2024-03-26T05:16:22.653 INFO:teuthology.orchestra.run.smithi132.stdout:snapset 0=[]:{} 2024-03-26T05:16:22.654 INFO:teuthology.orchestra.run.smithi132.stdout:write_pg epoch 320 info 4.d( v 254'31 (0'0,254'31] local-lis/les=69/70 n=8 ec=69/69 lis/c=69/69 les/c/f=70/70/0 sis=69) 2024-03-26T05:16:22.654 INFO:teuthology.orchestra.run.smithi132.stdout:Import successful 2024-03-26T05:16:22.695 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:22.727 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:22.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:22.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:22.829 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:22.861 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:22.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:22.877 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:16:22.877 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:16:22.962 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:23.001 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:23.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:23.072 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:16:23.102 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:23.135 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:23.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:23.237 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:23.301 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:23.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:23.402 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:23.442 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:23.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:23.544 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:23.573 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:23.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:23.675 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:23.687 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T05:16:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:23.708 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:23.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:23.740 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:23 smithi132 ceph-mon[33500]: pgmap v729: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 987 op/s; 60062/328536 objects degraded (18.282%); 6.7 MiB/s, 106 objects/s recovering 2024-03-26T05:16:23.809 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:23.839 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:23.850 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:23.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:23.951 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:23.977 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:23 smithi080 ceph-mon[29691]: pgmap v729: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 987 op/s; 60062/328536 objects degraded (18.282%); 6.7 MiB/s, 106 objects/s recovering 2024-03-26T05:16:23.977 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:23 smithi080 ceph-mon[40404]: pgmap v729: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 987 op/s; 60062/328536 objects degraded (18.282%); 6.7 MiB/s, 106 objects/s recovering 2024-03-26T05:16:23.986 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/exp.4.d.4 2024-03-26T05:16:23.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:23.991 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:23.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:24.090 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:16:24.092 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:24.134 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:24.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:24.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:24.236 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:24.277 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:24.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:24.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:24.286 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:16:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:16:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:16:24.379 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:24.413 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:24.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:24.515 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:24.548 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:24.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:24.650 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:24.681 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:24.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:24.783 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:24.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:24.849 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:24.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:24.878 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:16:24.950 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:24.990 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:24.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:24.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:25.092 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:25.143 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:25.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:25.245 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:25.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:25.287 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:25.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:25.388 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:25.433 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:25.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:25.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:25.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:25.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:25.568 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:25.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:25.669 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:25.685 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:25.710 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:25.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:25.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:25 smithi132 ceph-mon[33500]: pgmap v730: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 686 B/s rd, 44 MiB/s wr, 704 op/s; 60062/328536 objects degraded (18.282%); 4.2 MiB/s, 66 objects/s recovering 2024-03-26T05:16:25.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:16:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:16:25.811 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:25.822 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:25 smithi080 ceph-mon[40404]: pgmap v730: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 686 B/s rd, 44 MiB/s wr, 704 op/s; 60062/328536 objects degraded (18.282%); 4.2 MiB/s, 66 objects/s recovering 2024-03-26T05:16:25.823 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:25 smithi080 ceph-mon[29691]: pgmap v730: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 686 B/s rd, 44 MiB/s wr, 704 op/s; 60062/328536 objects degraded (18.282%); 4.2 MiB/s, 66 objects/s recovering 2024-03-26T05:16:25.864 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:25.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:25.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:25.966 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:26.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:26.003 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:26.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:26.104 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:26.133 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:26.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:26.235 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:26.268 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:26.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:26.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:26.369 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:26.413 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:26.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:26.514 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:26.562 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:26.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:26.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:26.664 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:26.706 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:26.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:26.774 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:26 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 60062/328536 objects degraded (18.282%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:26.775 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:26 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 60062/328536 objects degraded (18.282%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:26.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:26.840 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:26.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:26.942 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:26.977 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:26.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:27.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:26 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 60062/328536 objects degraded (18.282%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:27.079 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:27.112 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:27.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:27.213 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:27.244 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:27.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:27.346 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:27.374 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:27.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:27.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:27.476 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:27.511 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:27.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:27.613 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:27.652 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:27.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:27.659 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:16:27.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:27 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:27.688 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:27 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:27.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:27.793 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:27.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:27.895 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:27.914 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:27 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:27.914 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:27 smithi132 ceph-mon[33500]: pgmap v731: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 59 MiB/s wr, 1.26k op/s; 59477/327777 objects degraded (18.146%); 6.9 MiB/s, 110 objects/s recovering 2024-03-26T05:16:27.915 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:27 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3356697652' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:16:27.915 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:27 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:16:27.933 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:27.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:28.035 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:28.076 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:28.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:28.083 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:117727 10.9 10:94a9f952:::benchmark_data_smithi080_161990_object6004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:28.081803+0000", 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.010799659, 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010831192, 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 117727 2024-03-26T05:16:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.081803+0000", 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.081803+0000", 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.081804+0000", 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.403e-06 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.081809+0000", 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6249999999999998e-06 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.081824+0000", 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4885000000000001e-05 2024-03-26T05:16:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.081832+0000", 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2840000000000006e-06 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.086021+0000", 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0041885789999999996 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.086174+0000", 2024-03-26T05:16:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015374299999999999 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.086278+0000", 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103513 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.089975+0000", 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0036967889999999998 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPushReply(10.9 356/336 [PushReplyOp(10:9fa048c6:::benchmark_data_smithi080_161990_object51016:head)])", 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:28.090934+0000", 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0016683519999999999, 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0017879510000000001, 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.090934+0000", 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.090934+0000", 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.090935+0000", 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4900000000000004e-07 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.090942+0000", 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0530000000000001e-06 2024-03-26T05:16:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.090957+0000", 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4961e-05 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.090966+0000", 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.986e-06 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.091120+0000", 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015361800000000001 2024-03-26T05:16:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(client.15657.0:117727 10.9 e356/336)", 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:28.091177+0000", 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0014258439999999999, 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0015994609999999999, 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "queued for pg", 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.091177+0000", 2024-03-26T05:16:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.091177+0000", 2024-03-26T05:16:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.091177+0000", 2024-03-26T05:16:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4799999999999999e-07 2024-03-26T05:16:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.091181+0000", 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6720000000000002e-06 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.091189+0000", 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4849999999999992e-06 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.091195+0000", 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9220000000000002e-06 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:117722 10.9 10:982705ab:::benchmark_data_smithi080_161990_object5999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:28.079136+0000", 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.01346614, 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013695321, 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 117722 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.079136+0000", 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.079136+0000", 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.079138+0000", 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.522e-06 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.079142+0000", 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2520000000000001e-06 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.079151+0000", 2024-03-26T05:16:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3549999999999994e-06 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.079157+0000", 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0639999999999999e-06 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.082734+0000", 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0035770149999999998 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.082879+0000", 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014457200000000001 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.083039+0000", 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015993499999999999 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.089068+0000", 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0060292610000000002 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:28.091072+0000", 2024-03-26T05:16:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020033099999999999 2024-03-26T05:16:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-03-26T05:16:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 4 2024-03-26T05:16:28.101 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:16:28.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:27 smithi080 ceph-mon[40404]: pgmap v731: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 59 MiB/s wr, 1.26k op/s; 59477/327777 objects degraded (18.146%); 6.9 MiB/s, 110 objects/s recovering 2024-03-26T05:16:28.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:27 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3356697652' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:16:28.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:27 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:16:28.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:27 smithi080 ceph-mon[29691]: pgmap v731: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 59 MiB/s wr, 1.26k op/s; 59477/327777 objects degraded (18.146%); 6.9 MiB/s, 110 objects/s recovering 2024-03-26T05:16:28.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:27 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3356697652' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:16:28.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:27 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:16:28.178 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:28.216 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:28.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:28.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:28.317 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:28.357 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:28.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:28.459 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:28.502 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:28.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:28.603 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:28.637 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:28.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:28.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:28.739 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:28.775 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:28.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:28.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:28.795 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:28 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:16:28.795 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:28 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:16:28.796 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:28 smithi080 ceph-mon[40404]: osdmap e356: 8 total, 7 up, 8 in 2024-03-26T05:16:28.796 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:28 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:16:28.796 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:28 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:16:28.796 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:28 smithi080 ceph-mon[29691]: osdmap e356: 8 total, 7 up, 8 in 2024-03-26T05:16:28.878 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:28.916 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:28.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:29.018 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:29.033 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T05:16:29.033 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:16:29.034 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-03-26T05:16:29.034 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:16:29.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:28 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:16:29.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:28 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:16:29.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:28 smithi132 ceph-mon[33500]: osdmap e356: 8 total, 7 up, 8 in 2024-03-26T05:16:29.051 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:29.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:29.099 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T05:16:29.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:29.188 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:29.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:29.290 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:29.349 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:29.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:29.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:29.451 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:29.466 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:16:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:29.493 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:29.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:29.594 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:29.644 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:29.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:29.746 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:29.814 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:29.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:29.830 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:29.872 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:16:29.909 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:29 smithi132 ceph-mon[33500]: pgmap v733: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 42 MiB/s wr, 1.06k op/s; 59140/326631 objects degraded (18.106%); 8.1 MiB/s, 130 objects/s recovering 2024-03-26T05:16:29.915 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:29.916 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:16:29.916 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:29 smithi080 ceph-mon[29691]: pgmap v733: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 42 MiB/s wr, 1.06k op/s; 59140/326631 objects degraded (18.106%); 8.1 MiB/s, 130 objects/s recovering 2024-03-26T05:16:29.917 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:29 smithi080 ceph-mon[40404]: pgmap v733: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 42 MiB/s wr, 1.06k op/s; 59140/326631 objects degraded (18.106%); 8.1 MiB/s, 130 objects/s recovering 2024-03-26T05:16:29.955 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:29.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:29.960 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:16:30.000 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:16:30.042 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:16:30.057 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:30.086 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:16:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:30.100 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:30.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:30.123 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:16:30.123 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:16:30.123 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:16:30.123 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:16:30.123 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:16:30.123 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:16:30.123 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:16:30.123 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:16:30.123 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:16:30.201 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:30.234 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:30.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:30.336 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:30.367 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:30.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:30.469 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:30.509 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:30.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:30.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:30.611 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:30.648 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:30.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:30.750 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:30.784 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:30.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:30.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:30.886 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:30.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:30.927 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:30.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:31.028 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:31.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:30 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/766273153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:31.055 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:31.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:31.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:30 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/766273153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:31.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:30 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/766273153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:31.157 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:31.162 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:16:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:31.196 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:31.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:31.297 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:31.341 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:31.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:31.442 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:31.477 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:31.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:31.579 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:31.615 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:16:31.615 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:16:31.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:31.616 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:31.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:31.717 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:31.755 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:31.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:31.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:31.857 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:31.893 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:31.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:31.995 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:32.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:31 smithi080 ceph-mon[40404]: pgmap v734: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 726 B/s rd, 18 MiB/s wr, 939 op/s; 58117/319350 objects degraded (18.199%); 4.0 MiB/s, 64 objects/s recovering 2024-03-26T05:16:32.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:31 smithi080 ceph-mon[40404]: 3.1e scrub starts 2024-03-26T05:16:32.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:31 smithi080 ceph-mon[40404]: 3.1e scrub ok 2024-03-26T05:16:32.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:31 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 58117/319350 objects degraded (18.199%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:32.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:31 smithi080 ceph-mon[40404]: osdmap e357: 8 total, 7 up, 8 in 2024-03-26T05:16:32.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:31 smithi080 ceph-mon[29691]: pgmap v734: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 726 B/s rd, 18 MiB/s wr, 939 op/s; 58117/319350 objects degraded (18.199%); 4.0 MiB/s, 64 objects/s recovering 2024-03-26T05:16:32.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:31 smithi080 ceph-mon[29691]: 3.1e scrub starts 2024-03-26T05:16:32.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:31 smithi080 ceph-mon[29691]: 3.1e scrub ok 2024-03-26T05:16:32.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:31 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 58117/319350 objects degraded (18.199%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:32.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:31 smithi080 ceph-mon[29691]: osdmap e357: 8 total, 7 up, 8 in 2024-03-26T05:16:32.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:31 smithi132 ceph-mon[33500]: pgmap v734: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 726 B/s rd, 18 MiB/s wr, 939 op/s; 58117/319350 objects degraded (18.199%); 4.0 MiB/s, 64 objects/s recovering 2024-03-26T05:16:32.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:31 smithi132 ceph-mon[33500]: 3.1e scrub starts 2024-03-26T05:16:32.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:31 smithi132 ceph-mon[33500]: 3.1e scrub ok 2024-03-26T05:16:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:31 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 58117/319350 objects degraded (18.199%), 32 pgs degraded (PG_DEGRADED) 2024-03-26T05:16:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:31 smithi132 ceph-mon[33500]: osdmap e357: 8 total, 7 up, 8 in 2024-03-26T05:16:32.028 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:32.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:32.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:32.130 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:32.162 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:32.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:32.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:32.264 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:32.298 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:32.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:32.400 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:32.441 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:32.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:32.468 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T05:16:32.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:32.596 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:32.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:32.698 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:32.738 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:32.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:32.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:32.840 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:32.867 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:16:32.872 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:32.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:32.974 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:33.007 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:32 smithi132 ceph-mon[33500]: 3.18 scrub starts 2024-03-26T05:16:33.007 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:32 smithi132 ceph-mon[33500]: 3.18 scrub ok 2024-03-26T05:16:33.007 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:32 smithi132 ceph-mon[33500]: pgmap v736: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 19 MiB/s wr, 1.38k op/s; 55757/310080 objects degraded (17.981%); 9.2 MiB/s, 147 objects/s recovering 2024-03-26T05:16:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:33.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:33.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:32 smithi080 ceph-mon[40404]: 3.18 scrub starts 2024-03-26T05:16:33.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:32 smithi080 ceph-mon[40404]: 3.18 scrub ok 2024-03-26T05:16:33.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:32 smithi080 ceph-mon[40404]: pgmap v736: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 19 MiB/s wr, 1.38k op/s; 55757/310080 objects degraded (17.981%); 9.2 MiB/s, 147 objects/s recovering 2024-03-26T05:16:33.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:32 smithi080 ceph-mon[29691]: 3.18 scrub starts 2024-03-26T05:16:33.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:32 smithi080 ceph-mon[29691]: 3.18 scrub ok 2024-03-26T05:16:33.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:32 smithi080 ceph-mon[29691]: pgmap v736: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 19 MiB/s wr, 1.38k op/s; 55757/310080 objects degraded (17.981%); 9.2 MiB/s, 147 objects/s recovering 2024-03-26T05:16:33.118 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:33.156 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:33.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:33.257 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:33.298 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:33.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:33.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:33.400 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:33.427 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:33.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:33.529 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:33.562 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:33.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:33.664 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:33.706 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:33.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:33.808 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:33.839 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:33.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:33.941 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:33.964 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:33.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:34.023 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:33 smithi080 ceph-mon[40404]: 6.13 scrub starts 2024-03-26T05:16:34.023 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:33 smithi080 ceph-mon[40404]: 6.13 scrub ok 2024-03-26T05:16:34.023 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:33 smithi080 ceph-mon[29691]: 6.13 scrub starts 2024-03-26T05:16:34.023 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:33 smithi080 ceph-mon[29691]: 6.13 scrub ok 2024-03-26T05:16:34.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:33 smithi132 ceph-mon[33500]: 6.13 scrub starts 2024-03-26T05:16:34.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:33 smithi132 ceph-mon[33500]: 6.13 scrub ok 2024-03-26T05:16:34.066 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:34.098 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:34.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:34.200 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:34.240 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:34.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:34.341 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:34.367 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:16:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:16:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:16:34.373 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:34.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:34.475 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:34.509 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:34.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:34.611 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:34.654 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:34.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:34.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:34.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:34.798 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:34.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:34.900 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:34.943 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:34.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:34.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:34.997 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:16:34.998 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:16:34.999 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:34 smithi132 ceph-mon[33500]: 4.12 scrub starts 2024-03-26T05:16:34.999 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:34 smithi132 ceph-mon[33500]: 4.12 scrub ok 2024-03-26T05:16:34.999 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:34 smithi132 ceph-mon[33500]: pgmap v737: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 1.1 MiB/s wr, 774 op/s; 55757/310080 objects degraded (17.981%); 6.2 MiB/s, 99 objects/s recovering 2024-03-26T05:16:35.045 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:35.083 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:35.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:35.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:35.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:34 smithi080 ceph-mon[29691]: 4.12 scrub starts 2024-03-26T05:16:35.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:34 smithi080 ceph-mon[29691]: 4.12 scrub ok 2024-03-26T05:16:35.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:34 smithi080 ceph-mon[29691]: pgmap v737: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 1.1 MiB/s wr, 774 op/s; 55757/310080 objects degraded (17.981%); 6.2 MiB/s, 99 objects/s recovering 2024-03-26T05:16:35.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:34 smithi080 ceph-mon[40404]: 4.12 scrub starts 2024-03-26T05:16:35.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:34 smithi080 ceph-mon[40404]: 4.12 scrub ok 2024-03-26T05:16:35.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:34 smithi080 ceph-mon[40404]: pgmap v737: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 1.1 MiB/s wr, 774 op/s; 55757/310080 objects degraded (17.981%); 6.2 MiB/s, 99 objects/s recovering 2024-03-26T05:16:35.185 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:35.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:35.223 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:35.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:35.324 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:35.363 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:35.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:35.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:35.465 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:35.492 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:35.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:35.593 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:35.634 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:35.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:35.736 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:35.757 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T05:16:35.775 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:35.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:35.877 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:35.915 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:35.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:36.016 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:36.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:35 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:16:36.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:35 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:16:36.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:16:36.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:16:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:16:36.057 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:36.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:36.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:35 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:16:36.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:35 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:16:36.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:16:36.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:35 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:16:36.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:35 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:16:36.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:16:36.127 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:16:36.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:36.195 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:36.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:36.297 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:36.330 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:36.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:36.433 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:36.472 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:36.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:36.574 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:36.608 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:36.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:36.710 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:36.743 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:36.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:36.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:36.845 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:16:36.849 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:16:36.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:36.904 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:36.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:37.005 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:37.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[29691]: 5.12 scrub starts 2024-03-26T05:16:37.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[29691]: 5.12 scrub ok 2024-03-26T05:16:37.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:37.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:16:37.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:16:37.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[29691]: pgmap v738: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 52796/295302 objects degraded (17.879%); 8.3 MiB/s, 133 objects/s recovering 2024-03-26T05:16:37.015 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 55757/310080 objects degraded (17.981%), 32 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:37.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[40404]: 5.12 scrub starts 2024-03-26T05:16:37.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[40404]: 5.12 scrub ok 2024-03-26T05:16:37.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:37.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:16:37.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:16:37.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[40404]: pgmap v738: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 52796/295302 objects degraded (17.879%); 8.3 MiB/s, 133 objects/s recovering 2024-03-26T05:16:37.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:36 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 55757/310080 objects degraded (17.981%), 32 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:37.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:36 smithi132 ceph-mon[33500]: 5.12 scrub starts 2024-03-26T05:16:37.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:36 smithi132 ceph-mon[33500]: 5.12 scrub ok 2024-03-26T05:16:37.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:37.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:36 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:16:37.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:36 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:16:37.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:36 smithi132 ceph-mon[33500]: pgmap v738: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 52796/295302 objects degraded (17.879%); 8.3 MiB/s, 133 objects/s recovering 2024-03-26T05:16:37.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:36 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 55757/310080 objects degraded (17.981%), 32 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:37.043 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:37.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:37.145 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:37.183 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:37.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:37.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:37.285 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:37.321 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:37.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:37.423 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:37.456 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:37.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:37.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:37.558 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:37.567 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:37.590 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:37.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:37.692 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:37.734 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:37 smithi080 ceph-mon[40404]: osdmap e358: 8 total, 7 up, 8 in 2024-03-26T05:16:37.734 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:37 smithi080 ceph-mon[29691]: osdmap e358: 8 total, 7 up, 8 in 2024-03-26T05:16:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:37.740 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:37.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:37.841 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:37.891 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:37.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:37.980 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:37 smithi132 ceph-mon[33500]: osdmap e358: 8 total, 7 up, 8 in 2024-03-26T05:16:37.993 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:38.028 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:38.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:38.130 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:38.172 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:38.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:38.274 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:38.307 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:38.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:38.408 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:38.450 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:38.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:38.552 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:38.561 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:16:38.561 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:16:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:38.594 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:38.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:38.695 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:38.736 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:38.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:38.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:38.830 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:16:38.834 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:16:38.838 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:38.849 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:16:38.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:38 smithi080 ceph-mon[29691]: 4.10 scrub starts 2024-03-26T05:16:38.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:38 smithi080 ceph-mon[29691]: 4.10 scrub ok 2024-03-26T05:16:38.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:38 smithi080 ceph-mon[29691]: pgmap v740: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 52315/293250 objects degraded (17.840%); 9.9 MiB/s, 158 objects/s recovering 2024-03-26T05:16:38.850 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:38 smithi080 ceph-mon[40404]: 4.10 scrub starts 2024-03-26T05:16:38.850 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:38 smithi080 ceph-mon[40404]: 4.10 scrub ok 2024-03-26T05:16:38.851 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:38 smithi080 ceph-mon[40404]: pgmap v740: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 52315/293250 objects degraded (17.840%); 9.9 MiB/s, 158 objects/s recovering 2024-03-26T05:16:38.867 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:38.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:38.969 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:39.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:39.005 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:39.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:39.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:38 smithi132 ceph-mon[33500]: 4.10 scrub starts 2024-03-26T05:16:39.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:38 smithi132 ceph-mon[33500]: 4.10 scrub ok 2024-03-26T05:16:39.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:38 smithi132 ceph-mon[33500]: pgmap v740: 158 pgs: 37 active+undersized, 89 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 52315/293250 objects degraded (17.840%); 9.9 MiB/s, 158 objects/s recovering 2024-03-26T05:16:39.106 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:39.135 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:39.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:39.237 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:39.272 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:39.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:39.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:39.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:39.393 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T05:16:39.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:39.419 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:39.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:39.520 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:39.545 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:39.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:39.647 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:39.695 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:39.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:39.796 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:39.831 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:39.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:39.862 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:16:39.933 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:39.974 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:39.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:39.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:39.999 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:39 smithi132 ceph-mon[33500]: 10.f scrub starts 2024-03-26T05:16:39.999 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:39 smithi132 ceph-mon[33500]: from='client.25622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:16:40.076 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:40.114 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:40.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:40.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:39 smithi080 ceph-mon[40404]: 10.f scrub starts 2024-03-26T05:16:40.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:39 smithi080 ceph-mon[40404]: from='client.25622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:16:40.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:39 smithi080 ceph-mon[29691]: 10.f scrub starts 2024-03-26T05:16:40.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:39 smithi080 ceph-mon[29691]: from='client.25622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:16:40.215 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:40.247 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:40.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:40.349 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:40.383 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:40.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:40.484 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:40.518 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:40.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:40.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:40.621 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:40.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:40.659 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:40.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:40.759 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:40.793 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:40.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:40.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:40.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:40 smithi080 ceph-mon[29691]: 10.f scrub ok 2024-03-26T05:16:40.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:40 smithi080 ceph-mon[29691]: 10.a scrub starts 2024-03-26T05:16:40.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:40 smithi080 ceph-mon[29691]: pgmap v741: 158 pgs: 1 active+clean+scrubbing, 37 active+undersized, 88 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 605 B/s rd, 0 B/s wr, 966 op/s; 51233/285561 objects degraded (17.941%); 4.7 MiB/s, 74 objects/s recovering 2024-03-26T05:16:40.841 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:40 smithi080 ceph-mon[40404]: 10.f scrub ok 2024-03-26T05:16:40.841 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:40 smithi080 ceph-mon[40404]: 10.a scrub starts 2024-03-26T05:16:40.841 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:40 smithi080 ceph-mon[40404]: pgmap v741: 158 pgs: 1 active+clean+scrubbing, 37 active+undersized, 88 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 605 B/s rd, 0 B/s wr, 966 op/s; 51233/285561 objects degraded (17.941%); 4.7 MiB/s, 74 objects/s recovering 2024-03-26T05:16:40.896 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:40.930 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:40.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:40.959 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:41.031 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:41.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:40 smithi132 ceph-mon[33500]: 10.f scrub ok 2024-03-26T05:16:41.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:40 smithi132 ceph-mon[33500]: 10.a scrub starts 2024-03-26T05:16:41.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:40 smithi132 ceph-mon[33500]: pgmap v741: 158 pgs: 1 active+clean+scrubbing, 37 active+undersized, 88 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 605 B/s rd, 0 B/s wr, 966 op/s; 51233/285561 objects degraded (17.941%); 4.7 MiB/s, 74 objects/s recovering 2024-03-26T05:16:41.064 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:41.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:41.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:41.197 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:41.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:41.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:41.300 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:41.356 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:41.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:41.399 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":740,"stamp":"2024-03-26T05:16:38.081842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6391648990,"num_objects":97750,"num_object_clones":0,"num_object_copies":293250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97750,"num_whiteouts":0,"num_read":1684,"num_read_kb":1464,"num_write":126951,"num_write_kb":7151662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17087,"num_bytes_recovered":1113210692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89515,"ondisk_log_size":89515,"up":407,"acting":405,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22166200,"kb_used_data":18216824,"kb_used_omap":15,"kb_used_meta":3949040,"kb_avail":727795016,"statfs":{"total":767960285184,"available":745262096384,"internally_reserved":0,"allocated":18654027776,"data_stored":18650265719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":4043817742},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-570163200,"num_objects":-8700,"num_object_clones":0,"num_object_copies":-26100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5802,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8700,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8700,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1268,"num_bytes_recovered":83099648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003253"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":358,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.631339+0000","last_change":"2024-03-26T05:15:48.649557+0000","last_active":"2024-03-26T05:16:31.631339+0000","last_peered":"2024-03-26T05:16:31.631339+0000","last_clean":"2024-03-26T05:07:55.689482+0000","last_became_active":"2024-03-26T05:15:48.649557+0000","last_became_peered":"2024-03-26T05:15:48.649557+0000","last_unstale":"2024-03-26T05:16:31.631339+0000","last_undegraded":"2024-03-26T05:16:31.631339+0000","last_fullsized":"2024-03-26T05:15:48.628605+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:53.055153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:53.055153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":654,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.646016+0000","last_change":"2024-03-26T05:15:39.109021+0000","last_active":"2024-03-26T05:16:31.646016+0000","last_peered":"2024-03-26T05:16:31.646016+0000","last_clean":"2024-03-26T05:16:31.646016+0000","last_became_active":"2024-03-26T05:15:32.629027+0000","last_became_peered":"2024-03-26T05:15:32.629027+0000","last_unstale":"2024-03-26T05:16:31.646016+0000","last_undegraded":"2024-03-26T05:16:31.646016+0000","last_fullsized":"2024-03-26T05:16:31.646016+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:39.108879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:15:39.108879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:51.846119+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":664,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.635520+0000","last_change":"2024-03-26T05:14:20.034981+0000","last_active":"2024-03-26T05:16:31.635520+0000","last_peered":"2024-03-26T05:16:31.635520+0000","last_clean":"2024-03-26T05:16:31.635520+0000","last_became_active":"2024-03-26T05:07:59.994431+0000","last_became_peered":"2024-03-26T05:07:59.994431+0000","last_unstale":"2024-03-26T05:16:31.635520+0000","last_undegraded":"2024-03-26T05:16:31.635520+0000","last_fullsized":"2024-03-26T05:16:31.635520+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:20.034875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:20.034875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"254'21","reported_seq":487,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.641852+0000","last_change":"2024-03-26T05:15:48.639602+0000","last_active":"2024-03-26T05:16:31.641852+0000","last_peered":"2024-03-26T05:16:31.641852+0000","last_clean":"2024-03-26T05:10:30.455391+0000","last_became_active":"2024-03-26T05:15:48.639602+0000","last_became_peered":"2024-03-26T05:15:48.639602+0000","last_unstale":"2024-03-26T05:16:31.641852+0000","last_undegraded":"2024-03-26T05:15:48.632373+0000","last_fullsized":"2024-03-26T05:15:48.632263+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:15:04.002977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:15:04.002977+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:33.046483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":622,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.651688+0000","last_change":"2024-03-26T05:15:40.674528+0000","last_active":"2024-03-26T05:16:31.651688+0000","last_peered":"2024-03-26T05:16:31.651688+0000","last_clean":"2024-03-26T05:16:31.651688+0000","last_became_active":"2024-03-26T05:15:32.637717+0000","last_became_peered":"2024-03-26T05:15:32.637717+0000","last_unstale":"2024-03-26T05:16:31.651688+0000","last_undegraded":"2024-03-26T05:16:31.651688+0000","last_fullsized":"2024-03-26T05:16:31.651688+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:40.674397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:40.674397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:54.508111+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":37,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.641496+0000","last_change":"2024-03-26T05:15:48.639904+0000","last_active":"2024-03-26T05:16:31.641496+0000","last_peered":"2024-03-26T05:16:31.641496+0000","last_clean":"2024-03-26T05:00:25.933152+0000","last_became_active":"2024-03-26T05:15:48.639904+0000","last_became_peered":"2024-03-26T05:15:48.639904+0000","last_unstale":"2024-03-26T05:16:31.641496+0000","last_undegraded":"2024-03-26T05:16:31.641496+0000","last_fullsized":"2024-03-26T05:15:48.630870+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:59.872240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:13:59.872240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:32.373054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":651,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642159+0000","last_change":"2024-03-26T05:16:30.881766+0000","last_active":"2024-03-26T05:16:31.642159+0000","last_peered":"2024-03-26T05:16:31.642159+0000","last_clean":"2024-03-26T05:16:31.642159+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:16:31.642159+0000","last_undegraded":"2024-03-26T05:16:31.642159+0000","last_fullsized":"2024-03-26T05:16:31.642159+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:30.881617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:16:30.881617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:56.811155+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"254'51","reported_seq":728,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642723+0000","last_change":"2024-03-26T05:15:14.624289+0000","last_active":"2024-03-26T05:16:31.642723+0000","last_peered":"2024-03-26T05:16:31.642723+0000","last_clean":"2024-03-26T05:16:31.642723+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:16:31.642723+0000","last_undegraded":"2024-03-26T05:16:31.642723+0000","last_fullsized":"2024-03-26T05:16:31.642723+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'51","last_scrub_stamp":"2024-03-26T05:15:14.624135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:15:14.624135+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":640,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.647196+0000","last_change":"2024-03-26T05:15:32.640385+0000","last_active":"2024-03-26T05:16:31.647196+0000","last_peered":"2024-03-26T05:16:31.647196+0000","last_clean":"2024-03-26T05:16:31.647196+0000","last_became_active":"2024-03-26T05:15:32.640052+0000","last_became_peered":"2024-03-26T05:15:32.640052+0000","last_unstale":"2024-03-26T05:16:31.647196+0000","last_undegraded":"2024-03-26T05:16:31.647196+0000","last_fullsized":"2024-03-26T05:16:31.647196+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:04.037472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:04.037472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":324,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.630940+0000","last_change":"2024-03-26T05:15:32.622450+0000","last_active":"2024-03-26T05:16:31.630940+0000","last_peered":"2024-03-26T05:16:31.630940+0000","last_clean":"2024-03-26T05:16:31.630940+0000","last_became_active":"2024-03-26T05:15:32.622003+0000","last_became_peered":"2024-03-26T05:15:32.622003+0000","last_unstale":"2024-03-26T05:16:31.630940+0000","last_undegraded":"2024-03-26T05:16:31.630940+0000","last_fullsized":"2024-03-26T05:16:31.630940+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:04.643034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:04.643034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":326,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.631606+0000","last_change":"2024-03-26T05:15:48.636186+0000","last_active":"2024-03-26T05:16:31.631606+0000","last_peered":"2024-03-26T05:16:31.631606+0000","last_clean":"2024-03-26T05:15:46.631730+0000","last_became_active":"2024-03-26T05:15:48.636186+0000","last_became_peered":"2024-03-26T05:15:48.636186+0000","last_unstale":"2024-03-26T05:16:31.631606+0000","last_undegraded":"2024-03-26T05:16:31.631606+0000","last_fullsized":"2024-03-26T05:15:48.629342+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:17.792745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:17.792745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"254'27","reported_seq":499,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.642373+0000","last_change":"2024-03-26T05:15:48.647529+0000","last_active":"2024-03-26T05:16:31.642373+0000","last_peered":"2024-03-26T05:16:31.642373+0000","last_clean":"2024-03-26T05:10:30.566889+0000","last_became_active":"2024-03-26T05:15:48.647529+0000","last_became_peered":"2024-03-26T05:15:48.647529+0000","last_unstale":"2024-03-26T05:16:31.642373+0000","last_undegraded":"2024-03-26T05:15:48.629088+0000","last_fullsized":"2024-03-26T05:15:48.628959+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:13:22.026354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:22.026354+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:35.867348+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":323,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.631333+0000","last_change":"2024-03-26T05:14:10.840006+0000","last_active":"2024-03-26T05:16:31.631333+0000","last_peered":"2024-03-26T05:16:31.631333+0000","last_clean":"2024-03-26T05:16:31.631333+0000","last_became_active":"2024-03-26T05:07:59.998777+0000","last_became_peered":"2024-03-26T05:07:59.998777+0000","last_unstale":"2024-03-26T05:16:31.631333+0000","last_undegraded":"2024-03-26T05:16:31.631333+0000","last_fullsized":"2024-03-26T05:16:31.631333+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:10.839912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:10.839912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":562,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.624623+0000","last_change":"2024-03-26T05:15:48.652929+0000","last_active":"2024-03-26T05:16:31.624623+0000","last_peered":"2024-03-26T05:16:31.624623+0000","last_clean":"2024-03-26T05:15:46.623242+0000","last_became_active":"2024-03-26T05:15:48.652929+0000","last_became_peered":"2024-03-26T05:15:48.652929+0000","last_unstale":"2024-03-26T05:16:31.624623+0000","last_undegraded":"2024-03-26T05:16:31.624623+0000","last_fullsized":"2024-03-26T05:15:48.634670+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_clean_scrub_stamp":"2024-03-26T05:12:18.959687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":323,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.632125+0000","last_change":"2024-03-26T05:15:00.635456+0000","last_active":"2024-03-26T05:16:31.632125+0000","last_peered":"2024-03-26T05:16:31.632125+0000","last_clean":"2024-03-26T05:16:31.632125+0000","last_became_active":"2024-03-26T05:08:00.011262+0000","last_became_peered":"2024-03-26T05:08:00.011262+0000","last_unstale":"2024-03-26T05:16:31.632125+0000","last_undegraded":"2024-03-26T05:16:31.632125+0000","last_fullsized":"2024-03-26T05:16:31.632125+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:00.635308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:00.635308+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.635308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"321'16","reported_seq":623,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.651294+0000","last_change":"2024-03-26T05:15:48.646822+0000","last_active":"2024-03-26T05:16:31.651294+0000","last_peered":"2024-03-26T05:16:31.651294+0000","last_clean":"2024-03-26T05:15:00.994414+0000","last_became_active":"2024-03-26T05:15:48.646822+0000","last_became_peered":"2024-03-26T05:15:48.646822+0000","last_unstale":"2024-03-26T05:16:31.651294+0000","last_undegraded":"2024-03-26T05:15:48.630770+0000","last_fullsized":"2024-03-26T05:15:48.630637+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'16","last_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_deep_scrub":"321'16","last_deep_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.995735+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":366,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.644181+0000","last_change":"2024-03-26T05:15:48.646231+0000","last_active":"2024-03-26T05:16:31.644181+0000","last_peered":"2024-03-26T05:16:31.644181+0000","last_clean":"2024-03-26T05:07:55.689953+0000","last_became_active":"2024-03-26T05:15:48.646231+0000","last_became_peered":"2024-03-26T05:15:48.646231+0000","last_unstale":"2024-03-26T05:16:31.644181+0000","last_undegraded":"2024-03-26T05:16:31.644181+0000","last_fullsized":"2024-03-26T05:15:48.631854+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:05.787712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:14:05.787712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:32.373040+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":644,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.647791+0000","last_change":"2024-03-26T05:15:48.648471+0000","last_active":"2024-03-26T05:16:31.647791+0000","last_peered":"2024-03-26T05:16:31.647791+0000","last_clean":"2024-03-26T05:15:46.610002+0000","last_became_active":"2024-03-26T05:15:48.648471+0000","last_became_peered":"2024-03-26T05:15:48.648471+0000","last_unstale":"2024-03-26T05:16:31.647791+0000","last_undegraded":"2024-03-26T05:16:31.647791+0000","last_fullsized":"2024-03-26T05:15:48.637209+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:11.912203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:15:11.912203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:41.249734+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":655,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.640561+0000","last_change":"2024-03-26T05:15:48.652795+0000","last_active":"2024-03-26T05:16:31.640561+0000","last_peered":"2024-03-26T05:16:31.640561+0000","last_clean":"2024-03-26T05:15:46.621710+0000","last_became_active":"2024-03-26T05:15:48.652795+0000","last_became_peered":"2024-03-26T05:15:48.652795+0000","last_unstale":"2024-03-26T05:16:31.640561+0000","last_undegraded":"2024-03-26T05:16:31.640561+0000","last_fullsized":"2024-03-26T05:15:48.642501+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:04.980730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:12:04.980730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"254'27","reported_seq":696,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.652077+0000","last_change":"2024-03-26T05:14:53.363298+0000","last_active":"2024-03-26T05:16:31.652077+0000","last_peered":"2024-03-26T05:16:31.652077+0000","last_clean":"2024-03-26T05:16:31.652077+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:16:31.652077+0000","last_undegraded":"2024-03-26T05:16:31.652077+0000","last_fullsized":"2024-03-26T05:16:31.652077+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:14:53.363147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:53.363147+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":651,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.644094+0000","last_change":"2024-03-26T05:16:03.006651+0000","last_active":"2024-03-26T05:16:31.644094+0000","last_peered":"2024-03-26T05:16:31.644094+0000","last_clean":"2024-03-26T05:16:31.644094+0000","last_became_active":"2024-03-26T05:07:59.959705+0000","last_became_peered":"2024-03-26T05:07:59.959705+0000","last_unstale":"2024-03-26T05:16:31.644094+0000","last_undegraded":"2024-03-26T05:16:31.644094+0000","last_fullsized":"2024-03-26T05:16:31.644094+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:03.006537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:15.038608+0000","last_clean_scrub_stamp":"2024-03-26T05:16:03.006537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:03.648941+0000","scrub_duration":237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":37,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.642618+0000","last_change":"2024-03-26T05:15:48.638667+0000","last_active":"2024-03-26T05:16:31.642618+0000","last_peered":"2024-03-26T05:16:31.642618+0000","last_clean":"2024-03-26T05:00:25.933152+0000","last_became_active":"2024-03-26T05:15:48.638667+0000","last_became_peered":"2024-03-26T05:15:48.638667+0000","last_unstale":"2024-03-26T05:16:31.642618+0000","last_undegraded":"2024-03-26T05:16:31.642618+0000","last_fullsized":"2024-03-26T05:15:48.630074+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:04.830989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:04.830989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:35.867372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":37,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.647133+0000","last_change":"2024-03-26T05:15:48.650757+0000","last_active":"2024-03-26T05:16:31.647133+0000","last_peered":"2024-03-26T05:16:31.647133+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:15:48.650757+0000","last_became_peered":"2024-03-26T05:15:48.650757+0000","last_unstale":"2024-03-26T05:16:31.647133+0000","last_undegraded":"2024-03-26T05:16:31.647133+0000","last_fullsized":"2024-03-26T05:15:48.636772+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:03.047441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:15:03.047441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"322'32","reported_seq":692,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.644880+0000","last_change":"2024-03-26T05:15:48.643576+0000","last_active":"2024-03-26T05:16:31.644880+0000","last_peered":"2024-03-26T05:16:31.644880+0000","last_clean":"2024-03-26T05:15:46.628715+0000","last_became_active":"2024-03-26T05:15:48.643576+0000","last_became_peered":"2024-03-26T05:15:48.643576+0000","last_unstale":"2024-03-26T05:16:31.644880+0000","last_undegraded":"2024-03-26T05:15:48.629218+0000","last_fullsized":"2024-03-26T05:15:48.629123+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'32","last_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_deep_scrub":"322'32","last_deep_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_clean_scrub_stamp":"2024-03-26T05:15:10.715786+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":566,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.625721+0000","last_change":"2024-03-26T05:15:32.637389+0000","last_active":"2024-03-26T05:16:31.625721+0000","last_peered":"2024-03-26T05:16:31.625721+0000","last_clean":"2024-03-26T05:16:31.625721+0000","last_became_active":"2024-03-26T05:15:32.636624+0000","last_became_peered":"2024-03-26T05:15:32.636624+0000","last_unstale":"2024-03-26T05:16:31.625721+0000","last_undegraded":"2024-03-26T05:16:31.625721+0000","last_fullsized":"2024-03-26T05:16:31.625721+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:12:13.995144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:13.995144+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"343'11","reported_seq":870,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.649326+0000","last_change":"2024-03-26T05:15:48.644163+0000","last_active":"2024-03-26T05:16:31.649326+0000","last_peered":"2024-03-26T05:16:31.649326+0000","last_clean":"2024-03-26T05:15:46.619140+0000","last_became_active":"2024-03-26T05:15:48.644163+0000","last_became_peered":"2024-03-26T05:15:48.644163+0000","last_unstale":"2024-03-26T05:16:31.649326+0000","last_undegraded":"2024-03-26T05:15:48.636521+0000","last_fullsized":"2024-03-26T05:15:48.636379+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'10","last_scrub_stamp":"2024-03-26T05:15:38.516660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:15:38.516660+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:54.906930+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":653,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.646110+0000","last_change":"2024-03-26T05:15:48.647499+0000","last_active":"2024-03-26T05:16:31.646110+0000","last_peered":"2024-03-26T05:16:31.646110+0000","last_clean":"2024-03-26T05:15:46.609101+0000","last_became_active":"2024-03-26T05:15:48.647499+0000","last_became_peered":"2024-03-26T05:15:48.647499+0000","last_unstale":"2024-03-26T05:16:31.646110+0000","last_undegraded":"2024-03-26T05:15:48.634583+0000","last_fullsized":"2024-03-26T05:15:48.634474+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:49.948818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:49.948818+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'32","reported_seq":698,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.636064+0000","last_change":"2024-03-26T05:15:32.634867+0000","last_active":"2024-03-26T05:16:31.636064+0000","last_peered":"2024-03-26T05:16:31.636064+0000","last_clean":"2024-03-26T05:16:31.636064+0000","last_became_active":"2024-03-26T05:15:32.634216+0000","last_became_peered":"2024-03-26T05:15:32.634216+0000","last_unstale":"2024-03-26T05:16:31.636064+0000","last_undegraded":"2024-03-26T05:16:31.636064+0000","last_fullsized":"2024-03-26T05:16:31.636064+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'32","last_scrub_stamp":"2024-03-26T05:12:15.970313+0000","last_deep_scrub":"290'32","last_deep_scrub_stamp":"2024-03-26T05:12:15.970313+0000","last_clean_scrub_stamp":"2024-03-26T05:12:15.970313+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":652,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.648085+0000","last_change":"2024-03-26T05:15:48.644977+0000","last_active":"2024-03-26T05:16:31.648085+0000","last_peered":"2024-03-26T05:16:31.648085+0000","last_clean":"2024-03-26T05:15:46.610540+0000","last_became_active":"2024-03-26T05:15:48.644977+0000","last_became_peered":"2024-03-26T05:15:48.644977+0000","last_unstale":"2024-03-26T05:16:31.648085+0000","last_undegraded":"2024-03-26T05:16:31.648085+0000","last_fullsized":"2024-03-26T05:15:48.636267+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:00.151355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.151355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":638,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.636323+0000","last_change":"2024-03-26T05:15:32.635171+0000","last_active":"2024-03-26T05:16:31.636323+0000","last_peered":"2024-03-26T05:16:31.636323+0000","last_clean":"2024-03-26T05:16:31.636323+0000","last_became_active":"2024-03-26T05:15:32.634716+0000","last_became_peered":"2024-03-26T05:15:32.634716+0000","last_unstale":"2024-03-26T05:16:31.636323+0000","last_undegraded":"2024-03-26T05:16:31.636323+0000","last_fullsized":"2024-03-26T05:16:31.636323+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:27.080959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:27.080959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":649,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.868683+0000","last_change":"2024-03-26T05:16:31.868719+0000","last_active":"2024-03-26T05:16:31.868683+0000","last_peered":"2024-03-26T05:16:31.868683+0000","last_clean":"2024-03-26T05:16:31.868683+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:16:31.868683+0000","last_undegraded":"2024-03-26T05:16:31.868683+0000","last_fullsized":"2024-03-26T05:16:31.868683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:31.868603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:16:31.868603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:45.432783+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"254'25","reported_seq":772,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:34.220484+0000","last_change":"2024-03-26T05:16:34.220528+0000","last_active":"2024-03-26T05:16:34.220484+0000","last_peered":"2024-03-26T05:16:34.220484+0000","last_clean":"2024-03-26T05:16:34.220484+0000","last_became_active":"2024-03-26T05:07:59.962007+0000","last_became_peered":"2024-03-26T05:07:59.962007+0000","last_unstale":"2024-03-26T05:16:34.220484+0000","last_undegraded":"2024-03-26T05:16:34.220484+0000","last_fullsized":"2024-03-26T05:16:34.220484+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:16:34.220406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:16:34.220406+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:56.927766+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":547,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.624393+0000","last_change":"2024-03-26T05:15:48.641318+0000","last_active":"2024-03-26T05:16:31.624393+0000","last_peered":"2024-03-26T05:16:31.624393+0000","last_clean":"2024-03-26T05:15:46.623140+0000","last_became_active":"2024-03-26T05:15:48.641318+0000","last_became_peered":"2024-03-26T05:15:48.641318+0000","last_unstale":"2024-03-26T05:16:31.624393+0000","last_undegraded":"2024-03-26T05:16:31.624393+0000","last_fullsized":"2024-03-26T05:15:48.634268+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:10.020572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:10.020572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":644,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.641545+0000","last_change":"2024-03-26T05:15:48.643994+0000","last_active":"2024-03-26T05:16:31.641545+0000","last_peered":"2024-03-26T05:16:31.641545+0000","last_clean":"2024-03-26T05:15:46.616068+0000","last_became_active":"2024-03-26T05:15:48.643994+0000","last_became_peered":"2024-03-26T05:15:48.643994+0000","last_unstale":"2024-03-26T05:16:31.641545+0000","last_undegraded":"2024-03-26T05:16:31.641545+0000","last_fullsized":"2024-03-26T05:15:48.631480+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.693592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.693592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:32.372989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":671,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.637399+0000","last_change":"2024-03-26T05:15:11.070456+0000","last_active":"2024-03-26T05:16:31.637399+0000","last_peered":"2024-03-26T05:16:31.637399+0000","last_clean":"2024-03-26T05:16:31.637399+0000","last_became_active":"2024-03-26T05:07:59.993996+0000","last_became_peered":"2024-03-26T05:07:59.993996+0000","last_unstale":"2024-03-26T05:16:31.637399+0000","last_undegraded":"2024-03-26T05:16:31.637399+0000","last_fullsized":"2024-03-26T05:16:31.637399+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:11.070341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:11.070341+0000","last_clean_scrub_stamp":"2024-03-26T05:15:11.070341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"254'10","reported_seq":651,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.641922+0000","last_change":"2024-03-26T05:14:58.814632+0000","last_active":"2024-03-26T05:16:31.641922+0000","last_peered":"2024-03-26T05:16:31.641922+0000","last_clean":"2024-03-26T05:16:31.641922+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:16:31.641922+0000","last_undegraded":"2024-03-26T05:16:31.641922+0000","last_fullsized":"2024-03-26T05:16:31.641922+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'10","last_scrub_stamp":"2024-03-26T05:14:58.814461+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:14:58.814461+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":637,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:33.064359+0000","last_change":"2024-03-26T05:16:33.064381+0000","last_active":"2024-03-26T05:16:33.064359+0000","last_peered":"2024-03-26T05:16:33.064359+0000","last_clean":"2024-03-26T05:16:33.064359+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:16:33.064359+0000","last_undegraded":"2024-03-26T05:16:33.064359+0000","last_fullsized":"2024-03-26T05:16:33.064359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:33.064299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_clean_scrub_stamp":"2024-03-26T05:16:33.064299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:48.093037+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":563,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.630190+0000","last_change":"2024-03-26T05:15:48.640423+0000","last_active":"2024-03-26T05:16:31.630190+0000","last_peered":"2024-03-26T05:16:31.630190+0000","last_clean":"2024-03-26T05:15:46.625977+0000","last_became_active":"2024-03-26T05:15:48.640423+0000","last_became_peered":"2024-03-26T05:15:48.640423+0000","last_unstale":"2024-03-26T05:16:31.630190+0000","last_undegraded":"2024-03-26T05:16:31.630190+0000","last_fullsized":"2024-03-26T05:15:48.633132+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:23.957542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:11:23.957542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":316,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.632534+0000","last_change":"2024-03-26T05:15:32.620927+0000","last_active":"2024-03-26T05:16:31.632534+0000","last_peered":"2024-03-26T05:16:31.632534+0000","last_clean":"2024-03-26T05:16:31.632534+0000","last_became_active":"2024-03-26T05:15:32.620559+0000","last_became_peered":"2024-03-26T05:15:32.620559+0000","last_unstale":"2024-03-26T05:16:31.632534+0000","last_undegraded":"2024-03-26T05:16:31.632534+0000","last_fullsized":"2024-03-26T05:16:31.632534+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:19.513035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:11:19.513035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"254'25","reported_seq":681,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642619+0000","last_change":"2024-03-26T05:15:50.723419+0000","last_active":"2024-03-26T05:16:31.642619+0000","last_peered":"2024-03-26T05:16:31.642619+0000","last_clean":"2024-03-26T05:16:31.642619+0000","last_became_active":"2024-03-26T05:15:32.636359+0000","last_became_peered":"2024-03-26T05:15:32.636359+0000","last_unstale":"2024-03-26T05:16:31.642619+0000","last_undegraded":"2024-03-26T05:16:31.642619+0000","last_fullsized":"2024-03-26T05:16:31.642619+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:15:50.723325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:15:50.723325+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:56.830309+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":659,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.639656+0000","last_change":"2024-03-26T05:14:46.024625+0000","last_active":"2024-03-26T05:16:31.639656+0000","last_peered":"2024-03-26T05:16:31.639656+0000","last_clean":"2024-03-26T05:16:31.639656+0000","last_became_active":"2024-03-26T05:07:59.994210+0000","last_became_peered":"2024-03-26T05:07:59.994210+0000","last_unstale":"2024-03-26T05:16:31.639656+0000","last_undegraded":"2024-03-26T05:16:31.639656+0000","last_fullsized":"2024-03-26T05:16:31.639656+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:46.024526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:46.024526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":633,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.651600+0000","last_change":"2024-03-26T05:15:09.726646+0000","last_active":"2024-03-26T05:16:31.651600+0000","last_peered":"2024-03-26T05:16:31.651600+0000","last_clean":"2024-03-26T05:16:31.651600+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:16:31.651600+0000","last_undegraded":"2024-03-26T05:16:31.651600+0000","last_fullsized":"2024-03-26T05:16:31.651600+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:09.726518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:15:09.726518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":526,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.625358+0000","last_change":"2024-03-26T05:15:48.639023+0000","last_active":"2024-03-26T05:16:31.625358+0000","last_peered":"2024-03-26T05:16:31.625358+0000","last_clean":"2024-03-26T05:15:46.624142+0000","last_became_active":"2024-03-26T05:15:48.639023+0000","last_became_peered":"2024-03-26T05:15:48.639023+0000","last_unstale":"2024-03-26T05:16:31.625358+0000","last_undegraded":"2024-03-26T05:16:31.625358+0000","last_fullsized":"2024-03-26T05:15:48.631660+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:33.016072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:11:33.016072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"254'15","reported_seq":678,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:33.261601+0000","last_change":"2024-03-26T05:16:33.261812+0000","last_active":"2024-03-26T05:16:33.261601+0000","last_peered":"2024-03-26T05:16:33.261601+0000","last_clean":"2024-03-26T05:16:33.261601+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:16:33.261601+0000","last_undegraded":"2024-03-26T05:16:33.261601+0000","last_fullsized":"2024-03-26T05:16:33.261601+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'15","last_scrub_stamp":"2024-03-26T05:16:33.261286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:16:33.261286+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:54.535677+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":641,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642169+0000","last_change":"2024-03-26T05:16:03.797756+0000","last_active":"2024-03-26T05:16:31.642169+0000","last_peered":"2024-03-26T05:16:31.642169+0000","last_clean":"2024-03-26T05:16:31.642169+0000","last_became_active":"2024-03-26T05:07:59.999703+0000","last_became_peered":"2024-03-26T05:07:59.999703+0000","last_unstale":"2024-03-26T05:16:31.642169+0000","last_undegraded":"2024-03-26T05:16:31.642169+0000","last_fullsized":"2024-03-26T05:16:31.642169+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:03.797558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:16:03.797558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:25.381688+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":669,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:35.331871+0000","last_change":"2024-03-26T05:16:35.331912+0000","last_active":"2024-03-26T05:16:35.331871+0000","last_peered":"2024-03-26T05:16:35.331871+0000","last_clean":"2024-03-26T05:16:35.331871+0000","last_became_active":"2024-03-26T05:08:00.001601+0000","last_became_peered":"2024-03-26T05:08:00.001601+0000","last_unstale":"2024-03-26T05:16:35.331871+0000","last_undegraded":"2024-03-26T05:16:35.331871+0000","last_fullsized":"2024-03-26T05:16:35.331871+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:35.331752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:16:35.331752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:18:01.671300+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":622,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.625424+0000","last_change":"2024-03-26T05:15:48.642442+0000","last_active":"2024-03-26T05:16:31.625424+0000","last_peered":"2024-03-26T05:16:31.625424+0000","last_clean":"2024-03-26T05:12:19.733579+0000","last_became_active":"2024-03-26T05:15:48.642442+0000","last_became_peered":"2024-03-26T05:15:48.642442+0000","last_unstale":"2024-03-26T05:16:31.625424+0000","last_undegraded":"2024-03-26T05:16:31.625424+0000","last_fullsized":"2024-03-26T05:15:48.631873+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:21.515294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:21.515294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"254'24","reported_seq":486,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.641437+0000","last_change":"2024-03-26T05:15:48.645289+0000","last_active":"2024-03-26T05:16:31.641437+0000","last_peered":"2024-03-26T05:16:31.641437+0000","last_clean":"2024-03-26T05:10:30.628152+0000","last_became_active":"2024-03-26T05:15:48.645289+0000","last_became_peered":"2024-03-26T05:15:48.645289+0000","last_unstale":"2024-03-26T05:16:31.641437+0000","last_undegraded":"2024-03-26T05:15:48.632524+0000","last_fullsized":"2024-03-26T05:15:48.632421+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'24","last_scrub_stamp":"2024-03-26T05:13:56.875610+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:13:56.875610+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:32.373026+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":640,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.635695+0000","last_change":"2024-03-26T05:15:32.631774+0000","last_active":"2024-03-26T05:16:31.635695+0000","last_peered":"2024-03-26T05:16:31.635695+0000","last_clean":"2024-03-26T05:16:31.635695+0000","last_became_active":"2024-03-26T05:15:32.631444+0000","last_became_peered":"2024-03-26T05:15:32.631444+0000","last_unstale":"2024-03-26T05:16:31.635695+0000","last_undegraded":"2024-03-26T05:16:31.635695+0000","last_fullsized":"2024-03-26T05:16:31.635695+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:17.975697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_clean_scrub_stamp":"2024-03-26T05:12:17.975697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"216'10","reported_seq":430,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.631551+0000","last_change":"2024-03-26T05:14:20.704552+0000","last_active":"2024-03-26T05:16:31.631551+0000","last_peered":"2024-03-26T05:16:31.631551+0000","last_clean":"2024-03-26T05:16:31.631551+0000","last_became_active":"2024-03-26T05:08:00.005648+0000","last_became_peered":"2024-03-26T05:08:00.005648+0000","last_unstale":"2024-03-26T05:16:31.631551+0000","last_undegraded":"2024-03-26T05:16:31.631551+0000","last_fullsized":"2024-03-26T05:16:31.631551+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:14:20.704431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:20.704431+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":554,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.624787+0000","last_change":"2024-03-26T05:15:48.641873+0000","last_active":"2024-03-26T05:16:31.624787+0000","last_peered":"2024-03-26T05:16:31.624787+0000","last_clean":"2024-03-26T05:15:46.623430+0000","last_became_active":"2024-03-26T05:15:48.641873+0000","last_became_peered":"2024-03-26T05:15:48.641873+0000","last_unstale":"2024-03-26T05:16:31.624787+0000","last_undegraded":"2024-03-26T05:16:31.624787+0000","last_fullsized":"2024-03-26T05:15:48.632198+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:01.962048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:01.962048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"345'39","reported_seq":707,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642480+0000","last_change":"2024-03-26T05:15:56.674563+0000","last_active":"2024-03-26T05:16:31.642480+0000","last_peered":"2024-03-26T05:16:31.642480+0000","last_clean":"2024-03-26T05:16:31.642480+0000","last_became_active":"2024-03-26T05:15:32.626448+0000","last_became_peered":"2024-03-26T05:15:32.626448+0000","last_unstale":"2024-03-26T05:16:31.642480+0000","last_undegraded":"2024-03-26T05:16:31.642480+0000","last_fullsized":"2024-03-26T05:16:31.642480+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'39","last_scrub_stamp":"2024-03-26T05:15:56.674423+0000","last_deep_scrub":"345'39","last_deep_scrub_stamp":"2024-03-26T05:15:56.674423+0000","last_clean_scrub_stamp":"2024-03-26T05:15:56.674423+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:59.060766+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":358,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.651613+0000","last_change":"2024-03-26T05:15:48.641096+0000","last_active":"2024-03-26T05:16:31.651613+0000","last_peered":"2024-03-26T05:16:31.651613+0000","last_clean":"2024-03-26T05:07:55.690747+0000","last_became_active":"2024-03-26T05:15:48.641096+0000","last_became_peered":"2024-03-26T05:15:48.641096+0000","last_unstale":"2024-03-26T05:16:31.651613+0000","last_undegraded":"2024-03-26T05:16:31.651613+0000","last_fullsized":"2024-03-26T05:15:48.634411+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:06.096138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:06.096138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"336'11","reported_seq":843,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.643247+0000","last_change":"2024-03-26T05:15:41.588979+0000","last_active":"2024-03-26T05:16:31.643247+0000","last_peered":"2024-03-26T05:16:31.643247+0000","last_clean":"2024-03-26T05:16:31.643247+0000","last_became_active":"2024-03-26T05:15:32.636392+0000","last_became_peered":"2024-03-26T05:15:32.636392+0000","last_unstale":"2024-03-26T05:16:31.643247+0000","last_undegraded":"2024-03-26T05:16:31.643247+0000","last_fullsized":"2024-03-26T05:16:31.643247+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-03-26T05:15:41.588817+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:15:41.588817+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:56.696493+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":652,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.638178+0000","last_change":"2024-03-26T05:15:32.632805+0000","last_active":"2024-03-26T05:16:31.638178+0000","last_peered":"2024-03-26T05:16:31.638178+0000","last_clean":"2024-03-26T05:16:31.638178+0000","last_became_active":"2024-03-26T05:15:32.632387+0000","last_became_peered":"2024-03-26T05:15:32.632387+0000","last_unstale":"2024-03-26T05:16:31.638178+0000","last_undegraded":"2024-03-26T05:16:31.638178+0000","last_fullsized":"2024-03-26T05:16:31.638178+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:59.989341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:59.989341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"254'26","reported_seq":345,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.631354+0000","last_change":"2024-03-26T05:15:32.621508+0000","last_active":"2024-03-26T05:16:31.631354+0000","last_peered":"2024-03-26T05:16:31.631354+0000","last_clean":"2024-03-26T05:16:31.631354+0000","last_became_active":"2024-03-26T05:15:32.621049+0000","last_became_peered":"2024-03-26T05:15:32.621049+0000","last_unstale":"2024-03-26T05:16:31.631354+0000","last_undegraded":"2024-03-26T05:16:31.631354+0000","last_fullsized":"2024-03-26T05:16:31.631354+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:12:07.599746+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.599746+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":649,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.652350+0000","last_change":"2024-03-26T05:15:48.639802+0000","last_active":"2024-03-26T05:16:31.652350+0000","last_peered":"2024-03-26T05:16:31.652350+0000","last_clean":"2024-03-26T05:15:46.626518+0000","last_became_active":"2024-03-26T05:15:48.639802+0000","last_became_peered":"2024-03-26T05:15:48.639802+0000","last_unstale":"2024-03-26T05:16:31.652350+0000","last_undegraded":"2024-03-26T05:16:31.652350+0000","last_fullsized":"2024-03-26T05:15:48.633216+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:13.142714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:13.142714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":634,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.644004+0000","last_change":"2024-03-26T05:15:48.701885+0000","last_active":"2024-03-26T05:16:31.644004+0000","last_peered":"2024-03-26T05:16:31.644004+0000","last_clean":"2024-03-26T05:16:31.644004+0000","last_became_active":"2024-03-26T05:15:32.641031+0000","last_became_peered":"2024-03-26T05:15:32.641031+0000","last_unstale":"2024-03-26T05:16:31.644004+0000","last_undegraded":"2024-03-26T05:16:31.644004+0000","last_fullsized":"2024-03-26T05:16:31.644004+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:48.701610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:15:48.701610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:05.188763+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":520,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.629064+0000","last_change":"2024-03-26T05:15:48.646040+0000","last_active":"2024-03-26T05:16:31.629064+0000","last_peered":"2024-03-26T05:16:31.629064+0000","last_clean":"2024-03-26T05:15:46.624763+0000","last_became_active":"2024-03-26T05:15:48.646040+0000","last_became_peered":"2024-03-26T05:15:48.646040+0000","last_unstale":"2024-03-26T05:16:31.629064+0000","last_undegraded":"2024-03-26T05:16:31.629064+0000","last_fullsized":"2024-03-26T05:15:48.635818+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.939576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.939576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"254'27","reported_seq":697,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.636148+0000","last_change":"2024-03-26T05:15:32.635064+0000","last_active":"2024-03-26T05:16:31.636148+0000","last_peered":"2024-03-26T05:16:31.636148+0000","last_clean":"2024-03-26T05:16:31.636148+0000","last_became_active":"2024-03-26T05:15:32.634434+0000","last_became_peered":"2024-03-26T05:15:32.634434+0000","last_unstale":"2024-03-26T05:16:31.636148+0000","last_undegraded":"2024-03-26T05:16:31.636148+0000","last_fullsized":"2024-03-26T05:16:31.636148+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:12:11.885908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:12:11.885908+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":546,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.625179+0000","last_change":"2024-03-26T05:15:48.643985+0000","last_active":"2024-03-26T05:16:31.625179+0000","last_peered":"2024-03-26T05:16:31.625179+0000","last_clean":"2024-03-26T05:15:46.624057+0000","last_became_active":"2024-03-26T05:15:48.643985+0000","last_became_peered":"2024-03-26T05:15:48.643985+0000","last_unstale":"2024-03-26T05:16:31.625179+0000","last_undegraded":"2024-03-26T05:16:31.625179+0000","last_fullsized":"2024-03-26T05:15:48.635169+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:07.006139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.006139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":323,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.632484+0000","last_change":"2024-03-26T05:14:04.756574+0000","last_active":"2024-03-26T05:16:31.632484+0000","last_peered":"2024-03-26T05:16:31.632484+0000","last_clean":"2024-03-26T05:16:31.632484+0000","last_became_active":"2024-03-26T05:07:59.997434+0000","last_became_peered":"2024-03-26T05:07:59.997434+0000","last_unstale":"2024-03-26T05:16:31.632484+0000","last_undegraded":"2024-03-26T05:16:31.632484+0000","last_fullsized":"2024-03-26T05:16:31.632484+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:04.756452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:04.756452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":691,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642136+0000","last_change":"2024-03-26T05:15:51.440953+0000","last_active":"2024-03-26T05:16:31.642136+0000","last_peered":"2024-03-26T05:16:31.642136+0000","last_clean":"2024-03-26T05:16:31.642136+0000","last_became_active":"2024-03-26T05:07:59.992217+0000","last_became_peered":"2024-03-26T05:07:59.992217+0000","last_unstale":"2024-03-26T05:16:31.642136+0000","last_undegraded":"2024-03-26T05:16:31.642136+0000","last_fullsized":"2024-03-26T05:16:31.642136+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:15:51.440867+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_clean_scrub_stamp":"2024-03-26T05:15:51.440867+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:03.691023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"254'18","reported_seq":668,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.644187+0000","last_change":"2024-03-26T05:15:46.643207+0000","last_active":"2024-03-26T05:16:31.644187+0000","last_peered":"2024-03-26T05:16:31.644187+0000","last_clean":"2024-03-26T05:16:31.644187+0000","last_became_active":"2024-03-26T05:15:32.640871+0000","last_became_peered":"2024-03-26T05:15:32.640871+0000","last_unstale":"2024-03-26T05:16:31.644187+0000","last_undegraded":"2024-03-26T05:16:31.644187+0000","last_fullsized":"2024-03-26T05:16:31.644187+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'18","last_scrub_stamp":"2024-03-26T05:15:46.643068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:15:46.643068+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:12.872459+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"357'7717","reported_seq":7812,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:35.389044+0000","last_change":"2024-03-26T05:15:48.647232+0000","last_active":"2024-03-26T05:16:35.389044+0000","last_peered":"2024-03-26T05:16:35.389044+0000","last_clean":"2024-03-26T05:15:47.585964+0000","last_became_active":"2024-03-26T05:15:48.647232+0000","last_became_peered":"2024-03-26T05:15:48.647232+0000","last_unstale":"2024-03-26T05:16:35.389044+0000","last_undegraded":"2024-03-26T05:15:48.629455+0000","last_fullsized":"2024-03-26T05:15:48.629339+0000","mapping_epoch":342,"log_start":"321'2400","ondisk_log_start":"321'2400","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5317,"log_dups_size":0,"ondisk_log_size":5317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394067968,"num_objects":6013,"num_object_clones":0,"num_object_copies":18039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":439360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":6013}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":327,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.632232+0000","last_change":"2024-03-26T05:14:54.651241+0000","last_active":"2024-03-26T05:16:31.632232+0000","last_peered":"2024-03-26T05:16:31.632232+0000","last_clean":"2024-03-26T05:16:31.632232+0000","last_became_active":"2024-03-26T05:08:00.000166+0000","last_became_peered":"2024-03-26T05:08:00.000166+0000","last_unstale":"2024-03-26T05:16:31.632232+0000","last_undegraded":"2024-03-26T05:16:31.632232+0000","last_fullsized":"2024-03-26T05:16:31.632232+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:14:54.651113+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:14:54.651113+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":553,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.625735+0000","last_change":"2024-03-26T05:15:48.651187+0000","last_active":"2024-03-26T05:16:31.625735+0000","last_peered":"2024-03-26T05:16:31.625735+0000","last_clean":"2024-03-26T05:15:46.625050+0000","last_became_active":"2024-03-26T05:15:48.651187+0000","last_became_peered":"2024-03-26T05:15:48.651187+0000","last_unstale":"2024-03-26T05:16:31.625735+0000","last_undegraded":"2024-03-26T05:15:48.633222+0000","last_fullsized":"2024-03-26T05:15:48.633108+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.959120+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"254'48","reported_seq":371,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.632658+0000","last_change":"2024-03-26T05:15:32.620944+0000","last_active":"2024-03-26T05:16:31.632658+0000","last_peered":"2024-03-26T05:16:31.632658+0000","last_clean":"2024-03-26T05:16:31.632658+0000","last_became_active":"2024-03-26T05:15:32.620512+0000","last_became_peered":"2024-03-26T05:15:32.620512+0000","last_unstale":"2024-03-26T05:16:31.632658+0000","last_undegraded":"2024-03-26T05:16:31.632658+0000","last_fullsized":"2024-03-26T05:16:31.632658+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'48","last_scrub_stamp":"2024-03-26T05:12:01.634109+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:12:01.634109+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"343'11","reported_seq":712,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.628867+0000","last_change":"2024-03-26T05:15:48.645351+0000","last_active":"2024-03-26T05:16:31.628867+0000","last_peered":"2024-03-26T05:16:31.628867+0000","last_clean":"2024-03-26T05:15:46.624511+0000","last_became_active":"2024-03-26T05:15:48.645351+0000","last_became_peered":"2024-03-26T05:15:48.645351+0000","last_unstale":"2024-03-26T05:16:31.628867+0000","last_undegraded":"2024-03-26T05:15:48.635536+0000","last_fullsized":"2024-03-26T05:15:48.635413+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:12:03.961391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:03.961391+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"357'7857","reported_seq":8011,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:34.755157+0000","last_change":"2024-03-26T05:15:57.473599+0000","last_active":"2024-03-26T05:16:34.755157+0000","last_peered":"2024-03-26T05:16:34.755157+0000","last_clean":"2024-03-26T05:16:34.755157+0000","last_became_active":"2024-03-26T05:14:41.670908+0000","last_became_peered":"2024-03-26T05:14:41.670908+0000","last_unstale":"2024-03-26T05:16:34.755157+0000","last_undegraded":"2024-03-26T05:16:34.755157+0000","last_fullsized":"2024-03-26T05:16:34.755157+0000","mapping_epoch":316,"log_start":"321'2300","ondisk_log_start":"321'2300","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'5611","last_scrub_stamp":"2024-03-26T05:15:57.473473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:15:57.473473+0000","objects_scrubbed":5604,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:21.352453+0000","scrub_duration":994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404291584,"num_objects":6169,"num_object_clones":0,"num_object_copies":18507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":448832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":639,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.651790+0000","last_change":"2024-03-26T05:15:48.646428+0000","last_active":"2024-03-26T05:16:31.651790+0000","last_peered":"2024-03-26T05:16:31.651790+0000","last_clean":"2024-03-26T05:15:46.626191+0000","last_became_active":"2024-03-26T05:15:48.646428+0000","last_became_peered":"2024-03-26T05:15:48.646428+0000","last_unstale":"2024-03-26T05:16:31.651790+0000","last_undegraded":"2024-03-26T05:16:31.651790+0000","last_fullsized":"2024-03-26T05:15:48.631162+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:51.030982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:51.030982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":564,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.625365+0000","last_change":"2024-03-26T05:15:48.650267+0000","last_active":"2024-03-26T05:16:31.625365+0000","last_peered":"2024-03-26T05:16:31.625365+0000","last_clean":"2024-03-26T05:15:46.624664+0000","last_became_active":"2024-03-26T05:15:48.650267+0000","last_became_peered":"2024-03-26T05:15:48.650267+0000","last_unstale":"2024-03-26T05:16:31.625365+0000","last_undegraded":"2024-03-26T05:16:31.625365+0000","last_fullsized":"2024-03-26T05:15:48.633889+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:20.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:20.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"254'37","reported_seq":513,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.646399+0000","last_change":"2024-03-26T05:15:48.645453+0000","last_active":"2024-03-26T05:16:31.646399+0000","last_peered":"2024-03-26T05:16:31.646399+0000","last_clean":"2024-03-26T05:10:30.473392+0000","last_became_active":"2024-03-26T05:15:48.645453+0000","last_became_peered":"2024-03-26T05:15:48.645453+0000","last_unstale":"2024-03-26T05:16:31.646399+0000","last_undegraded":"2024-03-26T05:15:48.634195+0000","last_fullsized":"2024-03-26T05:15:48.634068+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-26T05:14:55.963773+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_clean_scrub_stamp":"2024-03-26T05:14:55.963773+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":642,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.653368+0000","last_change":"2024-03-26T05:14:01.337189+0000","last_active":"2024-03-26T05:16:31.653368+0000","last_peered":"2024-03-26T05:16:31.653368+0000","last_clean":"2024-03-26T05:16:31.653368+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:16:31.653368+0000","last_undegraded":"2024-03-26T05:16:31.653368+0000","last_fullsized":"2024-03-26T05:16:31.653368+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:01.336959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.432732+0000","last_clean_scrub_stamp":"2024-03-26T05:14:01.336959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"357'7818","reported_seq":7912,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:36.391674+0000","last_change":"2024-03-26T05:15:48.654809+0000","last_active":"2024-03-26T05:16:36.391674+0000","last_peered":"2024-03-26T05:16:36.391674+0000","last_clean":"2024-03-26T05:15:47.586253+0000","last_became_active":"2024-03-26T05:15:48.654809+0000","last_became_peered":"2024-03-26T05:15:48.654809+0000","last_unstale":"2024-03-26T05:16:36.391674+0000","last_undegraded":"2024-03-26T05:15:48.642769+0000","last_fullsized":"2024-03-26T05:15:48.642069+0000","mapping_epoch":342,"log_start":"322'2800","ondisk_log_start":"322'2800","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389939200,"num_objects":5950,"num_object_clones":0,"num_object_copies":17850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7818,"num_write_kb":440576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":5950}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":542,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.629867+0000","last_change":"2024-03-26T05:15:32.638404+0000","last_active":"2024-03-26T05:16:31.629867+0000","last_peered":"2024-03-26T05:16:31.629867+0000","last_clean":"2024-03-26T05:16:31.629867+0000","last_became_active":"2024-03-26T05:15:32.633481+0000","last_became_peered":"2024-03-26T05:15:32.633481+0000","last_unstale":"2024-03-26T05:16:31.629867+0000","last_undegraded":"2024-03-26T05:16:31.629867+0000","last_fullsized":"2024-03-26T05:16:31.629867+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:05.986499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_clean_scrub_stamp":"2024-03-26T05:12:05.986499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":662,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.653068+0000","last_change":"2024-03-26T05:14:10.201428+0000","last_active":"2024-03-26T05:16:31.653068+0000","last_peered":"2024-03-26T05:16:31.653068+0000","last_clean":"2024-03-26T05:16:31.653068+0000","last_became_active":"2024-03-26T05:08:00.000009+0000","last_became_peered":"2024-03-26T05:08:00.000009+0000","last_unstale":"2024-03-26T05:16:31.653068+0000","last_undegraded":"2024-03-26T05:16:31.653068+0000","last_fullsized":"2024-03-26T05:16:31.653068+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:10.201320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:10.201320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":531,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.625587+0000","last_change":"2024-03-26T05:15:32.641510+0000","last_active":"2024-03-26T05:16:31.625587+0000","last_peered":"2024-03-26T05:16:31.625587+0000","last_clean":"2024-03-26T05:16:31.625587+0000","last_became_active":"2024-03-26T05:15:32.640861+0000","last_became_peered":"2024-03-26T05:15:32.640861+0000","last_unstale":"2024-03-26T05:16:31.625587+0000","last_undegraded":"2024-03-26T05:16:31.625587+0000","last_fullsized":"2024-03-26T05:16:31.625587+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:27.963495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:27.963495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"343'51","reported_seq":731,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642171+0000","last_change":"2024-03-26T05:15:49.436480+0000","last_active":"2024-03-26T05:16:31.642171+0000","last_peered":"2024-03-26T05:16:31.642171+0000","last_clean":"2024-03-26T05:16:31.642171+0000","last_became_active":"2024-03-26T05:15:32.626640+0000","last_became_peered":"2024-03-26T05:15:32.626640+0000","last_unstale":"2024-03-26T05:16:31.642171+0000","last_undegraded":"2024-03-26T05:16:31.642171+0000","last_fullsized":"2024-03-26T05:16:31.642171+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'51","last_scrub_stamp":"2024-03-26T05:15:49.436355+0000","last_deep_scrub":"343'51","last_deep_scrub_stamp":"2024-03-26T05:15:49.436355+0000","last_clean_scrub_stamp":"2024-03-26T05:15:49.436355+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:54.873891+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"357'7885","reported_seq":7977,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:35.418265+0000","last_change":"2024-03-26T05:14:41.674537+0000","last_active":"2024-03-26T05:16:35.418265+0000","last_peered":"2024-03-26T05:16:35.418265+0000","last_clean":"2024-03-26T05:16:35.418265+0000","last_became_active":"2024-03-26T05:14:41.674204+0000","last_became_peered":"2024-03-26T05:14:41.674204+0000","last_unstale":"2024-03-26T05:16:35.418265+0000","last_undegraded":"2024-03-26T05:16:35.418265+0000","last_fullsized":"2024-03-26T05:16:35.418265+0000","mapping_epoch":316,"log_start":"321'2000","ondisk_log_start":"321'2000","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5885,"log_dups_size":0,"ondisk_log_size":5885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401276928,"num_objects":6123,"num_object_clones":0,"num_object_copies":18369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7885,"num_write_kb":448256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":633,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.636785+0000","last_change":"2024-03-26T05:15:32.640388+0000","last_active":"2024-03-26T05:16:31.636785+0000","last_peered":"2024-03-26T05:16:31.636785+0000","last_clean":"2024-03-26T05:16:31.636785+0000","last_became_active":"2024-03-26T05:15:32.639943+0000","last_became_peered":"2024-03-26T05:15:32.639943+0000","last_unstale":"2024-03-26T05:16:31.636785+0000","last_undegraded":"2024-03-26T05:16:31.636785+0000","last_fullsized":"2024-03-26T05:16:31.636785+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:30.164093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:30.164093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":653,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.648342+0000","last_change":"2024-03-26T05:15:48.638309+0000","last_active":"2024-03-26T05:16:31.648342+0000","last_peered":"2024-03-26T05:16:31.648342+0000","last_clean":"2024-03-26T05:15:46.625678+0000","last_became_active":"2024-03-26T05:15:48.638309+0000","last_became_peered":"2024-03-26T05:15:48.638309+0000","last_unstale":"2024-03-26T05:16:31.648342+0000","last_undegraded":"2024-03-26T05:16:31.648342+0000","last_fullsized":"2024-03-26T05:15:48.631169+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:52.371799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:52.371799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":323,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.632042+0000","last_change":"2024-03-26T05:14:00.685481+0000","last_active":"2024-03-26T05:16:31.632042+0000","last_peered":"2024-03-26T05:16:31.632042+0000","last_clean":"2024-03-26T05:16:31.632042+0000","last_became_active":"2024-03-26T05:07:59.959978+0000","last_became_peered":"2024-03-26T05:07:59.959978+0000","last_unstale":"2024-03-26T05:16:31.632042+0000","last_undegraded":"2024-03-26T05:16:31.632042+0000","last_fullsized":"2024-03-26T05:16:31.632042+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:00.685364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:00.685364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"254'21","reported_seq":688,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.640797+0000","last_change":"2024-03-26T05:15:48.655699+0000","last_active":"2024-03-26T05:16:31.640797+0000","last_peered":"2024-03-26T05:16:31.640797+0000","last_clean":"2024-03-26T05:15:46.621863+0000","last_became_active":"2024-03-26T05:15:48.655699+0000","last_became_peered":"2024-03-26T05:15:48.655699+0000","last_unstale":"2024-03-26T05:16:31.640797+0000","last_undegraded":"2024-03-26T05:15:48.645108+0000","last_fullsized":"2024-03-26T05:15:48.644023+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:14:16.094249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.094249+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"88'128","reported_seq":446,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.630763+0000","last_change":"2024-03-26T05:15:22.621657+0000","last_active":"2024-03-26T05:16:31.630763+0000","last_peered":"2024-03-26T05:16:31.630763+0000","last_clean":"2024-03-26T05:16:31.630763+0000","last_became_active":"2024-03-26T05:15:22.621431+0000","last_became_peered":"2024-03-26T05:15:22.621431+0000","last_unstale":"2024-03-26T05:16:31.630763+0000","last_undegraded":"2024-03-26T05:16:31.630763+0000","last_fullsized":"2024-03-26T05:16:31.630763+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:34.396777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"357'7999","reported_seq":8090,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:34.756301+0000","last_change":"2024-03-26T05:14:41.671380+0000","last_active":"2024-03-26T05:16:34.756301+0000","last_peered":"2024-03-26T05:16:34.756301+0000","last_clean":"2024-03-26T05:16:34.756301+0000","last_became_active":"2024-03-26T05:14:41.671031+0000","last_became_peered":"2024-03-26T05:14:41.671031+0000","last_unstale":"2024-03-26T05:16:34.756301+0000","last_undegraded":"2024-03-26T05:16:34.756301+0000","last_fullsized":"2024-03-26T05:16:34.756301+0000","mapping_epoch":316,"log_start":"321'2400","ondisk_log_start":"321'2400","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409141248,"num_objects":6243,"num_object_clones":0,"num_object_copies":18729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7999,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":626,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.643077+0000","last_change":"2024-03-26T05:15:48.637195+0000","last_active":"2024-03-26T05:16:31.643077+0000","last_peered":"2024-03-26T05:16:31.643077+0000","last_clean":"2024-03-26T05:15:46.628089+0000","last_became_active":"2024-03-26T05:15:48.637195+0000","last_became_peered":"2024-03-26T05:15:48.637195+0000","last_unstale":"2024-03-26T05:16:31.643077+0000","last_undegraded":"2024-03-26T05:16:31.643077+0000","last_fullsized":"2024-03-26T05:15:48.630113+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:31.518660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:31.518660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:35.867333+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":328,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.631659+0000","last_change":"2024-03-26T05:14:47.525396+0000","last_active":"2024-03-26T05:16:31.631659+0000","last_peered":"2024-03-26T05:16:31.631659+0000","last_clean":"2024-03-26T05:16:31.631659+0000","last_became_active":"2024-03-26T05:07:59.998511+0000","last_became_peered":"2024-03-26T05:07:59.998511+0000","last_unstale":"2024-03-26T05:16:31.631659+0000","last_undegraded":"2024-03-26T05:16:31.631659+0000","last_fullsized":"2024-03-26T05:16:31.631659+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:14:47.525284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:14:47.525284+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":37,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.640285+0000","last_change":"2024-03-26T05:15:48.646887+0000","last_active":"2024-03-26T05:16:31.640285+0000","last_peered":"2024-03-26T05:16:31.640285+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:15:48.646887+0000","last_became_peered":"2024-03-26T05:15:48.646887+0000","last_unstale":"2024-03-26T05:16:31.640285+0000","last_undegraded":"2024-03-26T05:16:31.640285+0000","last_fullsized":"2024-03-26T05:15:48.641087+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:02.881800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:02.881800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"254'46","reported_seq":930,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.647160+0000","last_change":"2024-03-26T05:15:52.070480+0000","last_active":"2024-03-26T05:16:31.647160+0000","last_peered":"2024-03-26T05:16:31.647160+0000","last_clean":"2024-03-26T05:16:31.647160+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:16:31.647160+0000","last_undegraded":"2024-03-26T05:16:31.647160+0000","last_fullsized":"2024-03-26T05:16:31.647160+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'46","last_scrub_stamp":"2024-03-26T05:15:52.070295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:15:52.070295+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:15.255095+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":647,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.638662+0000","last_change":"2024-03-26T05:15:48.645632+0000","last_active":"2024-03-26T05:16:31.638662+0000","last_peered":"2024-03-26T05:16:31.638662+0000","last_clean":"2024-03-26T05:15:46.622114+0000","last_became_active":"2024-03-26T05:15:48.645632+0000","last_became_peered":"2024-03-26T05:15:48.645632+0000","last_unstale":"2024-03-26T05:16:31.638662+0000","last_undegraded":"2024-03-26T05:16:31.638662+0000","last_fullsized":"2024-03-26T05:15:48.638911+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:14.131427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:14.131427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"88'128","reported_seq":818,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.624588+0000","last_change":"2024-03-26T05:15:48.652567+0000","last_active":"2024-03-26T05:16:31.624588+0000","last_peered":"2024-03-26T05:16:31.624588+0000","last_clean":"2024-03-26T05:15:46.623182+0000","last_became_active":"2024-03-26T05:15:48.652567+0000","last_became_peered":"2024-03-26T05:15:48.652567+0000","last_unstale":"2024-03-26T05:16:31.624588+0000","last_undegraded":"2024-03-26T05:15:48.634567+0000","last_fullsized":"2024-03-26T05:15:48.634420+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:36.319420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"357'8031","reported_seq":15984,"reported_epoch":357,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T05:16:35.387156+0000","last_change":"2024-03-26T05:15:33.840678+0000","last_active":"2024-03-26T05:16:35.387156+0000","last_peered":"2024-03-26T05:16:35.387156+0000","last_clean":"2024-03-26T05:14:40.645004+0000","last_became_active":"2024-03-26T05:15:33.779613+0000","last_became_peered":"2024-03-26T05:15:33.779613+0000","last_unstale":"2024-03-26T05:16:35.387156+0000","last_undegraded":"2024-03-26T05:15:33.651478+0000","last_fullsized":"2024-03-26T05:15:33.615358+0000","mapping_epoch":336,"log_start":"321'2700","ondisk_log_start":"321'2700","created":316,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406913024,"num_objects":6209,"num_object_clones":0,"num_object_copies":18627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":455680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3930,"num_bytes_recovered":257556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5332}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":510,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.625087+0000","last_change":"2024-03-26T05:15:32.638631+0000","last_active":"2024-03-26T05:16:31.625087+0000","last_peered":"2024-03-26T05:16:31.625087+0000","last_clean":"2024-03-26T05:16:31.625087+0000","last_became_active":"2024-03-26T05:15:32.633633+0000","last_became_peered":"2024-03-26T05:15:32.633633+0000","last_unstale":"2024-03-26T05:16:31.625087+0000","last_undegraded":"2024-03-26T05:16:31.625087+0000","last_fullsized":"2024-03-26T05:16:31.625087+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:20.054579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:20.054579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":651,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.651499+0000","last_change":"2024-03-26T05:15:48.007235+0000","last_active":"2024-03-26T05:16:31.651499+0000","last_peered":"2024-03-26T05:16:31.651499+0000","last_clean":"2024-03-26T05:16:31.651499+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:16:31.651499+0000","last_undegraded":"2024-03-26T05:16:31.651499+0000","last_fullsized":"2024-03-26T05:16:31.651499+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:48.007148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:15:48.007148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:52.349065+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":667,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.649296+0000","last_change":"2024-03-26T05:15:00.402058+0000","last_active":"2024-03-26T05:16:31.649296+0000","last_peered":"2024-03-26T05:16:31.649296+0000","last_clean":"2024-03-26T05:16:31.649296+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:16:31.649296+0000","last_undegraded":"2024-03-26T05:16:31.649296+0000","last_fullsized":"2024-03-26T05:16:31.649296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:00.401943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.401943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"254'23","reported_seq":464,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.651797+0000","last_change":"2024-03-26T05:14:55.315443+0000","last_active":"2024-03-26T05:16:31.651797+0000","last_peered":"2024-03-26T05:16:31.651797+0000","last_clean":"2024-03-26T05:16:31.651797+0000","last_became_active":"2024-03-26T05:08:00.001398+0000","last_became_peered":"2024-03-26T05:08:00.001398+0000","last_unstale":"2024-03-26T05:16:31.651797+0000","last_undegraded":"2024-03-26T05:16:31.651797+0000","last_fullsized":"2024-03-26T05:16:31.651797+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'23","last_scrub_stamp":"2024-03-26T05:14:55.315296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:55.315296+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":663,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.653076+0000","last_change":"2024-03-26T05:15:32.625481+0000","last_active":"2024-03-26T05:16:31.653076+0000","last_peered":"2024-03-26T05:16:31.653076+0000","last_clean":"2024-03-26T05:16:31.653076+0000","last_became_active":"2024-03-26T05:15:32.625188+0000","last_became_peered":"2024-03-26T05:15:32.625188+0000","last_unstale":"2024-03-26T05:16:31.653076+0000","last_undegraded":"2024-03-26T05:16:31.653076+0000","last_fullsized":"2024-03-26T05:16:31.653076+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:07.171121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.171121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"357'7796","reported_seq":7951,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:34.724032+0000","last_change":"2024-03-26T05:15:56.584965+0000","last_active":"2024-03-26T05:16:34.724032+0000","last_peered":"2024-03-26T05:16:34.724032+0000","last_clean":"2024-03-26T05:16:34.724032+0000","last_became_active":"2024-03-26T05:14:41.674748+0000","last_became_peered":"2024-03-26T05:14:41.674748+0000","last_unstale":"2024-03-26T05:16:34.724032+0000","last_undegraded":"2024-03-26T05:16:34.724032+0000","last_fullsized":"2024-03-26T05:16:34.724032+0000","mapping_epoch":316,"log_start":"321'2200","ondisk_log_start":"321'2200","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'5516","last_scrub_stamp":"2024-03-26T05:15:56.584807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:15:56.584807+0000","objects_scrubbed":5516,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:12.034118+0000","scrub_duration":1184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394133504,"num_objects":6014,"num_object_clones":0,"num_object_copies":18042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":441920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":334,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.632987+0000","last_change":"2024-03-26T05:15:48.643476+0000","last_active":"2024-03-26T05:16:31.632987+0000","last_peered":"2024-03-26T05:16:31.632987+0000","last_clean":"2024-03-26T05:15:46.630583+0000","last_became_active":"2024-03-26T05:15:48.643476+0000","last_became_peered":"2024-03-26T05:15:48.643476+0000","last_unstale":"2024-03-26T05:16:31.632987+0000","last_undegraded":"2024-03-26T05:16:31.632987+0000","last_fullsized":"2024-03-26T05:15:48.632538+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:14.528992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:14.528992+0000","last_clean_scrub_stamp":"2024-03-26T05:15:14.528992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:41.492962+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":817,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.630223+0000","last_change":"2024-03-26T05:15:32.635542+0000","last_active":"2024-03-26T05:16:31.630223+0000","last_peered":"2024-03-26T05:16:31.630223+0000","last_clean":"2024-03-26T05:16:31.630223+0000","last_became_active":"2024-03-26T05:15:32.632929+0000","last_became_peered":"2024-03-26T05:15:32.632929+0000","last_unstale":"2024-03-26T05:16:31.630223+0000","last_undegraded":"2024-03-26T05:16:31.630223+0000","last_fullsized":"2024-03-26T05:16:31.630223+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:36.319400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":662,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642203+0000","last_change":"2024-03-26T05:16:05.780331+0000","last_active":"2024-03-26T05:16:31.642203+0000","last_peered":"2024-03-26T05:16:31.642203+0000","last_clean":"2024-03-26T05:16:31.642203+0000","last_became_active":"2024-03-26T05:07:59.960061+0000","last_became_peered":"2024-03-26T05:07:59.960061+0000","last_unstale":"2024-03-26T05:16:31.642203+0000","last_undegraded":"2024-03-26T05:16:31.642203+0000","last_fullsized":"2024-03-26T05:16:31.642203+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:05.780217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:16:05.780217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:25.544931+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"343'39","reported_seq":911,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.644493+0000","last_change":"2024-03-26T05:15:48.640970+0000","last_active":"2024-03-26T05:16:31.644493+0000","last_peered":"2024-03-26T05:16:31.644493+0000","last_clean":"2024-03-26T05:15:46.625517+0000","last_became_active":"2024-03-26T05:15:48.640970+0000","last_became_peered":"2024-03-26T05:15:48.640970+0000","last_unstale":"2024-03-26T05:16:31.644493+0000","last_undegraded":"2024-03-26T05:15:48.633766+0000","last_fullsized":"2024-03-26T05:15:48.633639+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'38","last_scrub_stamp":"2024-03-26T05:15:46.434771+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:15:46.434771+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:02.187525+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":570,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.625136+0000","last_change":"2024-03-26T05:15:32.638848+0000","last_active":"2024-03-26T05:16:31.625136+0000","last_peered":"2024-03-26T05:16:31.625136+0000","last_clean":"2024-03-26T05:16:31.625136+0000","last_became_active":"2024-03-26T05:15:32.637320+0000","last_became_peered":"2024-03-26T05:15:32.637320+0000","last_unstale":"2024-03-26T05:16:31.625136+0000","last_undegraded":"2024-03-26T05:16:31.625136+0000","last_fullsized":"2024-03-26T05:16:31.625136+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:14.948209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:14.948209+0000","last_clean_scrub_stamp":"2024-03-26T05:12:14.948209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"88'128","reported_seq":458,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.648193+0000","last_change":"2024-03-26T05:15:48.647616+0000","last_active":"2024-03-26T05:16:31.648193+0000","last_peered":"2024-03-26T05:16:31.648193+0000","last_clean":"2024-03-26T05:15:46.610234+0000","last_became_active":"2024-03-26T05:15:48.647616+0000","last_became_peered":"2024-03-26T05:15:48.647616+0000","last_unstale":"2024-03-26T05:16:31.648193+0000","last_undegraded":"2024-03-26T05:16:31.648193+0000","last_fullsized":"2024-03-26T05:15:48.634886+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:15:44.054471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:15:44.054471+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:07.469497+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"357'7878","reported_seq":11848,"reported_epoch":357,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T05:16:36.382273+0000","last_change":"2024-03-26T05:15:56.101290+0000","last_active":"2024-03-26T05:16:36.382273+0000","last_peered":"2024-03-26T05:16:36.382273+0000","last_clean":"2024-03-26T05:14:40.645004+0000","last_became_active":"2024-03-26T05:15:33.767160+0000","last_became_peered":"2024-03-26T05:15:33.767160+0000","last_unstale":"2024-03-26T05:16:36.382273+0000","last_undegraded":"2024-03-26T05:15:33.637259+0000","last_fullsized":"2024-03-26T05:15:33.610587+0000","mapping_epoch":336,"log_start":"322'2800","ondisk_log_start":"322'2800","created":316,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391249920,"num_objects":5970,"num_object_clones":0,"num_object_copies":17910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2737,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7878,"num_write_kb":443136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1937,"num_bytes_recovered":126943232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3233}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":658,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.646804+0000","last_change":"2024-03-26T05:15:47.053940+0000","last_active":"2024-03-26T05:16:31.646804+0000","last_peered":"2024-03-26T05:16:31.646804+0000","last_clean":"2024-03-26T05:16:31.646804+0000","last_became_active":"2024-03-26T05:08:00.001657+0000","last_became_peered":"2024-03-26T05:08:00.001657+0000","last_unstale":"2024-03-26T05:16:31.646804+0000","last_undegraded":"2024-03-26T05:16:31.646804+0000","last_fullsized":"2024-03-26T05:16:31.646804+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:47.053815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:47.053815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:14.798534+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":370,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.636388+0000","last_change":"2024-03-26T05:15:48.659891+0000","last_active":"2024-03-26T05:16:31.636388+0000","last_peered":"2024-03-26T05:16:31.636388+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:15:48.659891+0000","last_became_peered":"2024-03-26T05:15:48.659891+0000","last_unstale":"2024-03-26T05:16:31.636388+0000","last_undegraded":"2024-03-26T05:16:31.636388+0000","last_fullsized":"2024-03-26T05:15:48.638273+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:16.863389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.863389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":1323,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:33.860927+0000","last_change":"2024-03-26T05:15:32.637254+0000","last_active":"2024-03-26T05:16:33.860927+0000","last_peered":"2024-03-26T05:16:33.860927+0000","last_clean":"2024-03-26T05:16:33.860927+0000","last_became_active":"2024-03-26T05:15:32.633061+0000","last_became_peered":"2024-03-26T05:15:32.633061+0000","last_unstale":"2024-03-26T05:16:33.860927+0000","last_undegraded":"2024-03-26T05:16:33.860927+0000","last_fullsized":"2024-03-26T05:16:33.860927+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:12:02.961845+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:12:02.961845+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":128,"num_read_kb":123,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"254'30","reported_seq":730,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.635785+0000","last_change":"2024-03-26T05:14:10.147920+0000","last_active":"2024-03-26T05:16:31.635785+0000","last_peered":"2024-03-26T05:16:31.635785+0000","last_clean":"2024-03-26T05:16:31.635785+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:16:31.635785+0000","last_undegraded":"2024-03-26T05:16:31.635785+0000","last_fullsized":"2024-03-26T05:16:31.635785+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'30","last_scrub_stamp":"2024-03-26T05:14:10.147766+0000","last_deep_scrub":"187'18","last_deep_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_clean_scrub_stamp":"2024-03-26T05:14:10.147766+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":323,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.632802+0000","last_change":"2024-03-26T05:14:12.823709+0000","last_active":"2024-03-26T05:16:31.632802+0000","last_peered":"2024-03-26T05:16:31.632802+0000","last_clean":"2024-03-26T05:16:31.632802+0000","last_became_active":"2024-03-26T05:07:59.999990+0000","last_became_peered":"2024-03-26T05:07:59.999990+0000","last_unstale":"2024-03-26T05:16:31.632802+0000","last_undegraded":"2024-03-26T05:16:31.632802+0000","last_fullsized":"2024-03-26T05:16:31.632802+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:12.823589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:12.823589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"88'128","reported_seq":386,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.643905+0000","last_change":"2024-03-26T05:15:39.548464+0000","last_active":"2024-03-26T05:16:31.643905+0000","last_peered":"2024-03-26T05:16:31.643905+0000","last_clean":"2024-03-26T05:16:31.643905+0000","last_became_active":"2024-03-26T05:15:22.624234+0000","last_became_peered":"2024-03-26T05:15:22.624234+0000","last_unstale":"2024-03-26T05:16:31.643905+0000","last_undegraded":"2024-03-26T05:16:31.643905+0000","last_fullsized":"2024-03-26T05:16:31.643905+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:15:39.548321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:15:39.548321+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:44.213018+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"357'7957","reported_seq":8051,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:34.756404+0000","last_change":"2024-03-26T05:15:48.643432+0000","last_active":"2024-03-26T05:16:34.756404+0000","last_peered":"2024-03-26T05:16:34.756404+0000","last_clean":"2024-03-26T05:15:47.616159+0000","last_became_active":"2024-03-26T05:15:48.643432+0000","last_became_peered":"2024-03-26T05:15:48.643432+0000","last_unstale":"2024-03-26T05:16:34.756404+0000","last_undegraded":"2024-03-26T05:15:48.633364+0000","last_fullsized":"2024-03-26T05:15:48.633251+0000","mapping_epoch":342,"log_start":"321'2400","ondisk_log_start":"321'2400","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407502872,"num_objects":6219,"num_object_clones":0,"num_object_copies":18657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7957,"num_write_kb":453569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":6219}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":638,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.645738+0000","last_change":"2024-03-26T05:15:51.091029+0000","last_active":"2024-03-26T05:16:31.645738+0000","last_peered":"2024-03-26T05:16:31.645738+0000","last_clean":"2024-03-26T05:16:31.645738+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:16:31.645738+0000","last_undegraded":"2024-03-26T05:16:31.645738+0000","last_fullsized":"2024-03-26T05:16:31.645738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:51.090933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:15:51.090933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:00.889185+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":670,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.647029+0000","last_change":"2024-03-26T05:16:06.271601+0000","last_active":"2024-03-26T05:16:31.647029+0000","last_peered":"2024-03-26T05:16:31.647029+0000","last_clean":"2024-03-26T05:16:31.647029+0000","last_became_active":"2024-03-26T05:08:00.003099+0000","last_became_peered":"2024-03-26T05:08:00.003099+0000","last_unstale":"2024-03-26T05:16:31.647029+0000","last_undegraded":"2024-03-26T05:16:31.647029+0000","last_fullsized":"2024-03-26T05:16:31.647029+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:06.271319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:11:29.514915+0000","last_clean_scrub_stamp":"2024-03-26T05:16:06.271319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:09.592650+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"254'44","reported_seq":773,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.640471+0000","last_change":"2024-03-26T05:15:32.634333+0000","last_active":"2024-03-26T05:16:31.640471+0000","last_peered":"2024-03-26T05:16:31.640471+0000","last_clean":"2024-03-26T05:16:31.640471+0000","last_became_active":"2024-03-26T05:15:32.634000+0000","last_became_peered":"2024-03-26T05:15:32.634000+0000","last_unstale":"2024-03-26T05:16:31.640471+0000","last_undegraded":"2024-03-26T05:16:31.640471+0000","last_fullsized":"2024-03-26T05:16:31.640471+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'44","last_scrub_stamp":"2024-03-26T05:11:29.126183+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.126183+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":669,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642289+0000","last_change":"2024-03-26T05:15:11.334177+0000","last_active":"2024-03-26T05:16:31.642289+0000","last_peered":"2024-03-26T05:16:31.642289+0000","last_clean":"2024-03-26T05:16:31.642289+0000","last_became_active":"2024-03-26T05:07:59.991484+0000","last_became_peered":"2024-03-26T05:07:59.991484+0000","last_unstale":"2024-03-26T05:16:31.642289+0000","last_undegraded":"2024-03-26T05:16:31.642289+0000","last_fullsized":"2024-03-26T05:16:31.642289+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:11.334011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:15:11.334011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"88'128","reported_seq":392,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.644598+0000","last_change":"2024-03-26T05:15:40.586982+0000","last_active":"2024-03-26T05:16:31.644598+0000","last_peered":"2024-03-26T05:16:31.644598+0000","last_clean":"2024-03-26T05:16:31.644598+0000","last_became_active":"2024-03-26T05:15:33.617538+0000","last_became_peered":"2024-03-26T05:15:33.617538+0000","last_unstale":"2024-03-26T05:16:31.644598+0000","last_undegraded":"2024-03-26T05:16:31.644598+0000","last_fullsized":"2024-03-26T05:16:31.644598+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:15:40.586852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:15:40.586852+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:48.179714+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"357'8036","reported_seq":19500,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:35.745158+0000","last_change":"2024-03-26T05:16:01.519823+0000","last_active":"2024-03-26T05:16:35.745158+0000","last_peered":"2024-03-26T05:16:35.745158+0000","last_clean":"2024-03-26T05:14:40.645004+0000","last_became_active":"2024-03-26T05:16:01.519823+0000","last_became_peered":"2024-03-26T05:16:01.519823+0000","last_unstale":"2024-03-26T05:16:35.745158+0000","last_undegraded":"2024-03-26T05:16:01.505259+0000","last_fullsized":"2024-03-26T05:16:01.505132+0000","mapping_epoch":347,"log_start":"322'3000","ondisk_log_start":"322'3000","created":316,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403308544,"num_objects":6154,"num_object_clones":0,"num_object_copies":18462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8036,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5683,"num_bytes_recovered":372441088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":6154}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":641,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.639811+0000","last_change":"2024-03-26T05:14:07.165543+0000","last_active":"2024-03-26T05:16:31.639811+0000","last_peered":"2024-03-26T05:16:31.639811+0000","last_clean":"2024-03-26T05:16:31.639811+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:16:31.639811+0000","last_undegraded":"2024-03-26T05:16:31.639811+0000","last_fullsized":"2024-03-26T05:16:31.639811+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:07.165430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:07.165430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":324,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.631452+0000","last_change":"2024-03-26T05:14:15.770146+0000","last_active":"2024-03-26T05:16:31.631452+0000","last_peered":"2024-03-26T05:16:31.631452+0000","last_clean":"2024-03-26T05:16:31.631452+0000","last_became_active":"2024-03-26T05:07:59.960768+0000","last_became_peered":"2024-03-26T05:07:59.960768+0000","last_unstale":"2024-03-26T05:16:31.631452+0000","last_undegraded":"2024-03-26T05:16:31.631452+0000","last_fullsized":"2024-03-26T05:16:31.631452+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:15.770027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:15.770027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"254'26","reported_seq":702,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.648677+0000","last_change":"2024-03-26T05:15:48.638710+0000","last_active":"2024-03-26T05:16:31.648677+0000","last_peered":"2024-03-26T05:16:31.648677+0000","last_clean":"2024-03-26T05:15:46.626893+0000","last_became_active":"2024-03-26T05:15:48.638710+0000","last_became_peered":"2024-03-26T05:15:48.638710+0000","last_unstale":"2024-03-26T05:16:31.648677+0000","last_undegraded":"2024-03-26T05:15:48.631213+0000","last_fullsized":"2024-03-26T05:15:48.631104+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:14:05.219767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:05.219767+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"216'11","reported_seq":877,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.641071+0000","last_change":"2024-03-26T05:14:48.976961+0000","last_active":"2024-03-26T05:16:31.641071+0000","last_peered":"2024-03-26T05:16:31.641071+0000","last_clean":"2024-03-26T05:16:31.641071+0000","last_became_active":"2024-03-26T05:07:59.994682+0000","last_became_peered":"2024-03-26T05:07:59.994682+0000","last_unstale":"2024-03-26T05:16:31.641071+0000","last_undegraded":"2024-03-26T05:16:31.641071+0000","last_fullsized":"2024-03-26T05:16:31.641071+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:14:48.976873+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:14:48.976873+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"88'128","reported_seq":460,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642158+0000","last_change":"2024-03-26T05:15:39.542943+0000","last_active":"2024-03-26T05:16:31.642158+0000","last_peered":"2024-03-26T05:16:31.642158+0000","last_clean":"2024-03-26T05:16:31.642158+0000","last_became_active":"2024-03-26T05:15:33.617734+0000","last_became_peered":"2024-03-26T05:15:33.617734+0000","last_unstale":"2024-03-26T05:16:31.642158+0000","last_undegraded":"2024-03-26T05:16:31.642158+0000","last_fullsized":"2024-03-26T05:16:31.642158+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:15:39.542698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:15:39.542698+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:05.116328+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"357'7579","reported_seq":7671,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:34.765108+0000","last_change":"2024-03-26T05:14:41.668609+0000","last_active":"2024-03-26T05:16:34.765108+0000","last_peered":"2024-03-26T05:16:34.765108+0000","last_clean":"2024-03-26T05:16:34.765108+0000","last_became_active":"2024-03-26T05:14:41.668381+0000","last_became_peered":"2024-03-26T05:14:41.668381+0000","last_unstale":"2024-03-26T05:16:34.765108+0000","last_undegraded":"2024-03-26T05:16:34.765108+0000","last_fullsized":"2024-03-26T05:16:34.765108+0000","mapping_epoch":316,"log_start":"321'2000","ondisk_log_start":"321'2000","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390660096,"num_objects":5961,"num_object_clones":0,"num_object_copies":17883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7579,"num_write_kb":433280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":549,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.624692+0000","last_change":"2024-03-26T05:15:32.630784+0000","last_active":"2024-03-26T05:16:31.624692+0000","last_peered":"2024-03-26T05:16:31.624692+0000","last_clean":"2024-03-26T05:16:31.624692+0000","last_became_active":"2024-03-26T05:15:32.630135+0000","last_became_peered":"2024-03-26T05:15:32.630135+0000","last_unstale":"2024-03-26T05:16:31.624692+0000","last_undegraded":"2024-03-26T05:16:31.624692+0000","last_fullsized":"2024-03-26T05:16:31.624692+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:22.910006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:22.910006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":664,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.635879+0000","last_change":"2024-03-26T05:15:48.646902+0000","last_active":"2024-03-26T05:16:31.635879+0000","last_peered":"2024-03-26T05:16:31.635879+0000","last_clean":"2024-03-26T05:15:46.622472+0000","last_became_active":"2024-03-26T05:15:48.646902+0000","last_became_peered":"2024-03-26T05:15:48.646902+0000","last_unstale":"2024-03-26T05:16:31.635879+0000","last_undegraded":"2024-03-26T05:15:48.640683+0000","last_fullsized":"2024-03-26T05:15:48.640556+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:12.113993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:12.113993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"254'34","reported_seq":552,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.631356+0000","last_change":"2024-03-26T05:15:48.656188+0000","last_active":"2024-03-26T05:16:31.631356+0000","last_peered":"2024-03-26T05:16:31.631356+0000","last_clean":"2024-03-26T05:10:30.652936+0000","last_became_active":"2024-03-26T05:15:48.656188+0000","last_became_peered":"2024-03-26T05:15:48.656188+0000","last_unstale":"2024-03-26T05:16:31.631356+0000","last_undegraded":"2024-03-26T05:15:48.628982+0000","last_fullsized":"2024-03-26T05:15:48.628847+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'34","last_scrub_stamp":"2024-03-26T05:14:59.971454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:59.971454+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"343'9","reported_seq":833,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.642376+0000","last_change":"2024-03-26T05:15:48.648617+0000","last_active":"2024-03-26T05:16:31.642376+0000","last_peered":"2024-03-26T05:16:31.642376+0000","last_clean":"2024-03-26T05:15:46.625052+0000","last_became_active":"2024-03-26T05:15:48.648617+0000","last_became_peered":"2024-03-26T05:15:48.648617+0000","last_unstale":"2024-03-26T05:16:31.642376+0000","last_undegraded":"2024-03-26T05:15:48.628872+0000","last_fullsized":"2024-03-26T05:15:48.628757+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:13:56.721250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:13:56.721250+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:35.867386+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"88'128","reported_seq":417,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.648120+0000","last_change":"2024-03-26T05:15:48.644026+0000","last_active":"2024-03-26T05:16:31.648120+0000","last_peered":"2024-03-26T05:16:31.648120+0000","last_clean":"2024-03-26T05:01:05.801809+0000","last_became_active":"2024-03-26T05:15:48.644026+0000","last_became_peered":"2024-03-26T05:15:48.644026+0000","last_unstale":"2024-03-26T05:16:31.648120+0000","last_undegraded":"2024-03-26T05:16:31.648120+0000","last_fullsized":"2024-03-26T05:15:48.633510+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:35.014148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.e","version":"357'7727","reported_seq":7822,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:34.757086+0000","last_change":"2024-03-26T05:15:48.637088+0000","last_active":"2024-03-26T05:16:34.757086+0000","last_peered":"2024-03-26T05:16:34.757086+0000","last_clean":"2024-03-26T05:15:47.581062+0000","last_became_active":"2024-03-26T05:15:48.637088+0000","last_became_peered":"2024-03-26T05:15:48.637088+0000","last_unstale":"2024-03-26T05:16:34.757086+0000","last_undegraded":"2024-03-26T05:15:48.631634+0000","last_fullsized":"2024-03-26T05:15:48.631526+0000","mapping_epoch":342,"log_start":"321'2100","ondisk_log_start":"321'2100","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:39.202425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397082624,"num_objects":6059,"num_object_clones":0,"num_object_copies":18177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7727,"num_write_kb":441152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":6059}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":38,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.642656+0000","last_change":"2024-03-26T05:15:48.635004+0000","last_active":"2024-03-26T05:16:31.642656+0000","last_peered":"2024-03-26T05:16:31.642656+0000","last_clean":"2024-03-26T05:00:27.943471+0000","last_became_active":"2024-03-26T05:15:48.635004+0000","last_became_peered":"2024-03-26T05:15:48.635004+0000","last_unstale":"2024-03-26T05:16:31.642656+0000","last_undegraded":"2024-03-26T05:16:31.642656+0000","last_fullsized":"2024-03-26T05:15:48.630635+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:54.923744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:13:54.923744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:35.867360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":643,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.643767+0000","last_change":"2024-03-26T05:15:58.725078+0000","last_active":"2024-03-26T05:16:31.643767+0000","last_peered":"2024-03-26T05:16:31.643767+0000","last_clean":"2024-03-26T05:16:31.643767+0000","last_became_active":"2024-03-26T05:15:32.636983+0000","last_became_peered":"2024-03-26T05:15:32.636983+0000","last_unstale":"2024-03-26T05:16:31.643767+0000","last_undegraded":"2024-03-26T05:16:31.643767+0000","last_fullsized":"2024-03-26T05:16:31.643767+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:58.724953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:15:58.724953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:14.259681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"254'52","reported_seq":732,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642206+0000","last_change":"2024-03-26T05:15:54.719734+0000","last_active":"2024-03-26T05:16:31.642206+0000","last_peered":"2024-03-26T05:16:31.642206+0000","last_clean":"2024-03-26T05:16:31.642206+0000","last_became_active":"2024-03-26T05:15:32.636466+0000","last_became_peered":"2024-03-26T05:15:32.636466+0000","last_unstale":"2024-03-26T05:16:31.642206+0000","last_undegraded":"2024-03-26T05:16:31.642206+0000","last_fullsized":"2024-03-26T05:16:31.642206+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'52","last_scrub_stamp":"2024-03-26T05:15:54.719604+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:15:54.719604+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:03.234291+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":615,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.642422+0000","last_change":"2024-03-26T05:15:48.640538+0000","last_active":"2024-03-26T05:16:31.642422+0000","last_peered":"2024-03-26T05:16:31.642422+0000","last_clean":"2024-03-26T05:12:19.732211+0000","last_became_active":"2024-03-26T05:15:48.640538+0000","last_became_peered":"2024-03-26T05:15:48.640538+0000","last_unstale":"2024-03-26T05:16:31.642422+0000","last_undegraded":"2024-03-26T05:16:31.642422+0000","last_fullsized":"2024-03-26T05:15:48.628482+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.411139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.411139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:35.867318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"88'128","reported_seq":447,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.631230+0000","last_change":"2024-03-26T05:15:22.623013+0000","last_active":"2024-03-26T05:16:31.631230+0000","last_peered":"2024-03-26T05:16:31.631230+0000","last_clean":"2024-03-26T05:16:31.631230+0000","last_became_active":"2024-03-26T05:15:22.622542+0000","last_became_peered":"2024-03-26T05:15:22.622542+0000","last_unstale":"2024-03-26T05:16:31.631230+0000","last_undegraded":"2024-03-26T05:16:31.631230+0000","last_fullsized":"2024-03-26T05:16:31.631230+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:34.396797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1","version":"357'7894","reported_seq":18823,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:35.768606+0000","last_change":"2024-03-26T05:15:56.662302+0000","last_active":"2024-03-26T05:16:35.768606+0000","last_peered":"2024-03-26T05:16:35.768606+0000","last_clean":"2024-03-26T05:14:40.645004+0000","last_became_active":"2024-03-26T05:15:56.662302+0000","last_became_peered":"2024-03-26T05:15:56.662302+0000","last_unstale":"2024-03-26T05:16:35.768606+0000","last_undegraded":"2024-03-26T05:15:56.611572+0000","last_fullsized":"2024-03-26T05:15:56.611443+0000","mapping_epoch":344,"log_start":"322'2800","ondisk_log_start":"322'2800","created":316,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394395648,"num_objects":6018,"num_object_clones":0,"num_object_copies":18054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7894,"num_write_kb":445184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5415,"num_bytes_recovered":354877440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":6018}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":323,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.631257+0000","last_change":"2024-03-26T05:15:02.690388+0000","last_active":"2024-03-26T05:16:31.631257+0000","last_peered":"2024-03-26T05:16:31.631257+0000","last_clean":"2024-03-26T05:16:31.631257+0000","last_became_active":"2024-03-26T05:08:00.005676+0000","last_became_peered":"2024-03-26T05:08:00.005676+0000","last_unstale":"2024-03-26T05:16:31.631257+0000","last_undegraded":"2024-03-26T05:16:31.631257+0000","last_fullsized":"2024-03-26T05:16:31.631257+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:02.690252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_clean_scrub_stamp":"2024-03-26T05:15:02.690252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"343'11","reported_seq":478,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.632411+0000","last_change":"2024-03-26T05:15:48.648004+0000","last_active":"2024-03-26T05:16:31.632411+0000","last_peered":"2024-03-26T05:16:31.632411+0000","last_clean":"2024-03-26T05:07:58.484584+0000","last_became_active":"2024-03-26T05:15:48.648004+0000","last_became_peered":"2024-03-26T05:15:48.648004+0000","last_unstale":"2024-03-26T05:16:31.632411+0000","last_undegraded":"2024-03-26T05:15:48.629532+0000","last_fullsized":"2024-03-26T05:15:48.629408+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:14:03.822423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:03.822423+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":643,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642008+0000","last_change":"2024-03-26T05:15:42.617357+0000","last_active":"2024-03-26T05:16:31.642008+0000","last_peered":"2024-03-26T05:16:31.642008+0000","last_clean":"2024-03-26T05:16:31.642008+0000","last_became_active":"2024-03-26T05:15:32.626265+0000","last_became_peered":"2024-03-26T05:15:32.626265+0000","last_unstale":"2024-03-26T05:16:31.642008+0000","last_undegraded":"2024-03-26T05:16:31.642008+0000","last_fullsized":"2024-03-26T05:16:31.642008+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:42.617232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:15:42.617232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:59.396125+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"292'34","reported_seq":707,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.648392+0000","last_change":"2024-03-26T05:15:48.644946+0000","last_active":"2024-03-26T05:16:31.648392+0000","last_peered":"2024-03-26T05:16:31.648392+0000","last_clean":"2024-03-26T05:15:46.609501+0000","last_became_active":"2024-03-26T05:15:48.644946+0000","last_became_peered":"2024-03-26T05:15:48.644946+0000","last_unstale":"2024-03-26T05:16:31.648392+0000","last_undegraded":"2024-03-26T05:15:48.636746+0000","last_fullsized":"2024-03-26T05:15:48.636639+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'34","last_scrub_stamp":"2024-03-26T05:15:43.090549+0000","last_deep_scrub":"292'34","last_deep_scrub_stamp":"2024-03-26T05:12:23.778630+0000","last_clean_scrub_stamp":"2024-03-26T05:15:43.090549+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:47.195160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"357'7808","reported_seq":7903,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:34.756187+0000","last_change":"2024-03-26T05:15:48.648899+0000","last_active":"2024-03-26T05:16:34.756187+0000","last_peered":"2024-03-26T05:16:34.756187+0000","last_clean":"2024-03-26T05:15:47.606254+0000","last_became_active":"2024-03-26T05:15:48.648899+0000","last_became_peered":"2024-03-26T05:15:48.648899+0000","last_unstale":"2024-03-26T05:16:34.756187+0000","last_undegraded":"2024-03-26T05:15:48.630729+0000","last_fullsized":"2024-03-26T05:15:48.630611+0000","mapping_epoch":342,"log_start":"321'2200","ondisk_log_start":"321'2200","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5608,"log_dups_size":0,"ondisk_log_size":5608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:39.202415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400556032,"num_objects":6112,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7808,"num_write_kb":445440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":6112}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":642,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.642387+0000","last_change":"2024-03-26T05:14:59.850924+0000","last_active":"2024-03-26T05:16:31.642387+0000","last_peered":"2024-03-26T05:16:31.642387+0000","last_clean":"2024-03-26T05:16:31.642387+0000","last_became_active":"2024-03-26T05:07:59.957995+0000","last_became_peered":"2024-03-26T05:07:59.957995+0000","last_unstale":"2024-03-26T05:16:31.642387+0000","last_undegraded":"2024-03-26T05:16:31.642387+0000","last_fullsized":"2024-03-26T05:16:31.642387+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:59.850805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:59.850805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":326,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.632774+0000","last_change":"2024-03-26T05:15:48.643225+0000","last_active":"2024-03-26T05:16:31.632774+0000","last_peered":"2024-03-26T05:16:31.632774+0000","last_clean":"2024-03-26T05:15:46.630428+0000","last_became_active":"2024-03-26T05:15:48.643225+0000","last_became_peered":"2024-03-26T05:15:48.643225+0000","last_unstale":"2024-03-26T05:16:31.632774+0000","last_undegraded":"2024-03-26T05:16:31.632774+0000","last_fullsized":"2024-03-26T05:15:48.630436+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:56.632109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:13:07.883082+0000","last_clean_scrub_stamp":"2024-03-26T05:14:56.632109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":630,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.649057+0000","last_change":"2024-03-26T05:15:48.641615+0000","last_active":"2024-03-26T05:16:31.649057+0000","last_peered":"2024-03-26T05:16:31.649057+0000","last_clean":"2024-03-26T05:12:19.732296+0000","last_became_active":"2024-03-26T05:15:48.641615+0000","last_became_peered":"2024-03-26T05:15:48.641615+0000","last_unstale":"2024-03-26T05:16:31.649057+0000","last_undegraded":"2024-03-26T05:16:31.649057+0000","last_fullsized":"2024-03-26T05:15:48.632821+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:29.465975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.465975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"254'25","reported_seq":493,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.644376+0000","last_change":"2024-03-26T05:15:48.646310+0000","last_active":"2024-03-26T05:16:31.644376+0000","last_peered":"2024-03-26T05:16:31.644376+0000","last_clean":"2024-03-26T05:10:30.649037+0000","last_became_active":"2024-03-26T05:15:48.646310+0000","last_became_peered":"2024-03-26T05:15:48.646310+0000","last_unstale":"2024-03-26T05:16:31.644376+0000","last_undegraded":"2024-03-26T05:15:48.632255+0000","last_fullsized":"2024-03-26T05:15:48.632148+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:13:04.195003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:04.195003+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:32.373010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":6}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"88'128","reported_seq":447,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.632482+0000","last_change":"2024-03-26T05:15:22.617319+0000","last_active":"2024-03-26T05:16:31.632482+0000","last_peered":"2024-03-26T05:16:31.632482+0000","last_clean":"2024-03-26T05:16:31.632482+0000","last_became_active":"2024-03-26T05:15:22.616949+0000","last_became_peered":"2024-03-26T05:15:22.616949+0000","last_unstale":"2024-03-26T05:16:31.632482+0000","last_undegraded":"2024-03-26T05:16:31.632482+0000","last_fullsized":"2024-03-26T05:16:31.632482+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:34.396754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":423697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"357'8066","reported_seq":8158,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:35.432483+0000","last_change":"2024-03-26T05:14:41.667178+0000","last_active":"2024-03-26T05:16:35.432483+0000","last_peered":"2024-03-26T05:16:35.432483+0000","last_clean":"2024-03-26T05:16:35.432483+0000","last_became_active":"2024-03-26T05:14:41.666892+0000","last_became_peered":"2024-03-26T05:14:41.666892+0000","last_unstale":"2024-03-26T05:16:35.432483+0000","last_undegraded":"2024-03-26T05:16:35.432483+0000","last_fullsized":"2024-03-26T05:16:35.432483+0000","mapping_epoch":316,"log_start":"321'2100","ondisk_log_start":"321'2100","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407896064,"num_objects":6224,"num_object_clones":0,"num_object_copies":18672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8066,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":641,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.652851+0000","last_change":"2024-03-26T05:15:48.640264+0000","last_active":"2024-03-26T05:16:31.652851+0000","last_peered":"2024-03-26T05:16:31.652851+0000","last_clean":"2024-03-26T05:15:46.627568+0000","last_became_active":"2024-03-26T05:15:48.640264+0000","last_became_peered":"2024-03-26T05:15:48.640264+0000","last_unstale":"2024-03-26T05:16:31.652851+0000","last_undegraded":"2024-03-26T05:16:31.652851+0000","last_fullsized":"2024-03-26T05:15:48.632563+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:19.269035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:19.269035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":645,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.647270+0000","last_change":"2024-03-26T05:15:48.649219+0000","last_active":"2024-03-26T05:16:31.647270+0000","last_peered":"2024-03-26T05:16:31.647270+0000","last_clean":"2024-03-26T05:15:46.609940+0000","last_became_active":"2024-03-26T05:15:48.649219+0000","last_became_peered":"2024-03-26T05:15:48.649219+0000","last_unstale":"2024-03-26T05:16:31.647270+0000","last_undegraded":"2024-03-26T05:16:31.647270+0000","last_fullsized":"2024-03-26T05:15:48.637386+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:21.059907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:14:21.059907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":679,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.641892+0000","last_change":"2024-03-26T05:15:52.465360+0000","last_active":"2024-03-26T05:16:31.641892+0000","last_peered":"2024-03-26T05:16:31.641892+0000","last_clean":"2024-03-26T05:16:31.641892+0000","last_became_active":"2024-03-26T05:06:06.521150+0000","last_became_peered":"2024-03-26T05:06:06.521150+0000","last_unstale":"2024-03-26T05:16:31.641892+0000","last_undegraded":"2024-03-26T05:16:31.641892+0000","last_fullsized":"2024-03-26T05:16:31.641892+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:52.465232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:15:52.465232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:11.202094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"254'31","reported_seq":499,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.648614+0000","last_change":"2024-03-26T05:15:48.650607+0000","last_active":"2024-03-26T05:16:31.648614+0000","last_peered":"2024-03-26T05:16:31.648614+0000","last_clean":"2024-03-26T05:10:30.646940+0000","last_became_active":"2024-03-26T05:15:48.650607+0000","last_became_peered":"2024-03-26T05:15:48.650607+0000","last_unstale":"2024-03-26T05:16:31.648614+0000","last_undegraded":"2024-03-26T05:15:48.634034+0000","last_fullsized":"2024-03-26T05:15:48.633929+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'31","last_scrub_stamp":"2024-03-26T05:14:50.944513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:50.944513+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"88'128","reported_seq":815,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.628596+0000","last_change":"2024-03-26T05:15:33.617248+0000","last_active":"2024-03-26T05:16:31.628596+0000","last_peered":"2024-03-26T05:16:31.628596+0000","last_clean":"2024-03-26T05:16:31.628596+0000","last_became_active":"2024-03-26T05:15:33.616505+0000","last_became_peered":"2024-03-26T05:15:33.616505+0000","last_unstale":"2024-03-26T05:16:31.628596+0000","last_undegraded":"2024-03-26T05:16:31.628596+0000","last_fullsized":"2024-03-26T05:16:31.628596+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:33.256965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2","version":"357'7875","reported_seq":7970,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:35.889596+0000","last_change":"2024-03-26T05:15:48.655789+0000","last_active":"2024-03-26T05:16:35.889596+0000","last_peered":"2024-03-26T05:16:35.889596+0000","last_clean":"2024-03-26T05:15:47.613860+0000","last_became_active":"2024-03-26T05:15:48.655789+0000","last_became_peered":"2024-03-26T05:15:48.655789+0000","last_unstale":"2024-03-26T05:16:35.889596+0000","last_undegraded":"2024-03-26T05:15:48.643094+0000","last_fullsized":"2024-03-26T05:15:48.637125+0000","mapping_epoch":342,"log_start":"322'2700","ondisk_log_start":"322'2700","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:38.248334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398524416,"num_objects":6081,"num_object_clones":0,"num_object_copies":18243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7875,"num_write_kb":446592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":6081}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":38,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-03-26T05:16:31.647958+0000","last_change":"2024-03-26T05:15:48.650651+0000","last_active":"2024-03-26T05:16:31.647958+0000","last_peered":"2024-03-26T05:16:31.647958+0000","last_clean":"2024-03-26T05:00:27.943471+0000","last_became_active":"2024-03-26T05:15:48.650651+0000","last_became_peered":"2024-03-26T05:15:48.650651+0000","last_unstale":"2024-03-26T05:16:31.647958+0000","last_undegraded":"2024-03-26T05:16:31.647958+0000","last_fullsized":"2024-03-26T05:15:48.641490+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:54.067506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:54.067506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"336'13","reported_seq":870,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.648500+0000","last_change":"2024-03-26T05:15:32.632679+0000","last_active":"2024-03-26T05:16:31.648500+0000","last_peered":"2024-03-26T05:16:31.648500+0000","last_clean":"2024-03-26T05:16:31.648500+0000","last_became_active":"2024-03-26T05:15:32.632283+0000","last_became_peered":"2024-03-26T05:15:32.632283+0000","last_unstale":"2024-03-26T05:16:31.648500+0000","last_undegraded":"2024-03-26T05:16:31.648500+0000","last_fullsized":"2024-03-26T05:16:31.648500+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:12:08.228910+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:12:08.228910+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":656,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-03-26T05:16:31.651507+0000","last_change":"2024-03-26T05:15:12.643401+0000","last_active":"2024-03-26T05:16:31.651507+0000","last_peered":"2024-03-26T05:16:31.651507+0000","last_clean":"2024-03-26T05:16:31.651507+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:16:31.651507+0000","last_undegraded":"2024-03-26T05:16:31.651507+0000","last_fullsized":"2024-03-26T05:16:31.651507+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:12.642910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:15:12.642910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:16:31.713475+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"254'20","reported_seq":480,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:16:31.642782+0000","last_change":"2024-03-26T05:15:48.638251+0000","last_active":"2024-03-26T05:16:31.642782+0000","last_peered":"2024-03-26T05:16:31.642782+0000","last_clean":"2024-03-26T05:10:30.608013+0000","last_became_active":"2024-03-26T05:15:48.638251+0000","last_became_peered":"2024-03-26T05:15:48.638251+0000","last_unstale":"2024-03-26T05:16:31.642782+0000","last_undegraded":"2024-03-26T05:15:48.630289+0000","last_fullsized":"2024-03-26T05:15:48.630171+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'20","last_scrub_stamp":"2024-03-26T05:14:56.989928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:56.989928+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6390939672,"num_objects":97519,"num_object_clones":0,"num_object_copies":292557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97519,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125923,"num_write_kb":7150081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16965,"num_bytes_recovered":1111818240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18523299840,"data_stored":18523291696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87023,"ondisk_log_size":87023,"up":40,"acting":38,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":703368,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":186,"num_write":114,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1389754,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":30,"acting":30,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":82,"acting":82,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1405,"num_read_kb":1130,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":1870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":976,"ondisk_log_size":976,"up":82,"acting":82,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":81,"acting":81,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":83,"acting":83,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":335,"seq":1438814044174,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506236,"kb_used_data":1336372,"kb_used_omap":4,"kb_used_meta":169851,"kb_avail":92238916,"statfs":{"total":95995035648,"available":94452649984,"internally_reserved":0,"allocated":1368444928,"data_stored":1367971632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":173928104},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332300,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3174320,"kb_used_data":2755816,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":90570832,"statfs":{"total":95995035648,"available":92744531968,"internally_reserved":0,"allocated":2821955584,"data_stored":2821493084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":428538310},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575078,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1717780,"kb_used_data":1163212,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92027372,"statfs":{"total":95995035648,"available":94236028928,"internally_reserved":0,"allocated":1191129088,"data_stored":1190630634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738603,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4501452,"kb_used_data":3878852,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":89243700,"statfs":{"total":95995035648,"available":91385548800,"internally_reserved":0,"allocated":3971944448,"data_stored":3971454945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411553,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2813172,"kb_used_data":2352812,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":90931980,"statfs":{"total":95995035648,"available":93114347520,"internally_reserved":0,"allocated":2409279488,"data_stored":2408813556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117211,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2586508,"kb_used_data":1981956,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91158644,"statfs":{"total":95995035648,"available":93346451456,"internally_reserved":0,"allocated":2029522944,"data_stored":2029032772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":619051463},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920973,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3139256,"kb_used_data":2751280,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":90605896,"statfs":{"total":95995035648,"available":92780437504,"internally_reserved":0,"allocated":2817310720,"data_stored":2816894270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659345,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2727476,"kb_used_data":1996524,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":91017676,"statfs":{"total":95995035648,"available":93202100224,"internally_reserved":0,"allocated":2044440576,"data_stored":2043974826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3940093952,"data_stored":3940089880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1181941760,"data_stored":1181941760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2391801856,"data_stored":2391801856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2009595904,"data_stored":2009595904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2819096576,"data_stored":2819096576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2015756288,"data_stored":2015756288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2802913280,"data_stored":2802909208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1362100224,"data_stored":1362100224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:16:41.407 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-26T05:16:41.407 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 26 2024-03-26T05:16:41.457 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:41.501 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:41.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:41.602 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:41.631 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:16:41.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:41.683 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:41.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:41.784 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:41.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:41 smithi132 ceph-mon[33500]: 6.c scrub starts 2024-03-26T05:16:41.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:41 smithi132 ceph-mon[33500]: 6.c scrub ok 2024-03-26T05:16:41.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:41 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3613363560' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:41.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:41 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 51233/285561 objects degraded (17.941%), 32 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:41.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:41 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:16:41.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:41 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3613363560' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:16:41.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:41 smithi132 ceph-mon[33500]: osdmap e359: 8 total, 7 up, 8 in 2024-03-26T05:16:41.822 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:41.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:41.924 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:41.955 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:41.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:42.057 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:42.092 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:42.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:42.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[29691]: 6.c scrub starts 2024-03-26T05:16:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[29691]: 6.c scrub ok 2024-03-26T05:16:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3613363560' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 51233/285561 objects degraded (17.941%), 32 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:16:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3613363560' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:16:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[29691]: osdmap e359: 8 total, 7 up, 8 in 2024-03-26T05:16:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[40404]: 6.c scrub starts 2024-03-26T05:16:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[40404]: 6.c scrub ok 2024-03-26T05:16:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3613363560' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 51233/285561 objects degraded (17.941%), 32 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:16:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3613363560' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:16:42.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:41 smithi080 ceph-mon[40404]: osdmap e359: 8 total, 7 up, 8 in 2024-03-26T05:16:42.193 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:42.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:42.234 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:42.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:42.267 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:16:42.267 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:16:42.335 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:42.362 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:42.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:42.464 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:42.500 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:42.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:42.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:42.602 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:42.633 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:42.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:42.735 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:42.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:42.775 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:42.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:42.876 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:42.901 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:42.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:43.003 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:43.027 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:43.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:43.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:42 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:16:43.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:42 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:16:43.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:42 smithi132 ceph-mon[33500]: 10.a scrub ok 2024-03-26T05:16:43.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:42 smithi132 ceph-mon[33500]: pgmap v743: 158 pgs: 1 active+clean+scrubbing, 37 active+undersized, 88 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 48607/275781 objects degraded (17.625%); 11 MiB/s, 180 objects/s recovering 2024-03-26T05:16:43.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:42 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:16:43.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:42 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:16:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:42 smithi080 ceph-mon[29691]: 10.a scrub ok 2024-03-26T05:16:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:42 smithi080 ceph-mon[29691]: pgmap v743: 158 pgs: 1 active+clean+scrubbing, 37 active+undersized, 88 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 48607/275781 objects degraded (17.625%); 11 MiB/s, 180 objects/s recovering 2024-03-26T05:16:43.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:42 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:16:43.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:42 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:16:43.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:42 smithi080 ceph-mon[40404]: 10.a scrub ok 2024-03-26T05:16:43.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:42 smithi080 ceph-mon[40404]: pgmap v743: 158 pgs: 1 active+clean+scrubbing, 37 active+undersized, 88 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 48607/275781 objects degraded (17.625%); 11 MiB/s, 180 objects/s recovering 2024-03-26T05:16:43.128 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:43.153 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:43.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:43.254 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:43.279 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:43.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:43.381 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:43.405 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:43.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:43.507 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:43.534 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:43.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:43.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:43.636 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:43.660 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:43.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:43.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:43.761 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:43.789 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:43.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:43.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:43.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:43 smithi080 ceph-mon[29691]: osdmap e360: 8 total, 7 up, 8 in 2024-03-26T05:16:43.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:43 smithi080 ceph-mon[40404]: osdmap e360: 8 total, 7 up, 8 in 2024-03-26T05:16:43.891 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:43.929 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:43.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:44.032 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:44.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:43 smithi132 ceph-mon[33500]: osdmap e360: 8 total, 7 up, 8 in 2024-03-26T05:16:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:44.070 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:44.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:44.171 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:44.198 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:44.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:44.300 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:44.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:44.345 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:44.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:44.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:16:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:16:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:16:44.446 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:44.485 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:44.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:44.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:44.632 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:44.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:44.733 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:44.778 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:44.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:44.880 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:44.924 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:44.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:45.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:45.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:44 smithi132 ceph-mon[33500]: 3.16 scrub starts 2024-03-26T05:16:45.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:44 smithi132 ceph-mon[33500]: 3.16 scrub ok 2024-03-26T05:16:45.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:44 smithi132 ceph-mon[33500]: osdmap e361: 8 total, 7 up, 8 in 2024-03-26T05:16:45.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:44 smithi132 ceph-mon[33500]: pgmap v746: 158 pgs: 1 active+clean+scrubbing, 37 active+undersized, 88 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 970 op/s; 48607/275781 objects degraded (17.625%); 8.5 MiB/s, 135 objects/s recovering 2024-03-26T05:16:45.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:44 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:16:45.052 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:45.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:44 smithi080 ceph-mon[29691]: 3.16 scrub starts 2024-03-26T05:16:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:44 smithi080 ceph-mon[29691]: 3.16 scrub ok 2024-03-26T05:16:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:44 smithi080 ceph-mon[29691]: osdmap e361: 8 total, 7 up, 8 in 2024-03-26T05:16:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:44 smithi080 ceph-mon[29691]: pgmap v746: 158 pgs: 1 active+clean+scrubbing, 37 active+undersized, 88 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 970 op/s; 48607/275781 objects degraded (17.625%); 8.5 MiB/s, 135 objects/s recovering 2024-03-26T05:16:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:44 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:16:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:44 smithi080 ceph-mon[40404]: 3.16 scrub starts 2024-03-26T05:16:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:44 smithi080 ceph-mon[40404]: 3.16 scrub ok 2024-03-26T05:16:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:44 smithi080 ceph-mon[40404]: osdmap e361: 8 total, 7 up, 8 in 2024-03-26T05:16:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:44 smithi080 ceph-mon[40404]: pgmap v746: 158 pgs: 1 active+clean+scrubbing, 37 active+undersized, 88 active+clean, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 970 op/s; 48607/275781 objects degraded (17.625%); 8.5 MiB/s, 135 objects/s recovering 2024-03-26T05:16:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:44 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:16:45.154 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:45.178 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:45.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:45.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:45.309 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:45.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:45.411 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:45.470 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:45.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:45.572 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:45.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:45.617 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:45.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:45.718 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:45.760 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:45.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:45.862 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:45.893 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:45.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:45.994 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:46.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:16:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:16:46.059 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:46.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:46.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:46.161 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:46.189 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:46.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:46.291 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:46.334 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:46.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:46.413 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T05:16:46.435 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:46.442 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:46.484 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:46.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:46.552 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:16:46.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:46.596 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:16:46.634 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:46.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:46.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:46.638 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:16:46.688 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:16:46.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:46 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:46.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:46 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:46.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:46 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3756661857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:46.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:46 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:46.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:46 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:46.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:46 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3756661857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:46.729 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:16:46.730 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:16:46.737 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:46.773 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:16:46.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:46.783 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:46.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:46.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:46 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:46.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:46 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:46.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:46 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3756661857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:46.818 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:16:46.818 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:16:46.818 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:16:46.818 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:16:46.819 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:16:46.819 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:16:46.819 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:16:46.819 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:16:46.819 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:16:46.884 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:46.912 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:46.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:47.014 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:47.047 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:47.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:47.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:47.149 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:47.174 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:47.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:47.275 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:47.319 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:47.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:47.422 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:47.459 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:47.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:47.560 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:47.603 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:47.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:47.704 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:47.734 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:47.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:16:47.763 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:16:47.836 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:47.850 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:16:47.866 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:47.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:47.889 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:47.889 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[40404]: pgmap v747: 158 pgs: 37 active+undersized, 90 active+clean, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s; 46434/264147 objects degraded (17.579%); 11 MiB/s, 176 objects/s recovering 2024-03-26T05:16:47.889 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[40404]: 1.8 scrub starts 2024-03-26T05:16:47.889 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[40404]: 1.8 scrub ok 2024-03-26T05:16:47.890 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 48607/275781 objects degraded (17.625%), 32 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:47.890 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:16:47.890 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:16:47.890 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:47.890 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:47.890 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[29691]: pgmap v747: 158 pgs: 37 active+undersized, 90 active+clean, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s; 46434/264147 objects degraded (17.579%); 11 MiB/s, 176 objects/s recovering 2024-03-26T05:16:47.890 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[29691]: 1.8 scrub starts 2024-03-26T05:16:47.890 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[29691]: 1.8 scrub ok 2024-03-26T05:16:47.890 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 48607/275781 objects degraded (17.625%), 32 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:47.890 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:16:47.891 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:16:47.891 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:47 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:47.968 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:48.010 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:48.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:47 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:47 smithi132 ceph-mon[33500]: pgmap v747: 158 pgs: 37 active+undersized, 90 active+clean, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s; 46434/264147 objects degraded (17.579%); 11 MiB/s, 176 objects/s recovering 2024-03-26T05:16:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:47 smithi132 ceph-mon[33500]: 1.8 scrub starts 2024-03-26T05:16:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:47 smithi132 ceph-mon[33500]: 1.8 scrub ok 2024-03-26T05:16:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:47 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 48607/275781 objects degraded (17.625%), 32 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:47 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:16:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:47 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:16:48.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:47 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:48.111 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:48.161 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:48.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:48.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:48.263 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:48.298 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:48.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:48.400 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:48.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:48.457 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:48.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:48.558 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:48.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:48.615 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:48.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:48.716 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:48.762 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:48.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:48.864 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:48.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:48 smithi080 ceph-mon[40404]: 1.6 scrub starts 2024-03-26T05:16:48.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:48 smithi080 ceph-mon[40404]: 1.6 scrub ok 2024-03-26T05:16:48.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:48 smithi080 ceph-mon[40404]: 1.1 scrub starts 2024-03-26T05:16:48.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:48 smithi080 ceph-mon[40404]: 1.1 scrub ok 2024-03-26T05:16:48.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:48 smithi080 ceph-mon[29691]: 1.6 scrub starts 2024-03-26T05:16:48.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:48 smithi080 ceph-mon[29691]: 1.6 scrub ok 2024-03-26T05:16:48.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:48 smithi080 ceph-mon[29691]: 1.1 scrub starts 2024-03-26T05:16:48.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:48 smithi080 ceph-mon[29691]: 1.1 scrub ok 2024-03-26T05:16:48.904 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:48.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:49.005 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:49.006 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:49.033 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:49.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:49.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:48 smithi132 ceph-mon[33500]: 1.6 scrub starts 2024-03-26T05:16:49.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:48 smithi132 ceph-mon[33500]: 1.6 scrub ok 2024-03-26T05:16:49.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:48 smithi132 ceph-mon[33500]: 1.1 scrub starts 2024-03-26T05:16:49.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:48 smithi132 ceph-mon[33500]: 1.1 scrub ok 2024-03-26T05:16:49.135 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:49.185 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:49.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:49.286 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:49.339 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:49.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:49.439 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:49.480 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:49.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:49.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:49.582 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:49.612 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:49.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:49.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:49.640 INFO:teuthology.orchestra.run.smithi080.stderr:set pool 10 pg_num to 26 2024-03-26T05:16:49.713 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:49.714 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:49 smithi080 ceph-mon[40404]: pgmap v748: 158 pgs: 37 active+undersized, 90 active+clean, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 792 B/s rd, 0 B/s wr, 676 op/s; 45806/262692 objects degraded (17.437%); 5.9 MiB/s, 94 objects/s recovering 2024-03-26T05:16:49.715 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:49 smithi080 ceph-mon[40404]: 1.a scrub starts 2024-03-26T05:16:49.715 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:49 smithi080 ceph-mon[40404]: 1.a scrub ok 2024-03-26T05:16:49.715 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:49 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2955850788' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-26T05:16:49.715 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:49 smithi080 ceph-mon[29691]: pgmap v748: 158 pgs: 37 active+undersized, 90 active+clean, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 792 B/s rd, 0 B/s wr, 676 op/s; 45806/262692 objects degraded (17.437%); 5.9 MiB/s, 94 objects/s recovering 2024-03-26T05:16:49.715 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:49 smithi080 ceph-mon[29691]: 1.a scrub starts 2024-03-26T05:16:49.715 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:49 smithi080 ceph-mon[29691]: 1.a scrub ok 2024-03-26T05:16:49.715 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:49 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2955850788' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-26T05:16:49.748 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:49.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:49.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:49.764 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:16:49.850 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:49.898 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:49.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:49.999 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:49 smithi132 ceph-mon[33500]: pgmap v748: 158 pgs: 37 active+undersized, 90 active+clean, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 792 B/s rd, 0 B/s wr, 676 op/s; 45806/262692 objects degraded (17.437%); 5.9 MiB/s, 94 objects/s recovering 2024-03-26T05:16:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:49 smithi132 ceph-mon[33500]: 1.a scrub starts 2024-03-26T05:16:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:49 smithi132 ceph-mon[33500]: 1.a scrub ok 2024-03-26T05:16:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:49 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2955850788' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-26T05:16:50.048 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:50.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:50.150 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:50.180 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:50.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:50.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:50.282 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:50.313 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:50.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:50.415 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:50.459 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:50.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:50.561 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:50.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:50.596 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:50.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:50.697 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:50.738 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:50.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:50.839 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:50.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:50 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2955850788' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-26T05:16:50.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:50 smithi080 ceph-mon[29691]: osdmap e362: 8 total, 7 up, 8 in 2024-03-26T05:16:50.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T05:16:50.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:50.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:16:50.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:50 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2955850788' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-26T05:16:50.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:50 smithi080 ceph-mon[40404]: osdmap e362: 8 total, 7 up, 8 in 2024-03-26T05:16:50.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T05:16:50.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:50.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:16:50.988 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:50 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2955850788' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-26T05:16:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:50 smithi132 ceph-mon[33500]: osdmap e362: 8 total, 7 up, 8 in 2024-03-26T05:16:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T05:16:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:16:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:16:51.694 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:51.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:51.696 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:16:51.796 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:51.835 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:51.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:51.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[29691]: pgmap v750: 158 pgs: 37 active+undersized, 90 active+clean, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 700 B/s rd, 0 B/s wr, 911 op/s; 44823/255786 objects degraded (17.524%); 5.2 MiB/s, 83 objects/s recovering 2024-03-26T05:16:51.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-26T05:16:51.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[29691]: osdmap e363: 8 total, 7 up, 8 in 2024-03-26T05:16:51.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[29691]: 4.3 scrub starts 2024-03-26T05:16:51.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[29691]: 4.3 scrub ok 2024-03-26T05:16:51.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 44823/255786 objects degraded (17.524%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:51.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4016770995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:16:51.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:16:51.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[40404]: pgmap v750: 158 pgs: 37 active+undersized, 90 active+clean, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 700 B/s rd, 0 B/s wr, 911 op/s; 44823/255786 objects degraded (17.524%); 5.2 MiB/s, 83 objects/s recovering 2024-03-26T05:16:51.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-26T05:16:51.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[40404]: osdmap e363: 8 total, 7 up, 8 in 2024-03-26T05:16:51.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[40404]: 4.3 scrub starts 2024-03-26T05:16:51.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[40404]: 4.3 scrub ok 2024-03-26T05:16:51.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 44823/255786 objects degraded (17.524%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:51.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4016770995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:16:51.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:51 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:16:51.937 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:51.965 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:51.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:51 smithi132 ceph-mon[33500]: pgmap v750: 158 pgs: 37 active+undersized, 90 active+clean, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 700 B/s rd, 0 B/s wr, 911 op/s; 44823/255786 objects degraded (17.524%); 5.2 MiB/s, 83 objects/s recovering 2024-03-26T05:16:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-26T05:16:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:51 smithi132 ceph-mon[33500]: osdmap e363: 8 total, 7 up, 8 in 2024-03-26T05:16:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:51 smithi132 ceph-mon[33500]: 4.3 scrub starts 2024-03-26T05:16:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:51 smithi132 ceph-mon[33500]: 4.3 scrub ok 2024-03-26T05:16:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:51 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 44823/255786 objects degraded (17.524%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:51 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4016770995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:16:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:51 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:16:52.066 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:52.090 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:52.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:52.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:52.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:52.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:52.216 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:52.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:52.317 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:52.342 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:52.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:52.443 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:52.468 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:52.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:52.570 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:52.593 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:52.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:52.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:52.695 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:52.720 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:52.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:52.823 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:52.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:52.848 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:52.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:52.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[40404]: 4.15 scrub starts 2024-03-26T05:16:52.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[40404]: 4.15 scrub ok 2024-03-26T05:16:52.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[40404]: 1.9 scrub starts 2024-03-26T05:16:52.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[40404]: 1.9 scrub ok 2024-03-26T05:16:52.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:16:52.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:16:52.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[40404]: osdmap e364: 8 total, 7 up, 8 in 2024-03-26T05:16:52.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[40404]: pgmap v753: 168 pgs: 1 remapped+peering, 3 peering, 10 unknown, 37 active+undersized, 90 active+clean, 27 active+undersized+degraded; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 25766/247191 objects degraded (10.424%); 5.6 MiB/s, 91 objects/s recovering 2024-03-26T05:16:52.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T05:16:52.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[29691]: 4.15 scrub starts 2024-03-26T05:16:52.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[29691]: 4.15 scrub ok 2024-03-26T05:16:52.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[29691]: 1.9 scrub starts 2024-03-26T05:16:52.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[29691]: 1.9 scrub ok 2024-03-26T05:16:52.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:16:52.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:16:52.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[29691]: osdmap e364: 8 total, 7 up, 8 in 2024-03-26T05:16:52.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[29691]: pgmap v753: 168 pgs: 1 remapped+peering, 3 peering, 10 unknown, 37 active+undersized, 90 active+clean, 27 active+undersized+degraded; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 25766/247191 objects degraded (10.424%); 5.6 MiB/s, 91 objects/s recovering 2024-03-26T05:16:52.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:52 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T05:16:52.949 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:52.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:52.975 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:52.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:52.978 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:52 smithi132 ceph-mon[33500]: 4.15 scrub starts 2024-03-26T05:16:52.978 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:52 smithi132 ceph-mon[33500]: 4.15 scrub ok 2024-03-26T05:16:52.978 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:52 smithi132 ceph-mon[33500]: 1.9 scrub starts 2024-03-26T05:16:52.978 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:52 smithi132 ceph-mon[33500]: 1.9 scrub ok 2024-03-26T05:16:52.979 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:52 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:16:52.979 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:52 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:16:52.979 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:52 smithi132 ceph-mon[33500]: osdmap e364: 8 total, 7 up, 8 in 2024-03-26T05:16:52.979 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:52 smithi132 ceph-mon[33500]: pgmap v753: 168 pgs: 1 remapped+peering, 3 peering, 10 unknown, 37 active+undersized, 90 active+clean, 27 active+undersized+degraded; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 25766/247191 objects degraded (10.424%); 5.6 MiB/s, 91 objects/s recovering 2024-03-26T05:16:52.979 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:52 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T05:16:53.075 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:53.100 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:53.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:53.201 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:53.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:53.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:53.327 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:53.353 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:53.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:53.419 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:16:53.419 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:16:53.419 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:16:53.419 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:16:53.419 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.419 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198054 8.9 8:99f5ccfa:::benchmark_data_smithi080_109213_object83537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:16:53.419 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:53.895008+0000", 2024-03-26T05:16:53.420 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 419.53246953600001, 2024-03-26T05:16:53.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20899105000000001, 2024-03-26T05:16:53.420 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.420 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.420 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.420 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:16:53.420 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:53.420 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198054 2024-03-26T05:16:53.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.420 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895008+0000", 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895008+0000", 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895010+0000", 2024-03-26T05:16:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.72e-06 2024-03-26T05:16:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895014+0000", 2024-03-26T05:16:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.69e-06 2024-03-26T05:16:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895028+0000", 2024-03-26T05:16:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3808000000000001e-05 2024-03-26T05:16:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895036+0000", 2024-03-26T05:16:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6459999999999993e-06 2024-03-26T05:16:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.981756+0000", 2024-03-26T05:16:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.086720452000000003 2024-03-26T05:16:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.981876+0000", 2024-03-26T05:16:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119364 2024-03-26T05:16:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:16:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.982000+0000", 2024-03-26T05:16:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012400099999999999 2024-03-26T05:16:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992267+0000", 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010267642 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992404+0000", 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000136713 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.103815+0000", 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.111411407 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.103923+0000", 2024-03-26T05:16:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000107909 2024-03-26T05:16:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.103999+0000", 2024-03-26T05:16:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5698000000000006e-05 2024-03-26T05:16:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198055 8.9 8:9376f681:::benchmark_data_smithi080_109213_object83538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:16:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:53.895393+0000", 2024-03-26T05:16:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 419.53208429300003, 2024-03-26T05:16:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.208813532, 2024-03-26T05:16:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:16:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198055 2024-03-26T05:16:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895393+0000", 2024-03-26T05:16:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895393+0000", 2024-03-26T05:16:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895394+0000", 2024-03-26T05:16:53.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6400000000000002e-07 2024-03-26T05:16:53.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895397+0000", 2024-03-26T05:16:53.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5499999999999999e-06 2024-03-26T05:16:53.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895410+0000", 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2989e-05 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.895417+0000", 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4330000000000001e-06 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.984852+0000", 2024-03-26T05:16:53.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.089435597000000006 2024-03-26T05:16:53.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.984975+0000", 2024-03-26T05:16:53.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000122328 2024-03-26T05:16:53.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:16:53.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.985100+0000", 2024-03-26T05:16:53.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000125513 2024-03-26T05:16:53.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:53.992272+0000", 2024-03-26T05:16:53.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0071720559999999996 2024-03-26T05:16:53.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.002218+0000", 2024-03-26T05:16:53.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.009945934 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104058+0000", 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.101839334 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104151+0000", 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3468999999999996e-05 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.104207+0000", 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5665000000000003e-05 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198508 8.9 8:986ea8e3:::benchmark_data_smithi080_109213_object83991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.353239+0000", 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 419.07423866099998, 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20569538500000001, 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.434 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.435 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.435 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:16:53.435 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:53.435 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198508 2024-03-26T05:16:53.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.435 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353239+0000", 2024-03-26T05:16:53.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353239+0000", 2024-03-26T05:16:53.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353240+0000", 2024-03-26T05:16:53.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4100000000000001e-06 2024-03-26T05:16:53.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353245+0000", 2024-03-26T05:16:53.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2000000000000002e-06 2024-03-26T05:16:53.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353258+0000", 2024-03-26T05:16:53.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2106000000000001e-05 2024-03-26T05:16:53.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353272+0000", 2024-03-26T05:16:53.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4893e-05 2024-03-26T05:16:53.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.406729+0000", 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.053456469 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.406825+0000", 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6075000000000004e-05 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.406903+0000", 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8450999999999998e-05 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.409021+0000", 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002117819 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.410391+0000", 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001370008 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.558821+0000", 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.148429591 2024-03-26T05:16:53.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.558877+0000", 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6178000000000003e-05 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.558934+0000", 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7185000000000002e-05 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.442 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198509 8.9 8:914b5413:::benchmark_data_smithi080_109213_object83992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:16:53.442 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.353322+0000", 2024-03-26T05:16:53.442 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 419.07415556799998, 2024-03-26T05:16:53.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.205746507, 2024-03-26T05:16:53.442 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.442 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.442 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.442 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:16:53.442 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:53.442 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198509 2024-03-26T05:16:53.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.443 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353322+0000", 2024-03-26T05:16:53.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353322+0000", 2024-03-26T05:16:53.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353322+0000", 2024-03-26T05:16:53.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9500000000000003e-07 2024-03-26T05:16:53.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353325+0000", 2024-03-26T05:16:53.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.909e-06 2024-03-26T05:16:53.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353333+0000", 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3699999999999997e-06 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.353340+0000", 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1969999999999999e-06 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.410429+0000", 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.057089411999999999 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.410567+0000", 2024-03-26T05:16:53.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000137405 2024-03-26T05:16:53.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.446 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:16:53.446 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.410721+0000", 2024-03-26T05:16:53.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015416099999999999 2024-03-26T05:16:53.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.446 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.446 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416309+0000", 2024-03-26T05:16:53.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0055884469999999999 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416351+0000", 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2101999999999998e-05 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.558984+0000", 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.14263231300000001 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.559015+0000", 2024-03-26T05:16:53.447 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.1498000000000002e-05 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.559068+0000", 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3198000000000001e-05 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198524 8.9 8:97161286:::benchmark_data_smithi080_109213_object84007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.366616+0000", 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 419.06086114599998, 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20856360500000001, 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.448 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198524 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366616+0000", 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366616+0000", 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366618+0000", 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4479999999999999e-06 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366623+0000", 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3310000000000003e-06 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366639+0000", 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.59e-05 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.366648+0000", 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5639999999999997e-06 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.412563+0000", 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.045915688000000003 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.412676+0000", 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000113008 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.412809+0000", 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013275799999999999 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416343+0000", 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0035344500000000002 2024-03-26T05:16:53.452 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.453 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416354+0000", 2024-03-26T05:16:53.453 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0794e-05 2024-03-26T05:16:53.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.453 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575077+0000", 2024-03-26T05:16:53.453 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.15872312899999999 2024-03-26T05:16:53.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575136+0000", 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8783000000000003e-05 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575180+0000", 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3751999999999997e-05 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.454 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:198530 8.9 8:9a064d51:::benchmark_data_smithi080_109213_object84013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:16:53.455 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:54.370993+0000", 2024-03-26T05:16:53.455 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 419.05648410200001, 2024-03-26T05:16:53.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20430893899999999, 2024-03-26T05:16:53.455 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.455 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.455 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.455 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:16:53.455 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:53.455 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 198530 2024-03-26T05:16:53.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.370993+0000", 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.370993+0000", 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.370995+0000", 2024-03-26T05:16:53.456 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3349999999999999e-06 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.371001+0000", 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2369999999999997e-06 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.371012+0000", 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0692e-05 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.371018+0000", 2024-03-26T05:16:53.457 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9390000000000002e-06 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.414470+0000", 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043452706000000001 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.414555+0000", 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4563999999999999e-05 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:16:53.458 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.414637+0000", 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2443999999999999e-05 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416408+0000", 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0017704999999999999 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.416541+0000", 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000133337 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575224+0000", 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.15868285500000001 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575262+0000", 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7988999999999998e-05 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:54.575302+0000", 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0340999999999999e-05 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.460 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:199631 8.9 8:951c68a4:::benchmark_data_smithi080_109213_object85114:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:55.624693+0000", 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 417.80278449100001, 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20792925800000001, 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 199631 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.461 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624693+0000", 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624693+0000", 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624695+0000", 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5770000000000001e-06 2024-03-26T05:16:53.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624699+0000", 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6999999999999999e-06 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624710+0000", 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0747000000000001e-05 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.624717+0000", 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1489999999999997e-06 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.655682+0000", 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.030964476000000001 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.655762+0000", 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0160999999999994e-05 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.464 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.655849+0000", 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7125000000000004e-05 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.660996+0000", 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0051472660000000002 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.663215+0000", 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0022190790000000001 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.465 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832537+0000", 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.169322062 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832580+0000", 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3037999999999997e-05 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:55.832622+0000", 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1878000000000001e-05 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.466 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15093.0:203255 8.9 8:9507306b:::benchmark_data_smithi080_109213_object88738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:09:59.393074+0000", 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 414.034403666, 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.21242334900000001, 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15093", 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 203255 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393074+0000", 2024-03-26T05:16:53.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393074+0000", 2024-03-26T05:16:53.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393075+0000", 2024-03-26T05:16:53.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0079999999999999e-06 2024-03-26T05:16:53.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393079+0000", 2024-03-26T05:16:53.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8909999999999997e-06 2024-03-26T05:16:53.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393092+0000", 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2927e-05 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.393097+0000", 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7459999999999997e-06 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.475807+0000", 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.082709258999999993 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.470 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.475886+0000", 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9282000000000004e-05 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,5", 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.476011+0000", 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000125425 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.482040+0000", 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0060289230000000003 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.486423+0000", 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043832439999999997 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.605408+0000", 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.11898417 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.605444+0000", 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6127999999999997e-05 2024-03-26T05:16:53.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.473 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.473 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:09:59.605497+0000", 2024-03-26T05:16:53.473 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3346000000000001e-05 2024-03-26T05:16:53.473 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.473 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.473 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.473 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24356.0:3175 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94789128796928 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:16:53.474 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:12:26.361163+0000", 2024-03-26T05:16:53.474 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 267.06631475699999, 2024-03-26T05:16:53.474 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.43763786900000001, 2024-03-26T05:16:53.474 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.474 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:16:53.474 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.474 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24356", 2024-03-26T05:16:53.474 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:16:53.474 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 3175 2024-03-26T05:16:53.474 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.474 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361163+0000", 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361163+0000", 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361165+0000", 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.9269999999999999e-06 2024-03-26T05:16:53.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361170+0000", 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7490000000000004e-06 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361187+0000", 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6344000000000001e-05 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361198+0000", 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0868e-05 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.476 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361267+0000", 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9015000000000002e-05 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361278+0000", 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1096e-05 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.796862+0000", 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.43558413099999999 2024-03-26T05:16:53.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.797030+0000", 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016853500000000001 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.798801+0000", 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001770204 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.479 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:53.480 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24356.0:3176 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94789134008320 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:16:53.480 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:12:26.361223+0000", 2024-03-26T05:16:53.480 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 267.06625444700001, 2024-03-26T05:16:53.480 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.43790922500000001, 2024-03-26T05:16:53.480 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.480 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:16:53.480 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24356", 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 3176 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361223+0000", 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361223+0000", 2024-03-26T05:16:53.481 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361224+0000", 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2e-07 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361226+0000", 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.7360000000000001e-06 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361234+0000", 2024-03-26T05:16:53.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8050000000000007e-06 2024-03-26T05:16:53.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.483 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361241+0000", 2024-03-26T05:16:53.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9949999999999999e-06 2024-03-26T05:16:53.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.483 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361297+0000", 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6232e-05 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361306+0000", 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3659999999999992e-06 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.797188+0000", 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.43588267200000003 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.797316+0000", 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012806199999999999 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.799132+0000", 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001815837 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24356.0:3177 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94789128803840 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:16:53.485 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:12:26.361253+0000", 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 267.06622465200002, 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.44081309600000002, 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24356", 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 3177 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.486 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361253+0000", 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361253+0000", 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361253+0000", 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8000000000000001e-07 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361256+0000", 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.3819999999999998e-06 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361263+0000", 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.379e-06 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361268+0000", 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.31e-06 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.488 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361324+0000", 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6106000000000001e-05 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.361335+0000", 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0271e-05 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.799570+0000", 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.43823521100000001 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.489 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.799717+0000", 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014688200000000001 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.802066+0000", 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0023491749999999998 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.490 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141736 10.d 10:b5d9e068:::benchmark_data_smithi080_161990_object30013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.675956+0000", 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.7515215629999998, 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.21765442800000001, 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141736 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.491 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.675956+0000", 2024-03-26T05:16:53.492 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.492 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.492 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.492 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.492 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.675956+0000", 2024-03-26T05:16:53.492 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.492 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.492 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.492 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.492 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.675957+0000", 2024-03-26T05:16:53.492 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2279999999999999e-06 2024-03-26T05:16:53.492 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.675965+0000", 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8539999999999997e-06 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.675977+0000", 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2349e-05 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.675985+0000", 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.1110000000000008e-06 2024-03-26T05:16:53.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.494 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.676047+0000", 2024-03-26T05:16:53.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1660000000000003e-05 2024-03-26T05:16:53.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.494 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.676159+0000", 2024-03-26T05:16:53.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011217000000000001 2024-03-26T05:16:53.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.676290+0000", 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013086099999999999 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.696789+0000", 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020498663 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.893511+0000", 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.196722376 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.893551+0000", 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9579000000000002e-05 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.893610+0000", 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9577000000000002e-05 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141743 10.d 10:bbf4562a:::benchmark_data_smithi080_161990_object30020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.679951+0000", 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.7475264049999999, 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.21379651499999999, 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141743 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.497 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.679951+0000", 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.679951+0000", 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.679952+0000", 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2640000000000001e-06 2024-03-26T05:16:53.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.679958+0000", 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5589999999999996e-06 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.679973+0000", 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4683e-05 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.679981+0000", 2024-03-26T05:16:53.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9789999999999993e-06 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.680049+0000", 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8518999999999999e-05 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.680154+0000", 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104561 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.680255+0000", 2024-03-26T05:16:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000101346 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.700051+0000", 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.019796286999999999 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.893654+0000", 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.19360229900000001 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.893690+0000", 2024-03-26T05:16:53.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6908e-05 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.893748+0000", 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.711e-05 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141744 10.d 10:b6e94735:::benchmark_data_smithi080_161990_object30021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.681445+0000", 2024-03-26T05:16:53.502 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.7460321560000001, 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.21241697700000001, 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141744 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.681445+0000", 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.681445+0000", 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.681447+0000", 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2610000000000001e-06 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.681452+0000", 2024-03-26T05:16:53.504 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1150000000000002e-06 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.681464+0000", 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2218000000000001e-05 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.681472+0000", 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9789999999999993e-06 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.505 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.506 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.684360+0000", 2024-03-26T05:16:53.506 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0028882069999999998 2024-03-26T05:16:53.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.506 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.506 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.506 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.684490+0000", 2024-03-26T05:16:53.506 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013007200000000001 2024-03-26T05:16:53.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.506 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.506 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:16:53.506 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.684618+0000", 2024-03-26T05:16:53.506 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012830200000000001 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.700059+0000", 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.015440406 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.893782+0000", 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.19372286999999999 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.507 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.893812+0000", 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0549999999999997e-05 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.893862+0000", 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9997000000000003e-05 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141746 10.d 10:b7ec8eaf:::benchmark_data_smithi080_161990_object30023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.684677+0000", 2024-03-26T05:16:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.7428006270000003, 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.20930870400000001, 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141746 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.684677+0000", 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.684677+0000", 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.684678+0000", 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1680000000000001e-06 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.684683+0000", 2024-03-26T05:16:53.510 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1819999999999997e-06 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.684694+0000", 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1297999999999999e-05 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.684702+0000", 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.807e-06 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.511 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.688169+0000", 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0034664790000000002 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.688321+0000", 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000151877 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.692282+0000", 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0039610859999999999 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.512 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.704419+0000", 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.012137112 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.893895+0000", 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.18947619900000001 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.893938+0000", 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3347999999999999e-05 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.514 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.514 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.893986+0000", 2024-03-26T05:16:53.514 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7148000000000003e-05 2024-03-26T05:16:53.514 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.514 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.514 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.514 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.514 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141753 10.10 10:0cd8b062:::benchmark_data_smithi080_161990_object30030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:16:53.514 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.871167+0000", 2024-03-26T05:16:53.515 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.5563102020000001, 2024-03-26T05:16:53.515 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.85659952900000003, 2024-03-26T05:16:53.515 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.515 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.515 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.515 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:16:53.515 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:16:53.515 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141753 2024-03-26T05:16:53.515 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.516 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.516 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.516 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.516 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871167+0000", 2024-03-26T05:16:53.516 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.516 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.516 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.516 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871167+0000", 2024-03-26T05:16:53.516 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871168+0000", 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.1999999999999998e-07 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871172+0000", 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5999999999999998e-06 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871180+0000", 2024-03-26T05:16:53.517 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2700000000000004e-06 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871186+0000", 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6189999999999999e-06 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871237+0000", 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1038999999999997e-05 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.518 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871247+0000", 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0285999999999999e-05 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.722770+0000", 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.85152329999999998 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.722921+0000", 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000150297 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:16:53.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.723205+0000", 2024-03-26T05:16:53.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00028435299999999999 2024-03-26T05:16:53.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.520 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.726976+0000", 2024-03-26T05:16:53.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0037705690000000001 2024-03-26T05:16:53.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.520 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727633+0000", 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065709700000000004 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727682+0000", 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9518000000000003e-05 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727767+0000", 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4660999999999995e-05 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141755 10.0 10:0360d020:::benchmark_data_smithi080_161990_object30032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.881338+0000", 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.5461391940000002, 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.84627915200000003, 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141755 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881338+0000", 2024-03-26T05:16:53.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881338+0000", 2024-03-26T05:16:53.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881339+0000", 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2399999999999999e-07 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881340+0000", 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.57e-06 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881345+0000", 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8890000000000001e-06 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881348+0000", 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0960000000000001e-06 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881381+0000", 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2922000000000001e-05 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881387+0000", 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.711e-06 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.708793+0000", 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.82740590999999997 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.708904+0000", 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000111308 2024-03-26T05:16:53.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.709245+0000", 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034106199999999999 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.724788+0000", 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.015542861 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727472+0000", 2024-03-26T05:16:53.527 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0026844220000000001 2024-03-26T05:16:53.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.528 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.528 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727527+0000", 2024-03-26T05:16:53.528 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4558999999999997e-05 2024-03-26T05:16:53.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.528 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.528 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727617+0000", 2024-03-26T05:16:53.528 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0518000000000003e-05 2024-03-26T05:16:53.528 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.528 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.529 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.529 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141758 10.17 10:e86fd74d:::benchmark_data_smithi080_161990_object30035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:16:53.529 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.894305+0000", 2024-03-26T05:16:53.529 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.5331722889999999, 2024-03-26T05:16:53.529 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.833540261, 2024-03-26T05:16:53.529 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.529 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.529 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141758 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894305+0000", 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894305+0000", 2024-03-26T05:16:53.530 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894306+0000", 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4600000000000004e-07 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894309+0000", 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.083e-06 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.531 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894319+0000", 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7410000000000004e-06 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894327+0000", 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9840000000000001e-06 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894369+0000", 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1891000000000003e-05 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.532 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894376+0000", 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3869999999999997e-06 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.724007+0000", 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.82963071399999999 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.724136+0000", 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012902200000000001 2024-03-26T05:16:53.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,5", 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.724390+0000", 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000254813 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727187+0000", 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002796345 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727401+0000", 2024-03-26T05:16:53.534 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021470699999999999 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727758+0000", 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035681899999999997 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727808+0000", 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9663999999999997e-05 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.535 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727845+0000", 2024-03-26T05:16:53.536 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7444999999999998e-05 2024-03-26T05:16:53.536 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.536 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.536 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.536 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.536 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.536 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141763 10.10 10:08b1d820:::benchmark_data_smithi080_161990_object30040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:16:53.536 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.942376+0000", 2024-03-26T05:16:53.536 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.4851011980000002, 2024-03-26T05:16:53.536 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.79749960399999997, 2024-03-26T05:16:53.537 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.537 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.537 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.537 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:16:53.537 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:16:53.537 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141763 2024-03-26T05:16:53.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.537 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.537 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.537 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.537 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942376+0000", 2024-03-26T05:16:53.538 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.538 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.538 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942376+0000", 2024-03-26T05:16:53.538 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.538 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.538 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942378+0000", 2024-03-26T05:16:53.538 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4869999999999999e-06 2024-03-26T05:16:53.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.539 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.539 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.539 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942385+0000", 2024-03-26T05:16:53.539 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.469e-06 2024-03-26T05:16:53.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.539 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.539 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.539 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942408+0000", 2024-03-26T05:16:53.539 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.2993000000000002e-05 2024-03-26T05:16:53.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.539 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942423+0000", 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.431e-05 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942573+0000", 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000150483 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942602+0000", 2024-03-26T05:16:53.540 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.8705999999999999e-05 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.725127+0000", 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.78252492399999996 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.725246+0000", 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119441 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:16:53.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.725452+0000", 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00020625900000000001 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727779+0000", 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0023271519999999999 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739726+0000", 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011946417000000001 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.543 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.543 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739783+0000", 2024-03-26T05:16:53.543 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6626000000000003e-05 2024-03-26T05:16:53.543 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.543 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.543 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.543 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739876+0000", 2024-03-26T05:16:53.543 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3337000000000004e-05 2024-03-26T05:16:53.543 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.543 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141764 10.7 10:e6576bba:::benchmark_data_smithi080_161990_object30041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.949318+0000", 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.4781596170000002, 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.77638933799999998, 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:16:53.544 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141764 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949318+0000", 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949318+0000", 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.545 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:16:53.546 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949319+0000", 2024-03-26T05:16:53.546 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.85e-07 2024-03-26T05:16:53.546 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.546 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.546 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:16:53.546 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949322+0000", 2024-03-26T05:16:53.546 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0400000000000001e-06 2024-03-26T05:16:53.546 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.546 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.546 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:16:53.546 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949331+0000", 2024-03-26T05:16:53.546 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.234e-06 2024-03-26T05:16:53.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.547 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:16:53.547 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949337+0000", 2024-03-26T05:16:53.547 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9000000000000003e-06 2024-03-26T05:16:53.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.547 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.547 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949393+0000", 2024-03-26T05:16:53.547 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6372999999999997e-05 2024-03-26T05:16:53.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949402+0000", 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6770000000000001e-06 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.710744+0000", 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.76134257900000002 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.710890+0000", 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000145816 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,5", 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.711254+0000", 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000364068 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.713197+0000", 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001942444 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.713483+0000", 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00028582 2024-03-26T05:16:53.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.725630+0000", 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.012147269 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.725665+0000", 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.4888999999999997e-05 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:16:53.550 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.725707+0000", 2024-03-26T05:16:53.551 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2444000000000002e-05 2024-03-26T05:16:53.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.551 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:16:53.551 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:16:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:16:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:53.575 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:53.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:53.676 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:53.707 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:53.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:53.808 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:53.831 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:53.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:53.843 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:53 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-26T05:16:53.843 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:53 smithi080 ceph-mon[40404]: osdmap e365: 8 total, 7 up, 8 in 2024-03-26T05:16:53.843 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:53 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-26T05:16:53.843 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:53 smithi080 ceph-mon[29691]: osdmap e365: 8 total, 7 up, 8 in 2024-03-26T05:16:53.933 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:53.958 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:53.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:53.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:53.975 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:16:53.999 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:16:54.023 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:16:54.038 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:53 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-26T05:16:54.038 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:53 smithi132 ceph-mon[33500]: osdmap e365: 8 total, 7 up, 8 in 2024-03-26T05:16:54.044 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:16:54.059 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:54.064 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:16:54.084 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:54.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:54.086 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:16:54.110 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:16:54.110 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:16:54.110 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:16:54.111 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:16:54.111 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:16:54.111 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:16:54.111 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:16:54.111 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:16:54.111 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:16:54.111 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:16:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:16:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:16:54.186 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:54.210 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:54.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:54.312 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:54.337 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:54.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:54.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:54.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T05:16:54.439 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:54.463 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:54.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:54.565 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:54.589 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:54.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:54.691 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:54.703 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:54 smithi132 ceph-mon[33500]: osdmap e366: 8 total, 7 up, 8 in 2024-03-26T05:16:54.714 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:16:54.719 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:54.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:54.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:54.822 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:54.850 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:54.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:54.882 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:16:54.882 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:16:54.900 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:16:54.952 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:54.983 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:54.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:54.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:54.989 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:54 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2744429071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:54.989 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:54 smithi132 ceph-mon[33500]: pgmap v756: 168 pgs: 1 remapped+peering, 3 peering, 10 unknown, 37 active+undersized, 90 active+clean, 27 active+undersized+degraded; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 703 op/s; 25766/247191 objects degraded (10.424%); 2.7 MiB/s, 46 objects/s recovering 2024-03-26T05:16:55.085 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:55.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:54 smithi080 ceph-mon[29691]: osdmap e366: 8 total, 7 up, 8 in 2024-03-26T05:16:55.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:54 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2744429071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:55.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:54 smithi080 ceph-mon[29691]: pgmap v756: 168 pgs: 1 remapped+peering, 3 peering, 10 unknown, 37 active+undersized, 90 active+clean, 27 active+undersized+degraded; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 703 op/s; 25766/247191 objects degraded (10.424%); 2.7 MiB/s, 46 objects/s recovering 2024-03-26T05:16:55.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:54 smithi080 ceph-mon[40404]: osdmap e366: 8 total, 7 up, 8 in 2024-03-26T05:16:55.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:54 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2744429071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:16:55.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:54 smithi080 ceph-mon[40404]: pgmap v756: 168 pgs: 1 remapped+peering, 3 peering, 10 unknown, 37 active+undersized, 90 active+clean, 27 active+undersized+degraded; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 703 op/s; 25766/247191 objects degraded (10.424%); 2.7 MiB/s, 46 objects/s recovering 2024-03-26T05:16:55.134 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:55.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:55.236 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:55.285 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:55.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:55.337 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:55.386 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:55.429 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:55.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:55.494 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T05:16:55.508 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:16:55.508 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-26T05:16:55.509 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T05:16:55.532 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:55.592 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:55.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:55.693 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:55.709 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 362'8744 2024-03-26T05:16:55.709 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 364'9054 2024-03-26T05:16:55.709 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 364'9066 2024-03-26T05:16:55.709 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: osdmap e367: 8 total, 7 up, 8 in 2024-03-26T05:16:55.709 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 364'8846 2024-03-26T05:16:55.710 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 364'8858 2024-03-26T05:16:55.710 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 364'8800 2024-03-26T05:16:55.710 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 364'8725 2024-03-26T05:16:55.710 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 364'8826 2024-03-26T05:16:55.710 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 364'8695 2024-03-26T05:16:55.710 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 364'9054 2024-03-26T05:16:55.710 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 362'8744 2024-03-26T05:16:55.711 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 364'9054 2024-03-26T05:16:55.712 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 364'9066 2024-03-26T05:16:55.712 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: osdmap e367: 8 total, 7 up, 8 in 2024-03-26T05:16:55.712 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 364'8846 2024-03-26T05:16:55.712 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 364'8858 2024-03-26T05:16:55.712 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 362'8744 2024-03-26T05:16:55.712 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 364'9066 2024-03-26T05:16:55.712 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 364'8725 2024-03-26T05:16:55.712 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 364'8800 2024-03-26T05:16:55.712 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 364'8695 2024-03-26T05:16:55.712 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[29691]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 364'8800 2024-03-26T05:16:55.740 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:55.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:55.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:55.751 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 364'8858 2024-03-26T05:16:55.751 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 364'8800 2024-03-26T05:16:55.751 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 364'8725 2024-03-26T05:16:55.751 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 364'8826 2024-03-26T05:16:55.752 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 364'8695 2024-03-26T05:16:55.752 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 364'9054 2024-03-26T05:16:55.752 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 364'8858 2024-03-26T05:16:55.752 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 362'8744 2024-03-26T05:16:55.752 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 364'9066 2024-03-26T05:16:55.752 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 364'8725 2024-03-26T05:16:55.752 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 364'8800 2024-03-26T05:16:55.752 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 364'8695 2024-03-26T05:16:55.752 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:55 smithi080 ceph-mon[40404]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 364'8800 2024-03-26T05:16:55.842 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:55.873 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:55.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:55.891 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 362'8744 2024-03-26T05:16:55.891 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 364'9054 2024-03-26T05:16:55.891 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 364'9066 2024-03-26T05:16:55.891 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: osdmap e367: 8 total, 7 up, 8 in 2024-03-26T05:16:55.891 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 364'8846 2024-03-26T05:16:55.891 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 364'8858 2024-03-26T05:16:55.891 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 364'8800 2024-03-26T05:16:55.891 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 364'8725 2024-03-26T05:16:55.891 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 364'8826 2024-03-26T05:16:55.891 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 364'8695 2024-03-26T05:16:55.892 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 364'9054 2024-03-26T05:16:55.892 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 364'8858 2024-03-26T05:16:55.892 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 362'8744 2024-03-26T05:16:55.892 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 364'9066 2024-03-26T05:16:55.892 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 364'8725 2024-03-26T05:16:55.892 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 364'8800 2024-03-26T05:16:55.892 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 364'8695 2024-03-26T05:16:55.892 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-mon[33500]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 364'8800 2024-03-26T05:16:55.892 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:16:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:16:55.978 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:56.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:56.020 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:56.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:56.120 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:56.171 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:56.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:56.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:56.273 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:56.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:56.307 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:56.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:56.408 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:56.440 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:56.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:56.542 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:56.574 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:56.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:56.646 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:16:56.646 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:16:56.676 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:56.721 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:56.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:56.823 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:56.853 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:56.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:56.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:16:56.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:16:56.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:16:56.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:16:56.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:56.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[29691]: osdmap e368: 8 total, 7 up, 8 in 2024-03-26T05:16:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[29691]: pgmap v759: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 1 remapped+peering, 1 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 unknown, 37 active+undersized, 1 active+recovering+degraded+remapped, 1 active+remapped, 90 active+clean, 1 active+remapped+backfill_wait, 29 active+undersized+degraded; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 28821/236463 objects degraded (12.188%); 23796/236463 objects misplaced (10.063%); 2.4 MiB/s, 39 objects/s recovering 2024-03-26T05:16:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 25766/247191 objects degraded (10.424%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:56.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[29691]: osdmap e369: 8 total, 7 up, 8 in 2024-03-26T05:16:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:16:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:16:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:16:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:16:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[40404]: osdmap e368: 8 total, 7 up, 8 in 2024-03-26T05:16:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[40404]: pgmap v759: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 1 remapped+peering, 1 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 unknown, 37 active+undersized, 1 active+recovering+degraded+remapped, 1 active+remapped, 90 active+clean, 1 active+remapped+backfill_wait, 29 active+undersized+degraded; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 28821/236463 objects degraded (12.188%); 23796/236463 objects misplaced (10.063%); 2.4 MiB/s, 39 objects/s recovering 2024-03-26T05:16:56.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 25766/247191 objects degraded (10.424%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:56.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:56 smithi080 ceph-mon[40404]: osdmap e369: 8 total, 7 up, 8 in 2024-03-26T05:16:56.883 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:16:56.955 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:57.032 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:57.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:56 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:16:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:56 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:16:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:56 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:16:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:56 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:16:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:56 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:16:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:56 smithi132 ceph-mon[33500]: osdmap e368: 8 total, 7 up, 8 in 2024-03-26T05:16:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:56 smithi132 ceph-mon[33500]: pgmap v759: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 1 remapped+peering, 1 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 unknown, 37 active+undersized, 1 active+recovering+degraded+remapped, 1 active+remapped, 90 active+clean, 1 active+remapped+backfill_wait, 29 active+undersized+degraded; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 28821/236463 objects degraded (12.188%); 23796/236463 objects misplaced (10.063%); 2.4 MiB/s, 39 objects/s recovering 2024-03-26T05:16:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:56 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 25766/247191 objects degraded (10.424%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T05:16:57.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:56 smithi132 ceph-mon[33500]: osdmap e369: 8 total, 7 up, 8 in 2024-03-26T05:16:57.132 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:57.195 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:57.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:57.296 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:57.347 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:57.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:57.378 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:57.389 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T05:16:57.449 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:57.510 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:57.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:57.611 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:57.671 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:57.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:57.706 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:16:57.772 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:57.813 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:57 smithi080 ceph-mon[29691]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:16:57.813 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:57 smithi080 ceph-mon[40404]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:16:57.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:57.819 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:57.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:57.919 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:57.951 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:57.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:57.997 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:57 smithi132 ceph-mon[33500]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:16:58.054 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:16:58.086 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:58.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:16:58.135 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:16:58.188 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:58.221 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:58.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:58.321 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:58.363 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:58.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:58.465 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:58.498 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:58.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:58.600 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:16:58.637 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:58.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:58.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:16:58.739 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:58.769 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:16:58.801 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:58.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:58.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:58.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2327215398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:58.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:58 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:58.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:58 smithi080 ceph-mon[29691]: pgmap v761: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 1 active+recovering+degraded+remapped, 1 active+remapped, 91 active+clean, 1 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 323 op/s; 40633/242874 objects degraded (16.730%); 35998/242874 objects misplaced (14.822%); 25 MiB/s, 398 objects/s recovering 2024-03-26T05:16:58.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2327215398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:58.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:58 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:58.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:58 smithi080 ceph-mon[40404]: pgmap v761: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 1 active+recovering+degraded+remapped, 1 active+remapped, 91 active+clean, 1 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 323 op/s; 40633/242874 objects degraded (16.730%); 35998/242874 objects misplaced (14.822%); 25 MiB/s, 398 objects/s recovering 2024-03-26T05:16:58.903 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:16:58.934 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:58.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:16:58.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2327215398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:58.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:58 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:16:58.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:58 smithi132 ceph-mon[33500]: pgmap v761: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 1 active+recovering+degraded+remapped, 1 active+remapped, 91 active+clean, 1 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 323 op/s; 40633/242874 objects degraded (16.730%); 35998/242874 objects misplaced (14.822%); 25 MiB/s, 398 objects/s recovering 2024-03-26T05:16:59.039 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:59.078 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:59.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:59.179 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:59.216 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:59.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:59.317 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:59.360 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:59.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:16:59.462 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:16:59.500 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:16:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:59.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:16:59.602 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:16:59.645 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:59.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:59.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:16:59.664 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:16:59.664 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:16:59.747 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:16:59.782 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:59.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:59.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:16:59.883 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:16:59.928 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:16:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:16:59.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:00.030 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:00.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:59 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:17:00.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:59 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:17:00.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:16:59 smithi132 ceph-mon[33500]: osdmap e370: 8 total, 7 up, 8 in 2024-03-26T05:17:00.062 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:00.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:00.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:59 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:17:00.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:59 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:17:00.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:16:59 smithi080 ceph-mon[29691]: osdmap e370: 8 total, 7 up, 8 in 2024-03-26T05:17:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:59 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:17:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:59 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:17:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:16:59 smithi080 ceph-mon[40404]: osdmap e370: 8 total, 7 up, 8 in 2024-03-26T05:17:00.164 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:00.198 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:00.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:00.300 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:00.336 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:00.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:00.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:00.401 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T05:17:00.438 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:00.484 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:00.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:00.585 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:00.617 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:00.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:00.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:00.719 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:00.749 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:17:00.757 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:00.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:00.859 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:00.882 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:00.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:00.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:01.018 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:01.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:01.030 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:00 smithi132 ceph-mon[33500]: pgmap v763: 168 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 1 active+recovering+degraded+remapped, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 39799/235602 objects degraded (16.892%); 35428/235602 objects misplaced (15.037%); 20 MiB/s, 325 objects/s recovering 2024-03-26T05:17:01.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:00 smithi080 ceph-mon[29691]: pgmap v763: 168 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 1 active+recovering+degraded+remapped, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 39799/235602 objects degraded (16.892%); 35428/235602 objects misplaced (15.037%); 20 MiB/s, 325 objects/s recovering 2024-03-26T05:17:01.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:00 smithi080 ceph-mon[40404]: pgmap v763: 168 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 1 active+recovering+degraded+remapped, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 39799/235602 objects degraded (16.892%); 35428/235602 objects misplaced (15.037%); 20 MiB/s, 325 objects/s recovering 2024-03-26T05:17:01.119 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:01.157 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:01.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:01.259 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:01.296 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:01.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:01.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:01.398 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:01.430 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:01.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:01.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:01.532 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:01.573 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:01.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:01.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:01.675 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:01.713 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:01.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:01.814 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:01.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:01.864 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:01.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:01.965 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:01.981 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:01 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 39799/235602 objects degraded (16.892%), 36 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:01.981 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:01 smithi132 ceph-mon[33500]: osdmap e371: 8 total, 7 up, 8 in 2024-03-26T05:17:02.003 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:02.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:02.105 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:01 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 39799/235602 objects degraded (16.892%), 36 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:01 smithi080 ceph-mon[29691]: osdmap e371: 8 total, 7 up, 8 in 2024-03-26T05:17:02.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:01 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 39799/235602 objects degraded (16.892%), 36 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:02.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:01 smithi080 ceph-mon[40404]: osdmap e371: 8 total, 7 up, 8 in 2024-03-26T05:17:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:02.149 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:02.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:02.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:02.290 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:02.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:02.391 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:02.430 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:02.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:02.529 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:17:02.529 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:17:02.529 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 371, 2024-03-26T05:17:02.529 INFO:teuthology.orchestra.run.smithi080.stdout: "fsid": "dd09e9cc-eb2c-11ee-95d0-87774f69a715", 2024-03-26T05:17:02.529 INFO:teuthology.orchestra.run.smithi080.stdout: "created": "2024-03-26T04:54:51.106954+0000", 2024-03-26T05:17:02.529 INFO:teuthology.orchestra.run.smithi080.stdout: "modified": "2024-03-26T05:17:01.636201+0000", 2024-03-26T05:17:02.529 INFO:teuthology.orchestra.run.smithi080.stdout: "last_up_change": "2024-03-26T05:15:47.612973+0000", 2024-03-26T05:17:02.529 INFO:teuthology.orchestra.run.smithi080.stdout: "last_in_change": "2024-03-26T05:07:58.901005+0000", 2024-03-26T05:17:02.529 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T05:17:02.529 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_num": 5806080, 2024-03-26T05:17:02.530 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_set": [ 2024-03-26T05:17:02.530 INFO:teuthology.orchestra.run.smithi080.stdout: "nodeep-scrub", 2024-03-26T05:17:02.530 INFO:teuthology.orchestra.run.smithi080.stdout: "noscrub", 2024-03-26T05:17:02.530 INFO:teuthology.orchestra.run.smithi080.stdout: "pglog_hardlimit", 2024-03-26T05:17:02.530 INFO:teuthology.orchestra.run.smithi080.stdout: "purged_snapdirs", 2024-03-26T05:17:02.530 INFO:teuthology.orchestra.run.smithi080.stdout: "recovery_deletes", 2024-03-26T05:17:02.530 INFO:teuthology.orchestra.run.smithi080.stdout: "sortbitwise" 2024-03-26T05:17:02.530 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:17:02.530 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_version": 19, 2024-03-26T05:17:02.530 INFO:teuthology.orchestra.run.smithi080.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T05:17:02.530 INFO:teuthology.orchestra.run.smithi080.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T05:17:02.530 INFO:teuthology.orchestra.run.smithi080.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T05:17:02.531 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_snapshot": "", 2024-03-26T05:17:02.531 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_max": 10, 2024-03-26T05:17:02.531 INFO:teuthology.orchestra.run.smithi080.stdout: "max_osd": 8, 2024-03-26T05:17:02.531 INFO:teuthology.orchestra.run.smithi080.stdout: "require_min_compat_client": "luminous", 2024-03-26T05:17:02.531 INFO:teuthology.orchestra.run.smithi080.stdout: "min_compat_client": "luminous", 2024-03-26T05:17:02.531 INFO:teuthology.orchestra.run.smithi080.stdout: "require_osd_release": "squid", 2024-03-26T05:17:02.531 INFO:teuthology.orchestra.run.smithi080.stdout: "allow_crimson": false, 2024-03-26T05:17:02.531 INFO:teuthology.orchestra.run.smithi080.stdout: "pools": [ 2024-03-26T05:17:02.531 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.531 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 1, 2024-03-26T05:17:02.531 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": ".mgr", 2024-03-26T05:17:02.531 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T04:58:06.876819+0000", 2024-03-26T05:17:02.532 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:17:02.532 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:17:02.532 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:17:02.532 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:17:02.532 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:17:02.532 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:17:02.532 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:17:02.532 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:17:02.532 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:17:02.532 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:17:02.532 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:17:02.533 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:17:02.533 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 11, 2024-03-26T05:17:02.533 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 11, 2024-03-26T05:17:02.533 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 11, 2024-03-26T05:17:02.533 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 11, 2024-03-26T05:17:02.533 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 11, 2024-03-26T05:17:02.533 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:17:02.533 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:17:02.533 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:17:02.533 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:17:02.533 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:17:02.534 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:17:02.534 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:17:02.534 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.534 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "326", 2024-03-26T05:17:02.534 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:17:02.534 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:17:02.534 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "324", 2024-03-26T05:17:02.534 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:17:02.534 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:17:02.534 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:17:02.534 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:17:02.534 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:17:02.535 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:17:02.536 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: "options": { 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_max": 32, 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_min": 1 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": {} 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:17:02.537 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:17:02.538 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.1700000762939453, 2024-03-26T05:17:02.538 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.1700000762939453, 2024-03-26T05:17:02.538 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T05:17:02.538 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.9099999666213989, 2024-03-26T05:17:02.538 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.9099999666213989, 2024-03-26T05:17:02.538 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:17:02.538 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:17:02.538 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:17:02.538 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.538 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.538 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.538 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 2, 2024-03-26T05:17:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "datapool", 2024-03-26T05:17:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:20.705417+0000", 2024-03-26T05:17:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 8193, 2024-03-26T05:17:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T05:17:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:17:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:17:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:17:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:17:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:17:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:17:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:17:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 3, 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 3, 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 3, 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 3, 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 3, 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:17:02.540 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:17:02.541 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:17:02.541 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.541 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "71", 2024-03-26T05:17:02.541 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:17:02.541 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:17:02.541 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:17:02.541 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:17:02.541 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:17:02.541 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 2, 2024-03-26T05:17:02.541 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 71, 2024-03-26T05:17:02.541 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:17:02.541 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:17:02.542 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:17:02.542 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:17:02.542 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:17:02.542 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:17:02.542 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:17:02.542 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:17:02.542 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:17:02.542 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:17:02.542 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:17:02.542 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:17:02.542 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:17:02.542 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:17:02.543 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:17:02.543 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:17:02.543 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:17:02.543 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:17:02.543 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:17:02.543 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.543 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:17:02.543 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:17:02.543 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:17:02.543 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:17:02.543 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:17:02.543 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:17:02.544 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:17:02.544 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:17:02.544 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:17:02.544 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:17:02.544 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:17:02.544 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:17:02.544 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:17:02.544 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd": {} 2024-03-26T05:17:02.544 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.544 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:17:02.544 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:17:02.544 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.6500000953674316, 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.6500000953674316, 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 3, 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": ".rgw.root", 2024-03-26T05:17:02.545 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:21.150368+0000", 2024-03-26T05:17:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:17:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:17:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:17:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:17:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:17:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:17:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:17:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:17:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:17:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:17:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:17:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:17:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "69", 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:17:02.548 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:17:02.549 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:17:02.550 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.550 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:17:02.550 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:17:02.550 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:17:02.550 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:17:02.550 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:17:02.550 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:17:02.550 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:17:02.550 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:17:02.550 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:17:02.550 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:17:02.551 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:17:02.551 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:17:02.551 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:17:02.551 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:17:02.551 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.551 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:17:02.551 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:17:02.551 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.7400000095367432, 2024-03-26T05:17:02.551 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.7400000095367432, 2024-03-26T05:17:02.551 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T05:17:02.552 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T05:17:02.552 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T05:17:02.552 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:17:02.552 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:17:02.552 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:17:02.552 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.552 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.552 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.552 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 4, 2024-03-26T05:17:02.552 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.log", 2024-03-26T05:17:02.552 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:23.638065+0000", 2024-03-26T05:17:02.552 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:17:02.553 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:17:02.553 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:17:02.553 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:17:02.553 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:17:02.553 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:17:02.553 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:17:02.553 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:17:02.553 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:17:02.553 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:17:02.553 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:17:02.553 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:17:02.553 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:17:02.554 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:17:02.554 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:17:02.554 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:17:02.554 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:17:02.554 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:17:02.554 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:17:02.554 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:17:02.554 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:17:02.554 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:17:02.554 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:17:02.554 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:17:02.555 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.555 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "71", 2024-03-26T05:17:02.555 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:17:02.555 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:17:02.555 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:17:02.555 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:17:02.555 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:17:02.555 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:17:02.555 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:17:02.555 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:17:02.555 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:17:02.556 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:17:02.556 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:17:02.556 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:17:02.556 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:17:02.556 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:17:02.556 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:17:02.556 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:17:02.556 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:17:02.556 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:17:02.556 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:17:02.556 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:17:02.556 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:17:02.557 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:17:02.558 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:17:02.558 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:17:02.558 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:17:02.558 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:17:02.558 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:17:02.558 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:17:02.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.558 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:17:02.558 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:17:02.558 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.4900000095367432, 2024-03-26T05:17:02.558 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.4900000095367432, 2024-03-26T05:17:02.558 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T05:17:02.559 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2.190000057220459, 2024-03-26T05:17:02.559 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2.190000057220459, 2024-03-26T05:17:02.559 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:17:02.559 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:17:02.559 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:17:02.559 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.559 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.559 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.559 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 5, 2024-03-26T05:17:02.559 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.control", 2024-03-26T05:17:02.559 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:25.181000+0000", 2024-03-26T05:17:02.559 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:17:02.560 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:17:02.560 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:17:02.560 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:17:02.560 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:17:02.560 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:17:02.560 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:17:02.560 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:17:02.560 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:17:02.560 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:17:02.560 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:17:02.560 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:17:02.560 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:17:02.561 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:17:02.561 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:17:02.561 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:17:02.561 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:17:02.561 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:17:02.561 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:17:02.561 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:17:02.561 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:17:02.561 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:17:02.561 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:17:02.561 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:17:02.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.562 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "73", 2024-03-26T05:17:02.562 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:17:02.562 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:17:02.562 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:17:02.562 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:17:02.562 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:17:02.562 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:17:02.562 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:17:02.562 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:17:02.562 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:17:02.562 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:17:02.562 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:17:02.563 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:17:02.563 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:17:02.563 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:17:02.563 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:17:02.563 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:17:02.563 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:17:02.563 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:17:02.563 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:17:02.563 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:17:02.563 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:17:02.563 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:17:02.563 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:17:02.564 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:17:02.564 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:17:02.564 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:17:02.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.564 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:17:02.564 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:17:02.564 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:17:02.564 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:17:02.564 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:17:02.564 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:17:02.564 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:17:02.565 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:17:02.565 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:17:02.565 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:17:02.565 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:17:02.565 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:17:02.565 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:17:02.565 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:17:02.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.565 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:17:02.565 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:17:02.565 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.4900000095367432, 2024-03-26T05:17:02.566 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.4900000095367432, 2024-03-26T05:17:02.566 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T05:17:02.566 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.309999942779541, 2024-03-26T05:17:02.566 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.309999942779541, 2024-03-26T05:17:02.566 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:17:02.566 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:02.567 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:17:02.567 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:17:02.567 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.567 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.568 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 6, 2024-03-26T05:17:02.568 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.meta", 2024-03-26T05:17:02.568 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:27.249998+0000", 2024-03-26T05:17:02.568 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:17:02.568 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:17:02.568 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:17:02.568 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:17:02.568 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:17:02.568 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:17:02.568 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:17:02.568 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:17:02.568 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:17:02.569 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "75", 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:17:02.570 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:17:02.571 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:17:02.572 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:17:02.572 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:17:02.572 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:17:02.572 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:17:02.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.572 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:17:02.572 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:17:02.572 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:17:02.572 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:17:02.572 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:17:02.572 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:17:02.572 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:17:02.573 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:17:02.573 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:17:02.573 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:17:02.573 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:17:02.573 INFO:teuthology.orchestra.run.smithi080.stdout: "options": { 2024-03-26T05:17:02.573 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_bias": 4 2024-03-26T05:17:02.573 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.573 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:17:02.573 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:17:02.573 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.573 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:17:02.574 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:17:02.574 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.9900000095367432, 2024-03-26T05:17:02.574 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.9900000095367432, 2024-03-26T05:17:02.574 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T05:17:02.574 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.75, 2024-03-26T05:17:02.574 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.75, 2024-03-26T05:17:02.574 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:17:02.574 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:17:02.574 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:17:02.574 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.574 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.575 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.575 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 10, 2024-03-26T05:17:02.575 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "unique_pool_3", 2024-03-26T05:17:02.575 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:14:40.545427+0000", 2024-03-26T05:17:02.575 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:17:02.575 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:17:02.575 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:17:02.575 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:17:02.575 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:17:02.575 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:17:02.575 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:17:02.576 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:17:02.576 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:17:02.576 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:17:02.576 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:17:02.576 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:17:02.576 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 26, 2024-03-26T05:17:02.576 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 26, 2024-03-26T05:17:02.576 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 26, 2024-03-26T05:17:02.576 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 26, 2024-03-26T05:17:02.576 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 26, 2024-03-26T05:17:02.576 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:17:02.576 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:17:02.577 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:17:02.577 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:17:02.577 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:17:02.577 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:17:02.577 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:17:02.577 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.577 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "365", 2024-03-26T05:17:02.577 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:17:02.577 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:17:02.577 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "363", 2024-03-26T05:17:02.577 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:17:02.578 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:17:02.578 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:17:02.578 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:17:02.578 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:17:02.578 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:17:02.578 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:17:02.578 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:17:02.578 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:17:02.578 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:17:02.579 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: "rados": {} 2024-03-26T05:17:02.580 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.1400001049041748, 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.75, 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2.4200000762939453, 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.581 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 0, 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366", 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 8, 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 367, 2024-03-26T05:17:02.582 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6802", 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6803", 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.583 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6804", 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6805", 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.584 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6808", 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6809", 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.585 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6806", 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6807", 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:17:02.586 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6803/1840686622", 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6805/1840686622", 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6809/1840686622", 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6807/1840686622", 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 1, 2024-03-26T05:17:02.587 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23", 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 13, 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 366, 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.588 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6810", 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6811", 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.589 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6812", 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6813", 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.590 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6816", 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6817", 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.591 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6814", 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6815", 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6811/1726833996", 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6813/1726833996", 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6817/1726833996", 2024-03-26T05:17:02.592 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6815/1726833996", 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 2, 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "536bec9b-9636-4e67-b452-ed1a55c22ab6", 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:17:02.593 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 18, 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 366, 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6818", 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.594 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6819", 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6820", 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.595 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6821", 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6824", 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.596 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6825", 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6822", 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.597 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6823", 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6819/128713757", 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6821/128713757", 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6825/128713757", 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6823/128713757", 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:17:02.598 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 3, 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "640cfdb6-b9c5-446d-8312-504ae12c7818", 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 27, 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 366, 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:17:02.599 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6826", 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6827", 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6828", 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6829", 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6832", 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6833", 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.602 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6830", 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6831", 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.603 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.604 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6827/418771441", 2024-03-26T05:17:02.604 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6829/418771441", 2024-03-26T05:17:02.604 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6833/418771441", 2024-03-26T05:17:02.604 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6831/418771441", 2024-03-26T05:17:02.605 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:17:02.605 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:17:02.605 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:17:02.605 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.605 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.605 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.605 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 4, 2024-03-26T05:17:02.605 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639", 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 0, 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 33, 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 336, 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 342, 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.606 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6800", 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6801", 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.607 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6802", 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6803", 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.608 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6806", 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6807", 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.609 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6804", 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6805", 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6801/1441071423", 2024-03-26T05:17:02.610 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6803/1441071423", 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6807/1441071423", 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6805/1441071423", 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: "exists" 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 5, 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc", 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:17:02.611 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 41, 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 365, 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6808", 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.612 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6809", 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6810", 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.613 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6811", 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6814", 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.614 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6815", 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6812", 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.615 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6813", 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6809/794504625", 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6811/794504625", 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6815/794504625", 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6813/794504625", 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.616 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 6, 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5", 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 51, 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 365, 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:17:02.617 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6816", 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6817", 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.618 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6818", 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6819", 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.619 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6822", 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6823", 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:17:02.620 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6820", 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6821", 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6817/2209078972", 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6819/2209078972", 2024-03-26T05:17:02.621 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6823/2209078972", 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6821/2209078972", 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 7, 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d", 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:17:02.622 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 61, 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 292, 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 335, 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 365, 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 293, 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6824", 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.623 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.624 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.624 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6825", 2024-03-26T05:17:02.624 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:17:02.624 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.624 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.624 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.624 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:17:02.624 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.624 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.625 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6826", 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6827", 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.626 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6830", 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6831", 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:17:02.627 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6828", 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6829", 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6825/1165391760", 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6827/1165391760", 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6831/1165391760", 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6829/1165391760", 2024-03-26T05:17:02.628 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_xinfo": [ 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 0, 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:17:02.629 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:16:55.644324+0000", 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 1, 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:16:18.890820+0000", 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.630 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 2, 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:16:12.486905+0000", 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 3, 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:17:02.631 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:16:35.897907+0000", 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 4, 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:15:24.108106+0000", 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 341 2024-03-26T05:17:02.632 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 5, 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:17:01.398613+0000", 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 6, 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:17:02.633 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:15:58.953198+0000", 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 7, 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:17:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:16:55.705441+0000", 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 292 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap": [ 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.1", 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: 4, 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: 5 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap_items": [ 2024-03-26T05:17:02.635 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.a", 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 4, 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 0 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.b", 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 4, 2024-03-26T05:17:02.636 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 1 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.1c", 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 4, 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.637 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.2", 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.7", 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:17:02.638 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.1f", 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "5.10", 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:17:02.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 6, 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 3 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "6.b", 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 7, 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 4 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.640 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 3 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap_primaries": [], 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_temp": [ 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "10.10", 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: 1, 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: 6 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.641 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "10.11", 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: 7 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "10.12", 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: 1 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.642 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "10.13", 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: 2, 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: 3, 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: 5 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "10.14", 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: 3 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.643 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "10.15", 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: 3, 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: 2 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "10.16", 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: 5 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.644 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "10.17", 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: 3, 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: 6, 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: 5 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "10.18", 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: 2, 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:17:02.645 INFO:teuthology.orchestra.run.smithi080.stdout: 7 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "10.19", 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: 2, 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: 5, 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: 7 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_temp": [], 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: "blocklist": { 2024-03-26T05:17:02.646 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3992539385": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/2770889532": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/2466284312": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/490056571": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1986135300": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3796611170": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1993138634": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1832498181": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/2466284312": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1650870847": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/1418683843": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3147301987": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/2770889532": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/571905941": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1451012560": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1235125152": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:17:02.647 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/1418683843": "2024-03-27T05:01:03.689739+0000" 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "range_blocklist": {}, 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profiles": { 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "default": { 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "crush-failure-domain": "osd", 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "k": "2", 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "m": "1", 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "plugin": "jerasure", 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "technique": "reed_sol_van" 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps_queue": [], 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "new_removed_snaps": [], 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "new_purged_snaps": [], 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_node_flags": {}, 2024-03-26T05:17:02.648 INFO:teuthology.orchestra.run.smithi080.stdout: "device_class_flags": {}, 2024-03-26T05:17:02.649 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode": { 2024-03-26T05:17:02.649 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode_enabled": false, 2024-03-26T05:17:02.649 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_bucket_count": 0, 2024-03-26T05:17:02.649 INFO:teuthology.orchestra.run.smithi080.stdout: "degraded_stretch_mode": 0, 2024-03-26T05:17:02.649 INFO:teuthology.orchestra.run.smithi080.stdout: "recovering_stretch_mode": 0, 2024-03-26T05:17:02.649 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode_bucket": 0 2024-03-26T05:17:02.649 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:02.649 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:17:02.670 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:02.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:02.682 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:17:02.683 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:17:02.772 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:02.817 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:02.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:02.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:02.919 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:02.957 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:02.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:03.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:02 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:17:03.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:02 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:17:03.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:02 smithi132 ceph-mon[33500]: pgmap v765: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 1 active+recovering+degraded+remapped, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 36 MiB/s wr, 1.75k op/s; 36019/225609 objects degraded (15.965%); 31472/225609 objects misplaced (13.950%); 41 MiB/s, 655 objects/s recovering 2024-03-26T05:17:03.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:02 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3283193045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:17:03.059 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:03.101 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:03.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:03.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:02 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:17:03.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:02 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:17:03.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:02 smithi080 ceph-mon[40404]: pgmap v765: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 1 active+recovering+degraded+remapped, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 36 MiB/s wr, 1.75k op/s; 36019/225609 objects degraded (15.965%); 31472/225609 objects misplaced (13.950%); 41 MiB/s, 655 objects/s recovering 2024-03-26T05:17:03.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:02 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3283193045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:17:03.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:02 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:17:03.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:02 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:17:03.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:02 smithi080 ceph-mon[29691]: pgmap v765: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 1 active+recovering+degraded+remapped, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 36 MiB/s wr, 1.75k op/s; 36019/225609 objects degraded (15.965%); 31472/225609 objects misplaced (13.950%); 41 MiB/s, 655 objects/s recovering 2024-03-26T05:17:03.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:02 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3283193045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:17:03.203 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:03.238 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:03.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:03.340 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:03.378 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:03.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:03.390 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T05:17:03.480 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:03.498 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:17:03.509 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:03.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:03.543 INFO:tasks.thrashosds.thrasher:j is {'epoch': 371, 'fsid': 'dd09e9cc-eb2c-11ee-95d0-87774f69a715', 'created': '2024-03-26T04:54:51.106954+0000', 'modified': '2024-03-26T05:17:01.636201+0000', 'last_up_change': '2024-03-26T05:15:47.612973+0000', 'last_in_change': '2024-03-26T05:07:58.901005+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T04:58:06.876819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '326', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '324', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T05:00:20.705417+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T05:00:21.150368+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T05:00:23.638065+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T05:00:25.181000+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T05:00:27.249998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-26T05:14:40.545427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '365', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '363', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.140000104904175, 'score_stable': 2.75, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 2.4200000762939453, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '35a52544-0b0b-47d6-8d6a-571c48ceb366', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 367, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6802', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6803', 'nonce': 1840686622}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6804', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6805', 'nonce': 1840686622}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6808', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6809', 'nonce': 1840686622}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6806', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6807', 'nonce': 1840686622}]}, 'public_addr': '172.21.15.80:6803/1840686622', 'cluster_addr': '172.21.15.80:6805/1840686622', 'heartbeat_back_addr': '172.21.15.80:6809/1840686622', 'heartbeat_front_addr': '172.21.15.80:6807/1840686622', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efa5ea33-4f11-49b7-a3f3-f60d5329db23', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6810', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6811', 'nonce': 1726833996}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6812', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6813', 'nonce': 1726833996}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6816', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6817', 'nonce': 1726833996}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6814', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6815', 'nonce': 1726833996}]}, 'public_addr': '172.21.15.80:6811/1726833996', 'cluster_addr': '172.21.15.80:6813/1726833996', 'heartbeat_back_addr': '172.21.15.80:6817/1726833996', 'heartbeat_front_addr': '172.21.15.80:6815/1726833996', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '536bec9b-9636-4e67-b452-ed1a55c22ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6818', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6819', 'nonce': 128713757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6820', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6821', 'nonce': 128713757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6824', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6825', 'nonce': 128713757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6822', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6823', 'nonce': 128713757}]}, 'public_addr': '172.21.15.80:6819/128713757', 'cluster_addr': '172.21.15.80:6821/128713757', 'heartbeat_back_addr': '172.21.15.80:6825/128713757', 'heartbeat_front_addr': '172.21.15.80:6823/128713757', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '640cfdb6-b9c5-446d-8312-504ae12c7818', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6826', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6827', 'nonce': 418771441}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6828', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6829', 'nonce': 418771441}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6832', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6833', 'nonce': 418771441}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6830', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6831', 'nonce': 418771441}]}, 'public_addr': '172.21.15.80:6827/418771441', 'cluster_addr': '172.21.15.80:6829/418771441', 'heartbeat_back_addr': '172.21.15.80:6833/418771441', 'heartbeat_front_addr': '172.21.15.80:6831/418771441', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0cc7822a-ed73-4c79-a30d-1e151ddd6639', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 336, 'down_at': 342, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6800', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6801', 'nonce': 1441071423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6802', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6803', 'nonce': 1441071423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6806', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6807', 'nonce': 1441071423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6804', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6805', 'nonce': 1441071423}]}, 'public_addr': '172.21.15.132:6801/1441071423', 'cluster_addr': '172.21.15.132:6803/1441071423', 'heartbeat_back_addr': '172.21.15.132:6807/1441071423', 'heartbeat_front_addr': '172.21.15.132:6805/1441071423', 'state': ['exists']}, {'osd': 5, 'uuid': 'acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 365, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6808', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6809', 'nonce': 794504625}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6810', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6811', 'nonce': 794504625}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6814', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6815', 'nonce': 794504625}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6812', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6813', 'nonce': 794504625}]}, 'public_addr': '172.21.15.132:6809/794504625', 'cluster_addr': '172.21.15.132:6811/794504625', 'heartbeat_back_addr': '172.21.15.132:6815/794504625', 'heartbeat_front_addr': '172.21.15.132:6813/794504625', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2970bbab-11b2-4d1b-b57e-037bdac089b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 365, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6816', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6817', 'nonce': 2209078972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6818', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6819', 'nonce': 2209078972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6822', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6823', 'nonce': 2209078972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6820', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6821', 'nonce': 2209078972}]}, 'public_addr': '172.21.15.132:6817/2209078972', 'cluster_addr': '172.21.15.132:6819/2209078972', 'heartbeat_back_addr': '172.21.15.132:6823/2209078972', 'heartbeat_front_addr': '172.21.15.132:6821/2209078972', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '12f4b466-28f9-469b-8fd0-65b91c6dc22d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 61, 'last_clean_end': 292, 'up_from': 335, 'up_thru': 365, 'down_at': 293, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6824', 'nonce': 1165391760}, {'type': 'v1', 'addr': '172.21.15.132:6825', 'nonce': 1165391760}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6826', 'nonce': 1165391760}, {'type': 'v1', 'addr': '172.21.15.132:6827', 'nonce': 1165391760}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6830', 'nonce': 1165391760}, {'type': 'v1', 'addr': '172.21.15.132:6831', 'nonce': 1165391760}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6828', 'nonce': 1165391760}, {'type': 'v1', 'addr': '172.21.15.132:6829', 'nonce': 1165391760}]}, 'public_addr': '172.21.15.132:6825/1165391760', 'cluster_addr': '172.21.15.132:6827/1165391760', 'heartbeat_back_addr': '172.21.15.132:6831/1165391760', 'heartbeat_front_addr': '172.21.15.132:6829/1165391760', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:16:55.644324+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:16:18.890820+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:16:12.486905+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:16:35.897907+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:15:24.108106+0000', 'dead_epoch': 341}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:17:01.398613+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:15:58.953198+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:16:55.705441+0000', 'dead_epoch': 292}], 'pg_upmap': [{'pgid': '3.1', 'osds': [4, 0, 5]}], 'pg_upmap_items': [{'pgid': '3.a', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '3.b', 'mappings': [{'from': 4, 'to': 1}]}, {'pgid': '3.1c', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.10', 'mappings': [{'from': 6, 'to': 3}]}, {'pgid': '6.b', 'mappings': [{'from': 7, 'to': 4}, {'from': 1, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '10.10', 'osds': [1, 6]}, {'pgid': '10.11', 'osds': [0, 7]}, {'pgid': '10.12', 'osds': [0, 1]}, {'pgid': '10.13', 'osds': [2, 3, 5]}, {'pgid': '10.14', 'osds': [0, 3]}, {'pgid': '10.15', 'osds': [3, 0, 2]}, {'pgid': '10.16', 'osds': [0, 5]}, {'pgid': '10.17', 'osds': [3, 6, 5]}, {'pgid': '10.18', 'osds': [2, 0, 7]}, {'pgid': '10.19', 'osds': [2, 5, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.80:0/3992539385': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/2770889532': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:6801/2466284312': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/490056571': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1986135300': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/3796611170': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1993138634': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1832498181': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/2466284312': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1650870847': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6801/1418683843': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:0/3147301987': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6801/2770889532': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/571905941': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1451012560': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1235125152': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/1418683843': '2024-03-27T05:01:03.689739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T05:17:03.544 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:17:03.612 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:03.645 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:03.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:03.750 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:03.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:03.806 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:03.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:03.907 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:03.933 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:03.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:03.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:04.035 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:04.041 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:17:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=sqlstore.transactions t=2024-03-26T05:17:03.738755656Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T05:17:04.041 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:17:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=sqlstore.transactions t=2024-03-26T05:17:03.749335347Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-26T05:17:04.063 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:04.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:04.164 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:04.227 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:04.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:04.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:04.248 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:04.258 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:04.266 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:04.280 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:17:04.280 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:17:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:17:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:17:04.327 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:17:04.329 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:04.360 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:04.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:04.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:04.366 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:17:04.412 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:17:04.456 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:17:04.462 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:04.500 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:17:04.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:04.515 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:04.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:04.540 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:17:04.540 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:17:04.540 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:17:04.540 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:17:04.540 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:17:04.540 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:17:04.540 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:17:04.540 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:17:04.540 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:17:04.616 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:04.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:04.655 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:04.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:04.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:04.789 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:04.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:04.891 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:04.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:05.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:05.061 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:05.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:05.163 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:05.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:05.204 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:05.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:05.305 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:05.349 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:05.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:05.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:05 smithi080 ceph-mon[29691]: pgmap v766: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 1 active+recovering+degraded+remapped, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 686 B/s rd, 0 B/s wr, 772 op/s; 36019/225609 objects degraded (15.965%); 31472/225609 objects misplaced (13.950%); 20 MiB/s, 314 objects/s recovering 2024-03-26T05:17:05.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:05 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2432857836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:17:05.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:05 smithi080 ceph-mon[40404]: pgmap v766: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 1 active+recovering+degraded+remapped, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 686 B/s rd, 0 B/s wr, 772 op/s; 36019/225609 objects degraded (15.965%); 31472/225609 objects misplaced (13.950%); 20 MiB/s, 314 objects/s recovering 2024-03-26T05:17:05.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:05 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2432857836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:17:05.451 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:05.488 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:05.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:05.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:05.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:05 smithi132 ceph-mon[33500]: pgmap v766: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 1 active+recovering+degraded+remapped, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 686 B/s rd, 0 B/s wr, 772 op/s; 36019/225609 objects degraded (15.965%); 31472/225609 objects misplaced (13.950%); 20 MiB/s, 314 objects/s recovering 2024-03-26T05:17:05.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:05 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2432857836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:17:05.590 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:05.621 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:05.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:05.723 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:05.750 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:05.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:05.852 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:05.879 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:05.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:05.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:05.980 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:06.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:06.020 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:06.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:06.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:17:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:17:06.122 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:06.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:06.147 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:06.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:06.248 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:06.288 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:06.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:06.390 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:06.430 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:06.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:06.532 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:06.562 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:06.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:06.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:06.664 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:06.706 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:06.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:06.709 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:06.709 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:17:06.709 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:06.709 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:17:06.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:06.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:17:06.809 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:06.847 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:06.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:06.948 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:06.979 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:06.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:07.081 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:07.120 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:07.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:07.222 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:07.252 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:07.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:07.354 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:07.382 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:07.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:07.483 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:07.513 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:07.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:07.614 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:07.640 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:07.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:07.743 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:07.783 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:07.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:07.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:07 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:07.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:07 smithi132 ceph-mon[33500]: pgmap v767: 168 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 33697/211716 objects degraded (15.916%); 32619/211716 objects misplaced (15.407%); 26 MiB/s, 422 objects/s recovering 2024-03-26T05:17:07.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:07 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:17:07.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:07 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:17:07.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:07 smithi132 ceph-mon[33500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:07.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:07 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 33697/211716 objects degraded (15.916%), 35 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:07.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:07 smithi132 ceph-mon[33500]: osdmap e372: 8 total, 7 up, 8 in 2024-03-26T05:17:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[40404]: pgmap v767: 168 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 33697/211716 objects degraded (15.916%); 32619/211716 objects misplaced (15.407%); 26 MiB/s, 422 objects/s recovering 2024-03-26T05:17:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:17:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:17:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[40404]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 33697/211716 objects degraded (15.916%), 35 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:07.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[40404]: osdmap e372: 8 total, 7 up, 8 in 2024-03-26T05:17:07.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:07.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[29691]: pgmap v767: 168 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 33697/211716 objects degraded (15.916%); 32619/211716 objects misplaced (15.407%); 26 MiB/s, 422 objects/s recovering 2024-03-26T05:17:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:17:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:17:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[29691]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 33697/211716 objects degraded (15.916%), 35 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:07 smithi080 ceph-mon[29691]: osdmap e372: 8 total, 7 up, 8 in 2024-03-26T05:17:07.884 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:07.910 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:07.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:08.012 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:08.043 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:08.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:08.144 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:08.170 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:08.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:08.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:08.272 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:08.301 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:08.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:08.403 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:08.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:08.443 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:08.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:08.544 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:08.575 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:08.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:08.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:08.676 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:08.709 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:08.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:08.810 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:08.851 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:08.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:08.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:08.953 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:08.976 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:08.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:09.078 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:09.123 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:09.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:09.225 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:09.256 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:09.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:09.358 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:09.397 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:09.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:09.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:09.499 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:09.529 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:09.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:09.631 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:09.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:09.673 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:09.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:09.774 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:09.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:09 smithi132 ceph-mon[33500]: pgmap v769: 168 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 31861/208689 objects degraded (15.267%); 31415/208689 objects misplaced (15.054%); 36 MiB/s, 571 objects/s recovering 2024-03-26T05:17:09.803 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:09.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:09.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:09.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:09 smithi080 ceph-mon[29691]: pgmap v769: 168 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 31861/208689 objects degraded (15.267%); 31415/208689 objects misplaced (15.054%); 36 MiB/s, 571 objects/s recovering 2024-03-26T05:17:09.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:09 smithi080 ceph-mon[40404]: pgmap v769: 168 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 31861/208689 objects degraded (15.267%); 31415/208689 objects misplaced (15.054%); 36 MiB/s, 571 objects/s recovering 2024-03-26T05:17:09.905 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:09.937 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:09.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:10.039 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:10.070 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:10.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:10.173 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:10.197 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:10.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:10.299 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:10.324 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:10.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:10.426 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:10.459 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:10.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:10.560 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:10.583 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:10.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:10.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:10.685 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:10.722 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:10.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:10.824 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:10.849 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:10.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:10.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:10 smithi080 ceph-mon[29691]: osdmap e373: 8 total, 7 up, 8 in 2024-03-26T05:17:10.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:10 smithi080 ceph-mon[40404]: osdmap e373: 8 total, 7 up, 8 in 2024-03-26T05:17:10.950 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:10.991 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:10.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:11.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:10 smithi132 ceph-mon[33500]: osdmap e373: 8 total, 7 up, 8 in 2024-03-26T05:17:11.094 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:11.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:11.123 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:11.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:11.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:11.252 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:11.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:11.354 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:11.396 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:11.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:11.498 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:11.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:11.537 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:11.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:11.638 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:11.663 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:11.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:11.765 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:11.793 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:11.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:11.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:11.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:11 smithi080 ceph-mon[29691]: pgmap v771: 168 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 989 op/s; 31040/201864 objects degraded (15.377%); 30928/201864 objects misplaced (15.321%); 17 MiB/s, 279 objects/s recovering 2024-03-26T05:17:11.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:11 smithi080 ceph-mon[29691]: osdmap e374: 8 total, 7 up, 8 in 2024-03-26T05:17:11.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:11 smithi080 ceph-mon[40404]: pgmap v771: 168 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 989 op/s; 31040/201864 objects degraded (15.377%); 30928/201864 objects misplaced (15.321%); 17 MiB/s, 279 objects/s recovering 2024-03-26T05:17:11.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:11 smithi080 ceph-mon[40404]: osdmap e374: 8 total, 7 up, 8 in 2024-03-26T05:17:11.895 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:11.932 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:11.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:12.034 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:11 smithi132 ceph-mon[33500]: pgmap v771: 168 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 989 op/s; 31040/201864 objects degraded (15.377%); 30928/201864 objects misplaced (15.321%); 17 MiB/s, 279 objects/s recovering 2024-03-26T05:17:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:11 smithi132 ceph-mon[33500]: osdmap e374: 8 total, 7 up, 8 in 2024-03-26T05:17:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:12.060 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:12.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:12.161 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:12.190 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:12.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:12.292 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:12.321 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:12.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:12.422 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:12.466 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:12.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:12.567 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:12.591 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:12.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:12.693 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:12.734 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:12.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:12.836 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:12.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:12 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 31040/201864 objects degraded (15.377%), 35 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:12.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:12 smithi080 ceph-mon[29691]: osdmap e375: 8 total, 7 up, 8 in 2024-03-26T05:17:12.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:12 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 31040/201864 objects degraded (15.377%), 35 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:12.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:12 smithi080 ceph-mon[40404]: osdmap e375: 8 total, 7 up, 8 in 2024-03-26T05:17:12.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:12.867 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:12.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:12.968 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:13.001 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:13.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:13.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:12 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 31040/201864 objects degraded (15.377%), 35 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:13.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:12 smithi132 ceph-mon[33500]: osdmap e375: 8 total, 7 up, 8 in 2024-03-26T05:17:13.102 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:13.132 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:13.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:13.234 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:13.261 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:13.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:13.364 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:13.392 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:13.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:13.494 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:13.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:13.527 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:13.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:13.628 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:13.667 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:13.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:13.768 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:13.800 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:13.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:13.902 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:13.942 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:13.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:13.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:14.022 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:13 smithi080 ceph-mon[40404]: pgmap v774: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 28736/194409 objects degraded (14.781%); 26614/194409 objects misplaced (13.690%); 28 MiB/s, 455 objects/s recovering 2024-03-26T05:17:14.023 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:13 smithi080 ceph-mon[29691]: pgmap v774: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 28736/194409 objects degraded (14.781%); 26614/194409 objects misplaced (13.690%); 28 MiB/s, 455 objects/s recovering 2024-03-26T05:17:14.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:13 smithi132 ceph-mon[33500]: pgmap v774: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 28736/194409 objects degraded (14.781%); 26614/194409 objects misplaced (13.690%); 28 MiB/s, 455 objects/s recovering 2024-03-26T05:17:14.044 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:14.088 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:14.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:14.190 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:14.222 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:14.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:14.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:14.324 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:14.338 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:17:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:17:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:17:14.366 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:14.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:14.469 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:14.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:14.511 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:14.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:14.612 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:14.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:14.645 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:14.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:14.746 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:14.775 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:14.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:14.877 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:14.921 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:14.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:14.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:15.023 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:15.052 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:15.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:15.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:15.154 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:15.193 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:15.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:15.295 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:15.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:15.322 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:15.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:15.423 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:15.461 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:15.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:15.562 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:15.600 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:15.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:15.702 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:15.725 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:15.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:15.827 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:15.865 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:15.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:15.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:15.995 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:15.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:16.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:15 smithi132 ceph-mon[33500]: pgmap v775: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 793 op/s; 28736/194409 objects degraded (14.781%); 26614/194409 objects misplaced (13.690%); 22 MiB/s, 350 objects/s recovering 2024-03-26T05:17:16.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:17:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:17:16.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:16.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:15 smithi080 ceph-mon[40404]: pgmap v775: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 793 op/s; 28736/194409 objects degraded (14.781%); 26614/194409 objects misplaced (13.690%); 22 MiB/s, 350 objects/s recovering 2024-03-26T05:17:16.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:15 smithi080 ceph-mon[29691]: pgmap v775: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 91 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 793 op/s; 28736/194409 objects degraded (14.781%); 26614/194409 objects misplaced (13.690%); 22 MiB/s, 350 objects/s recovering 2024-03-26T05:17:16.135 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:16.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:16.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:16.237 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:16.269 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:16.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:16.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:16.402 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:16.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:16.504 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:16.540 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:16.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:16.642 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:16.681 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:16.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:16.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:16.783 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:16.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:16.816 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:16.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:16.917 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:16.945 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:16.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:17.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:16 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 28736/194409 objects degraded (14.781%), 34 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:17.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:16 smithi132 ceph-mon[33500]: osdmap e376: 8 total, 7 up, 8 in 2024-03-26T05:17:17.046 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:17.076 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:17.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:17.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:16 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 28736/194409 objects degraded (14.781%), 34 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:16 smithi080 ceph-mon[29691]: osdmap e376: 8 total, 7 up, 8 in 2024-03-26T05:17:17.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:16 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 28736/194409 objects degraded (14.781%), 34 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:17.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:16 smithi080 ceph-mon[40404]: osdmap e376: 8 total, 7 up, 8 in 2024-03-26T05:17:17.178 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:17.209 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:17.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:17.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:17.311 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:17.337 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:17.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:17.439 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:17.463 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:17.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:17.564 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:17.588 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:17.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:17.689 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:17.720 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:17.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:17.822 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:17.850 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:17.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:17.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:17.951 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:17.979 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:17.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:18.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:17 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:18.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:17 smithi132 ceph-mon[33500]: pgmap v776: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 92 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.37k op/s; 27026/181752 objects degraded (14.870%); 23580/181752 objects misplaced (12.974%); 28 MiB/s, 451 objects/s recovering 2024-03-26T05:17:18.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:17 smithi132 ceph-mon[33500]: osdmap e377: 8 total, 7 up, 8 in 2024-03-26T05:17:18.081 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:18.105 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:18.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:18.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:17 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:18.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:17 smithi080 ceph-mon[29691]: pgmap v776: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 92 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.37k op/s; 27026/181752 objects degraded (14.870%); 23580/181752 objects misplaced (12.974%); 28 MiB/s, 451 objects/s recovering 2024-03-26T05:17:18.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:17 smithi080 ceph-mon[29691]: osdmap e377: 8 total, 7 up, 8 in 2024-03-26T05:17:18.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:17 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:18.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:17 smithi080 ceph-mon[40404]: pgmap v776: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 92 active+clean, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.37k op/s; 27026/181752 objects degraded (14.870%); 23580/181752 objects misplaced (12.974%); 28 MiB/s, 451 objects/s recovering 2024-03-26T05:17:18.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:17 smithi080 ceph-mon[40404]: osdmap e377: 8 total, 7 up, 8 in 2024-03-26T05:17:18.207 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:18.231 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:18.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:18.333 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:18.361 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:18.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:18.462 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:18.491 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:18.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:18.593 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:18.617 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:18.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:18.719 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:18.756 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:18.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:18.858 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:18.897 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:18.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:18.999 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:19.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:19.041 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:19.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:19.142 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:19.167 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:19.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:19.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:19.268 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:19.298 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:19.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:19.400 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:19.430 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:19.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:19.532 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:19.560 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:19.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:19.662 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:19.708 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:19.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:19.810 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:19.851 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:19.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:19.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:19.953 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:19.980 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:19 smithi080 ceph-mon[40404]: pgmap v779: 168 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 92 active+clean, 1 active+undersized+remapped, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 792 B/s rd, 0 B/s wr, 785 op/s; 26126/179208 objects degraded (14.579%); 22507/179208 objects misplaced (12.559%); 12 MiB/s, 192 objects/s recovering 2024-03-26T05:17:19.980 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:19 smithi080 ceph-mon[40404]: osdmap e378: 8 total, 7 up, 8 in 2024-03-26T05:17:19.980 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:19 smithi080 ceph-mon[29691]: pgmap v779: 168 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 92 active+clean, 1 active+undersized+remapped, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 792 B/s rd, 0 B/s wr, 785 op/s; 26126/179208 objects degraded (14.579%); 22507/179208 objects misplaced (12.559%); 12 MiB/s, 192 objects/s recovering 2024-03-26T05:17:19.980 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:19 smithi080 ceph-mon[29691]: osdmap e378: 8 total, 7 up, 8 in 2024-03-26T05:17:19.983 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:19.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:19.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:20.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:19 smithi132 ceph-mon[33500]: pgmap v779: 168 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 92 active+clean, 1 active+undersized+remapped, 2 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 792 B/s rd, 0 B/s wr, 785 op/s; 26126/179208 objects degraded (14.579%); 22507/179208 objects misplaced (12.559%); 12 MiB/s, 192 objects/s recovering 2024-03-26T05:17:20.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:19 smithi132 ceph-mon[33500]: osdmap e378: 8 total, 7 up, 8 in 2024-03-26T05:17:20.086 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:20.109 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:20.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:20.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:20.211 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:20.242 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:20.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:20.344 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:20.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:20.384 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:20.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:20.485 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:20.527 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:20.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:20.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:20.630 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:20.672 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:20.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:20.774 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:20.797 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:20.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:20.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:20.899 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:20.929 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:20.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:21.031 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:21.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:20 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:17:21.060 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:21.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:21.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:21.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:20 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:17:21.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:20 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:17:21.162 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:21.185 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:21.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:21.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:21.287 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:21.312 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:21.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:21.413 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:21.440 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:21.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:21.542 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:21.568 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:21.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:21.669 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:21.701 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:21.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:21.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:21.803 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:21.830 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:21.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:21.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:21.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:21 smithi080 ceph-mon[29691]: pgmap v781: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 93 active+clean, 1 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s; 25511/173811 objects degraded (14.677%); 22156/173811 objects misplaced (12.747%); 13 MiB/s, 207 objects/s recovering 2024-03-26T05:17:21.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:21 smithi080 ceph-mon[29691]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:21.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:21 smithi080 ceph-mon[29691]: osdmap e379: 8 total, 7 up, 8 in 2024-03-26T05:17:21.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:21 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 25511/173811 objects degraded (14.677%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:21.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:21 smithi080 ceph-mon[40404]: pgmap v781: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 93 active+clean, 1 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s; 25511/173811 objects degraded (14.677%); 22156/173811 objects misplaced (12.747%); 13 MiB/s, 207 objects/s recovering 2024-03-26T05:17:21.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:21 smithi080 ceph-mon[40404]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:21.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:21 smithi080 ceph-mon[40404]: osdmap e379: 8 total, 7 up, 8 in 2024-03-26T05:17:21.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:21 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 25511/173811 objects degraded (14.677%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:21.933 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:21.960 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:21.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:22.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:21 smithi132 ceph-mon[33500]: pgmap v781: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 93 active+clean, 1 active+remapped+backfill_wait, 30 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s; 25511/173811 objects degraded (14.677%); 22156/173811 objects misplaced (12.747%); 13 MiB/s, 207 objects/s recovering 2024-03-26T05:17:22.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:21 smithi132 ceph-mon[33500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:22.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:21 smithi132 ceph-mon[33500]: osdmap e379: 8 total, 7 up, 8 in 2024-03-26T05:17:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:21 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 25511/173811 objects degraded (14.677%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:22.061 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:22.091 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:22.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:22.193 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:22.223 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:22.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:22.325 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:22.358 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:22.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:22.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:22.460 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:22.500 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:22.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:22.602 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:22.641 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:22.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:22.743 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:22.766 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:22.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:22.868 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:22.899 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:22.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:23.001 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:23.040 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:23.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:23.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:22 smithi132 ceph-mon[33500]: osdmap e380: 8 total, 7 up, 8 in 2024-03-26T05:17:23.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:22 smithi080 ceph-mon[40404]: osdmap e380: 8 total, 7 up, 8 in 2024-03-26T05:17:23.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:22 smithi080 ceph-mon[29691]: osdmap e380: 8 total, 7 up, 8 in 2024-03-26T05:17:23.141 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:23.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:23.171 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:23.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:23.272 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:23.316 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:23.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:23.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:23.418 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:23.455 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:23.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:23.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:23.557 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:23.592 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:23.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:23.694 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:23.731 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:23.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:23.833 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:23.872 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:23.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:23.974 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:24.012 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:24.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:24.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:23 smithi080 ceph-mon[29691]: pgmap v784: 168 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 93 active+clean, 30 active+undersized+degraded; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 893 op/s; 24408/167286 objects degraded (14.591%); 17565/167286 objects misplaced (10.500%); 24 MiB/s, 385 objects/s recovering 2024-03-26T05:17:24.016 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:23 smithi080 ceph-mon[40404]: pgmap v784: 168 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 93 active+clean, 30 active+undersized+degraded; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 893 op/s; 24408/167286 objects degraded (14.591%); 17565/167286 objects misplaced (10.500%); 24 MiB/s, 385 objects/s recovering 2024-03-26T05:17:24.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:23 smithi132 ceph-mon[33500]: pgmap v784: 168 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 93 active+clean, 30 active+undersized+degraded; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 893 op/s; 24408/167286 objects degraded (14.591%); 17565/167286 objects misplaced (10.500%); 24 MiB/s, 385 objects/s recovering 2024-03-26T05:17:24.114 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:24.144 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:24.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:24.246 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:24.270 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:24.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:24.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:24.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:17:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:17:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:17:24.372 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:24.403 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:24.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:24.505 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:24.528 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:24.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:24.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:24.660 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:24.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:24.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:24.762 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:24.792 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:24.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:24.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:24.894 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:24.934 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:24.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:25.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:25.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:25.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:25.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:25.170 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:25.207 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:25.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:25.309 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:25.340 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:25.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:25.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:25.442 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:25.486 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:25.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:25.587 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:25.617 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:25.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:25.719 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:25.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:25.758 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:25.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:25.859 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:25.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:25 smithi080 ceph-mon[29691]: pgmap v785: 168 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 93 active+clean, 30 active+undersized+degraded; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 662 op/s; 24408/167286 objects degraded (14.591%); 17565/167286 objects misplaced (10.500%); 18 MiB/s, 286 objects/s recovering 2024-03-26T05:17:25.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:25 smithi080 ceph-mon[40404]: pgmap v785: 168 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 93 active+clean, 30 active+undersized+degraded; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 662 op/s; 24408/167286 objects degraded (14.591%); 17565/167286 objects misplaced (10.500%); 18 MiB/s, 286 objects/s recovering 2024-03-26T05:17:25.889 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:25.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:25.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:17:25.967 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:17:25.992 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:26.020 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:26.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:26.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:26.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:25 smithi132 ceph-mon[33500]: pgmap v785: 168 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 93 active+clean, 30 active+undersized+degraded; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 662 op/s; 24408/167286 objects degraded (14.591%); 17565/167286 objects misplaced (10.500%); 18 MiB/s, 286 objects/s recovering 2024-03-26T05:17:26.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:17:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:17:26.123 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:26.168 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:26.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:26.269 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:26.319 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:26.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:26.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:26.452 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:26.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:26.554 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:26.583 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:26.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:26.685 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:26.730 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:26.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:26.831 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:26.885 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:26.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:26.987 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:27.026 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:27.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:27.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:27.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:26 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 24408/167286 objects degraded (14.591%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:27.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:26 smithi132 ceph-mon[33500]: osdmap e381: 8 total, 7 up, 8 in 2024-03-26T05:17:27.101 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:26 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 24408/167286 objects degraded (14.591%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:27.101 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:26 smithi080 ceph-mon[29691]: osdmap e381: 8 total, 7 up, 8 in 2024-03-26T05:17:27.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:26 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 24408/167286 objects degraded (14.591%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:27.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:26 smithi080 ceph-mon[40404]: osdmap e381: 8 total, 7 up, 8 in 2024-03-26T05:17:27.129 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:27.166 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:27.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:27.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:27.268 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:27.299 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:27.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:27.400 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:27.458 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:27.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:27.559 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:27.601 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:27.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:27.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:27.703 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:27.745 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:27.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:27.846 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:27.968 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:17:28.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:27 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:28.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:27 smithi132 ceph-mon[33500]: pgmap v786: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+clean+remapped, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 94 active+clean, 30 active+undersized+degraded; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 22800/154878 objects degraded (14.721%); 15268/154878 objects misplaced (9.858%); 20 MiB/s, 325 objects/s recovering 2024-03-26T05:17:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:28.097 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:28.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:28.098 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:27 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:28.098 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:27 smithi080 ceph-mon[29691]: pgmap v786: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+clean+remapped, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 94 active+clean, 30 active+undersized+degraded; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 22800/154878 objects degraded (14.721%); 15268/154878 objects misplaced (9.858%); 20 MiB/s, 325 objects/s recovering 2024-03-26T05:17:28.098 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:27 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:28.098 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:27 smithi080 ceph-mon[40404]: pgmap v786: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+clean+remapped, 3 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 94 active+clean, 30 active+undersized+degraded; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 22800/154878 objects degraded (14.721%); 15268/154878 objects misplaced (9.858%); 20 MiB/s, 325 objects/s recovering 2024-03-26T05:17:28.200 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:28.227 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:28.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:28.329 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:28.360 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:28.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:28.461 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:28.522 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:28.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:28.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:28.624 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:28.651 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:28.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:28.753 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:28.801 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:28.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:28.903 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:28.934 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:17:28.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:28.937 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:28.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:28.938 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:17:29.038 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:29.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:28 smithi132 ceph-mon[33500]: pgmap v788: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 94 active+clean, 30 active+undersized+degraded; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 891 op/s; 22141/152814 objects degraded (14.489%); 14298/152814 objects misplaced (9.356%); 22 MiB/s, 344 objects/s recovering 2024-03-26T05:17:29.069 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:29.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:29.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:29.086 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:28 smithi080 ceph-mon[40404]: pgmap v788: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 94 active+clean, 30 active+undersized+degraded; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 891 op/s; 22141/152814 objects degraded (14.489%); 14298/152814 objects misplaced (9.356%); 22 MiB/s, 344 objects/s recovering 2024-03-26T05:17:29.086 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:28 smithi080 ceph-mon[29691]: pgmap v788: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 94 active+clean, 30 active+undersized+degraded; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 891 op/s; 22141/152814 objects degraded (14.489%); 14298/152814 objects misplaced (9.356%); 22 MiB/s, 344 objects/s recovering 2024-03-26T05:17:29.171 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:29.204 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:29.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:29.307 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:29.341 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:29.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:29.443 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:29.472 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:29.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:29.574 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:29.600 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:29.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:29.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:29.701 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:29.731 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:17:29.731 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:29.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:29.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:29.833 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:29.864 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:29.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:29.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:29.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:29 smithi080 ceph-mon[40404]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:29.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:29 smithi080 ceph-mon[40404]: osdmap e382: 8 total, 7 up, 8 in 2024-03-26T05:17:29.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:29 smithi080 ceph-mon[40404]: from='client.25673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:17:29.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:29 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1559186546' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:17:29.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:29 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:17:29.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:29 smithi080 ceph-mon[29691]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:29.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:29 smithi080 ceph-mon[29691]: osdmap e382: 8 total, 7 up, 8 in 2024-03-26T05:17:29.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:29 smithi080 ceph-mon[29691]: from='client.25673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:17:29.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:29 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1559186546' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:17:29.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:29 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:17:29.966 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:30.008 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:30.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:30.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:29 smithi132 ceph-mon[33500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:30.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:29 smithi132 ceph-mon[33500]: osdmap e382: 8 total, 7 up, 8 in 2024-03-26T05:17:30.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:29 smithi132 ceph-mon[33500]: from='client.25673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:17:30.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:29 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1559186546' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:17:30.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:29 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:17:30.109 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:30.141 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:30.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:30.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:30.242 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:30.281 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:30.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:30.322 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":789,"stamp":"2024-03-26T05:17:28.736869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3323825767,"num_objects":50938,"num_object_clones":0,"num_object_copies":152814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22141,"num_objects_misplaced":14298,"num_objects_unfound":0,"num_objects_dirty":50938,"num_whiteouts":0,"num_read":1662,"num_read_kb":1408,"num_write":173698,"num_write_kb":7150785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30939,"num_bytes_recovered":2020141407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161081,"ondisk_log_size":161081,"up":433,"acting":434,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":515,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15933612,"kb_used_data":11610920,"kb_used_omap":15,"kb_used_meta":4322480,"kb_avail":734027604,"statfs":{"total":767960285184,"available":751644266496,"internally_reserved":0,"allocated":11889582080,"data_stored":11884787012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":4426220302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,9,5,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-316145664,"num_objects":-4824,"num_object_clones":0,"num_object_copies":-14472,"num_objects_missing_on_primary":0,"num_objects_missing":-1,"num_objects_degraded":-2267,"num_objects_misplaced":-3267,"num_objects_unfound":0,"num_objects_dirty":-4824,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4824,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":849,"num_bytes_recovered":55574528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.057867"},"pg_stats":[{"pgid":"4.c","version":"254'20","reported_seq":526,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:21.690424+0000","last_change":"2024-03-26T05:15:48.638251+0000","last_active":"2024-03-26T05:17:21.690424+0000","last_peered":"2024-03-26T05:17:21.690424+0000","last_clean":"2024-03-26T05:10:30.608013+0000","last_became_active":"2024-03-26T05:15:48.638251+0000","last_became_peered":"2024-03-26T05:15:48.638251+0000","last_unstale":"2024-03-26T05:17:21.690424+0000","last_undegraded":"2024-03-26T05:15:48.630289+0000","last_fullsized":"2024-03-26T05:15:48.630171+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'20","last_scrub_stamp":"2024-03-26T05:14:56.989928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:56.989928+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":702,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.693979+0000","last_change":"2024-03-26T05:15:12.643401+0000","last_active":"2024-03-26T05:17:21.693979+0000","last_peered":"2024-03-26T05:17:21.693979+0000","last_clean":"2024-03-26T05:17:21.693979+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:17:21.693979+0000","last_undegraded":"2024-03-26T05:17:21.693979+0000","last_fullsized":"2024-03-26T05:17:21.693979+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:12.642910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:15:12.642910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"336'13","reported_seq":926,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.686899+0000","last_change":"2024-03-26T05:15:32.632679+0000","last_active":"2024-03-26T05:17:21.686899+0000","last_peered":"2024-03-26T05:17:21.686899+0000","last_clean":"2024-03-26T05:17:21.686899+0000","last_became_active":"2024-03-26T05:15:32.632283+0000","last_became_peered":"2024-03-26T05:15:32.632283+0000","last_unstale":"2024-03-26T05:17:21.686899+0000","last_undegraded":"2024-03-26T05:17:21.686899+0000","last_fullsized":"2024-03-26T05:17:21.686899+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:12:08.228910+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:12:08.228910+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":84,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.682445+0000","last_change":"2024-03-26T05:15:48.650651+0000","last_active":"2024-03-26T05:17:21.682445+0000","last_peered":"2024-03-26T05:17:21.682445+0000","last_clean":"2024-03-26T05:00:27.943471+0000","last_became_active":"2024-03-26T05:15:48.650651+0000","last_became_peered":"2024-03-26T05:15:48.650651+0000","last_unstale":"2024-03-26T05:17:21.682445+0000","last_undegraded":"2024-03-26T05:17:21.682445+0000","last_fullsized":"2024-03-26T05:15:48.641490+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:54.067506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:54.067506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"380'9760","reported_seq":9908,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:25.903129+0000","last_change":"2024-03-26T05:16:51.717431+0000","last_active":"2024-03-26T05:17:25.903129+0000","last_peered":"2024-03-26T05:17:25.903129+0000","last_clean":"2024-03-26T05:15:47.613860+0000","last_became_active":"2024-03-26T05:16:51.717431+0000","last_became_peered":"2024-03-26T05:16:51.717431+0000","last_unstale":"2024-03-26T05:17:25.903129+0000","last_undegraded":"2024-03-26T05:16:51.691715+0000","last_fullsized":"2024-03-26T05:16:51.691588+0000","mapping_epoch":342,"log_start":"326'3600","ondisk_log_start":"326'3600","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6160,"log_dups_size":0,"ondisk_log_size":6160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:27.472698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105480192,"num_objects":1609,"num_object_clones":0,"num_object_copies":4827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5368,"num_write_kb":223296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1609}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"88'128","reported_seq":869,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.062525+0000","last_change":"2024-03-26T05:16:51.299599+0000","last_active":"2024-03-26T05:17:22.062525+0000","last_peered":"2024-03-26T05:17:22.062525+0000","last_clean":"2024-03-26T05:17:22.062525+0000","last_became_active":"2024-03-26T05:15:33.616505+0000","last_became_peered":"2024-03-26T05:15:33.616505+0000","last_unstale":"2024-03-26T05:17:22.062525+0000","last_undegraded":"2024-03-26T05:17:22.062525+0000","last_fullsized":"2024-03-26T05:17:22.062525+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:16:51.299504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:16:51.299504+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:18:19.972781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"254'31","reported_seq":545,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:21.687889+0000","last_change":"2024-03-26T05:15:48.650607+0000","last_active":"2024-03-26T05:17:21.687889+0000","last_peered":"2024-03-26T05:17:21.687889+0000","last_clean":"2024-03-26T05:10:30.646940+0000","last_became_active":"2024-03-26T05:15:48.650607+0000","last_became_peered":"2024-03-26T05:15:48.650607+0000","last_unstale":"2024-03-26T05:17:21.687889+0000","last_undegraded":"2024-03-26T05:15:48.634034+0000","last_fullsized":"2024-03-26T05:15:48.633929+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'31","last_scrub_stamp":"2024-03-26T05:14:50.944513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:50.944513+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":725,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.087656+0000","last_change":"2024-03-26T05:15:52.465360+0000","last_active":"2024-03-26T05:17:22.087656+0000","last_peered":"2024-03-26T05:17:22.087656+0000","last_clean":"2024-03-26T05:17:22.087656+0000","last_became_active":"2024-03-26T05:06:06.521150+0000","last_became_peered":"2024-03-26T05:06:06.521150+0000","last_unstale":"2024-03-26T05:17:22.087656+0000","last_undegraded":"2024-03-26T05:17:22.087656+0000","last_fullsized":"2024-03-26T05:17:22.087656+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:52.465232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:15:52.465232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":691,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.682940+0000","last_change":"2024-03-26T05:15:48.649219+0000","last_active":"2024-03-26T05:17:21.682940+0000","last_peered":"2024-03-26T05:17:21.682940+0000","last_clean":"2024-03-26T05:15:46.609940+0000","last_became_active":"2024-03-26T05:15:48.649219+0000","last_became_peered":"2024-03-26T05:15:48.649219+0000","last_unstale":"2024-03-26T05:17:21.682940+0000","last_undegraded":"2024-03-26T05:17:21.682940+0000","last_fullsized":"2024-03-26T05:15:48.637386+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:21.059907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:14:21.059907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":687,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.687644+0000","last_change":"2024-03-26T05:15:48.640264+0000","last_active":"2024-03-26T05:17:21.687644+0000","last_peered":"2024-03-26T05:17:21.687644+0000","last_clean":"2024-03-26T05:15:46.627568+0000","last_became_active":"2024-03-26T05:15:48.640264+0000","last_became_peered":"2024-03-26T05:15:48.640264+0000","last_unstale":"2024-03-26T05:17:21.687644+0000","last_undegraded":"2024-03-26T05:17:21.687644+0000","last_fullsized":"2024-03-26T05:15:48.632563+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:19.269035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:19.269035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"380'10056","reported_seq":10204,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:25.799426+0000","last_change":"2024-03-26T05:16:51.706396+0000","last_active":"2024-03-26T05:17:25.799426+0000","last_peered":"2024-03-26T05:17:25.799426+0000","last_clean":"2024-03-26T05:17:25.799426+0000","last_became_active":"2024-03-26T05:16:51.696073+0000","last_became_peered":"2024-03-26T05:16:51.696073+0000","last_unstale":"2024-03-26T05:17:25.799426+0000","last_undegraded":"2024-03-26T05:17:25.799426+0000","last_fullsized":"2024-03-26T05:17:25.799426+0000","mapping_epoch":316,"log_start":"322'3100","ondisk_log_start":"322'3100","created":316,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6956,"log_dups_size":0,"ondisk_log_size":6956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:29.149923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105807872,"num_objects":1614,"num_object_clones":0,"num_object_copies":4842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5530,"num_write_kb":228640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"88'128","reported_seq":501,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.680574+0000","last_change":"2024-03-26T05:16:46.320882+0000","last_active":"2024-03-26T05:17:21.680574+0000","last_peered":"2024-03-26T05:17:21.680574+0000","last_clean":"2024-03-26T05:17:21.680574+0000","last_became_active":"2024-03-26T05:15:22.616949+0000","last_became_peered":"2024-03-26T05:15:22.616949+0000","last_unstale":"2024-03-26T05:17:21.680574+0000","last_undegraded":"2024-03-26T05:17:21.680574+0000","last_fullsized":"2024-03-26T05:17:21.680574+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:16:46.320738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:16:46.320738+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:18:08.300389+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"254'25","reported_seq":539,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:22.088813+0000","last_change":"2024-03-26T05:15:48.646310+0000","last_active":"2024-03-26T05:17:22.088813+0000","last_peered":"2024-03-26T05:17:22.088813+0000","last_clean":"2024-03-26T05:10:30.649037+0000","last_became_active":"2024-03-26T05:15:48.646310+0000","last_became_peered":"2024-03-26T05:15:48.646310+0000","last_unstale":"2024-03-26T05:17:22.088813+0000","last_undegraded":"2024-03-26T05:15:48.632255+0000","last_fullsized":"2024-03-26T05:15:48.632148+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:13:04.195003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:04.195003+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":6}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":676,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.685068+0000","last_change":"2024-03-26T05:15:48.641615+0000","last_active":"2024-03-26T05:17:21.685068+0000","last_peered":"2024-03-26T05:17:21.685068+0000","last_clean":"2024-03-26T05:12:19.732296+0000","last_became_active":"2024-03-26T05:15:48.641615+0000","last_became_peered":"2024-03-26T05:15:48.641615+0000","last_unstale":"2024-03-26T05:17:21.685068+0000","last_undegraded":"2024-03-26T05:17:21.685068+0000","last_fullsized":"2024-03-26T05:15:48.632821+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:29.465975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.465975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":373,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.681815+0000","last_change":"2024-03-26T05:15:48.643225+0000","last_active":"2024-03-26T05:17:21.681815+0000","last_peered":"2024-03-26T05:17:21.681815+0000","last_clean":"2024-03-26T05:15:46.630428+0000","last_became_active":"2024-03-26T05:15:48.643225+0000","last_became_peered":"2024-03-26T05:15:48.643225+0000","last_unstale":"2024-03-26T05:17:21.681815+0000","last_undegraded":"2024-03-26T05:17:21.681815+0000","last_fullsized":"2024-03-26T05:15:48.630436+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:56.632109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:13:07.883082+0000","last_clean_scrub_stamp":"2024-03-26T05:14:56.632109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":696,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.690101+0000","last_change":"2024-03-26T05:16:41.068268+0000","last_active":"2024-03-26T05:17:21.690101+0000","last_peered":"2024-03-26T05:17:21.690101+0000","last_clean":"2024-03-26T05:17:21.690101+0000","last_became_active":"2024-03-26T05:07:59.957995+0000","last_became_peered":"2024-03-26T05:07:59.957995+0000","last_unstale":"2024-03-26T05:17:21.690101+0000","last_undegraded":"2024-03-26T05:17:21.690101+0000","last_fullsized":"2024-03-26T05:17:21.690101+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:41.068156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:16:41.068156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:58.651799+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"380'9756","reported_seq":9904,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:24.752543+0000","last_change":"2024-03-26T05:16:51.701677+0000","last_active":"2024-03-26T05:17:24.752543+0000","last_peered":"2024-03-26T05:17:24.752543+0000","last_clean":"2024-03-26T05:15:47.606254+0000","last_became_active":"2024-03-26T05:16:51.701677+0000","last_became_peered":"2024-03-26T05:16:51.701677+0000","last_unstale":"2024-03-26T05:17:24.752543+0000","last_undegraded":"2024-03-26T05:16:51.687625+0000","last_fullsized":"2024-03-26T05:16:51.687489+0000","mapping_epoch":342,"log_start":"322'3200","ondisk_log_start":"322'3200","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6556,"log_dups_size":0,"ondisk_log_size":6556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:25.143857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106037248,"num_objects":1618,"num_object_clones":0,"num_object_copies":4854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1618,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5342,"num_write_kb":222720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":1618}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"292'34","reported_seq":753,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:21.683007+0000","last_change":"2024-03-26T05:15:48.644946+0000","last_active":"2024-03-26T05:17:21.683007+0000","last_peered":"2024-03-26T05:17:21.683007+0000","last_clean":"2024-03-26T05:15:46.609501+0000","last_became_active":"2024-03-26T05:15:48.644946+0000","last_became_peered":"2024-03-26T05:15:48.644946+0000","last_unstale":"2024-03-26T05:17:21.683007+0000","last_undegraded":"2024-03-26T05:15:48.636746+0000","last_fullsized":"2024-03-26T05:15:48.636639+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'34","last_scrub_stamp":"2024-03-26T05:15:43.090549+0000","last_deep_scrub":"292'34","last_deep_scrub_stamp":"2024-03-26T05:12:23.778630+0000","last_clean_scrub_stamp":"2024-03-26T05:15:43.090549+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":689,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.688949+0000","last_change":"2024-03-26T05:15:42.617357+0000","last_active":"2024-03-26T05:17:21.688949+0000","last_peered":"2024-03-26T05:17:21.688949+0000","last_clean":"2024-03-26T05:17:21.688949+0000","last_became_active":"2024-03-26T05:15:32.626265+0000","last_became_peered":"2024-03-26T05:15:32.626265+0000","last_unstale":"2024-03-26T05:17:21.688949+0000","last_undegraded":"2024-03-26T05:17:21.688949+0000","last_fullsized":"2024-03-26T05:17:21.688949+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:42.617232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:15:42.617232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"343'11","reported_seq":534,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:21.681562+0000","last_change":"2024-03-26T05:15:48.648004+0000","last_active":"2024-03-26T05:17:21.681562+0000","last_peered":"2024-03-26T05:17:21.681562+0000","last_clean":"2024-03-26T05:07:58.484584+0000","last_became_active":"2024-03-26T05:15:48.648004+0000","last_became_peered":"2024-03-26T05:15:48.648004+0000","last_unstale":"2024-03-26T05:17:21.681562+0000","last_undegraded":"2024-03-26T05:15:48.629532+0000","last_fullsized":"2024-03-26T05:15:48.629408+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:14:03.822423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:03.822423+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":370,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681289+0000","last_change":"2024-03-26T05:15:02.690388+0000","last_active":"2024-03-26T05:17:21.681289+0000","last_peered":"2024-03-26T05:17:21.681289+0000","last_clean":"2024-03-26T05:17:21.681289+0000","last_became_active":"2024-03-26T05:08:00.005676+0000","last_became_peered":"2024-03-26T05:08:00.005676+0000","last_unstale":"2024-03-26T05:17:21.681289+0000","last_undegraded":"2024-03-26T05:17:21.681289+0000","last_fullsized":"2024-03-26T05:17:21.681289+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:02.690252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_clean_scrub_stamp":"2024-03-26T05:15:02.690252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1","version":"380'9801","reported_seq":20782,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:25.776530+0000","last_change":"2024-03-26T05:16:51.719352+0000","last_active":"2024-03-26T05:17:25.776530+0000","last_peered":"2024-03-26T05:17:25.776530+0000","last_clean":"2024-03-26T05:14:40.645004+0000","last_became_active":"2024-03-26T05:16:51.719352+0000","last_became_peered":"2024-03-26T05:16:51.719352+0000","last_unstale":"2024-03-26T05:17:25.776530+0000","last_undegraded":"2024-03-26T05:16:51.687190+0000","last_fullsized":"2024-03-26T05:16:51.687045+0000","mapping_epoch":344,"log_start":"327'3800","ondisk_log_start":"327'3800","created":316,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6001,"log_dups_size":0,"ondisk_log_size":6001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:30.387753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102236160,"num_objects":1560,"num_object_clones":0,"num_object_copies":4680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5396,"num_write_kb":222592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2707,"num_bytes_recovered":177438720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1560}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"88'128","reported_seq":501,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.680793+0000","last_change":"2024-03-26T05:16:48.345452+0000","last_active":"2024-03-26T05:17:21.680793+0000","last_peered":"2024-03-26T05:17:21.680793+0000","last_clean":"2024-03-26T05:17:21.680793+0000","last_became_active":"2024-03-26T05:15:22.622542+0000","last_became_peered":"2024-03-26T05:15:22.622542+0000","last_unstale":"2024-03-26T05:17:21.680793+0000","last_undegraded":"2024-03-26T05:17:21.680793+0000","last_fullsized":"2024-03-26T05:17:21.680793+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:16:48.345307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:16:48.345307+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:18:06.922101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":661,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.689171+0000","last_change":"2024-03-26T05:15:48.640538+0000","last_active":"2024-03-26T05:17:21.689171+0000","last_peered":"2024-03-26T05:17:21.689171+0000","last_clean":"2024-03-26T05:12:19.732211+0000","last_became_active":"2024-03-26T05:15:48.640538+0000","last_became_peered":"2024-03-26T05:15:48.640538+0000","last_unstale":"2024-03-26T05:17:21.689171+0000","last_undegraded":"2024-03-26T05:17:21.689171+0000","last_fullsized":"2024-03-26T05:15:48.628482+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.411139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.411139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"254'52","reported_seq":778,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.689147+0000","last_change":"2024-03-26T05:15:54.719734+0000","last_active":"2024-03-26T05:17:21.689147+0000","last_peered":"2024-03-26T05:17:21.689147+0000","last_clean":"2024-03-26T05:17:21.689147+0000","last_became_active":"2024-03-26T05:15:32.636466+0000","last_became_peered":"2024-03-26T05:15:32.636466+0000","last_unstale":"2024-03-26T05:17:21.689147+0000","last_undegraded":"2024-03-26T05:17:21.689147+0000","last_fullsized":"2024-03-26T05:17:21.689147+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'52","last_scrub_stamp":"2024-03-26T05:15:54.719604+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:15:54.719604+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":689,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.691384+0000","last_change":"2024-03-26T05:15:58.725078+0000","last_active":"2024-03-26T05:17:21.691384+0000","last_peered":"2024-03-26T05:17:21.691384+0000","last_clean":"2024-03-26T05:17:21.691384+0000","last_became_active":"2024-03-26T05:15:32.636983+0000","last_became_peered":"2024-03-26T05:15:32.636983+0000","last_unstale":"2024-03-26T05:17:21.691384+0000","last_undegraded":"2024-03-26T05:17:21.691384+0000","last_fullsized":"2024-03-26T05:17:21.691384+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:58.724953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:15:58.724953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":84,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.689725+0000","last_change":"2024-03-26T05:15:48.635004+0000","last_active":"2024-03-26T05:17:21.689725+0000","last_peered":"2024-03-26T05:17:21.689725+0000","last_clean":"2024-03-26T05:00:27.943471+0000","last_became_active":"2024-03-26T05:15:48.635004+0000","last_became_peered":"2024-03-26T05:15:48.635004+0000","last_unstale":"2024-03-26T05:17:21.689725+0000","last_undegraded":"2024-03-26T05:17:21.689725+0000","last_fullsized":"2024-03-26T05:15:48.630635+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:54.923744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:13:54.923744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"380'10568","reported_seq":10712,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:24.746107+0000","last_change":"2024-03-26T05:15:48.637088+0000","last_active":"2024-03-26T05:17:24.746107+0000","last_peered":"2024-03-26T05:17:24.746107+0000","last_clean":"2024-03-26T05:15:47.581062+0000","last_became_active":"2024-03-26T05:15:48.637088+0000","last_became_peered":"2024-03-26T05:15:48.637088+0000","last_unstale":"2024-03-26T05:17:24.746107+0000","last_undegraded":"2024-03-26T05:15:48.631634+0000","last_fullsized":"2024-03-26T05:15:48.631526+0000","mapping_epoch":342,"log_start":"340'5316","ondisk_log_start":"340'5316","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5252,"log_dups_size":0,"ondisk_log_size":5252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:27.134601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210894848,"num_objects":3218,"num_object_clones":0,"num_object_copies":9654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10568,"num_write_kb":441152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":3218}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"88'128","reported_seq":463,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.686189+0000","last_change":"2024-03-26T05:15:48.644026+0000","last_active":"2024-03-26T05:17:21.686189+0000","last_peered":"2024-03-26T05:17:21.686189+0000","last_clean":"2024-03-26T05:01:05.801809+0000","last_became_active":"2024-03-26T05:15:48.644026+0000","last_became_peered":"2024-03-26T05:15:48.644026+0000","last_unstale":"2024-03-26T05:17:21.686189+0000","last_undegraded":"2024-03-26T05:17:21.686189+0000","last_fullsized":"2024-03-26T05:15:48.633510+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:23.172131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"343'9","reported_seq":890,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:21.689347+0000","last_change":"2024-03-26T05:15:48.648617+0000","last_active":"2024-03-26T05:17:21.689347+0000","last_peered":"2024-03-26T05:17:21.689347+0000","last_clean":"2024-03-26T05:15:46.625052+0000","last_became_active":"2024-03-26T05:15:48.648617+0000","last_became_peered":"2024-03-26T05:15:48.648617+0000","last_unstale":"2024-03-26T05:17:21.689347+0000","last_undegraded":"2024-03-26T05:15:48.628872+0000","last_fullsized":"2024-03-26T05:15:48.628757+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:13:56.721250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:13:56.721250+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"254'34","reported_seq":598,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:21.681120+0000","last_change":"2024-03-26T05:15:48.656188+0000","last_active":"2024-03-26T05:17:21.681120+0000","last_peered":"2024-03-26T05:17:21.681120+0000","last_clean":"2024-03-26T05:10:30.652936+0000","last_became_active":"2024-03-26T05:15:48.656188+0000","last_became_peered":"2024-03-26T05:15:48.656188+0000","last_unstale":"2024-03-26T05:17:21.681120+0000","last_undegraded":"2024-03-26T05:15:48.628982+0000","last_fullsized":"2024-03-26T05:15:48.628847+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'34","last_scrub_stamp":"2024-03-26T05:14:59.971454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:59.971454+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":710,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:22.095287+0000","last_change":"2024-03-26T05:15:48.646902+0000","last_active":"2024-03-26T05:17:22.095287+0000","last_peered":"2024-03-26T05:17:22.095287+0000","last_clean":"2024-03-26T05:15:46.622472+0000","last_became_active":"2024-03-26T05:15:48.646902+0000","last_became_peered":"2024-03-26T05:15:48.646902+0000","last_unstale":"2024-03-26T05:17:22.095287+0000","last_undegraded":"2024-03-26T05:15:48.640683+0000","last_fullsized":"2024-03-26T05:15:48.640556+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:12.113993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:12.113993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":595,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.067292+0000","last_change":"2024-03-26T05:15:32.630784+0000","last_active":"2024-03-26T05:17:22.067292+0000","last_peered":"2024-03-26T05:17:22.067292+0000","last_clean":"2024-03-26T05:17:22.067292+0000","last_became_active":"2024-03-26T05:15:32.630135+0000","last_became_peered":"2024-03-26T05:15:32.630135+0000","last_unstale":"2024-03-26T05:17:22.067292+0000","last_undegraded":"2024-03-26T05:17:22.067292+0000","last_fullsized":"2024-03-26T05:17:22.067292+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:22.910006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:22.910006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"380'10457","reported_seq":10664,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:24.774489+0000","last_change":"2024-03-26T05:16:40.056489+0000","last_active":"2024-03-26T05:17:24.774489+0000","last_peered":"2024-03-26T05:17:24.774489+0000","last_clean":"2024-03-26T05:17:24.774489+0000","last_became_active":"2024-03-26T05:14:41.668381+0000","last_became_peered":"2024-03-26T05:14:41.668381+0000","last_unstale":"2024-03-26T05:17:24.774489+0000","last_undegraded":"2024-03-26T05:17:24.774489+0000","last_fullsized":"2024-03-26T05:17:24.774489+0000","mapping_epoch":316,"log_start":"340'5274","ondisk_log_start":"340'5274","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'7950","last_scrub_stamp":"2024-03-26T05:16:40.056228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:16:40.056228+0000","objects_scrubbed":5662,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:41.487437+0000","scrub_duration":1936,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202047488,"num_objects":3083,"num_object_clones":0,"num_object_copies":9249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":433280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"88'128","reported_seq":506,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.688846+0000","last_change":"2024-03-26T05:15:39.542943+0000","last_active":"2024-03-26T05:17:21.688846+0000","last_peered":"2024-03-26T05:17:21.688846+0000","last_clean":"2024-03-26T05:17:21.688846+0000","last_became_active":"2024-03-26T05:15:33.617734+0000","last_became_peered":"2024-03-26T05:15:33.617734+0000","last_unstale":"2024-03-26T05:17:21.688846+0000","last_undegraded":"2024-03-26T05:17:21.688846+0000","last_fullsized":"2024-03-26T05:17:21.688846+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:15:39.542698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:15:39.542698+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"216'11","reported_seq":934,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.094673+0000","last_change":"2024-03-26T05:14:48.976961+0000","last_active":"2024-03-26T05:17:22.094673+0000","last_peered":"2024-03-26T05:17:22.094673+0000","last_clean":"2024-03-26T05:17:22.094673+0000","last_became_active":"2024-03-26T05:07:59.994682+0000","last_became_peered":"2024-03-26T05:07:59.994682+0000","last_unstale":"2024-03-26T05:17:22.094673+0000","last_undegraded":"2024-03-26T05:17:22.094673+0000","last_fullsized":"2024-03-26T05:17:22.094673+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:14:48.976873+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:14:48.976873+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:28.138712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"254'26","reported_seq":748,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:21.686699+0000","last_change":"2024-03-26T05:15:48.638710+0000","last_active":"2024-03-26T05:17:21.686699+0000","last_peered":"2024-03-26T05:17:21.686699+0000","last_clean":"2024-03-26T05:15:46.626893+0000","last_became_active":"2024-03-26T05:15:48.638710+0000","last_became_peered":"2024-03-26T05:15:48.638710+0000","last_unstale":"2024-03-26T05:17:21.686699+0000","last_undegraded":"2024-03-26T05:15:48.631213+0000","last_fullsized":"2024-03-26T05:15:48.631104+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:14:05.219767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:05.219767+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":370,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681438+0000","last_change":"2024-03-26T05:14:15.770146+0000","last_active":"2024-03-26T05:17:21.681438+0000","last_peered":"2024-03-26T05:17:21.681438+0000","last_clean":"2024-03-26T05:17:21.681438+0000","last_became_active":"2024-03-26T05:07:59.960768+0000","last_became_peered":"2024-03-26T05:07:59.960768+0000","last_unstale":"2024-03-26T05:17:21.681438+0000","last_undegraded":"2024-03-26T05:17:21.681438+0000","last_fullsized":"2024-03-26T05:17:21.681438+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:15.770027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:15.770027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":687,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.092790+0000","last_change":"2024-03-26T05:14:07.165543+0000","last_active":"2024-03-26T05:17:22.092790+0000","last_peered":"2024-03-26T05:17:22.092790+0000","last_clean":"2024-03-26T05:17:22.092790+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:17:22.092790+0000","last_undegraded":"2024-03-26T05:17:22.092790+0000","last_fullsized":"2024-03-26T05:17:22.092790+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:07.165430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:07.165430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:22.094292+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"380'11003","reported_seq":22516,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:25.844161+0000","last_change":"2024-03-26T05:16:01.519823+0000","last_active":"2024-03-26T05:17:25.844161+0000","last_peered":"2024-03-26T05:17:25.844161+0000","last_clean":"2024-03-26T05:14:40.645004+0000","last_became_active":"2024-03-26T05:16:01.519823+0000","last_became_peered":"2024-03-26T05:16:01.519823+0000","last_unstale":"2024-03-26T05:17:25.844161+0000","last_undegraded":"2024-03-26T05:16:01.505259+0000","last_fullsized":"2024-03-26T05:16:01.505132+0000","mapping_epoch":347,"log_start":"350'6261","ondisk_log_start":"350'6261","created":316,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":4742,"log_dups_size":0,"ondisk_log_size":4742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:30.388187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208863232,"num_objects":3187,"num_object_clones":0,"num_object_copies":9561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11003,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5683,"num_bytes_recovered":372441088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":3187}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"88'128","reported_seq":438,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.088454+0000","last_change":"2024-03-26T05:15:40.586982+0000","last_active":"2024-03-26T05:17:22.088454+0000","last_peered":"2024-03-26T05:17:22.088454+0000","last_clean":"2024-03-26T05:17:22.088454+0000","last_became_active":"2024-03-26T05:15:33.617538+0000","last_became_peered":"2024-03-26T05:15:33.617538+0000","last_unstale":"2024-03-26T05:17:22.088454+0000","last_undegraded":"2024-03-26T05:17:22.088454+0000","last_fullsized":"2024-03-26T05:17:22.088454+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:15:40.586852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:15:40.586852+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":724,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.088025+0000","last_change":"2024-03-26T05:16:41.114415+0000","last_active":"2024-03-26T05:17:22.088025+0000","last_peered":"2024-03-26T05:17:22.088025+0000","last_clean":"2024-03-26T05:17:22.088025+0000","last_became_active":"2024-03-26T05:07:59.991484+0000","last_became_peered":"2024-03-26T05:07:59.991484+0000","last_unstale":"2024-03-26T05:17:22.088025+0000","last_undegraded":"2024-03-26T05:17:22.088025+0000","last_fullsized":"2024-03-26T05:17:22.088025+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:41.114180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:16:41.114180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:51.320864+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"254'44","reported_seq":828,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.094122+0000","last_change":"2024-03-26T05:16:50.954039+0000","last_active":"2024-03-26T05:17:22.094122+0000","last_peered":"2024-03-26T05:17:22.094122+0000","last_clean":"2024-03-26T05:17:22.094122+0000","last_became_active":"2024-03-26T05:15:32.634000+0000","last_became_peered":"2024-03-26T05:15:32.634000+0000","last_unstale":"2024-03-26T05:17:22.094122+0000","last_undegraded":"2024-03-26T05:17:22.094122+0000","last_fullsized":"2024-03-26T05:17:22.094122+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'44","last_scrub_stamp":"2024-03-26T05:16:50.953849+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:16:50.953849+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:18:13.949042+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":717,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681991+0000","last_change":"2024-03-26T05:16:06.271601+0000","last_active":"2024-03-26T05:17:21.681991+0000","last_peered":"2024-03-26T05:17:21.681991+0000","last_clean":"2024-03-26T05:17:21.681991+0000","last_became_active":"2024-03-26T05:08:00.003099+0000","last_became_peered":"2024-03-26T05:08:00.003099+0000","last_unstale":"2024-03-26T05:17:21.681991+0000","last_undegraded":"2024-03-26T05:17:21.681991+0000","last_fullsized":"2024-03-26T05:17:21.681991+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:06.271319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:11:29.514915+0000","last_clean_scrub_stamp":"2024-03-26T05:16:06.271319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":685,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.680502+0000","last_change":"2024-03-26T05:15:51.091029+0000","last_active":"2024-03-26T05:17:21.680502+0000","last_peered":"2024-03-26T05:17:21.680502+0000","last_clean":"2024-03-26T05:17:21.680502+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:17:21.680502+0000","last_undegraded":"2024-03-26T05:17:21.680502+0000","last_fullsized":"2024-03-26T05:17:21.680502+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:51.090933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:15:51.090933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"380'10928","reported_seq":11072,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:24.774065+0000","last_change":"2024-03-26T05:15:48.643432+0000","last_active":"2024-03-26T05:17:24.774065+0000","last_peered":"2024-03-26T05:17:24.774065+0000","last_clean":"2024-03-26T05:15:47.616159+0000","last_became_active":"2024-03-26T05:15:48.643432+0000","last_became_peered":"2024-03-26T05:15:48.643432+0000","last_unstale":"2024-03-26T05:17:24.774065+0000","last_undegraded":"2024-03-26T05:15:48.633364+0000","last_fullsized":"2024-03-26T05:15:48.633251+0000","mapping_epoch":342,"log_start":"345'5665","ondisk_log_start":"345'5665","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":5263,"log_dups_size":0,"ondisk_log_size":5263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:27.135206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212795416,"num_objects":3248,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10928,"num_write_kb":453569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":3248}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"88'128","reported_seq":440,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.088380+0000","last_change":"2024-03-26T05:16:47.136752+0000","last_active":"2024-03-26T05:17:22.088380+0000","last_peered":"2024-03-26T05:17:22.088380+0000","last_clean":"2024-03-26T05:17:22.088380+0000","last_became_active":"2024-03-26T05:15:22.624234+0000","last_became_peered":"2024-03-26T05:15:22.624234+0000","last_unstale":"2024-03-26T05:17:22.088380+0000","last_undegraded":"2024-03-26T05:17:22.088380+0000","last_fullsized":"2024-03-26T05:17:22.088380+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:16:47.136641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:16:47.136641+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:55.841948+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":370,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681866+0000","last_change":"2024-03-26T05:14:12.823709+0000","last_active":"2024-03-26T05:17:21.681866+0000","last_peered":"2024-03-26T05:17:21.681866+0000","last_clean":"2024-03-26T05:17:21.681866+0000","last_became_active":"2024-03-26T05:07:59.999990+0000","last_became_peered":"2024-03-26T05:07:59.999990+0000","last_unstale":"2024-03-26T05:17:21.681866+0000","last_undegraded":"2024-03-26T05:17:21.681866+0000","last_fullsized":"2024-03-26T05:17:21.681866+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:12.823589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:12.823589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"254'30","reported_seq":776,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.095170+0000","last_change":"2024-03-26T05:14:10.147920+0000","last_active":"2024-03-26T05:17:22.095170+0000","last_peered":"2024-03-26T05:17:22.095170+0000","last_clean":"2024-03-26T05:17:22.095170+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:17:22.095170+0000","last_undegraded":"2024-03-26T05:17:22.095170+0000","last_fullsized":"2024-03-26T05:17:22.095170+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'30","last_scrub_stamp":"2024-03-26T05:14:10.147766+0000","last_deep_scrub":"187'18","last_deep_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_clean_scrub_stamp":"2024-03-26T05:14:10.147766+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":1419,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:23.922476+0000","last_change":"2024-03-26T05:15:32.637254+0000","last_active":"2024-03-26T05:17:23.922476+0000","last_peered":"2024-03-26T05:17:23.922476+0000","last_clean":"2024-03-26T05:17:23.922476+0000","last_became_active":"2024-03-26T05:15:32.633061+0000","last_became_peered":"2024-03-26T05:15:32.633061+0000","last_unstale":"2024-03-26T05:17:23.922476+0000","last_undegraded":"2024-03-26T05:17:23.922476+0000","last_fullsized":"2024-03-26T05:17:23.922476+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:12:02.961845+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:12:02.961845+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":178,"num_read_kb":173,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":416,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.093078+0000","last_change":"2024-03-26T05:15:48.659891+0000","last_active":"2024-03-26T05:17:22.093078+0000","last_peered":"2024-03-26T05:17:22.093078+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:15:48.659891+0000","last_became_peered":"2024-03-26T05:15:48.659891+0000","last_unstale":"2024-03-26T05:17:22.093078+0000","last_undegraded":"2024-03-26T05:17:22.093078+0000","last_fullsized":"2024-03-26T05:15:48.638273+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:16.863389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.863389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:22.094341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":705,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681797+0000","last_change":"2024-03-26T05:15:47.053940+0000","last_active":"2024-03-26T05:17:21.681797+0000","last_peered":"2024-03-26T05:17:21.681797+0000","last_clean":"2024-03-26T05:17:21.681797+0000","last_became_active":"2024-03-26T05:08:00.001657+0000","last_became_peered":"2024-03-26T05:08:00.001657+0000","last_unstale":"2024-03-26T05:17:21.681797+0000","last_undegraded":"2024-03-26T05:17:21.681797+0000","last_fullsized":"2024-03-26T05:17:21.681797+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:47.053815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:47.053815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"380'9777","reported_seq":17289,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:26.514664+0000","last_change":"2024-03-26T05:16:55.766002+0000","last_active":"2024-03-26T05:17:26.514664+0000","last_peered":"2024-03-26T05:17:26.514664+0000","last_clean":"2024-03-26T05:17:26.514664+0000","last_became_active":"2024-03-26T05:16:55.753413+0000","last_became_peered":"2024-03-26T05:16:55.753413+0000","last_unstale":"2024-03-26T05:17:26.514664+0000","last_undegraded":"2024-03-26T05:17:26.514664+0000","last_fullsized":"2024-03-26T05:17:26.514664+0000","mapping_epoch":367,"log_start":"327'3700","ondisk_log_start":"327'3700","created":316,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6077,"log_dups_size":0,"ondisk_log_size":6077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:27.147449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99549184,"num_objects":1519,"num_object_clones":0,"num_object_copies":4557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5405,"num_write_kb":221568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2341,"num_bytes_recovered":153452544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"88'128","reported_seq":504,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.681205+0000","last_change":"2024-03-26T05:15:48.647616+0000","last_active":"2024-03-26T05:17:21.681205+0000","last_peered":"2024-03-26T05:17:21.681205+0000","last_clean":"2024-03-26T05:15:46.610234+0000","last_became_active":"2024-03-26T05:15:48.647616+0000","last_became_peered":"2024-03-26T05:15:48.647616+0000","last_unstale":"2024-03-26T05:17:21.681205+0000","last_undegraded":"2024-03-26T05:17:21.681205+0000","last_fullsized":"2024-03-26T05:15:48.634886+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:15:44.054471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:15:44.054471+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":616,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.063171+0000","last_change":"2024-03-26T05:15:32.638848+0000","last_active":"2024-03-26T05:17:22.063171+0000","last_peered":"2024-03-26T05:17:22.063171+0000","last_clean":"2024-03-26T05:17:22.063171+0000","last_became_active":"2024-03-26T05:15:32.637320+0000","last_became_peered":"2024-03-26T05:15:32.637320+0000","last_unstale":"2024-03-26T05:17:22.063171+0000","last_undegraded":"2024-03-26T05:17:22.063171+0000","last_fullsized":"2024-03-26T05:17:22.063171+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:14.948209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:14.948209+0000","last_clean_scrub_stamp":"2024-03-26T05:12:14.948209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"343'39","reported_seq":968,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:22.088607+0000","last_change":"2024-03-26T05:15:48.640970+0000","last_active":"2024-03-26T05:17:22.088607+0000","last_peered":"2024-03-26T05:17:22.088607+0000","last_clean":"2024-03-26T05:15:46.625517+0000","last_became_active":"2024-03-26T05:15:48.640970+0000","last_became_peered":"2024-03-26T05:15:48.640970+0000","last_unstale":"2024-03-26T05:17:22.088607+0000","last_undegraded":"2024-03-26T05:15:48.633766+0000","last_fullsized":"2024-03-26T05:15:48.633639+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'38","last_scrub_stamp":"2024-03-26T05:15:46.434771+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:15:46.434771+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:27.135305+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":709,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.689374+0000","last_change":"2024-03-26T05:16:05.780331+0000","last_active":"2024-03-26T05:17:21.689374+0000","last_peered":"2024-03-26T05:17:21.689374+0000","last_clean":"2024-03-26T05:17:21.689374+0000","last_became_active":"2024-03-26T05:07:59.960061+0000","last_became_peered":"2024-03-26T05:07:59.960061+0000","last_unstale":"2024-03-26T05:17:21.689374+0000","last_undegraded":"2024-03-26T05:17:21.689374+0000","last_fullsized":"2024-03-26T05:17:21.689374+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:05.780217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:16:05.780217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:25.544931+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":863,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.063485+0000","last_change":"2024-03-26T05:15:32.635542+0000","last_active":"2024-03-26T05:17:22.063485+0000","last_peered":"2024-03-26T05:17:22.063485+0000","last_clean":"2024-03-26T05:17:22.063485+0000","last_became_active":"2024-03-26T05:15:32.632929+0000","last_became_peered":"2024-03-26T05:15:32.632929+0000","last_unstale":"2024-03-26T05:17:22.063485+0000","last_undegraded":"2024-03-26T05:17:22.063485+0000","last_fullsized":"2024-03-26T05:17:22.063485+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":381,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.681959+0000","last_change":"2024-03-26T05:15:48.643476+0000","last_active":"2024-03-26T05:17:21.681959+0000","last_peered":"2024-03-26T05:17:21.681959+0000","last_clean":"2024-03-26T05:15:46.630583+0000","last_became_active":"2024-03-26T05:15:48.643476+0000","last_became_peered":"2024-03-26T05:15:48.643476+0000","last_unstale":"2024-03-26T05:17:21.681959+0000","last_undegraded":"2024-03-26T05:17:21.681959+0000","last_fullsized":"2024-03-26T05:15:48.632538+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:14.528992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:14.528992+0000","last_clean_scrub_stamp":"2024-03-26T05:15:14.528992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"380'10681","reported_seq":10886,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:24.795814+0000","last_change":"2024-03-26T05:15:56.584965+0000","last_active":"2024-03-26T05:17:24.795814+0000","last_peered":"2024-03-26T05:17:24.795814+0000","last_clean":"2024-03-26T05:17:24.795814+0000","last_became_active":"2024-03-26T05:14:41.674748+0000","last_became_peered":"2024-03-26T05:14:41.674748+0000","last_unstale":"2024-03-26T05:17:24.795814+0000","last_undegraded":"2024-03-26T05:17:24.795814+0000","last_fullsized":"2024-03-26T05:17:24.795814+0000","mapping_epoch":316,"log_start":"341'5437","ondisk_log_start":"341'5437","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'5516","last_scrub_stamp":"2024-03-26T05:15:56.584807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:15:56.584807+0000","objects_scrubbed":5516,"log_size":5244,"log_dups_size":0,"ondisk_log_size":5244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:27.135378+0000","scrub_duration":1184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205062144,"num_objects":3129,"num_object_clones":0,"num_object_copies":9387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10681,"num_write_kb":441920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":709,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.687323+0000","last_change":"2024-03-26T05:15:32.625481+0000","last_active":"2024-03-26T05:17:21.687323+0000","last_peered":"2024-03-26T05:17:21.687323+0000","last_clean":"2024-03-26T05:17:21.687323+0000","last_became_active":"2024-03-26T05:15:32.625188+0000","last_became_peered":"2024-03-26T05:15:32.625188+0000","last_unstale":"2024-03-26T05:17:21.687323+0000","last_undegraded":"2024-03-26T05:17:21.687323+0000","last_fullsized":"2024-03-26T05:17:21.687323+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:07.171121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.171121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"254'23","reported_seq":511,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.687950+0000","last_change":"2024-03-26T05:14:55.315443+0000","last_active":"2024-03-26T05:17:21.687950+0000","last_peered":"2024-03-26T05:17:21.687950+0000","last_clean":"2024-03-26T05:17:21.687950+0000","last_became_active":"2024-03-26T05:08:00.001398+0000","last_became_peered":"2024-03-26T05:08:00.001398+0000","last_unstale":"2024-03-26T05:17:21.687950+0000","last_undegraded":"2024-03-26T05:17:21.687950+0000","last_fullsized":"2024-03-26T05:17:21.687950+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'23","last_scrub_stamp":"2024-03-26T05:14:55.315296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:55.315296+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":714,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.685978+0000","last_change":"2024-03-26T05:15:00.402058+0000","last_active":"2024-03-26T05:17:21.685978+0000","last_peered":"2024-03-26T05:17:21.685978+0000","last_clean":"2024-03-26T05:17:21.685978+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:17:21.685978+0000","last_undegraded":"2024-03-26T05:17:21.685978+0000","last_fullsized":"2024-03-26T05:17:21.685978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:00.401943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.401943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":697,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.680689+0000","last_change":"2024-03-26T05:15:48.007235+0000","last_active":"2024-03-26T05:17:21.680689+0000","last_peered":"2024-03-26T05:17:21.680689+0000","last_clean":"2024-03-26T05:17:21.680689+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:17:21.680689+0000","last_undegraded":"2024-03-26T05:17:21.680689+0000","last_fullsized":"2024-03-26T05:17:21.680689+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:48.007148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:15:48.007148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":556,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.063330+0000","last_change":"2024-03-26T05:15:32.638631+0000","last_active":"2024-03-26T05:17:22.063330+0000","last_peered":"2024-03-26T05:17:22.063330+0000","last_clean":"2024-03-26T05:17:22.063330+0000","last_became_active":"2024-03-26T05:15:32.633633+0000","last_became_peered":"2024-03-26T05:15:32.633633+0000","last_unstale":"2024-03-26T05:17:22.063330+0000","last_undegraded":"2024-03-26T05:17:22.063330+0000","last_fullsized":"2024-03-26T05:17:22.063330+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:20.054579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:20.054579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"380'10042","reported_seq":19809,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:25.331797+0000","last_change":"2024-03-26T05:16:51.733601+0000","last_active":"2024-03-26T05:17:25.331797+0000","last_peered":"2024-03-26T05:17:25.331797+0000","last_clean":"2024-03-26T05:17:25.331797+0000","last_became_active":"2024-03-26T05:16:51.719783+0000","last_became_peered":"2024-03-26T05:16:51.719783+0000","last_unstale":"2024-03-26T05:17:25.331797+0000","last_undegraded":"2024-03-26T05:17:25.331797+0000","last_fullsized":"2024-03-26T05:17:25.331797+0000","mapping_epoch":360,"log_start":"325'3700","ondisk_log_start":"325'3700","created":316,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6342,"log_dups_size":0,"ondisk_log_size":6342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:30.291427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104890368,"num_objects":1600,"num_object_clones":0,"num_object_copies":4800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":227840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2403,"num_bytes_recovered":157515776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"88'128","reported_seq":864,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:22.062704+0000","last_change":"2024-03-26T05:15:48.652567+0000","last_active":"2024-03-26T05:17:22.062704+0000","last_peered":"2024-03-26T05:17:22.062704+0000","last_clean":"2024-03-26T05:15:46.623182+0000","last_became_active":"2024-03-26T05:15:48.652567+0000","last_became_peered":"2024-03-26T05:15:48.652567+0000","last_unstale":"2024-03-26T05:17:22.062704+0000","last_undegraded":"2024-03-26T05:15:48.634567+0000","last_fullsized":"2024-03-26T05:15:48.634420+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":694,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.095113+0000","last_change":"2024-03-26T05:15:48.645632+0000","last_active":"2024-03-26T05:17:22.095113+0000","last_peered":"2024-03-26T05:17:22.095113+0000","last_clean":"2024-03-26T05:15:46.622114+0000","last_became_active":"2024-03-26T05:15:48.645632+0000","last_became_peered":"2024-03-26T05:15:48.645632+0000","last_unstale":"2024-03-26T05:17:22.095113+0000","last_undegraded":"2024-03-26T05:17:22.095113+0000","last_fullsized":"2024-03-26T05:15:48.638911+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:14.131427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:14.131427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"254'46","reported_seq":987,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.683028+0000","last_change":"2024-03-26T05:15:52.070480+0000","last_active":"2024-03-26T05:17:21.683028+0000","last_peered":"2024-03-26T05:17:21.683028+0000","last_clean":"2024-03-26T05:17:21.683028+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:17:21.683028+0000","last_undegraded":"2024-03-26T05:17:21.683028+0000","last_fullsized":"2024-03-26T05:17:21.683028+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'46","last_scrub_stamp":"2024-03-26T05:15:52.070295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:15:52.070295+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":83,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.093939+0000","last_change":"2024-03-26T05:15:48.646887+0000","last_active":"2024-03-26T05:17:22.093939+0000","last_peered":"2024-03-26T05:17:22.093939+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:15:48.646887+0000","last_became_peered":"2024-03-26T05:15:48.646887+0000","last_unstale":"2024-03-26T05:17:22.093939+0000","last_undegraded":"2024-03-26T05:17:22.093939+0000","last_fullsized":"2024-03-26T05:15:48.641087+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:02.881800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:02.881800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:22.094233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":374,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681297+0000","last_change":"2024-03-26T05:14:47.525396+0000","last_active":"2024-03-26T05:17:21.681297+0000","last_peered":"2024-03-26T05:17:21.681297+0000","last_clean":"2024-03-26T05:17:21.681297+0000","last_became_active":"2024-03-26T05:07:59.998511+0000","last_became_peered":"2024-03-26T05:07:59.998511+0000","last_unstale":"2024-03-26T05:17:21.681297+0000","last_undegraded":"2024-03-26T05:17:21.681297+0000","last_fullsized":"2024-03-26T05:17:21.681297+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:14:47.525284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:14:47.525284+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":672,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.690989+0000","last_change":"2024-03-26T05:15:48.637195+0000","last_active":"2024-03-26T05:17:21.690989+0000","last_peered":"2024-03-26T05:17:21.690989+0000","last_clean":"2024-03-26T05:15:46.628089+0000","last_became_active":"2024-03-26T05:15:48.637195+0000","last_became_peered":"2024-03-26T05:15:48.637195+0000","last_unstale":"2024-03-26T05:17:21.690989+0000","last_undegraded":"2024-03-26T05:17:21.690989+0000","last_fullsized":"2024-03-26T05:15:48.630113+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:31.518660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:31.518660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"380'11027","reported_seq":11236,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:24.786063+0000","last_change":"2024-03-26T05:16:41.705002+0000","last_active":"2024-03-26T05:17:24.786063+0000","last_peered":"2024-03-26T05:17:24.786063+0000","last_clean":"2024-03-26T05:17:24.786063+0000","last_became_active":"2024-03-26T05:14:41.671031+0000","last_became_peered":"2024-03-26T05:14:41.671031+0000","last_unstale":"2024-03-26T05:17:24.786063+0000","last_undegraded":"2024-03-26T05:17:24.786063+0000","last_fullsized":"2024-03-26T05:17:24.786063+0000","mapping_epoch":316,"log_start":"349'5824","ondisk_log_start":"349'5824","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'8504","last_scrub_stamp":"2024-03-26T05:16:41.704775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:16:41.704775+0000","objects_scrubbed":5798,"log_size":5203,"log_dups_size":0,"ondisk_log_size":5203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:55.672738+0000","scrub_duration":1647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210698240,"num_objects":3215,"num_object_clones":0,"num_object_copies":9645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11027,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"88'128","reported_seq":500,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.680580+0000","last_change":"2024-03-26T05:16:47.319821+0000","last_active":"2024-03-26T05:17:21.680580+0000","last_peered":"2024-03-26T05:17:21.680580+0000","last_clean":"2024-03-26T05:17:21.680580+0000","last_became_active":"2024-03-26T05:15:22.621431+0000","last_became_peered":"2024-03-26T05:15:22.621431+0000","last_unstale":"2024-03-26T05:17:21.680580+0000","last_undegraded":"2024-03-26T05:17:21.680580+0000","last_fullsized":"2024-03-26T05:17:21.680580+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:16:47.319697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:16:47.319697+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:18:02.123016+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"254'21","reported_seq":734,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:22.094436+0000","last_change":"2024-03-26T05:15:48.655699+0000","last_active":"2024-03-26T05:17:22.094436+0000","last_peered":"2024-03-26T05:17:22.094436+0000","last_clean":"2024-03-26T05:15:46.621863+0000","last_became_active":"2024-03-26T05:15:48.655699+0000","last_became_peered":"2024-03-26T05:15:48.655699+0000","last_unstale":"2024-03-26T05:17:22.094436+0000","last_undegraded":"2024-03-26T05:15:48.645108+0000","last_fullsized":"2024-03-26T05:15:48.644023+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:14:16.094249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.094249+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":370,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681246+0000","last_change":"2024-03-26T05:14:00.685481+0000","last_active":"2024-03-26T05:17:21.681246+0000","last_peered":"2024-03-26T05:17:21.681246+0000","last_clean":"2024-03-26T05:17:21.681246+0000","last_became_active":"2024-03-26T05:07:59.959978+0000","last_became_peered":"2024-03-26T05:07:59.959978+0000","last_unstale":"2024-03-26T05:17:21.681246+0000","last_undegraded":"2024-03-26T05:17:21.681246+0000","last_fullsized":"2024-03-26T05:17:21.681246+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:00.685364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:00.685364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":699,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.684789+0000","last_change":"2024-03-26T05:15:48.638309+0000","last_active":"2024-03-26T05:17:21.684789+0000","last_peered":"2024-03-26T05:17:21.684789+0000","last_clean":"2024-03-26T05:15:46.625678+0000","last_became_active":"2024-03-26T05:15:48.638309+0000","last_became_peered":"2024-03-26T05:15:48.638309+0000","last_unstale":"2024-03-26T05:17:21.684789+0000","last_undegraded":"2024-03-26T05:17:21.684789+0000","last_fullsized":"2024-03-26T05:15:48.631169+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:52.371799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:52.371799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":679,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.093529+0000","last_change":"2024-03-26T05:15:32.640388+0000","last_active":"2024-03-26T05:17:22.093529+0000","last_peered":"2024-03-26T05:17:22.093529+0000","last_clean":"2024-03-26T05:17:22.093529+0000","last_became_active":"2024-03-26T05:15:32.639943+0000","last_became_peered":"2024-03-26T05:15:32.639943+0000","last_unstale":"2024-03-26T05:17:22.093529+0000","last_undegraded":"2024-03-26T05:17:22.093529+0000","last_fullsized":"2024-03-26T05:17:22.093529+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:30.164093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:30.164093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:22.094142+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"380'9839","reported_seq":9987,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:25.798465+0000","last_change":"2024-03-26T05:16:51.736907+0000","last_active":"2024-03-26T05:17:25.798465+0000","last_peered":"2024-03-26T05:17:25.798465+0000","last_clean":"2024-03-26T05:17:25.798465+0000","last_became_active":"2024-03-26T05:16:51.722143+0000","last_became_peered":"2024-03-26T05:16:51.722143+0000","last_unstale":"2024-03-26T05:17:25.798465+0000","last_undegraded":"2024-03-26T05:17:25.798465+0000","last_fullsized":"2024-03-26T05:17:25.798465+0000","mapping_epoch":316,"log_start":"322'2900","ondisk_log_start":"322'2900","created":316,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6939,"log_dups_size":0,"ondisk_log_size":6939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:29.149902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104071168,"num_objects":1588,"num_object_clones":0,"num_object_copies":4764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5416,"num_write_kb":224128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"343'51","reported_seq":777,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.087802+0000","last_change":"2024-03-26T05:15:49.436480+0000","last_active":"2024-03-26T05:17:22.087802+0000","last_peered":"2024-03-26T05:17:22.087802+0000","last_clean":"2024-03-26T05:17:22.087802+0000","last_became_active":"2024-03-26T05:15:32.626640+0000","last_became_peered":"2024-03-26T05:15:32.626640+0000","last_unstale":"2024-03-26T05:17:22.087802+0000","last_undegraded":"2024-03-26T05:17:22.087802+0000","last_fullsized":"2024-03-26T05:17:22.087802+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'51","last_scrub_stamp":"2024-03-26T05:15:49.436355+0000","last_deep_scrub":"343'51","last_deep_scrub_stamp":"2024-03-26T05:15:49.436355+0000","last_clean_scrub_stamp":"2024-03-26T05:15:49.436355+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":577,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.063818+0000","last_change":"2024-03-26T05:15:32.641510+0000","last_active":"2024-03-26T05:17:22.063818+0000","last_peered":"2024-03-26T05:17:22.063818+0000","last_clean":"2024-03-26T05:17:22.063818+0000","last_became_active":"2024-03-26T05:15:32.640861+0000","last_became_peered":"2024-03-26T05:15:32.640861+0000","last_unstale":"2024-03-26T05:17:22.063818+0000","last_undegraded":"2024-03-26T05:17:22.063818+0000","last_fullsized":"2024-03-26T05:17:22.063818+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:27.963495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:27.963495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":708,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.686304+0000","last_change":"2024-03-26T05:14:10.201428+0000","last_active":"2024-03-26T05:17:21.686304+0000","last_peered":"2024-03-26T05:17:21.686304+0000","last_clean":"2024-03-26T05:17:21.686304+0000","last_became_active":"2024-03-26T05:08:00.000009+0000","last_became_peered":"2024-03-26T05:08:00.000009+0000","last_unstale":"2024-03-26T05:17:21.686304+0000","last_undegraded":"2024-03-26T05:17:21.686304+0000","last_fullsized":"2024-03-26T05:17:21.686304+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:10.201320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:10.201320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":588,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.062459+0000","last_change":"2024-03-26T05:15:32.638404+0000","last_active":"2024-03-26T05:17:22.062459+0000","last_peered":"2024-03-26T05:17:22.062459+0000","last_clean":"2024-03-26T05:17:22.062459+0000","last_became_active":"2024-03-26T05:15:32.633481+0000","last_became_peered":"2024-03-26T05:15:32.633481+0000","last_unstale":"2024-03-26T05:17:22.062459+0000","last_undegraded":"2024-03-26T05:17:22.062459+0000","last_fullsized":"2024-03-26T05:17:22.062459+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:05.986499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_clean_scrub_stamp":"2024-03-26T05:12:05.986499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"380'9720","reported_seq":9865,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:26.486589+0000","last_change":"2024-03-26T05:16:51.699133+0000","last_active":"2024-03-26T05:17:26.486589+0000","last_peered":"2024-03-26T05:17:26.486589+0000","last_clean":"2024-03-26T05:15:47.586253+0000","last_became_active":"2024-03-26T05:16:51.699133+0000","last_became_peered":"2024-03-26T05:16:51.699133+0000","last_unstale":"2024-03-26T05:17:26.486589+0000","last_undegraded":"2024-03-26T05:16:51.685340+0000","last_fullsized":"2024-03-26T05:16:51.685217+0000","mapping_epoch":342,"log_start":"327'3700","ondisk_log_start":"327'3700","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6020,"log_dups_size":0,"ondisk_log_size":6020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:29.188599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99483648,"num_objects":1518,"num_object_clones":0,"num_object_copies":4554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":220288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1518}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":688,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.684768+0000","last_change":"2024-03-26T05:14:01.337189+0000","last_active":"2024-03-26T05:17:21.684768+0000","last_peered":"2024-03-26T05:17:21.684768+0000","last_clean":"2024-03-26T05:17:21.684768+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:17:21.684768+0000","last_undegraded":"2024-03-26T05:17:21.684768+0000","last_fullsized":"2024-03-26T05:17:21.684768+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:01.336959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.432732+0000","last_clean_scrub_stamp":"2024-03-26T05:14:01.336959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"254'37","reported_seq":559,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:21.680935+0000","last_change":"2024-03-26T05:15:48.645453+0000","last_active":"2024-03-26T05:17:21.680935+0000","last_peered":"2024-03-26T05:17:21.680935+0000","last_clean":"2024-03-26T05:10:30.473392+0000","last_became_active":"2024-03-26T05:15:48.645453+0000","last_became_peered":"2024-03-26T05:15:48.645453+0000","last_unstale":"2024-03-26T05:17:21.680935+0000","last_undegraded":"2024-03-26T05:15:48.634195+0000","last_fullsized":"2024-03-26T05:15:48.634068+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-26T05:14:55.963773+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_clean_scrub_stamp":"2024-03-26T05:14:55.963773+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":610,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.063714+0000","last_change":"2024-03-26T05:15:48.650267+0000","last_active":"2024-03-26T05:17:22.063714+0000","last_peered":"2024-03-26T05:17:22.063714+0000","last_clean":"2024-03-26T05:15:46.624664+0000","last_became_active":"2024-03-26T05:15:48.650267+0000","last_became_peered":"2024-03-26T05:15:48.650267+0000","last_unstale":"2024-03-26T05:17:22.063714+0000","last_undegraded":"2024-03-26T05:17:22.063714+0000","last_fullsized":"2024-03-26T05:15:48.633889+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:20.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:20.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":685,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.694266+0000","last_change":"2024-03-26T05:15:48.646428+0000","last_active":"2024-03-26T05:17:21.694266+0000","last_peered":"2024-03-26T05:17:21.694266+0000","last_clean":"2024-03-26T05:15:46.626191+0000","last_became_active":"2024-03-26T05:15:48.646428+0000","last_became_peered":"2024-03-26T05:15:48.646428+0000","last_unstale":"2024-03-26T05:17:21.694266+0000","last_undegraded":"2024-03-26T05:17:21.694266+0000","last_fullsized":"2024-03-26T05:15:48.631162+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:51.030982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:51.030982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"380'9855","reported_seq":10067,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:24.748104+0000","last_change":"2024-03-26T05:16:51.713041+0000","last_active":"2024-03-26T05:17:24.748104+0000","last_peered":"2024-03-26T05:17:24.748104+0000","last_clean":"2024-03-26T05:17:24.748104+0000","last_became_active":"2024-03-26T05:16:51.702318+0000","last_became_peered":"2024-03-26T05:16:51.702318+0000","last_unstale":"2024-03-26T05:17:24.748104+0000","last_undegraded":"2024-03-26T05:17:24.748104+0000","last_fullsized":"2024-03-26T05:17:24.748104+0000","mapping_epoch":316,"log_start":"323'3300","ondisk_log_start":"323'3300","created":316,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'5611","last_scrub_stamp":"2024-03-26T05:15:57.473473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:15:57.473473+0000","objects_scrubbed":5604,"log_size":6555,"log_dups_size":0,"ondisk_log_size":6555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:25.143884+0000","scrub_duration":994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106463232,"num_objects":1624,"num_object_clones":0,"num_object_copies":4872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5388,"num_write_kb":224416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"343'11","reported_seq":768,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:22.062873+0000","last_change":"2024-03-26T05:15:48.645351+0000","last_active":"2024-03-26T05:17:22.062873+0000","last_peered":"2024-03-26T05:17:22.062873+0000","last_clean":"2024-03-26T05:15:46.624511+0000","last_became_active":"2024-03-26T05:15:48.645351+0000","last_became_peered":"2024-03-26T05:15:48.645351+0000","last_unstale":"2024-03-26T05:17:22.062873+0000","last_undegraded":"2024-03-26T05:15:48.635536+0000","last_fullsized":"2024-03-26T05:15:48.635413+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:12:03.961391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:03.961391+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"254'48","reported_seq":418,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.680734+0000","last_change":"2024-03-26T05:15:32.620944+0000","last_active":"2024-03-26T05:17:21.680734+0000","last_peered":"2024-03-26T05:17:21.680734+0000","last_clean":"2024-03-26T05:17:21.680734+0000","last_became_active":"2024-03-26T05:15:32.620512+0000","last_became_peered":"2024-03-26T05:15:32.620512+0000","last_unstale":"2024-03-26T05:17:21.680734+0000","last_undegraded":"2024-03-26T05:17:21.680734+0000","last_fullsized":"2024-03-26T05:17:21.680734+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'48","last_scrub_stamp":"2024-03-26T05:12:01.634109+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:12:01.634109+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":599,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:22.064097+0000","last_change":"2024-03-26T05:15:48.651187+0000","last_active":"2024-03-26T05:17:22.064097+0000","last_peered":"2024-03-26T05:17:22.064097+0000","last_clean":"2024-03-26T05:15:46.625050+0000","last_became_active":"2024-03-26T05:15:48.651187+0000","last_became_peered":"2024-03-26T05:15:48.651187+0000","last_unstale":"2024-03-26T05:17:22.064097+0000","last_undegraded":"2024-03-26T05:15:48.633222+0000","last_fullsized":"2024-03-26T05:15:48.633108+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.959120+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":374,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681135+0000","last_change":"2024-03-26T05:14:54.651241+0000","last_active":"2024-03-26T05:17:21.681135+0000","last_peered":"2024-03-26T05:17:21.681135+0000","last_clean":"2024-03-26T05:17:21.681135+0000","last_became_active":"2024-03-26T05:08:00.000166+0000","last_became_peered":"2024-03-26T05:08:00.000166+0000","last_unstale":"2024-03-26T05:17:21.681135+0000","last_undegraded":"2024-03-26T05:17:21.681135+0000","last_fullsized":"2024-03-26T05:17:21.681135+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:14:54.651113+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:14:54.651113+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"380'9634","reported_seq":9782,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:25.330941+0000","last_change":"2024-03-26T05:16:51.709382+0000","last_active":"2024-03-26T05:17:25.330941+0000","last_peered":"2024-03-26T05:17:25.330941+0000","last_clean":"2024-03-26T05:15:47.585964+0000","last_became_active":"2024-03-26T05:16:51.709382+0000","last_became_peered":"2024-03-26T05:16:51.709382+0000","last_unstale":"2024-03-26T05:17:25.330941+0000","last_undegraded":"2024-03-26T05:16:51.685661+0000","last_fullsized":"2024-03-26T05:16:51.684897+0000","mapping_epoch":342,"log_start":"324'3400","ondisk_log_start":"324'3400","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6234,"log_dups_size":0,"ondisk_log_size":6234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:30.291409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103055360,"num_objects":1572,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5292,"num_write_kb":219680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1572}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.19","version":"380'10027","reported_seq":26472,"reported_epoch":380,"state":"active+clean+remapped","last_fresh":"2024-03-26T05:17:25.861463+0000","last_change":"2024-03-26T05:17:18.722983+0000","last_active":"2024-03-26T05:17:25.861463+0000","last_peered":"2024-03-26T05:17:25.861463+0000","last_clean":"2024-03-26T05:17:25.861463+0000","last_became_active":"2024-03-26T05:17:18.699791+0000","last_became_peered":"2024-03-26T05:17:18.699791+0000","last_unstale":"2024-03-26T05:17:25.861463+0000","last_undegraded":"2024-03-26T05:17:25.861463+0000","last_fullsized":"2024-03-26T05:17:25.861463+0000","mapping_epoch":377,"log_start":"325'3700","ondisk_log_start":"325'3700","created":363,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6327,"log_dups_size":0,"ondisk_log_size":6327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:29.564490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105873408,"num_objects":1616,"num_object_clones":0,"num_object_copies":4848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1616,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5505,"num_write_kb":227840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4626,"num_bytes_recovered":303136768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":1616}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"254'18","reported_seq":715,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.691582+0000","last_change":"2024-03-26T05:15:46.643207+0000","last_active":"2024-03-26T05:17:21.691582+0000","last_peered":"2024-03-26T05:17:21.691582+0000","last_clean":"2024-03-26T05:17:21.691582+0000","last_became_active":"2024-03-26T05:15:32.640871+0000","last_became_peered":"2024-03-26T05:15:32.640871+0000","last_unstale":"2024-03-26T05:17:21.691582+0000","last_undegraded":"2024-03-26T05:17:21.691582+0000","last_fullsized":"2024-03-26T05:17:21.691582+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'18","last_scrub_stamp":"2024-03-26T05:15:46.643068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:15:46.643068+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":738,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.087322+0000","last_change":"2024-03-26T05:15:51.440953+0000","last_active":"2024-03-26T05:17:22.087322+0000","last_peered":"2024-03-26T05:17:22.087322+0000","last_clean":"2024-03-26T05:17:22.087322+0000","last_became_active":"2024-03-26T05:07:59.992217+0000","last_became_peered":"2024-03-26T05:07:59.992217+0000","last_unstale":"2024-03-26T05:17:22.087322+0000","last_undegraded":"2024-03-26T05:17:22.087322+0000","last_fullsized":"2024-03-26T05:17:22.087322+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:15:51.440867+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:06:50.546241+0000","last_clean_scrub_stamp":"2024-03-26T05:15:51.440867+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":369,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681614+0000","last_change":"2024-03-26T05:14:04.756574+0000","last_active":"2024-03-26T05:17:21.681614+0000","last_peered":"2024-03-26T05:17:21.681614+0000","last_clean":"2024-03-26T05:17:21.681614+0000","last_became_active":"2024-03-26T05:07:59.997434+0000","last_became_peered":"2024-03-26T05:07:59.997434+0000","last_unstale":"2024-03-26T05:17:21.681614+0000","last_undegraded":"2024-03-26T05:17:21.681614+0000","last_fullsized":"2024-03-26T05:17:21.681614+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:04.756452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:04.756452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":592,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.063376+0000","last_change":"2024-03-26T05:15:48.643985+0000","last_active":"2024-03-26T05:17:22.063376+0000","last_peered":"2024-03-26T05:17:22.063376+0000","last_clean":"2024-03-26T05:15:46.624057+0000","last_became_active":"2024-03-26T05:15:48.643985+0000","last_became_peered":"2024-03-26T05:15:48.643985+0000","last_unstale":"2024-03-26T05:17:22.063376+0000","last_undegraded":"2024-03-26T05:17:22.063376+0000","last_fullsized":"2024-03-26T05:15:48.635169+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:07.006139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.006139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.18","version":"380'9721","reported_seq":21696,"reported_epoch":380,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T05:17:25.835153+0000","last_change":"2024-03-26T05:17:16.169303+0000","last_active":"2024-03-26T05:17:25.835153+0000","last_peered":"2024-03-26T05:17:25.835153+0000","last_clean":"2024-03-26T05:14:40.645004+0000","last_became_active":"2024-03-26T05:16:54.837523+0000","last_became_peered":"2024-03-26T05:16:54.837523+0000","last_unstale":"2024-03-26T05:17:25.835153+0000","last_undegraded":"2024-03-26T05:17:25.835153+0000","last_fullsized":"2024-03-26T05:17:25.835153+0000","mapping_epoch":366,"log_start":"327'3700","ondisk_log_start":"327'3700","created":363,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6021,"log_dups_size":0,"ondisk_log_size":6021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103219200,"num_objects":1575,"num_object_clones":0,"num_object_copies":4725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":428,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5349,"num_write_kb":221568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3830,"num_bytes_recovered":251002880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,0,7],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2,7","objects":1575}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"254'27","reported_seq":744,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.095517+0000","last_change":"2024-03-26T05:15:32.635064+0000","last_active":"2024-03-26T05:17:22.095517+0000","last_peered":"2024-03-26T05:17:22.095517+0000","last_clean":"2024-03-26T05:17:22.095517+0000","last_became_active":"2024-03-26T05:15:32.634434+0000","last_became_peered":"2024-03-26T05:15:32.634434+0000","last_unstale":"2024-03-26T05:17:22.095517+0000","last_undegraded":"2024-03-26T05:17:22.095517+0000","last_fullsized":"2024-03-26T05:17:22.095517+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:12:11.885908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:12:11.885908+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":566,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.063128+0000","last_change":"2024-03-26T05:15:48.646040+0000","last_active":"2024-03-26T05:17:22.063128+0000","last_peered":"2024-03-26T05:17:22.063128+0000","last_clean":"2024-03-26T05:15:46.624763+0000","last_became_active":"2024-03-26T05:15:48.646040+0000","last_became_peered":"2024-03-26T05:15:48.646040+0000","last_unstale":"2024-03-26T05:17:22.063128+0000","last_undegraded":"2024-03-26T05:17:22.063128+0000","last_fullsized":"2024-03-26T05:15:48.635818+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.939576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.939576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":681,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.691530+0000","last_change":"2024-03-26T05:15:48.701885+0000","last_active":"2024-03-26T05:17:21.691530+0000","last_peered":"2024-03-26T05:17:21.691530+0000","last_clean":"2024-03-26T05:17:21.691530+0000","last_became_active":"2024-03-26T05:15:32.641031+0000","last_became_peered":"2024-03-26T05:15:32.641031+0000","last_unstale":"2024-03-26T05:17:21.691530+0000","last_undegraded":"2024-03-26T05:17:21.691530+0000","last_fullsized":"2024-03-26T05:17:21.691530+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:48.701610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:15:48.701610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":696,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.686305+0000","last_change":"2024-03-26T05:15:48.639802+0000","last_active":"2024-03-26T05:17:21.686305+0000","last_peered":"2024-03-26T05:17:21.686305+0000","last_clean":"2024-03-26T05:15:46.626518+0000","last_became_active":"2024-03-26T05:15:48.639802+0000","last_became_peered":"2024-03-26T05:15:48.639802+0000","last_unstale":"2024-03-26T05:17:21.686305+0000","last_undegraded":"2024-03-26T05:17:21.686305+0000","last_fullsized":"2024-03-26T05:15:48.633216+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:13.142714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:13.142714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"254'26","reported_seq":399,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681434+0000","last_change":"2024-03-26T05:16:51.291839+0000","last_active":"2024-03-26T05:17:21.681434+0000","last_peered":"2024-03-26T05:17:21.681434+0000","last_clean":"2024-03-26T05:17:21.681434+0000","last_became_active":"2024-03-26T05:15:32.621049+0000","last_became_peered":"2024-03-26T05:15:32.621049+0000","last_unstale":"2024-03-26T05:17:21.681434+0000","last_undegraded":"2024-03-26T05:17:21.681434+0000","last_fullsized":"2024-03-26T05:17:21.681434+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:16:51.291743+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:16:51.291743+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:18:13.101693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":699,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.094646+0000","last_change":"2024-03-26T05:15:32.632805+0000","last_active":"2024-03-26T05:17:22.094646+0000","last_peered":"2024-03-26T05:17:22.094646+0000","last_clean":"2024-03-26T05:17:22.094646+0000","last_became_active":"2024-03-26T05:15:32.632387+0000","last_became_peered":"2024-03-26T05:15:32.632387+0000","last_unstale":"2024-03-26T05:17:22.094646+0000","last_undegraded":"2024-03-26T05:17:22.094646+0000","last_fullsized":"2024-03-26T05:17:22.094646+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:59.989341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:59.989341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"336'11","reported_seq":900,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.690630+0000","last_change":"2024-03-26T05:15:41.588979+0000","last_active":"2024-03-26T05:17:21.690630+0000","last_peered":"2024-03-26T05:17:21.690630+0000","last_clean":"2024-03-26T05:17:21.690630+0000","last_became_active":"2024-03-26T05:15:32.636392+0000","last_became_peered":"2024-03-26T05:15:32.636392+0000","last_unstale":"2024-03-26T05:17:21.690630+0000","last_undegraded":"2024-03-26T05:17:21.690630+0000","last_fullsized":"2024-03-26T05:17:21.690630+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-03-26T05:15:41.588817+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:15:41.588817+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":404,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.687739+0000","last_change":"2024-03-26T05:15:48.641096+0000","last_active":"2024-03-26T05:17:21.687739+0000","last_peered":"2024-03-26T05:17:21.687739+0000","last_clean":"2024-03-26T05:07:55.690747+0000","last_became_active":"2024-03-26T05:15:48.641096+0000","last_became_peered":"2024-03-26T05:15:48.641096+0000","last_unstale":"2024-03-26T05:17:21.687739+0000","last_undegraded":"2024-03-26T05:17:21.687739+0000","last_fullsized":"2024-03-26T05:15:48.634411+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:06.096138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:06.096138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"345'39","reported_seq":754,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.690124+0000","last_change":"2024-03-26T05:15:56.674563+0000","last_active":"2024-03-26T05:17:21.690124+0000","last_peered":"2024-03-26T05:17:21.690124+0000","last_clean":"2024-03-26T05:17:21.690124+0000","last_became_active":"2024-03-26T05:15:32.626448+0000","last_became_peered":"2024-03-26T05:15:32.626448+0000","last_unstale":"2024-03-26T05:17:21.690124+0000","last_undegraded":"2024-03-26T05:17:21.690124+0000","last_fullsized":"2024-03-26T05:17:21.690124+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'39","last_scrub_stamp":"2024-03-26T05:15:56.674423+0000","last_deep_scrub":"345'39","last_deep_scrub_stamp":"2024-03-26T05:15:56.674423+0000","last_clean_scrub_stamp":"2024-03-26T05:15:56.674423+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":600,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.067386+0000","last_change":"2024-03-26T05:15:48.641873+0000","last_active":"2024-03-26T05:17:22.067386+0000","last_peered":"2024-03-26T05:17:22.067386+0000","last_clean":"2024-03-26T05:15:46.623430+0000","last_became_active":"2024-03-26T05:15:48.641873+0000","last_became_peered":"2024-03-26T05:15:48.641873+0000","last_unstale":"2024-03-26T05:17:22.067386+0000","last_undegraded":"2024-03-26T05:17:22.067386+0000","last_fullsized":"2024-03-26T05:15:48.632198+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:01.962048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:01.962048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"216'10","reported_seq":486,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681617+0000","last_change":"2024-03-26T05:14:20.704552+0000","last_active":"2024-03-26T05:17:21.681617+0000","last_peered":"2024-03-26T05:17:21.681617+0000","last_clean":"2024-03-26T05:17:21.681617+0000","last_became_active":"2024-03-26T05:08:00.005648+0000","last_became_peered":"2024-03-26T05:08:00.005648+0000","last_unstale":"2024-03-26T05:17:21.681617+0000","last_undegraded":"2024-03-26T05:17:21.681617+0000","last_fullsized":"2024-03-26T05:17:21.681617+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:14:20.704431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:20.704431+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":687,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.095112+0000","last_change":"2024-03-26T05:15:32.631774+0000","last_active":"2024-03-26T05:17:22.095112+0000","last_peered":"2024-03-26T05:17:22.095112+0000","last_clean":"2024-03-26T05:17:22.095112+0000","last_became_active":"2024-03-26T05:15:32.631444+0000","last_became_peered":"2024-03-26T05:15:32.631444+0000","last_unstale":"2024-03-26T05:17:22.095112+0000","last_undegraded":"2024-03-26T05:17:22.095112+0000","last_fullsized":"2024-03-26T05:17:22.095112+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:17.975697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:04.057358+0000","last_clean_scrub_stamp":"2024-03-26T05:12:17.975697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:28.138527+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"254'24","reported_seq":532,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:22.087930+0000","last_change":"2024-03-26T05:15:48.645289+0000","last_active":"2024-03-26T05:17:22.087930+0000","last_peered":"2024-03-26T05:17:22.087930+0000","last_clean":"2024-03-26T05:10:30.628152+0000","last_became_active":"2024-03-26T05:15:48.645289+0000","last_became_peered":"2024-03-26T05:15:48.645289+0000","last_unstale":"2024-03-26T05:17:22.087930+0000","last_undegraded":"2024-03-26T05:15:48.632524+0000","last_fullsized":"2024-03-26T05:15:48.632421+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'24","last_scrub_stamp":"2024-03-26T05:13:56.875610+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:13:56.875610+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":668,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.063418+0000","last_change":"2024-03-26T05:15:48.642442+0000","last_active":"2024-03-26T05:17:22.063418+0000","last_peered":"2024-03-26T05:17:22.063418+0000","last_clean":"2024-03-26T05:12:19.733579+0000","last_became_active":"2024-03-26T05:15:48.642442+0000","last_became_peered":"2024-03-26T05:15:48.642442+0000","last_unstale":"2024-03-26T05:17:22.063418+0000","last_undegraded":"2024-03-26T05:17:22.063418+0000","last_fullsized":"2024-03-26T05:15:48.631873+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:21.515294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:21.515294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":716,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681347+0000","last_change":"2024-03-26T05:16:35.331912+0000","last_active":"2024-03-26T05:17:21.681347+0000","last_peered":"2024-03-26T05:17:21.681347+0000","last_clean":"2024-03-26T05:17:21.681347+0000","last_became_active":"2024-03-26T05:08:00.001601+0000","last_became_peered":"2024-03-26T05:08:00.001601+0000","last_unstale":"2024-03-26T05:17:21.681347+0000","last_undegraded":"2024-03-26T05:17:21.681347+0000","last_fullsized":"2024-03-26T05:17:21.681347+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:35.331752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:30.777070+0000","last_clean_scrub_stamp":"2024-03-26T05:16:35.331752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:18:01.671300+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":688,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.688976+0000","last_change":"2024-03-26T05:16:03.797756+0000","last_active":"2024-03-26T05:17:21.688976+0000","last_peered":"2024-03-26T05:17:21.688976+0000","last_clean":"2024-03-26T05:17:21.688976+0000","last_became_active":"2024-03-26T05:07:59.999703+0000","last_became_peered":"2024-03-26T05:07:59.999703+0000","last_unstale":"2024-03-26T05:17:21.688976+0000","last_undegraded":"2024-03-26T05:17:21.688976+0000","last_fullsized":"2024-03-26T05:17:21.688976+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:03.797558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:16:03.797558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:25.381688+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"254'15","reported_seq":725,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681436+0000","last_change":"2024-03-26T05:16:33.261812+0000","last_active":"2024-03-26T05:17:21.681436+0000","last_peered":"2024-03-26T05:17:21.681436+0000","last_clean":"2024-03-26T05:17:21.681436+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:17:21.681436+0000","last_undegraded":"2024-03-26T05:17:21.681436+0000","last_fullsized":"2024-03-26T05:17:21.681436+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'15","last_scrub_stamp":"2024-03-26T05:16:33.261286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:16:33.261286+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:54.535677+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":572,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.063581+0000","last_change":"2024-03-26T05:15:48.639023+0000","last_active":"2024-03-26T05:17:22.063581+0000","last_peered":"2024-03-26T05:17:22.063581+0000","last_clean":"2024-03-26T05:15:46.624142+0000","last_became_active":"2024-03-26T05:15:48.639023+0000","last_became_peered":"2024-03-26T05:15:48.639023+0000","last_unstale":"2024-03-26T05:17:22.063581+0000","last_undegraded":"2024-03-26T05:17:22.063581+0000","last_fullsized":"2024-03-26T05:15:48.631660+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:33.016072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:11:33.016072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":680,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.694094+0000","last_change":"2024-03-26T05:15:09.726646+0000","last_active":"2024-03-26T05:17:21.694094+0000","last_peered":"2024-03-26T05:17:21.694094+0000","last_clean":"2024-03-26T05:17:21.694094+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:17:21.694094+0000","last_undegraded":"2024-03-26T05:17:21.694094+0000","last_fullsized":"2024-03-26T05:17:21.694094+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:09.726518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:15:09.726518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":706,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.092869+0000","last_change":"2024-03-26T05:14:46.024625+0000","last_active":"2024-03-26T05:17:22.092869+0000","last_peered":"2024-03-26T05:17:22.092869+0000","last_clean":"2024-03-26T05:17:22.092869+0000","last_became_active":"2024-03-26T05:07:59.994210+0000","last_became_peered":"2024-03-26T05:07:59.994210+0000","last_unstale":"2024-03-26T05:17:22.092869+0000","last_undegraded":"2024-03-26T05:17:22.092869+0000","last_fullsized":"2024-03-26T05:17:22.092869+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:46.024526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:46.024526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:22.094372+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"254'25","reported_seq":728,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.689890+0000","last_change":"2024-03-26T05:15:50.723419+0000","last_active":"2024-03-26T05:17:21.689890+0000","last_peered":"2024-03-26T05:17:21.689890+0000","last_clean":"2024-03-26T05:17:21.689890+0000","last_became_active":"2024-03-26T05:15:32.636359+0000","last_became_peered":"2024-03-26T05:15:32.636359+0000","last_unstale":"2024-03-26T05:17:21.689890+0000","last_undegraded":"2024-03-26T05:17:21.689890+0000","last_fullsized":"2024-03-26T05:17:21.689890+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:15:50.723325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:15:50.723325+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":370,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681660+0000","last_change":"2024-03-26T05:16:43.283206+0000","last_active":"2024-03-26T05:17:21.681660+0000","last_peered":"2024-03-26T05:17:21.681660+0000","last_clean":"2024-03-26T05:17:21.681660+0000","last_became_active":"2024-03-26T05:15:32.620559+0000","last_became_peered":"2024-03-26T05:15:32.620559+0000","last_unstale":"2024-03-26T05:17:21.681660+0000","last_undegraded":"2024-03-26T05:17:21.681660+0000","last_fullsized":"2024-03-26T05:17:21.681660+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:43.283128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:16:43.283128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:59.170591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":609,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.063180+0000","last_change":"2024-03-26T05:15:48.640423+0000","last_active":"2024-03-26T05:17:22.063180+0000","last_peered":"2024-03-26T05:17:22.063180+0000","last_clean":"2024-03-26T05:15:46.625977+0000","last_became_active":"2024-03-26T05:15:48.640423+0000","last_became_peered":"2024-03-26T05:15:48.640423+0000","last_unstale":"2024-03-26T05:17:22.063180+0000","last_undegraded":"2024-03-26T05:17:22.063180+0000","last_fullsized":"2024-03-26T05:15:48.633132+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:23.957542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:11:23.957542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":684,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.694436+0000","last_change":"2024-03-26T05:16:33.064381+0000","last_active":"2024-03-26T05:17:21.694436+0000","last_peered":"2024-03-26T05:17:21.694436+0000","last_clean":"2024-03-26T05:17:21.694436+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:17:21.694436+0000","last_undegraded":"2024-03-26T05:17:21.694436+0000","last_fullsized":"2024-03-26T05:17:21.694436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:33.064299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:49.225897+0000","last_clean_scrub_stamp":"2024-03-26T05:16:33.064299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:48.093037+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"254'10","reported_seq":706,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.688714+0000","last_change":"2024-03-26T05:16:38.058911+0000","last_active":"2024-03-26T05:17:21.688714+0000","last_peered":"2024-03-26T05:17:21.688714+0000","last_clean":"2024-03-26T05:17:21.688714+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:17:21.688714+0000","last_undegraded":"2024-03-26T05:17:21.688714+0000","last_fullsized":"2024-03-26T05:17:21.688714+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'10","last_scrub_stamp":"2024-03-26T05:16:38.058776+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:16:38.058776+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:51.859927+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":718,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.095224+0000","last_change":"2024-03-26T05:15:11.070456+0000","last_active":"2024-03-26T05:17:22.095224+0000","last_peered":"2024-03-26T05:17:22.095224+0000","last_clean":"2024-03-26T05:17:22.095224+0000","last_became_active":"2024-03-26T05:07:59.993996+0000","last_became_peered":"2024-03-26T05:07:59.993996+0000","last_unstale":"2024-03-26T05:17:22.095224+0000","last_undegraded":"2024-03-26T05:17:22.095224+0000","last_fullsized":"2024-03-26T05:17:22.095224+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:11.070341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:11.070341+0000","last_clean_scrub_stamp":"2024-03-26T05:15:11.070341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":691,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.087414+0000","last_change":"2024-03-26T05:15:48.643994+0000","last_active":"2024-03-26T05:17:22.087414+0000","last_peered":"2024-03-26T05:17:22.087414+0000","last_clean":"2024-03-26T05:15:46.616068+0000","last_became_active":"2024-03-26T05:15:48.643994+0000","last_became_peered":"2024-03-26T05:15:48.643994+0000","last_unstale":"2024-03-26T05:17:22.087414+0000","last_undegraded":"2024-03-26T05:17:22.087414+0000","last_fullsized":"2024-03-26T05:15:48.631480+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.693592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.693592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":593,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.062659+0000","last_change":"2024-03-26T05:15:48.641318+0000","last_active":"2024-03-26T05:17:22.062659+0000","last_peered":"2024-03-26T05:17:22.062659+0000","last_clean":"2024-03-26T05:15:46.623140+0000","last_became_active":"2024-03-26T05:15:48.641318+0000","last_became_peered":"2024-03-26T05:15:48.641318+0000","last_unstale":"2024-03-26T05:17:22.062659+0000","last_undegraded":"2024-03-26T05:17:22.062659+0000","last_fullsized":"2024-03-26T05:15:48.634268+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:10.020572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:10.020572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"254'25","reported_seq":819,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.088652+0000","last_change":"2024-03-26T05:16:34.220528+0000","last_active":"2024-03-26T05:17:22.088652+0000","last_peered":"2024-03-26T05:17:22.088652+0000","last_clean":"2024-03-26T05:17:22.088652+0000","last_became_active":"2024-03-26T05:07:59.962007+0000","last_became_peered":"2024-03-26T05:07:59.962007+0000","last_unstale":"2024-03-26T05:17:22.088652+0000","last_undegraded":"2024-03-26T05:17:22.088652+0000","last_fullsized":"2024-03-26T05:17:22.088652+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:16:34.220406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:16:34.220406+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:56.927766+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":695,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.688662+0000","last_change":"2024-03-26T05:16:31.868719+0000","last_active":"2024-03-26T05:17:21.688662+0000","last_peered":"2024-03-26T05:17:21.688662+0000","last_clean":"2024-03-26T05:17:21.688662+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:17:21.688662+0000","last_undegraded":"2024-03-26T05:17:21.688662+0000","last_fullsized":"2024-03-26T05:17:21.688662+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:31.868603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:16:31.868603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:45.432783+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":684,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.095632+0000","last_change":"2024-03-26T05:15:32.635171+0000","last_active":"2024-03-26T05:17:22.095632+0000","last_peered":"2024-03-26T05:17:22.095632+0000","last_clean":"2024-03-26T05:17:22.095632+0000","last_became_active":"2024-03-26T05:15:32.634716+0000","last_became_peered":"2024-03-26T05:15:32.634716+0000","last_unstale":"2024-03-26T05:17:22.095632+0000","last_undegraded":"2024-03-26T05:17:22.095632+0000","last_fullsized":"2024-03-26T05:17:22.095632+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:27.080959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:27.080959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":698,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.683419+0000","last_change":"2024-03-26T05:15:48.644977+0000","last_active":"2024-03-26T05:17:21.683419+0000","last_peered":"2024-03-26T05:17:21.683419+0000","last_clean":"2024-03-26T05:15:46.610540+0000","last_became_active":"2024-03-26T05:15:48.644977+0000","last_became_peered":"2024-03-26T05:15:48.644977+0000","last_unstale":"2024-03-26T05:17:21.683419+0000","last_undegraded":"2024-03-26T05:17:21.683419+0000","last_fullsized":"2024-03-26T05:15:48.636267+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:00.151355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.151355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"380'9805","reported_seq":25425,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:25.329259+0000","last_change":"2024-03-26T05:17:10.689171+0000","last_active":"2024-03-26T05:17:25.329259+0000","last_peered":"2024-03-26T05:17:25.329259+0000","last_clean":"2024-03-26T05:17:25.329259+0000","last_became_active":"2024-03-26T05:17:10.675942+0000","last_became_peered":"2024-03-26T05:17:10.675942+0000","last_unstale":"2024-03-26T05:17:25.329259+0000","last_undegraded":"2024-03-26T05:17:25.329259+0000","last_fullsized":"2024-03-26T05:17:25.329259+0000","mapping_epoch":373,"log_start":"327'3800","ondisk_log_start":"327'3800","created":363,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6005,"log_dups_size":0,"ondisk_log_size":6005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:28.219624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101974016,"num_objects":1556,"num_object_clones":0,"num_object_copies":4668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5400,"num_write_kb":222592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5022,"num_bytes_recovered":329089024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"290'32","reported_seq":744,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.095479+0000","last_change":"2024-03-26T05:15:32.634867+0000","last_active":"2024-03-26T05:17:22.095479+0000","last_peered":"2024-03-26T05:17:22.095479+0000","last_clean":"2024-03-26T05:17:22.095479+0000","last_became_active":"2024-03-26T05:15:32.634216+0000","last_became_peered":"2024-03-26T05:15:32.634216+0000","last_unstale":"2024-03-26T05:17:22.095479+0000","last_undegraded":"2024-03-26T05:17:22.095479+0000","last_fullsized":"2024-03-26T05:17:22.095479+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'32","last_scrub_stamp":"2024-03-26T05:12:15.970313+0000","last_deep_scrub":"290'32","last_deep_scrub_stamp":"2024-03-26T05:12:15.970313+0000","last_clean_scrub_stamp":"2024-03-26T05:12:15.970313+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":700,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:21.680821+0000","last_change":"2024-03-26T05:15:48.647499+0000","last_active":"2024-03-26T05:17:21.680821+0000","last_peered":"2024-03-26T05:17:21.680821+0000","last_clean":"2024-03-26T05:15:46.609101+0000","last_became_active":"2024-03-26T05:15:48.647499+0000","last_became_peered":"2024-03-26T05:15:48.647499+0000","last_unstale":"2024-03-26T05:17:21.680821+0000","last_undegraded":"2024-03-26T05:15:48.634583+0000","last_fullsized":"2024-03-26T05:15:48.634474+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:49.948818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:49.948818+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"343'11","reported_seq":927,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:22.088956+0000","last_change":"2024-03-26T05:15:48.644163+0000","last_active":"2024-03-26T05:17:22.088956+0000","last_peered":"2024-03-26T05:17:22.088956+0000","last_clean":"2024-03-26T05:15:46.619140+0000","last_became_active":"2024-03-26T05:15:48.644163+0000","last_became_peered":"2024-03-26T05:15:48.644163+0000","last_unstale":"2024-03-26T05:17:22.088956+0000","last_undegraded":"2024-03-26T05:15:48.636521+0000","last_fullsized":"2024-03-26T05:15:48.636379+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'10","last_scrub_stamp":"2024-03-26T05:15:38.516660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:15:38.516660+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:27.135294+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":612,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.062796+0000","last_change":"2024-03-26T05:15:32.637389+0000","last_active":"2024-03-26T05:17:22.062796+0000","last_peered":"2024-03-26T05:17:22.062796+0000","last_clean":"2024-03-26T05:17:22.062796+0000","last_became_active":"2024-03-26T05:15:32.636624+0000","last_became_peered":"2024-03-26T05:15:32.636624+0000","last_unstale":"2024-03-26T05:17:22.062796+0000","last_undegraded":"2024-03-26T05:17:22.062796+0000","last_fullsized":"2024-03-26T05:17:22.062796+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:12:13.995144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:13.995144+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"380'9837","reported_seq":9992,"reported_epoch":380,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-03-26T05:17:25.864825+0000","last_change":"2024-03-26T05:17:24.814169+0000","last_active":"2024-03-26T05:17:25.864825+0000","last_peered":"2024-03-26T05:17:25.864825+0000","last_clean":"2024-03-26T05:15:47.606254+0000","last_became_active":"2024-03-26T05:16:54.770459+0000","last_became_peered":"2024-03-26T05:16:54.770459+0000","last_unstale":"2024-03-26T05:17:25.864825+0000","last_undegraded":"2024-03-26T05:16:54.730195+0000","last_fullsized":"2024-03-26T05:16:54.717803+0000","mapping_epoch":366,"log_start":"322'3200","ondisk_log_start":"322'3200","created":363,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6637,"log_dups_size":0,"ondisk_log_size":6637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100728832,"num_objects":1537,"num_object_clones":0,"num_object_copies":4611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1537,"num_objects_misplaced":1537,"num_objects_unfound":0,"num_objects_dirty":1537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5423,"num_write_kb":222720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":1537}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"322'32","reported_seq":738,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:21.690910+0000","last_change":"2024-03-26T05:15:48.643576+0000","last_active":"2024-03-26T05:17:21.690910+0000","last_peered":"2024-03-26T05:17:21.690910+0000","last_clean":"2024-03-26T05:15:46.628715+0000","last_became_active":"2024-03-26T05:15:48.643576+0000","last_became_peered":"2024-03-26T05:15:48.643576+0000","last_unstale":"2024-03-26T05:17:21.690910+0000","last_undegraded":"2024-03-26T05:15:48.629218+0000","last_fullsized":"2024-03-26T05:15:48.629123+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'32","last_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_deep_scrub":"322'32","last_deep_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_clean_scrub_stamp":"2024-03-26T05:15:10.715786+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":83,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.681071+0000","last_change":"2024-03-26T05:15:48.650757+0000","last_active":"2024-03-26T05:17:21.681071+0000","last_peered":"2024-03-26T05:17:21.681071+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:15:48.650757+0000","last_became_peered":"2024-03-26T05:15:48.650757+0000","last_unstale":"2024-03-26T05:17:21.681071+0000","last_undegraded":"2024-03-26T05:17:21.681071+0000","last_fullsized":"2024-03-26T05:15:48.636772+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:03.047441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:15:03.047441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":83,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.690227+0000","last_change":"2024-03-26T05:15:48.638667+0000","last_active":"2024-03-26T05:17:21.690227+0000","last_peered":"2024-03-26T05:17:21.690227+0000","last_clean":"2024-03-26T05:00:25.933152+0000","last_became_active":"2024-03-26T05:15:48.638667+0000","last_became_peered":"2024-03-26T05:15:48.638667+0000","last_unstale":"2024-03-26T05:17:21.690227+0000","last_undegraded":"2024-03-26T05:17:21.690227+0000","last_fullsized":"2024-03-26T05:15:48.630074+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:04.830989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:04.830989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":697,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.691560+0000","last_change":"2024-03-26T05:16:03.006651+0000","last_active":"2024-03-26T05:17:21.691560+0000","last_peered":"2024-03-26T05:17:21.691560+0000","last_clean":"2024-03-26T05:17:21.691560+0000","last_became_active":"2024-03-26T05:07:59.959705+0000","last_became_peered":"2024-03-26T05:07:59.959705+0000","last_unstale":"2024-03-26T05:17:21.691560+0000","last_undegraded":"2024-03-26T05:17:21.691560+0000","last_fullsized":"2024-03-26T05:17:21.691560+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:03.006537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:15.038608+0000","last_clean_scrub_stamp":"2024-03-26T05:16:03.006537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"380'10023","reported_seq":15975,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:25.935705+0000","last_change":"2024-03-26T05:17:22.122608+0000","last_active":"2024-03-26T05:17:25.935705+0000","last_peered":"2024-03-26T05:17:25.935705+0000","last_clean":"2024-03-26T05:17:25.935705+0000","last_became_active":"2024-03-26T05:17:22.068437+0000","last_became_peered":"2024-03-26T05:17:22.068437+0000","last_unstale":"2024-03-26T05:17:25.935705+0000","last_undegraded":"2024-03-26T05:17:25.935705+0000","last_fullsized":"2024-03-26T05:17:25.935705+0000","mapping_epoch":379,"log_start":"322'3100","ondisk_log_start":"322'3100","created":363,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6923,"log_dups_size":0,"ondisk_log_size":6923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:27.472724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107970560,"num_objects":1648,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5498,"num_write_kb":228640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1928,"num_bytes_recovered":126287872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"254'27","reported_seq":742,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.686042+0000","last_change":"2024-03-26T05:14:53.363298+0000","last_active":"2024-03-26T05:17:21.686042+0000","last_peered":"2024-03-26T05:17:21.686042+0000","last_clean":"2024-03-26T05:17:21.686042+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:17:21.686042+0000","last_undegraded":"2024-03-26T05:17:21.686042+0000","last_fullsized":"2024-03-26T05:17:21.686042+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:14:53.363147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:53.363147+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":701,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.094217+0000","last_change":"2024-03-26T05:15:48.652795+0000","last_active":"2024-03-26T05:17:22.094217+0000","last_peered":"2024-03-26T05:17:22.094217+0000","last_clean":"2024-03-26T05:15:46.621710+0000","last_became_active":"2024-03-26T05:15:48.652795+0000","last_became_peered":"2024-03-26T05:15:48.652795+0000","last_unstale":"2024-03-26T05:17:22.094217+0000","last_undegraded":"2024-03-26T05:17:22.094217+0000","last_fullsized":"2024-03-26T05:15:48.642501+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:04.980730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:12:04.980730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":690,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.683429+0000","last_change":"2024-03-26T05:15:48.648471+0000","last_active":"2024-03-26T05:17:21.683429+0000","last_peered":"2024-03-26T05:17:21.683429+0000","last_clean":"2024-03-26T05:15:46.610002+0000","last_became_active":"2024-03-26T05:15:48.648471+0000","last_became_peered":"2024-03-26T05:15:48.648471+0000","last_unstale":"2024-03-26T05:17:21.683429+0000","last_undegraded":"2024-03-26T05:17:21.683429+0000","last_fullsized":"2024-03-26T05:15:48.637209+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:11.912203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:15:11.912203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":412,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.088787+0000","last_change":"2024-03-26T05:15:48.646231+0000","last_active":"2024-03-26T05:17:22.088787+0000","last_peered":"2024-03-26T05:17:22.088787+0000","last_clean":"2024-03-26T05:07:55.689953+0000","last_became_active":"2024-03-26T05:15:48.646231+0000","last_became_peered":"2024-03-26T05:15:48.646231+0000","last_unstale":"2024-03-26T05:17:22.088787+0000","last_undegraded":"2024-03-26T05:17:22.088787+0000","last_fullsized":"2024-03-26T05:15:48.631854+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:05.787712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:14:05.787712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.12","version":"380'9774","reported_seq":10492,"reported_epoch":380,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-03-26T05:17:26.573203+0000","last_change":"2024-03-26T05:17:24.877576+0000","last_active":"2024-03-26T05:17:26.573203+0000","last_peered":"2024-03-26T05:17:26.573203+0000","last_clean":"2024-03-26T05:15:47.613860+0000","last_became_active":"2024-03-26T05:16:54.839244+0000","last_became_peered":"2024-03-26T05:16:54.839244+0000","last_unstale":"2024-03-26T05:17:26.573203+0000","last_undegraded":"2024-03-26T05:16:54.802957+0000","last_fullsized":"2024-03-26T05:16:54.721147+0000","mapping_epoch":366,"log_start":"326'3600","ondisk_log_start":"326'3600","created":363,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104562688,"num_objects":1596,"num_object_clones":0,"num_object_copies":4788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1459,"num_objects_misplaced":2918,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5383,"num_write_kb":223296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":9175040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1595},{"shards":"5","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"321'16","reported_seq":669,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:21.690059+0000","last_change":"2024-03-26T05:15:48.646822+0000","last_active":"2024-03-26T05:17:21.690059+0000","last_peered":"2024-03-26T05:17:21.690059+0000","last_clean":"2024-03-26T05:15:00.994414+0000","last_became_active":"2024-03-26T05:15:48.646822+0000","last_became_peered":"2024-03-26T05:15:48.646822+0000","last_unstale":"2024-03-26T05:17:21.690059+0000","last_undegraded":"2024-03-26T05:15:48.630770+0000","last_fullsized":"2024-03-26T05:15:48.630637+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'16","last_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_deep_scrub":"321'16","last_deep_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.995735+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":369,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681277+0000","last_change":"2024-03-26T05:15:00.635456+0000","last_active":"2024-03-26T05:17:21.681277+0000","last_peered":"2024-03-26T05:17:21.681277+0000","last_clean":"2024-03-26T05:17:21.681277+0000","last_became_active":"2024-03-26T05:08:00.011262+0000","last_became_peered":"2024-03-26T05:08:00.011262+0000","last_unstale":"2024-03-26T05:17:21.681277+0000","last_undegraded":"2024-03-26T05:17:21.681277+0000","last_fullsized":"2024-03-26T05:17:21.681277+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:00.635308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:00.635308+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.635308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":608,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.062905+0000","last_change":"2024-03-26T05:15:48.652929+0000","last_active":"2024-03-26T05:17:22.062905+0000","last_peered":"2024-03-26T05:17:22.062905+0000","last_clean":"2024-03-26T05:15:46.623242+0000","last_became_active":"2024-03-26T05:15:48.652929+0000","last_became_peered":"2024-03-26T05:15:48.652929+0000","last_unstale":"2024-03-26T05:17:22.062905+0000","last_undegraded":"2024-03-26T05:17:22.062905+0000","last_fullsized":"2024-03-26T05:15:48.634670+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_clean_scrub_stamp":"2024-03-26T05:12:18.959687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":369,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681331+0000","last_change":"2024-03-26T05:14:10.840006+0000","last_active":"2024-03-26T05:17:21.681331+0000","last_peered":"2024-03-26T05:17:21.681331+0000","last_clean":"2024-03-26T05:17:21.681331+0000","last_became_active":"2024-03-26T05:07:59.998777+0000","last_became_peered":"2024-03-26T05:07:59.998777+0000","last_unstale":"2024-03-26T05:17:21.681331+0000","last_undegraded":"2024-03-26T05:17:21.681331+0000","last_fullsized":"2024-03-26T05:17:21.681331+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:10.839912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:10.839912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.15","version":"380'9838","reported_seq":10004,"reported_epoch":380,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T05:17:24.754153+0000","last_change":"2024-03-26T05:17:16.160364+0000","last_active":"2024-03-26T05:17:24.754153+0000","last_peered":"2024-03-26T05:17:24.754153+0000","last_clean":"2024-03-26T05:16:51.984683+0000","last_became_active":"2024-03-26T05:16:54.816828+0000","last_became_peered":"2024-03-26T05:16:54.816828+0000","last_unstale":"2024-03-26T05:17:24.754153+0000","last_undegraded":"2024-03-26T05:17:24.754153+0000","last_fullsized":"2024-03-26T05:17:24.754153+0000","mapping_epoch":366,"log_start":"322'2900","ondisk_log_start":"322'2900","created":363,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104136704,"num_objects":1589,"num_object_clones":0,"num_object_copies":4767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3174,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5415,"num_write_kb":224128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,0,2],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,2,3","objects":1589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"254'27","reported_seq":545,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:21.689502+0000","last_change":"2024-03-26T05:15:48.647529+0000","last_active":"2024-03-26T05:17:21.689502+0000","last_peered":"2024-03-26T05:17:21.689502+0000","last_clean":"2024-03-26T05:10:30.566889+0000","last_became_active":"2024-03-26T05:15:48.647529+0000","last_became_peered":"2024-03-26T05:15:48.647529+0000","last_unstale":"2024-03-26T05:17:21.689502+0000","last_undegraded":"2024-03-26T05:15:48.629088+0000","last_fullsized":"2024-03-26T05:15:48.628959+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:13:22.026354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:22.026354+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":372,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.681663+0000","last_change":"2024-03-26T05:15:48.636186+0000","last_active":"2024-03-26T05:17:21.681663+0000","last_peered":"2024-03-26T05:17:21.681663+0000","last_clean":"2024-03-26T05:15:46.631730+0000","last_became_active":"2024-03-26T05:15:48.636186+0000","last_became_peered":"2024-03-26T05:15:48.636186+0000","last_unstale":"2024-03-26T05:17:21.681663+0000","last_undegraded":"2024-03-26T05:17:21.681663+0000","last_fullsized":"2024-03-26T05:15:48.629342+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:17.792745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:17.792745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":370,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.681050+0000","last_change":"2024-03-26T05:15:32.622450+0000","last_active":"2024-03-26T05:17:21.681050+0000","last_peered":"2024-03-26T05:17:21.681050+0000","last_clean":"2024-03-26T05:17:21.681050+0000","last_became_active":"2024-03-26T05:15:32.622003+0000","last_became_peered":"2024-03-26T05:15:32.622003+0000","last_unstale":"2024-03-26T05:17:21.681050+0000","last_undegraded":"2024-03-26T05:17:21.681050+0000","last_fullsized":"2024-03-26T05:17:21.681050+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:04.643034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:04.643034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":687,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.093237+0000","last_change":"2024-03-26T05:15:32.640385+0000","last_active":"2024-03-26T05:17:22.093237+0000","last_peered":"2024-03-26T05:17:22.093237+0000","last_clean":"2024-03-26T05:17:22.093237+0000","last_became_active":"2024-03-26T05:15:32.640052+0000","last_became_peered":"2024-03-26T05:15:32.640052+0000","last_unstale":"2024-03-26T05:17:22.093237+0000","last_undegraded":"2024-03-26T05:17:22.093237+0000","last_fullsized":"2024-03-26T05:17:22.093237+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:04.037472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:04.037472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:28.138473+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.14","version":"380'9718","reported_seq":9872,"reported_epoch":380,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-03-26T05:17:26.550824+0000","last_change":"2024-03-26T05:17:24.875847+0000","last_active":"2024-03-26T05:17:26.550824+0000","last_peered":"2024-03-26T05:17:26.550824+0000","last_clean":"2024-03-26T05:15:47.586253+0000","last_became_active":"2024-03-26T05:16:54.846494+0000","last_became_peered":"2024-03-26T05:16:54.846494+0000","last_unstale":"2024-03-26T05:17:26.550824+0000","last_undegraded":"2024-03-26T05:16:54.786734+0000","last_fullsized":"2024-03-26T05:16:54.721610+0000","mapping_epoch":366,"log_start":"327'3700","ondisk_log_start":"327'3700","created":363,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6018,"log_dups_size":0,"ondisk_log_size":6018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99614720,"num_objects":1520,"num_object_clones":0,"num_object_copies":4560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1520,"num_objects_misplaced":3040,"num_objects_unfound":0,"num_objects_dirty":1520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5364,"num_write_kb":220288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"254'51","reported_seq":774,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.690250+0000","last_change":"2024-03-26T05:15:14.624289+0000","last_active":"2024-03-26T05:17:21.690250+0000","last_peered":"2024-03-26T05:17:21.690250+0000","last_clean":"2024-03-26T05:17:21.690250+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:17:21.690250+0000","last_undegraded":"2024-03-26T05:17:21.690250+0000","last_fullsized":"2024-03-26T05:17:21.690250+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'51","last_scrub_stamp":"2024-03-26T05:15:14.624135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:15:14.624135+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":697,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.690853+0000","last_change":"2024-03-26T05:16:30.881766+0000","last_active":"2024-03-26T05:17:21.690853+0000","last_peered":"2024-03-26T05:17:21.690853+0000","last_clean":"2024-03-26T05:17:21.690853+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:17:21.690853+0000","last_undegraded":"2024-03-26T05:17:21.690853+0000","last_fullsized":"2024-03-26T05:17:21.690853+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:30.881617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:16:30.881617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:56.811155+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":83,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:22.087480+0000","last_change":"2024-03-26T05:15:48.639904+0000","last_active":"2024-03-26T05:17:22.087480+0000","last_peered":"2024-03-26T05:17:22.087480+0000","last_clean":"2024-03-26T05:00:25.933152+0000","last_became_active":"2024-03-26T05:15:48.639904+0000","last_became_peered":"2024-03-26T05:15:48.639904+0000","last_unstale":"2024-03-26T05:17:22.087480+0000","last_undegraded":"2024-03-26T05:17:22.087480+0000","last_fullsized":"2024-03-26T05:15:48.630870+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:59.872240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:13:59.872240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":668,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.694174+0000","last_change":"2024-03-26T05:15:40.674528+0000","last_active":"2024-03-26T05:17:21.694174+0000","last_peered":"2024-03-26T05:17:21.694174+0000","last_clean":"2024-03-26T05:17:21.694174+0000","last_became_active":"2024-03-26T05:15:32.637717+0000","last_became_peered":"2024-03-26T05:15:32.637717+0000","last_unstale":"2024-03-26T05:17:21.694174+0000","last_undegraded":"2024-03-26T05:17:21.694174+0000","last_fullsized":"2024-03-26T05:17:21.694174+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:40.674397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:40.674397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.17","version":"380'9895","reported_seq":10108,"reported_epoch":380,"state":"active+clean+remapped","last_fresh":"2024-03-26T05:17:24.803332+0000","last_change":"2024-03-26T05:16:54.750399+0000","last_active":"2024-03-26T05:17:24.803332+0000","last_peered":"2024-03-26T05:17:24.803332+0000","last_clean":"2024-03-26T05:17:24.803332+0000","last_became_active":"2024-03-26T05:16:54.730064+0000","last_became_peered":"2024-03-26T05:16:54.730064+0000","last_unstale":"2024-03-26T05:17:24.803332+0000","last_undegraded":"2024-03-26T05:17:24.803332+0000","last_fullsized":"2024-03-26T05:17:24.803332+0000","mapping_epoch":366,"log_start":"323'3300","ondisk_log_start":"323'3300","created":363,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":5,"last_scrub":"345'5611","last_scrub_stamp":"2024-03-26T05:15:57.473473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:15:57.473473+0000","objects_scrubbed":5604,"log_size":6595,"log_dups_size":0,"ondisk_log_size":6595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:28.870960+0000","scrub_duration":994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103841792,"num_objects":1585,"num_object_clones":0,"num_object_copies":4755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1585,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5429,"num_write_kb":224416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"254'21","reported_seq":533,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:17:22.087708+0000","last_change":"2024-03-26T05:15:48.639602+0000","last_active":"2024-03-26T05:17:22.087708+0000","last_peered":"2024-03-26T05:17:22.087708+0000","last_clean":"2024-03-26T05:10:30.455391+0000","last_became_active":"2024-03-26T05:15:48.639602+0000","last_became_peered":"2024-03-26T05:15:48.639602+0000","last_unstale":"2024-03-26T05:17:22.087708+0000","last_undegraded":"2024-03-26T05:15:48.632373+0000","last_fullsized":"2024-03-26T05:15:48.632263+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:15:04.002977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:15:04.002977+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":711,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:22.095025+0000","last_change":"2024-03-26T05:14:20.034981+0000","last_active":"2024-03-26T05:17:22.095025+0000","last_peered":"2024-03-26T05:17:22.095025+0000","last_clean":"2024-03-26T05:17:22.095025+0000","last_became_active":"2024-03-26T05:07:59.994431+0000","last_became_peered":"2024-03-26T05:07:59.994431+0000","last_unstale":"2024-03-26T05:17:22.095025+0000","last_undegraded":"2024-03-26T05:17:22.095025+0000","last_fullsized":"2024-03-26T05:17:22.095025+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:20.034875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:20.034875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":700,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:21.680593+0000","last_change":"2024-03-26T05:15:39.109021+0000","last_active":"2024-03-26T05:17:21.680593+0000","last_peered":"2024-03-26T05:17:21.680593+0000","last_clean":"2024-03-26T05:17:21.680593+0000","last_became_active":"2024-03-26T05:15:32.629027+0000","last_became_peered":"2024-03-26T05:15:32.629027+0000","last_unstale":"2024-03-26T05:17:21.680593+0000","last_undegraded":"2024-03-26T05:17:21.680593+0000","last_fullsized":"2024-03-26T05:17:21.680593+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:39.108879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:15:39.108879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":404,"reported_epoch":380,"state":"active+undersized","last_fresh":"2024-03-26T05:17:21.680907+0000","last_change":"2024-03-26T05:15:48.649557+0000","last_active":"2024-03-26T05:17:21.680907+0000","last_peered":"2024-03-26T05:17:21.680907+0000","last_clean":"2024-03-26T05:07:55.689482+0000","last_became_active":"2024-03-26T05:15:48.649557+0000","last_became_peered":"2024-03-26T05:15:48.649557+0000","last_unstale":"2024-03-26T05:17:21.680907+0000","last_undegraded":"2024-03-26T05:17:21.680907+0000","last_fullsized":"2024-03-26T05:15:48.628605+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:53.055153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:53.055153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"380'9624","reported_seq":16193,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-03-26T05:17:24.798368+0000","last_change":"2024-03-26T05:17:17.712947+0000","last_active":"2024-03-26T05:17:24.798368+0000","last_peered":"2024-03-26T05:17:24.798368+0000","last_clean":"2024-03-26T05:17:24.798368+0000","last_became_active":"2024-03-26T05:17:17.663844+0000","last_became_peered":"2024-03-26T05:17:17.663844+0000","last_unstale":"2024-03-26T05:17:24.798368+0000","last_undegraded":"2024-03-26T05:17:24.798368+0000","last_fullsized":"2024-03-26T05:17:24.798368+0000","mapping_epoch":376,"log_start":"324'3400","ondisk_log_start":"324'3400","created":363,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:14:40.645004+0000","last_clean_scrub_stamp":"2024-03-26T05:14:40.645004+0000","objects_scrubbed":0,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:17:28.164131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103710720,"num_objects":1583,"num_object_clones":0,"num_object_copies":4749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5283,"num_write_kb":219680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2136,"num_bytes_recovered":139919360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":3323068440,"num_objects":50707,"num_object_clones":0,"num_object_copies":152121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22046,"num_objects_misplaced":14298,"num_objects_unfound":0,"num_objects_dirty":50707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172735,"num_write_kb":7150081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30819,"num_bytes_recovered":2019655680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11784364032,"data_stored":11784355888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158589,"ondisk_log_size":158589,"up":66,"acting":67,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":178,"num_read_kb":173,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":751377,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":80,"num_write":49,"num_write_kb":659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":483029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":30,"acting":30,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":82,"acting":82,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1405,"num_read_kb":1130,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":1870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":976,"ondisk_log_size":976,"up":82,"acting":82,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":81,"acting":81,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":83,"acting":83,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":335,"seq":1438814044184,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1046156,"kb_used_data":817988,"kb_used_omap":4,"kb_used_meta":228155,"kb_avail":92698996,"statfs":{"total":95995035648,"available":94923771904,"internally_reserved":0,"allocated":837619712,"data_stored":836998843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":233631400},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332310,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2028800,"kb_used_data":1550584,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":91716352,"statfs":{"total":95995035648,"available":93917544448,"internally_reserved":0,"allocated":1587798016,"data_stored":1587187943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":489683398},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575088,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1612708,"kb_used_data":1003996,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":92132444,"statfs":{"total":95995035648,"available":94343622656,"internally_reserved":0,"allocated":1028091904,"data_stored":1027445877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738613,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2660636,"kb_used_data":1989396,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":91084516,"statfs":{"total":95995035648,"available":93270544384,"internally_reserved":0,"allocated":2037141504,"data_stored":2036504428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411563,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1752196,"kb_used_data":1233724,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91992956,"statfs":{"total":95995035648,"available":94200786944,"internally_reserved":0,"allocated":1263333376,"data_stored":1262719871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117221,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1788000,"kb_used_data":1129236,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":91957152,"statfs":{"total":95995035648,"available":94164123648,"internally_reserved":0,"allocated":1156337664,"data_stored":1155699919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":674494919},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920973,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3139256,"kb_used_data":2751280,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":90605896,"statfs":{"total":95995035648,"available":92780437504,"internally_reserved":0,"allocated":2817310720,"data_stored":2816894270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659355,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1905860,"kb_used_data":1134716,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":91839292,"statfs":{"total":95995035648,"available":94043435008,"internally_reserved":0,"allocated":1161949184,"data_stored":1161335861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":789641669},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2016874496,"data_stored":2016870424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1016528896,"data_stored":1016528896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1242300416,"data_stored":1242300416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1139998720,"data_stored":1139998720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2819096576,"data_stored":2819096576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1157365760,"data_stored":1157365760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1566183424,"data_stored":1566179352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":826015744,"data_stored":826015744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:17:30.330 INFO:tasks.thrashosds.thrasher:Setting 4.b to [0, 4, 2] 2024-03-26T05:17:30.330 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.b', '0', '4', '2'] 2024-03-26T05:17:30.330 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pg-upmap 4.b 0 4 2 2024-03-26T05:17:30.384 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:30.409 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:30.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:30.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:30.511 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:30.554 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:30.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:30.655 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:30.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:30.701 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:30.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:30.802 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:30.833 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:30.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:30.935 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:30.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:30.973 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:30.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:31.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:30 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:17:31.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:30 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:17:31.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:30 smithi132 ceph-mon[33500]: osdmap e383: 8 total, 7 up, 8 in 2024-03-26T05:17:31.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:30 smithi132 ceph-mon[33500]: 4.19 scrub starts 2024-03-26T05:17:31.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:30 smithi132 ceph-mon[33500]: 4.19 scrub ok 2024-03-26T05:17:31.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:30 smithi132 ceph-mon[33500]: pgmap v791: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 2 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 94 active+clean, 30 active+undersized+degraded; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.24k op/s; 21253/144927 objects degraded (14.665%); 13786/144927 objects misplaced (9.512%); 8.8 MiB/s, 141 objects/s recovering 2024-03-26T05:17:31.074 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:31.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:30 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:17:31.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:30 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:17:31.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:30 smithi080 ceph-mon[40404]: osdmap e383: 8 total, 7 up, 8 in 2024-03-26T05:17:31.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:30 smithi080 ceph-mon[40404]: 4.19 scrub starts 2024-03-26T05:17:31.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:30 smithi080 ceph-mon[40404]: 4.19 scrub ok 2024-03-26T05:17:31.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:30 smithi080 ceph-mon[40404]: pgmap v791: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 2 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 94 active+clean, 30 active+undersized+degraded; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.24k op/s; 21253/144927 objects degraded (14.665%); 13786/144927 objects misplaced (9.512%); 8.8 MiB/s, 141 objects/s recovering 2024-03-26T05:17:31.102 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:30 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:17:31.102 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:30 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:17:31.102 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:30 smithi080 ceph-mon[29691]: osdmap e383: 8 total, 7 up, 8 in 2024-03-26T05:17:31.102 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:30 smithi080 ceph-mon[29691]: 4.19 scrub starts 2024-03-26T05:17:31.102 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:30 smithi080 ceph-mon[29691]: 4.19 scrub ok 2024-03-26T05:17:31.102 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:30 smithi080 ceph-mon[29691]: pgmap v791: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 2 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 94 active+clean, 30 active+undersized+degraded; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.24k op/s; 21253/144927 objects degraded (14.665%); 13786/144927 objects misplaced (9.512%); 8.8 MiB/s, 141 objects/s recovering 2024-03-26T05:17:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:31.103 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:31.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:31.205 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:31.240 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:31.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:31.342 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:31.379 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:31.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:31.481 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:31.513 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:31.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:31.560 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:17:31.560 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:17:31.560 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.560 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:179169 10.12 10:4f509d32:::benchmark_data_smithi080_161990_object67446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-26T05:17:31.560 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:31.099415+0000", 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0091191339999999992, 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0091528879999999996, 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 179169 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.099415+0000", 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.099415+0000", 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.099422+0000", 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0129999999999996e-06 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.099431+0000", 2024-03-26T05:17:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0960000000000007e-06 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.099452+0000", 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0800000000000001e-05 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.099459+0000", 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.25e-06 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.099790+0000", 2024-03-26T05:17:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00033102099999999997 2024-03-26T05:17:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:17:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.100187+0000", 2024-03-26T05:17:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00039672500000000001 2024-03-26T05:17:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,2,5,6", 2024-03-26T05:17:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.100535+0000", 2024-03-26T05:17:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00034823200000000002 2024-03-26T05:17:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.105226+0000", 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0046913199999999997 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.105503+0000", 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00027653999999999998 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15657.0:179177 10.11 e383/373 10:8d3de266:::benchmark_data_smithi080_161990_object67454:head v 383'9986, mlcod=383'9984)", 2024-03-26T05:17:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:31.106102+0000", 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.002432395, 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002561308, 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106102+0000", 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106102+0000", 2024-03-26T05:17:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106102+0000", 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6499999999999999e-07 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106109+0000", 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3600000000000001e-06 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106123+0000", 2024-03-26T05:17:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4644000000000001e-05 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106139+0000", 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5721000000000001e-05 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106236+0000", 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7048999999999999e-05 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:17:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106258+0000", 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1857000000000001e-05 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15657.0:179170 10.15 e383/366 10:af8f013d:::benchmark_data_smithi080_161990_object67447:head v 383'10061, mlcod=383'10059)", 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:31.106447+0000", 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0020870580000000001, 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022754839999999999, 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106447+0000", 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106447+0000", 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106448+0000", 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8100000000000002e-07 2024-03-26T05:17:31.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106453+0000", 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5060000000000003e-06 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106465+0000", 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1484000000000001e-05 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106472+0000", 2024-03-26T05:17:31.571 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0249999999999997e-06 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106574+0000", 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000102181 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106633+0000", 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9095999999999998e-05 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.572 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15657.0:179178 10.1 e383/363 10:831c23f1:::benchmark_data_smithi080_161990_object67455:head v 383'10000, mlcod=383'9998)", 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:31.106532+0000", 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0020019740000000001, 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002250475, 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106532+0000", 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:31.573 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106532+0000", 2024-03-26T05:17:31.574 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.574 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.574 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.574 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:31.574 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106533+0000", 2024-03-26T05:17:31.574 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2500000000000005e-07 2024-03-26T05:17:31.574 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.574 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.574 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:31.574 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106538+0000", 2024-03-26T05:17:31.574 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.711e-06 2024-03-26T05:17:31.574 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106548+0000", 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0073e-05 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106555+0000", 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2859999999999996e-06 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.108029+0000", 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014746869999999999 2024-03-26T05:17:31.575 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.108049+0000", 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9448000000000001e-05 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:179176 10.8 10:12b0f8e9:::benchmark_data_smithi080_161990_object67453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:31.106537+0000", 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.00199738, 2024-03-26T05:17:31.576 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0023179989999999998, 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 179176 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106537+0000", 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.577 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106537+0000", 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106538+0000", 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2359999999999999e-06 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106541+0000", 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6660000000000001e-06 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.578 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106551+0000", 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7829999999999993e-06 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106558+0000", 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8530000000000003e-06 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.107050+0000", 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00049240999999999998 2024-03-26T05:17:31.579 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.107173+0000", 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012211700000000001 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.107645+0000", 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00047249999999999999 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.580 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "MOSDPGPushReply(10.12 383/366 [PushReplyOp(10:4b696e4e:::benchmark_data_smithi080_161990_object101282:head)])", 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:31.106630+0000", 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.001903603, 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0023024690000000001, 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106630+0000", 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.581 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106630+0000", 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106631+0000", 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.98e-07 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106634+0000", 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5750000000000001e-06 2024-03-26T05:17:31.582 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106642+0000", 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6389999999999992e-06 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106648+0000", 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.246e-06 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:31.583 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.108876+0000", 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022283849999999998 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15657.0:179175 10.2 e383/363 10:4150bcbd:::benchmark_data_smithi080_161990_object67452:head v 383'9938, mlcod=383'9935)", 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:31.106663+0000", 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0018708349999999999, 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0023455199999999998, 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.584 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106663+0000", 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106663+0000", 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106663+0000", 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3099999999999999e-07 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.585 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106668+0000", 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2180000000000001e-06 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106675+0000", 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9090000000000003e-06 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.106680+0000", 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0520000000000001e-06 2024-03-26T05:17:31.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.587 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.587 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:31.587 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.107319+0000", 2024-03-26T05:17:31.587 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00063958899999999998 2024-03-26T05:17:31.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:31.587 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:31.587 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:17:31.587 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:31.107340+0000", 2024-03-26T05:17:31.587 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1019e-05 2024-03-26T05:17:31.587 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.588 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:31.588 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.588 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:31.588 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:17:31.588 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 7 2024-03-26T05:17:31.588 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:17:31.615 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:31.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:31.656 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:31.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:31.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:31.787 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:31.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:31.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:31 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 21253/144927 objects degraded (14.665%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:31.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:31 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 21253/144927 objects degraded (14.665%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:31.890 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:31.933 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:31.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:32.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:32.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:31 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 21253/144927 objects degraded (14.665%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:32.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:32.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:32.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:32.169 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:32.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:32.225 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:32.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:32.326 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:32.367 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:32.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:32.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:32.469 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:32.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:32.511 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:32.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:32.612 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:32.639 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:32.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:32.740 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:32.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:32.781 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:32.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:32.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:32 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:17:32.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:32 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:17:32.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:32 smithi080 ceph-mon[40404]: 5.13 scrub starts 2024-03-26T05:17:32.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:32 smithi080 ceph-mon[40404]: 5.13 scrub ok 2024-03-26T05:17:32.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:32 smithi080 ceph-mon[40404]: pgmap v792: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s; 19133/135750 objects degraded (14.094%); 11304/135750 objects misplaced (8.327%); 8.9 MiB/s, 142 objects/s recovering 2024-03-26T05:17:32.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:32 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:17:32.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:32 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:17:32.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:32 smithi080 ceph-mon[29691]: 5.13 scrub starts 2024-03-26T05:17:32.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:32 smithi080 ceph-mon[29691]: 5.13 scrub ok 2024-03-26T05:17:32.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:32 smithi080 ceph-mon[29691]: pgmap v792: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s; 19133/135750 objects degraded (14.094%); 11304/135750 objects misplaced (8.327%); 8.9 MiB/s, 142 objects/s recovering 2024-03-26T05:17:32.882 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:32.920 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:32.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:32.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:33.022 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:33.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:32 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:17:33.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:32 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:17:33.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:32 smithi132 ceph-mon[33500]: 5.13 scrub starts 2024-03-26T05:17:33.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:32 smithi132 ceph-mon[33500]: 5.13 scrub ok 2024-03-26T05:17:33.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:32 smithi132 ceph-mon[33500]: pgmap v792: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s; 19133/135750 objects degraded (14.094%); 11304/135750 objects misplaced (8.327%); 8.9 MiB/s, 142 objects/s recovering 2024-03-26T05:17:33.048 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:33.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:33.055 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T05:17:33.149 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:33.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:33.279 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:33.306 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:33.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:33.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:33.408 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:33.457 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:33.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:33.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:33.558 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:33.588 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:33.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:33.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:33.660 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:33.690 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:33.723 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:33.758 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:33.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:33.859 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:33.917 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:33.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:33.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:33.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:33 smithi080 ceph-mon[40404]: 1.0 scrub starts 2024-03-26T05:17:33.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:33 smithi080 ceph-mon[40404]: 1.0 scrub ok 2024-03-26T05:17:33.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:33 smithi080 ceph-mon[40404]: osdmap e384: 8 total, 7 up, 8 in 2024-03-26T05:17:33.987 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:33 smithi080 ceph-mon[29691]: 1.0 scrub starts 2024-03-26T05:17:33.987 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:33 smithi080 ceph-mon[29691]: 1.0 scrub ok 2024-03-26T05:17:33.988 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:33 smithi080 ceph-mon[29691]: osdmap e384: 8 total, 7 up, 8 in 2024-03-26T05:17:34.018 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:34.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:33 smithi132 ceph-mon[33500]: 1.0 scrub starts 2024-03-26T05:17:34.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:33 smithi132 ceph-mon[33500]: 1.0 scrub ok 2024-03-26T05:17:34.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:33 smithi132 ceph-mon[33500]: osdmap e384: 8 total, 7 up, 8 in 2024-03-26T05:17:34.062 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:34.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:34.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:34.164 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:34.205 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:34.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:34.299 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:17:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:17:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:17:34.307 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:34.345 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:34.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:34.447 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:34.489 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:34.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:34.590 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:34.632 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:34.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:34.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:34.734 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:34.765 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:34.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:34.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:34.867 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:34.893 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:34.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:34.995 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:35.023 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:35.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:35.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:34 smithi132 ceph-mon[33500]: 10.13 scrub starts 2024-03-26T05:17:35.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:34 smithi132 ceph-mon[33500]: 10.17 scrub starts 2024-03-26T05:17:35.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:34 smithi132 ceph-mon[33500]: pgmap v794: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 948 op/s; 19133/135750 objects degraded (14.094%); 11304/135750 objects misplaced (8.327%); 7.5 MiB/s, 119 objects/s recovering 2024-03-26T05:17:35.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:34 smithi132 ceph-mon[33500]: 10.17 scrub ok 2024-03-26T05:17:35.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:34 smithi080 ceph-mon[40404]: 10.13 scrub starts 2024-03-26T05:17:35.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:34 smithi080 ceph-mon[40404]: 10.17 scrub starts 2024-03-26T05:17:35.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:34 smithi080 ceph-mon[40404]: pgmap v794: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 948 op/s; 19133/135750 objects degraded (14.094%); 11304/135750 objects misplaced (8.327%); 7.5 MiB/s, 119 objects/s recovering 2024-03-26T05:17:35.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:34 smithi080 ceph-mon[40404]: 10.17 scrub ok 2024-03-26T05:17:35.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:34 smithi080 ceph-mon[29691]: 10.13 scrub starts 2024-03-26T05:17:35.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:34 smithi080 ceph-mon[29691]: 10.17 scrub starts 2024-03-26T05:17:35.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:34 smithi080 ceph-mon[29691]: pgmap v794: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 948 op/s; 19133/135750 objects degraded (14.094%); 11304/135750 objects misplaced (8.327%); 7.5 MiB/s, 119 objects/s recovering 2024-03-26T05:17:35.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:34 smithi080 ceph-mon[29691]: 10.17 scrub ok 2024-03-26T05:17:35.123 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:35.162 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:35.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:35.264 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:35.304 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:35.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:35.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:35.406 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:35.435 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:35.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:35.537 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:35.566 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:35.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:35.667 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:35.703 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:17:35.710 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:35.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:35.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:35.813 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:35.839 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:35.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:35.941 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:35.968 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:35.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:35.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:36.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:35 smithi132 ceph-mon[33500]: 10.13 scrub ok 2024-03-26T05:17:36.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:35 smithi132 ceph-mon[33500]: 3.b scrub starts 2024-03-26T05:17:36.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:35 smithi132 ceph-mon[33500]: 3.b scrub ok 2024-03-26T05:17:36.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:17:36.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:17:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:17:36.069 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:36.112 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:36.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:36.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:36.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:35 smithi080 ceph-mon[29691]: 10.13 scrub ok 2024-03-26T05:17:36.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:35 smithi080 ceph-mon[29691]: 3.b scrub starts 2024-03-26T05:17:36.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:35 smithi080 ceph-mon[29691]: 3.b scrub ok 2024-03-26T05:17:36.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:17:36.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:35 smithi080 ceph-mon[40404]: 10.13 scrub ok 2024-03-26T05:17:36.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:35 smithi080 ceph-mon[40404]: 3.b scrub starts 2024-03-26T05:17:36.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:35 smithi080 ceph-mon[40404]: 3.b scrub ok 2024-03-26T05:17:36.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:17:36.214 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:36.242 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:36.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:36.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:36.344 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:36.369 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:36.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:36.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:36.471 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:36.520 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:36.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:36.622 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:36.652 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:36.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:36.753 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:36.784 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:36.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:36.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:36.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[29691]: 10.19 scrub starts 2024-03-26T05:17:36.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:36.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[29691]: 6.1b scrub starts 2024-03-26T05:17:36.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[29691]: 6.1b scrub ok 2024-03-26T05:17:36.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[29691]: pgmap v795: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 16937/119208 objects degraded (14.208%); 10292/119208 objects misplaced (8.634%) 2024-03-26T05:17:36.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 19133/135750 objects degraded (14.094%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:36.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[29691]: osdmap e385: 8 total, 7 up, 8 in 2024-03-26T05:17:36.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[40404]: 10.19 scrub starts 2024-03-26T05:17:36.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:36.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[40404]: 6.1b scrub starts 2024-03-26T05:17:36.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[40404]: 6.1b scrub ok 2024-03-26T05:17:36.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[40404]: pgmap v795: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 16937/119208 objects degraded (14.208%); 10292/119208 objects misplaced (8.634%) 2024-03-26T05:17:36.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 19133/135750 objects degraded (14.094%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:36.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:36 smithi080 ceph-mon[40404]: osdmap e385: 8 total, 7 up, 8 in 2024-03-26T05:17:36.885 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:36.914 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:36.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:36.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:37.016 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:37.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:36 smithi132 ceph-mon[33500]: 10.19 scrub starts 2024-03-26T05:17:37.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:37.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:36 smithi132 ceph-mon[33500]: 6.1b scrub starts 2024-03-26T05:17:37.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:36 smithi132 ceph-mon[33500]: 6.1b scrub ok 2024-03-26T05:17:37.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:36 smithi132 ceph-mon[33500]: pgmap v795: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 16937/119208 objects degraded (14.208%); 10292/119208 objects misplaced (8.634%) 2024-03-26T05:17:37.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:36 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 19133/135750 objects degraded (14.094%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:37.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:36 smithi132 ceph-mon[33500]: osdmap e385: 8 total, 7 up, 8 in 2024-03-26T05:17:37.046 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:37.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:37.097 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:17:37.143 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:17:37.148 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:37.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:37.180 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:37.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:37.183 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:17:37.224 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:17:37.268 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:17:37.281 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:37.305 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:17:37.325 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:37.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:37.342 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:17:37.342 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:17:37.343 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:17:37.343 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:17:37.343 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:17:37.343 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:17:37.343 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:17:37.343 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:17:37.343 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:17:37.427 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:37.465 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:37.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:37.567 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:37.600 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:37.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:37.702 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:37.739 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:37.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:37.841 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:37.879 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:37.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:37.981 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:38.009 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:38.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:38.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:38.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:37 smithi132 ceph-mon[33500]: 10.19 scrub ok 2024-03-26T05:17:38.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:37 smithi132 ceph-mon[33500]: 5.19 scrub starts 2024-03-26T05:17:38.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:37 smithi132 ceph-mon[33500]: 5.19 scrub ok 2024-03-26T05:17:38.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:37 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/747316233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:17:38.112 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:38.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:37 smithi080 ceph-mon[40404]: 10.19 scrub ok 2024-03-26T05:17:38.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:37 smithi080 ceph-mon[40404]: 5.19 scrub starts 2024-03-26T05:17:38.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:37 smithi080 ceph-mon[40404]: 5.19 scrub ok 2024-03-26T05:17:38.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:37 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/747316233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:17:38.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:37 smithi080 ceph-mon[29691]: 10.19 scrub ok 2024-03-26T05:17:38.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:37 smithi080 ceph-mon[29691]: 5.19 scrub starts 2024-03-26T05:17:38.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:37 smithi080 ceph-mon[29691]: 5.19 scrub ok 2024-03-26T05:17:38.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:37 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/747316233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:17:38.148 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:38.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:38.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:38.296 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:38.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:38.398 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:38.439 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:38.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:38.541 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:38.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:38.581 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:38.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:38.682 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:38.708 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:38.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:38.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:38.810 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:38.829 INFO:teuthology.orchestra.run.smithi080.stderr:set 4.b pg_upmap mapping to [0,4,2] 2024-03-26T05:17:38.848 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:17:38.848 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:17:38.848 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[40404]: 3.0 scrub starts 2024-03-26T05:17:38.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[40404]: 3.0 scrub ok 2024-03-26T05:17:38.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[40404]: osdmap e386: 8 total, 7 up, 8 in 2024-03-26T05:17:38.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[40404]: 5.14 scrub starts 2024-03-26T05:17:38.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[40404]: 5.14 scrub ok 2024-03-26T05:17:38.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[40404]: pgmap v798: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 16096/116844 objects degraded (13.776%); 8996/116844 objects misplaced (7.699%) 2024-03-26T05:17:38.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3344851543' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 4, 2]}]: dispatch 2024-03-26T05:17:38.849 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 4, 2]}]: dispatch 2024-03-26T05:17:38.849 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:17:38.849 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:17:38.849 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[29691]: 3.0 scrub starts 2024-03-26T05:17:38.849 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[29691]: 3.0 scrub ok 2024-03-26T05:17:38.849 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[29691]: osdmap e386: 8 total, 7 up, 8 in 2024-03-26T05:17:38.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[29691]: 5.14 scrub starts 2024-03-26T05:17:38.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[29691]: 5.14 scrub ok 2024-03-26T05:17:38.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[29691]: pgmap v798: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 16096/116844 objects degraded (13.776%); 8996/116844 objects misplaced (7.699%) 2024-03-26T05:17:38.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3344851543' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 4, 2]}]: dispatch 2024-03-26T05:17:38.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:38 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 4, 2]}]: dispatch 2024-03-26T05:17:38.852 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:38.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:38.955 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:38.985 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:38.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:39.087 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:17:39.093 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:17:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:39.131 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:39.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:39.232 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:39.262 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:39.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:38 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:17:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:38 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:17:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:38 smithi132 ceph-mon[33500]: 3.0 scrub starts 2024-03-26T05:17:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:38 smithi132 ceph-mon[33500]: 3.0 scrub ok 2024-03-26T05:17:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:38 smithi132 ceph-mon[33500]: osdmap e386: 8 total, 7 up, 8 in 2024-03-26T05:17:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:38 smithi132 ceph-mon[33500]: 5.14 scrub starts 2024-03-26T05:17:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:38 smithi132 ceph-mon[33500]: 5.14 scrub ok 2024-03-26T05:17:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:38 smithi132 ceph-mon[33500]: pgmap v798: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 16096/116844 objects degraded (13.776%); 8996/116844 objects misplaced (7.699%) 2024-03-26T05:17:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:38 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3344851543' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 4, 2]}]: dispatch 2024-03-26T05:17:39.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:38 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 4, 2]}]: dispatch 2024-03-26T05:17:39.320 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:39.363 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:39.391 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:39.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:39.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:39.493 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:39.540 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:39.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:39.641 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:39.677 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:39.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:39.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:39.778 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:39.809 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:39.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:39.910 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:39.954 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:39.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:40.055 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:40.097 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:40.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:40.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:39 smithi080 ceph-mon[40404]: 6.5 scrub starts 2024-03-26T05:17:40.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:39 smithi080 ceph-mon[40404]: 6.5 scrub ok 2024-03-26T05:17:40.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:39 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 4, 2]}]': finished 2024-03-26T05:17:40.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:39 smithi080 ceph-mon[40404]: osdmap e387: 8 total, 7 up, 8 in 2024-03-26T05:17:40.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:39 smithi080 ceph-mon[40404]: 4.11 scrub starts 2024-03-26T05:17:40.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:39 smithi080 ceph-mon[40404]: 4.11 scrub ok 2024-03-26T05:17:40.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:39 smithi080 ceph-mon[29691]: 6.5 scrub starts 2024-03-26T05:17:40.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:39 smithi080 ceph-mon[29691]: 6.5 scrub ok 2024-03-26T05:17:40.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:39 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 4, 2]}]': finished 2024-03-26T05:17:40.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:39 smithi080 ceph-mon[29691]: osdmap e387: 8 total, 7 up, 8 in 2024-03-26T05:17:40.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:39 smithi080 ceph-mon[29691]: 4.11 scrub starts 2024-03-26T05:17:40.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:39 smithi080 ceph-mon[29691]: 4.11 scrub ok 2024-03-26T05:17:40.199 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:40.252 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:40.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:40.279 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:39 smithi132 ceph-mon[33500]: 6.5 scrub starts 2024-03-26T05:17:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:39 smithi132 ceph-mon[33500]: 6.5 scrub ok 2024-03-26T05:17:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:39 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 4, 2]}]': finished 2024-03-26T05:17:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:39 smithi132 ceph-mon[33500]: osdmap e387: 8 total, 7 up, 8 in 2024-03-26T05:17:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:39 smithi132 ceph-mon[33500]: 4.11 scrub starts 2024-03-26T05:17:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:39 smithi132 ceph-mon[33500]: 4.11 scrub ok 2024-03-26T05:17:40.354 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:40.384 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:40.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:40.486 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:40.518 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:40.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:40.620 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:40.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:40.650 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:40.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:40.751 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:40.794 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:40.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:40.895 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:40.939 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:40.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:41.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:40 smithi080 ceph-mon[40404]: 3.d scrub starts 2024-03-26T05:17:41.036 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:40 smithi080 ceph-mon[40404]: 3.d scrub ok 2024-03-26T05:17:41.037 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:40 smithi080 ceph-mon[40404]: osdmap e388: 8 total, 7 up, 8 in 2024-03-26T05:17:41.037 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:40 smithi080 ceph-mon[40404]: pgmap v801: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 15104/108390 objects degraded (13.935%); 8447/108390 objects misplaced (7.793%); 6.0 MiB/s, 95 objects/s recovering 2024-03-26T05:17:41.037 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:40 smithi080 ceph-mon[29691]: 3.d scrub starts 2024-03-26T05:17:41.037 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:40 smithi080 ceph-mon[29691]: 3.d scrub ok 2024-03-26T05:17:41.037 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:40 smithi080 ceph-mon[29691]: osdmap e388: 8 total, 7 up, 8 in 2024-03-26T05:17:41.037 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:40 smithi080 ceph-mon[29691]: pgmap v801: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 15104/108390 objects degraded (13.935%); 8447/108390 objects misplaced (7.793%); 6.0 MiB/s, 95 objects/s recovering 2024-03-26T05:17:41.040 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:41.072 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:41.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:41.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:41.094 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:17:41.175 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:41.222 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:41.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:41.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:40 smithi132 ceph-mon[33500]: 3.d scrub starts 2024-03-26T05:17:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:40 smithi132 ceph-mon[33500]: 3.d scrub ok 2024-03-26T05:17:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:40 smithi132 ceph-mon[33500]: osdmap e388: 8 total, 7 up, 8 in 2024-03-26T05:17:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:40 smithi132 ceph-mon[33500]: pgmap v801: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing, 37 active+undersized, 95 active+clean, 30 active+undersized+degraded; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 15104/108390 objects degraded (13.935%); 8447/108390 objects misplaced (7.793%); 6.0 MiB/s, 95 objects/s recovering 2024-03-26T05:17:41.324 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:41.347 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:41.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:41.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:41.449 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:41.473 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:41.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:41.574 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:41.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:41.599 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:41.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:41.700 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:41.724 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:41.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:41.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:41.826 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:41.854 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:41.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:41.957 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:41.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:41.990 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:41.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:42.091 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:42.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[40404]: 1.7 scrub starts 2024-03-26T05:17:42.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[40404]: 1.7 scrub ok 2024-03-26T05:17:42.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[40404]: osdmap e389: 8 total, 7 up, 8 in 2024-03-26T05:17:42.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[40404]: 4.14 scrub starts 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[40404]: 4.14 scrub ok 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[40404]: 10.5 scrub starts 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[40404]: 10.5 scrub ok 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[40404]: 5.1e scrub starts 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[40404]: 5.1e scrub ok 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 15104/108390 objects degraded (13.935%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[29691]: 1.7 scrub starts 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[29691]: 1.7 scrub ok 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[29691]: osdmap e389: 8 total, 7 up, 8 in 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[29691]: 4.14 scrub starts 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[29691]: 4.14 scrub ok 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[29691]: 10.5 scrub starts 2024-03-26T05:17:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[29691]: 10.5 scrub ok 2024-03-26T05:17:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[29691]: 5.1e scrub starts 2024-03-26T05:17:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[29691]: 5.1e scrub ok 2024-03-26T05:17:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:41 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 15104/108390 objects degraded (13.935%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:42.130 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:42.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:42.231 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:42.270 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:42.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:42.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:41 smithi132 ceph-mon[33500]: 1.7 scrub starts 2024-03-26T05:17:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:41 smithi132 ceph-mon[33500]: 1.7 scrub ok 2024-03-26T05:17:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:41 smithi132 ceph-mon[33500]: osdmap e389: 8 total, 7 up, 8 in 2024-03-26T05:17:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:41 smithi132 ceph-mon[33500]: 4.14 scrub starts 2024-03-26T05:17:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:41 smithi132 ceph-mon[33500]: 4.14 scrub ok 2024-03-26T05:17:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:41 smithi132 ceph-mon[33500]: 10.5 scrub starts 2024-03-26T05:17:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:41 smithi132 ceph-mon[33500]: 10.5 scrub ok 2024-03-26T05:17:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:41 smithi132 ceph-mon[33500]: 5.1e scrub starts 2024-03-26T05:17:42.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:41 smithi132 ceph-mon[33500]: 5.1e scrub ok 2024-03-26T05:17:42.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:41 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 15104/108390 objects degraded (13.935%), 33 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:42.372 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:42.405 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:42.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:42.507 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:42.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:42.553 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:42.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:42.654 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:42.689 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:42.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:42.791 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:42.830 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:42.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:42.931 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:42.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:42.975 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:42.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:43.076 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:43.082 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[40404]: 5.4 scrub starts 2024-03-26T05:17:43.082 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[40404]: 5.4 scrub ok 2024-03-26T05:17:43.082 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[40404]: osdmap e390: 8 total, 7 up, 8 in 2024-03-26T05:17:43.082 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:17:43.083 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:17:43.083 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[40404]: pgmap v804: 168 pgs: 3 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 95 active+clean, 29 active+undersized+degraded; 2.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 15 op/s; 13608/99036 objects degraded (13.740%); 7686/99036 objects misplaced (7.761%) 2024-03-26T05:17:43.083 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[40404]: 10.3 scrub starts 2024-03-26T05:17:43.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[29691]: 5.4 scrub starts 2024-03-26T05:17:43.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[29691]: 5.4 scrub ok 2024-03-26T05:17:43.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[29691]: osdmap e390: 8 total, 7 up, 8 in 2024-03-26T05:17:43.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:17:43.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:17:43.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[29691]: pgmap v804: 168 pgs: 3 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 95 active+clean, 29 active+undersized+degraded; 2.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 15 op/s; 13608/99036 objects degraded (13.740%); 7686/99036 objects misplaced (7.761%) 2024-03-26T05:17:43.083 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:42 smithi080 ceph-mon[29691]: 10.3 scrub starts 2024-03-26T05:17:43.124 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:43.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:43.226 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:43.255 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:43.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:43.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:43.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:42 smithi132 ceph-mon[33500]: 5.4 scrub starts 2024-03-26T05:17:43.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:42 smithi132 ceph-mon[33500]: 5.4 scrub ok 2024-03-26T05:17:43.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:42 smithi132 ceph-mon[33500]: osdmap e390: 8 total, 7 up, 8 in 2024-03-26T05:17:43.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:42 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:17:43.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:42 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:17:43.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:42 smithi132 ceph-mon[33500]: pgmap v804: 168 pgs: 3 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 95 active+clean, 29 active+undersized+degraded; 2.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 15 op/s; 13608/99036 objects degraded (13.740%); 7686/99036 objects misplaced (7.761%) 2024-03-26T05:17:43.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:42 smithi132 ceph-mon[33500]: 10.3 scrub starts 2024-03-26T05:17:43.357 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:43.386 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:43.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:43.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:43.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:43.533 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:43.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:43.635 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:43.683 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:43.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:43.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:43.785 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:43.813 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:17:43.813 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:17:43.821 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:43.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:43.923 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:43.965 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:43.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:43.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:44.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[29691]: 10.7 scrub starts 2024-03-26T05:17:44.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[29691]: 10.9 scrub starts 2024-03-26T05:17:44.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[29691]: 10.7 scrub ok 2024-03-26T05:17:44.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[29691]: 10.3 scrub ok 2024-03-26T05:17:44.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[29691]: 10.9 scrub ok 2024-03-26T05:17:44.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[29691]: osdmap e391: 8 total, 7 up, 8 in 2024-03-26T05:17:44.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[29691]: 3.8 scrub starts 2024-03-26T05:17:44.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[29691]: 3.8 scrub ok 2024-03-26T05:17:44.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[29691]: 10.18 scrub starts 2024-03-26T05:17:44.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[29691]: 10.8 scrub starts 2024-03-26T05:17:44.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[40404]: 10.7 scrub starts 2024-03-26T05:17:44.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[40404]: 10.9 scrub starts 2024-03-26T05:17:44.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[40404]: 10.7 scrub ok 2024-03-26T05:17:44.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[40404]: 10.3 scrub ok 2024-03-26T05:17:44.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[40404]: 10.9 scrub ok 2024-03-26T05:17:44.025 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[40404]: osdmap e391: 8 total, 7 up, 8 in 2024-03-26T05:17:44.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[40404]: 3.8 scrub starts 2024-03-26T05:17:44.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[40404]: 3.8 scrub ok 2024-03-26T05:17:44.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[40404]: 10.18 scrub starts 2024-03-26T05:17:44.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:43 smithi080 ceph-mon[40404]: 10.8 scrub starts 2024-03-26T05:17:44.067 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:44.069 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:44.123 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:44.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:44.225 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:44.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:44.266 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:44.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:44.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:43 smithi132 ceph-mon[33500]: 10.7 scrub starts 2024-03-26T05:17:44.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:43 smithi132 ceph-mon[33500]: 10.9 scrub starts 2024-03-26T05:17:44.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:43 smithi132 ceph-mon[33500]: 10.7 scrub ok 2024-03-26T05:17:44.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:43 smithi132 ceph-mon[33500]: 10.3 scrub ok 2024-03-26T05:17:44.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:43 smithi132 ceph-mon[33500]: 10.9 scrub ok 2024-03-26T05:17:44.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:43 smithi132 ceph-mon[33500]: osdmap e391: 8 total, 7 up, 8 in 2024-03-26T05:17:44.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:43 smithi132 ceph-mon[33500]: 3.8 scrub starts 2024-03-26T05:17:44.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:43 smithi132 ceph-mon[33500]: 3.8 scrub ok 2024-03-26T05:17:44.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:43 smithi132 ceph-mon[33500]: 10.18 scrub starts 2024-03-26T05:17:44.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:43 smithi132 ceph-mon[33500]: 10.8 scrub starts 2024-03-26T05:17:44.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:17:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:17:44.367 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:44.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:44.410 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:44.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:44.511 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:44.550 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T05:17:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:44.575 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:44.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:44.676 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:44.703 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:44.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:44.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:44.804 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:44.851 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:44.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:17:44.952 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:44.992 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:44.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:44.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:45.093 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[29691]: 10.12 scrub starts 2024-03-26T05:17:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[29691]: 10.12 scrub ok 2024-03-26T05:17:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[29691]: 10.18 scrub ok 2024-03-26T05:17:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[29691]: 6.1 scrub starts 2024-03-26T05:17:45.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[29691]: 10.8 scrub ok 2024-03-26T05:17:45.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[29691]: 6.1 scrub ok 2024-03-26T05:17:45.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[29691]: pgmap v806: 168 pgs: 3 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 95 active+clean, 29 active+undersized+degraded; 2.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 14 op/s; 13608/99036 objects degraded (13.740%); 7686/99036 objects misplaced (7.761%) 2024-03-26T05:17:45.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1715180358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:17:45.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[29691]: 6.9 scrub starts 2024-03-26T05:17:45.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[29691]: 6.9 scrub ok 2024-03-26T05:17:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[40404]: 10.12 scrub starts 2024-03-26T05:17:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[40404]: 10.12 scrub ok 2024-03-26T05:17:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[40404]: 10.18 scrub ok 2024-03-26T05:17:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[40404]: 6.1 scrub starts 2024-03-26T05:17:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[40404]: 10.8 scrub ok 2024-03-26T05:17:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[40404]: 6.1 scrub ok 2024-03-26T05:17:45.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[40404]: pgmap v806: 168 pgs: 3 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 95 active+clean, 29 active+undersized+degraded; 2.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 14 op/s; 13608/99036 objects degraded (13.740%); 7686/99036 objects misplaced (7.761%) 2024-03-26T05:17:45.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1715180358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:17:45.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[40404]: 6.9 scrub starts 2024-03-26T05:17:45.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:44 smithi080 ceph-mon[40404]: 6.9 scrub ok 2024-03-26T05:17:45.122 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:45.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:45.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:45.169 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T05:17:45.170 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:17:45.170 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-26T05:17:45.170 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T05:17:45.170 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T05:17:45.170 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-26T05:17:45.203 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:44 smithi132 ceph-mon[33500]: 10.12 scrub starts 2024-03-26T05:17:45.203 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:44 smithi132 ceph-mon[33500]: 10.12 scrub ok 2024-03-26T05:17:45.203 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:44 smithi132 ceph-mon[33500]: 10.18 scrub ok 2024-03-26T05:17:45.203 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:44 smithi132 ceph-mon[33500]: 6.1 scrub starts 2024-03-26T05:17:45.203 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:44 smithi132 ceph-mon[33500]: 10.8 scrub ok 2024-03-26T05:17:45.203 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:44 smithi132 ceph-mon[33500]: 6.1 scrub ok 2024-03-26T05:17:45.203 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:44 smithi132 ceph-mon[33500]: pgmap v806: 168 pgs: 3 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 95 active+clean, 29 active+undersized+degraded; 2.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 14 op/s; 13608/99036 objects degraded (13.740%); 7686/99036 objects misplaced (7.761%) 2024-03-26T05:17:45.204 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:44 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1715180358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:17:45.204 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:44 smithi132 ceph-mon[33500]: 6.9 scrub starts 2024-03-26T05:17:45.204 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:44 smithi132 ceph-mon[33500]: 6.9 scrub ok 2024-03-26T05:17:45.225 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:45.254 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:45.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:45.356 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:45.394 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:45.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:45.496 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:45.533 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:45.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:45.634 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:45.674 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:45.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:45.776 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:45.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:45.808 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:45.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:45.909 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:45.949 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:45.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:46.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:17:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:17:46.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:45 smithi132 ceph-mon[33500]: 10.16 scrub starts 2024-03-26T05:17:46.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:45 smithi132 ceph-mon[33500]: 10.11 scrub starts 2024-03-26T05:17:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:45 smithi132 ceph-mon[33500]: 10.16 scrub ok 2024-03-26T05:17:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:45 smithi132 ceph-mon[33500]: 10.11 scrub ok 2024-03-26T05:17:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:45 smithi132 ceph-mon[33500]: 3.4 scrub starts 2024-03-26T05:17:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:45 smithi132 ceph-mon[33500]: 3.4 scrub ok 2024-03-26T05:17:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:45 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:17:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:45 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1715180358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:17:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:45 smithi132 ceph-mon[33500]: osdmap e392: 8 total, 7 up, 8 in 2024-03-26T05:17:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:45 smithi132 ceph-mon[33500]: 3.12 deep-scrub starts 2024-03-26T05:17:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:45 smithi132 ceph-mon[33500]: 3.12 deep-scrub ok 2024-03-26T05:17:46.051 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:46.082 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:46.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:46.184 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:46.221 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:46.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:46.322 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:46.333 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:17:46.346 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:46.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:46.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:46.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[29691]: 10.16 scrub starts 2024-03-26T05:17:46.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[29691]: 10.11 scrub starts 2024-03-26T05:17:46.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[29691]: 10.16 scrub ok 2024-03-26T05:17:46.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[29691]: 10.11 scrub ok 2024-03-26T05:17:46.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[29691]: 3.4 scrub starts 2024-03-26T05:17:46.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[29691]: 3.4 scrub ok 2024-03-26T05:17:46.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:17:46.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1715180358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:17:46.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[29691]: osdmap e392: 8 total, 7 up, 8 in 2024-03-26T05:17:46.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[29691]: 3.12 deep-scrub starts 2024-03-26T05:17:46.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[29691]: 3.12 deep-scrub ok 2024-03-26T05:17:46.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[40404]: 10.16 scrub starts 2024-03-26T05:17:46.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[40404]: 10.11 scrub starts 2024-03-26T05:17:46.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[40404]: 10.16 scrub ok 2024-03-26T05:17:46.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[40404]: 10.11 scrub ok 2024-03-26T05:17:46.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[40404]: 3.4 scrub starts 2024-03-26T05:17:46.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[40404]: 3.4 scrub ok 2024-03-26T05:17:46.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:17:46.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1715180358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:17:46.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[40404]: osdmap e392: 8 total, 7 up, 8 in 2024-03-26T05:17:46.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[40404]: 3.12 deep-scrub starts 2024-03-26T05:17:46.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:45 smithi080 ceph-mon[40404]: 3.12 deep-scrub ok 2024-03-26T05:17:46.447 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:46.489 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:46.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:46.589 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:46.613 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:46.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:46.715 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:46.759 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:46.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:46.862 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:46.889 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:46.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:46.991 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:47.025 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:47.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:47.127 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:47.153 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:47.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:47.214 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:46 smithi132 ceph-mon[33500]: 4.a deep-scrub starts 2024-03-26T05:17:47.214 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:46 smithi132 ceph-mon[33500]: 4.a deep-scrub ok 2024-03-26T05:17:47.214 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:46 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:47.214 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:46 smithi132 ceph-mon[33500]: pgmap v808: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 11521/84684 objects degraded (13.605%); 6274/84684 objects misplaced (7.409%) 2024-03-26T05:17:47.214 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:46 smithi132 ceph-mon[33500]: 5.d scrub starts 2024-03-26T05:17:47.215 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:46 smithi132 ceph-mon[33500]: 5.d scrub ok 2024-03-26T05:17:47.215 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:46 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 13608/99036 objects degraded (13.740%), 32 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:47.215 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:46 smithi132 ceph-mon[33500]: osdmap e393: 8 total, 7 up, 8 in 2024-03-26T05:17:47.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:47.285 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:47.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:47.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[40404]: 4.a deep-scrub starts 2024-03-26T05:17:47.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[40404]: 4.a deep-scrub ok 2024-03-26T05:17:47.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:47.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[40404]: pgmap v808: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 11521/84684 objects degraded (13.605%); 6274/84684 objects misplaced (7.409%) 2024-03-26T05:17:47.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[40404]: 5.d scrub starts 2024-03-26T05:17:47.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[40404]: 5.d scrub ok 2024-03-26T05:17:47.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 13608/99036 objects degraded (13.740%), 32 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:47.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[40404]: osdmap e393: 8 total, 7 up, 8 in 2024-03-26T05:17:47.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[29691]: 4.a deep-scrub starts 2024-03-26T05:17:47.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[29691]: 4.a deep-scrub ok 2024-03-26T05:17:47.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:47.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[29691]: pgmap v808: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 11521/84684 objects degraded (13.605%); 6274/84684 objects misplaced (7.409%) 2024-03-26T05:17:47.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[29691]: 5.d scrub starts 2024-03-26T05:17:47.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[29691]: 5.d scrub ok 2024-03-26T05:17:47.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 13608/99036 objects degraded (13.740%), 32 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:47.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:46 smithi080 ceph-mon[29691]: osdmap e393: 8 total, 7 up, 8 in 2024-03-26T05:17:47.387 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:47.424 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:47.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:47.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:47.525 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:47.559 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:47.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:47.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:47.661 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:47.688 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:47.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:47.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:47.789 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:47.813 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:47.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:47.914 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:47.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:47.949 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:47.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:48.050 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:48.086 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:48.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:48.188 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:48.218 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:47 smithi132 ceph-mon[33500]: 2.0 scrub starts 2024-03-26T05:17:48.218 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:47 smithi132 ceph-mon[33500]: 2.0 scrub ok 2024-03-26T05:17:48.218 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:47 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:48.228 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:48.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:48.328 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:48.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:47 smithi080 ceph-mon[40404]: 2.0 scrub starts 2024-03-26T05:17:48.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:47 smithi080 ceph-mon[40404]: 2.0 scrub ok 2024-03-26T05:17:48.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:47 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:48.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:47 smithi080 ceph-mon[29691]: 2.0 scrub starts 2024-03-26T05:17:48.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:47 smithi080 ceph-mon[29691]: 2.0 scrub ok 2024-03-26T05:17:48.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:47 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:48.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:48.372 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:48.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:48.473 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:48.505 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:48.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:48.607 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:48.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:48.649 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:48.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:48.750 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:48.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:48.791 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:48.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:48.892 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:48.925 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:48.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:49.027 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:49.065 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:49.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:49.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:49.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:49.168 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:48 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:49.168 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:48 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:49.168 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:48 smithi132 ceph-mon[33500]: pgmap v810: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 11159/83052 objects degraded (13.436%); 5904/83052 objects misplaced (7.109%) 2024-03-26T05:17:49.186 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:17:49.186 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T05:17:49.186 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:17:49.209 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:49.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:49.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:49.311 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:49.354 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:49.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:49.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:48 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:49.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:48 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:49.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:48 smithi080 ceph-mon[40404]: pgmap v810: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 11159/83052 objects degraded (13.436%); 5904/83052 objects misplaced (7.109%) 2024-03-26T05:17:49.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:48 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:49.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:48 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:49.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:48 smithi080 ceph-mon[29691]: pgmap v810: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 11159/83052 objects degraded (13.436%); 5904/83052 objects misplaced (7.109%) 2024-03-26T05:17:49.455 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:49.496 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:49.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:49.598 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:49.632 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:49.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:49.734 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:49.764 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:49.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:49.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:49.866 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:49.897 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:49.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:49.999 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:50.033 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:50.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:50.135 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:50.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:50.194 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:50.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:50.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:50.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:50.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:49 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:50.291 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:50.292 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:50.292 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:49 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:50.292 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:50.292 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:50.292 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:49 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:50.295 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:50.324 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:50.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:50.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:50.426 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:50.455 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:50.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:50.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:50.538 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:50.557 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:50.617 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:50.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:50.660 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:17:50.701 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:17:50.719 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:50.752 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:17:50.754 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:50.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:50.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:50.800 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:17:50.844 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:17:50.856 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:50.894 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:17:50.898 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:50.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:50.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:50.944 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:17:50.945 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:17:50.945 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:17:50.945 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:17:50.945 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:17:50.945 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:17:50.945 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:17:50.945 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:17:50.945 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:17:51.001 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:51.033 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:51.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:51.136 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:51.177 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:51.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:51.278 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:51.314 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:51.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:51.415 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:51.451 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:51.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:51.552 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:51.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:51.687 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:51.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: pgmap v811: 168 pgs: 1 active+remapped+backfilling, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 706 B/s rd, 0 op/s; 10345/75237 objects degraded (13.750%); 4750/75237 objects misplaced (6.313%) 2024-03-26T05:17:51.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: 3.10 deep-scrub starts 2024-03-26T05:17:51.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: 3.10 deep-scrub ok 2024-03-26T05:17:51.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:51.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:17:51.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/833148228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:17:51.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: 6.1c scrub starts 2024-03-26T05:17:51.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: 6.1c scrub ok 2024-03-26T05:17:51.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:51.689 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: 5.6 scrub starts 2024-03-26T05:17:51.690 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: 5.6 scrub ok 2024-03-26T05:17:51.690 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: 4.1e scrub starts 2024-03-26T05:17:51.690 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[29691]: 4.1e scrub ok 2024-03-26T05:17:51.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: pgmap v811: 168 pgs: 1 active+remapped+backfilling, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 706 B/s rd, 0 op/s; 10345/75237 objects degraded (13.750%); 4750/75237 objects misplaced (6.313%) 2024-03-26T05:17:51.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: 3.10 deep-scrub starts 2024-03-26T05:17:51.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: 3.10 deep-scrub ok 2024-03-26T05:17:51.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:51.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:17:51.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/833148228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:17:51.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: 6.1c scrub starts 2024-03-26T05:17:51.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: 6.1c scrub ok 2024-03-26T05:17:51.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:51.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: 5.6 scrub starts 2024-03-26T05:17:51.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: 5.6 scrub ok 2024-03-26T05:17:51.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: 4.1e scrub starts 2024-03-26T05:17:51.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:51 smithi080 ceph-mon[40404]: 4.1e scrub ok 2024-03-26T05:17:51.719 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:51.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:51.801 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: pgmap v811: 168 pgs: 1 active+remapped+backfilling, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 706 B/s rd, 0 op/s; 10345/75237 objects degraded (13.750%); 4750/75237 objects misplaced (6.313%) 2024-03-26T05:17:51.801 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: 3.10 deep-scrub starts 2024-03-26T05:17:51.802 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: 3.10 deep-scrub ok 2024-03-26T05:17:51.802 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:17:51.802 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:17:51.802 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/833148228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:17:51.802 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: 6.1c scrub starts 2024-03-26T05:17:51.802 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: 6.1c scrub ok 2024-03-26T05:17:51.802 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:51.802 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: 5.6 scrub starts 2024-03-26T05:17:51.802 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: 5.6 scrub ok 2024-03-26T05:17:51.802 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: 4.1e scrub starts 2024-03-26T05:17:51.802 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:51 smithi132 ceph-mon[33500]: 4.1e scrub ok 2024-03-26T05:17:51.821 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:51.859 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:51.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:51.960 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:52.000 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:52.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:52.002 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:17:52.102 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:52.132 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:52.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:52.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:52.180 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:17:52.181 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:17:52.234 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:52.265 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:52.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:52.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:52.367 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:52.414 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:52.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:52.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:52.516 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:52.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:52.548 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:52.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:52.649 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:52.681 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:52.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:52.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:52.782 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:52.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:52 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 10345/75237 objects degraded (13.750%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:52.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:52 smithi080 ceph-mon[29691]: 5.8 scrub starts 2024-03-26T05:17:52.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:52 smithi080 ceph-mon[29691]: 5.8 scrub ok 2024-03-26T05:17:52.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:52 smithi080 ceph-mon[29691]: 6.18 scrub starts 2024-03-26T05:17:52.841 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:52 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 10345/75237 objects degraded (13.750%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:52.842 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:52 smithi080 ceph-mon[40404]: 5.8 scrub starts 2024-03-26T05:17:52.842 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:52 smithi080 ceph-mon[40404]: 5.8 scrub ok 2024-03-26T05:17:52.842 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:52 smithi080 ceph-mon[40404]: 6.18 scrub starts 2024-03-26T05:17:52.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:52.844 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:52.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:52.945 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:52.986 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:52.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:52.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:53.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:52 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 10345/75237 objects degraded (13.750%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:53.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:52 smithi132 ceph-mon[33500]: 5.8 scrub starts 2024-03-26T05:17:53.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:52 smithi132 ceph-mon[33500]: 5.8 scrub ok 2024-03-26T05:17:53.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:52 smithi132 ceph-mon[33500]: 6.18 scrub starts 2024-03-26T05:17:53.088 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:53.119 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:53.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:53.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:53.221 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:53.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:53.264 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:53.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:53.288 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:53.365 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:53.393 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:17:53.393 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:17:53.393 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:17:53.393 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:17:53.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.393 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:75288 10.c 10:39a3c057:::benchmark_data_smithi080_161990_object75287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e336)", 2024-03-26T05:17:53.393 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:15:32.612902+0000", 2024-03-26T05:17:53.393 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 140.78427535200001, 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.105485719, 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 75288 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612902+0000", 2024-03-26T05:17:53.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612902+0000", 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612903+0000", 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8899999999999999e-07 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612927+0000", 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4192000000000001e-05 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612936+0000", 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0410000000000003e-06 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612943+0000", 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.674e-06 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.396 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618645+0000", 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0057027739999999999 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618649+0000", 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8029999999999998e-06 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.711588+0000", 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0929387699999999 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.397 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.711690+0000", 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010237 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,7", 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.711799+0000", 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000108787 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.715595+0000", 2024-03-26T05:17:53.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0037958430000000001 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.717997+0000", 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0024021649999999999 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718266+0000", 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00026850799999999998 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.399 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718301+0000", 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5404999999999997e-05 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718388+0000", 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6798000000000005e-05 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:75295 10.c 10:34625f2b:::benchmark_data_smithi080_161990_object75294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e336)", 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:15:32.612952+0000", 2024-03-26T05:17:53.400 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 140.78422551200001, 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.112282223, 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 75295 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612952+0000", 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612952+0000", 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612952+0000", 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5999999999999999e-07 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.402 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612974+0000", 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1180000000000001e-05 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612981+0000", 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6260000000000003e-06 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612988+0000", 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5050000000000004e-06 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.403 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618673+0000", 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0056853609999999999 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618677+0000", 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5810000000000001e-06 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.713508+0000", 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.094830787 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.404 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.713636+0000", 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000128009 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,7", 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.713806+0000", 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017006899999999999 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718010+0000", 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0042043200000000001 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718223+0000", 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000213496 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725036+0000", 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0068121019999999996 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725109+0000", 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3539000000000002e-05 2024-03-26T05:17:53.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725234+0000", 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012518800000000001 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:75302 10.c 10:32bd27e8:::benchmark_data_smithi080_161990_object75301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e336)", 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:15:32.612996+0000", 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 140.78418157499999, 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.112536143, 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.407 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 75302 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612996+0000", 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612996+0000", 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612996+0000", 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7800000000000002e-07 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613092+0000", 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5562000000000004e-05 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613102+0000", 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0127e-05 2024-03-26T05:17:53.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613110+0000", 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4930000000000001e-06 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618701+0000", 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055912059999999996 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618704+0000", 2024-03-26T05:17:53.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5240000000000001e-06 2024-03-26T05:17:53.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.414 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.715630+0000", 2024-03-26T05:17:53.414 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0969253430000001 2024-03-26T05:17:53.414 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.414 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.414 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:17:53.414 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.715759+0000", 2024-03-26T05:17:53.414 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000129696 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,7", 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.715956+0000", 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019639200000000001 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718232+0000", 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022764870000000002 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718445+0000", 2024-03-26T05:17:53.415 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021288500000000001 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725356+0000", 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0069105149999999999 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725423+0000", 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6884000000000005e-05 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.416 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725532+0000", 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000109551 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:75303 10.c 10:3d000c38:::benchmark_data_smithi080_161990_object75302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e336)", 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:15:32.613121+0000", 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 140.78405701599999, 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.112658986, 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.417 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 75303 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613121+0000", 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613121+0000", 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613121+0000", 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1699999999999998e-07 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613144+0000", 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2555e-05 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613152+0000", 2024-03-26T05:17:53.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8869999999999992e-06 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613158+0000", 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2269999999999998e-06 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618720+0000", 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055622780000000004 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618724+0000", 2024-03-26T05:17:53.420 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3239999999999996e-06 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718491+0000", 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0997661860000001 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718602+0000", 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000111718 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,7", 2024-03-26T05:17:53.421 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718702+0000", 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9610000000000003e-05 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.724985+0000", 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0062834459999999998 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.724996+0000", 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0403e-05 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.422 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:17:53.423 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725598+0000", 2024-03-26T05:17:53.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00060190199999999997 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725664+0000", 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6790000000000003e-05 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725780+0000", 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115043 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.424 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141751 10.18 10.aa5d358 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:50.719618+0000", 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 62.677559703, 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.9921031060000001, 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141751 2024-03-26T05:17:53.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.719618+0000", 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.719618+0000", 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.426 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.719618+0000", 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5900000000000003e-07 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.719620+0000", 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0810000000000001e-06 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.719626+0000", 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5879999999999997e-06 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.427 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.719630+0000", 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.622e-06 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.909973+0000", 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.190342961 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.909981+0000", 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9889999999999992e-06 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710386+0000", 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8004045870000001 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710398+0000", 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2138e-05 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.711708+0000", 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.001309923 2024-03-26T05:17:53.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.711721+0000", 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2858e-05 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141765 10.18 10.fd02f5d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.726425+0000", 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.670752446000002, 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9870093049999999, 2024-03-26T05:17:53.430 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141765 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.726425+0000", 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.431 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.726425+0000", 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.726426+0000", 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3969999999999999e-06 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.726431+0000", 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5050000000000001e-06 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.432 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.726441+0000", 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0202e-05 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.726455+0000", 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3762e-05 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.734757+0000", 2024-03-26T05:17:53.433 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0083015550000000004 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.734771+0000", 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.482e-05 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710425+0000", 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97565345599999997 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710432+0000", 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2409999999999998e-06 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713413+0000", 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0029810889999999 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713434+0000", 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1277999999999999e-05 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.435 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141814 10.18 10.1bf2e6f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.770677+0000", 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.626500937000003, 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9427979639999999, 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141814 2024-03-26T05:17:53.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770677+0000", 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770677+0000", 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770678+0000", 2024-03-26T05:17:53.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.869e-06 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770682+0000", 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.703e-06 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770692+0000", 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.022e-05 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770700+0000", 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2339999999999997e-06 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770888+0000", 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000188703 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770897+0000", 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4579999999999993e-06 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710457+0000", 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93956030400000001 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710464+0000", 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5899999999999996e-06 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713466+0000", 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.003002395 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713475+0000", 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4880000000000007e-06 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141838 10.18 10.28ed5a58 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.789128+0000", 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.608049749999999, 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9243787189999999, 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.441 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141838 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.789128+0000", 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.442 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.789128+0000", 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.789129+0000", 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6600000000000005e-07 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.789134+0000", 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8449999999999999e-06 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.789148+0000", 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3341e-05 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.789248+0000", 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010049599999999999 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.792405+0000", 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0031570399999999998 2024-03-26T05:17:53.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.792416+0000", 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0246e-05 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710487+0000", 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.918070948 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710493+0000", 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9290000000000001e-06 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713498+0000", 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030049729999999 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713507+0000", 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0350000000000001e-06 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.446 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141870 10.18 10.f77d5298 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.808382+0000", 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.588795095000002, 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9051565770000001, 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141870 2024-03-26T05:17:53.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.808382+0000", 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.808382+0000", 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.808384+0000", 2024-03-26T05:17:53.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.598e-06 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.808390+0000", 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8250000000000001e-06 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.808402+0000", 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2278e-05 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.808411+0000", 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.439e-06 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.810206+0000", 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0017952210000000001 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.810215+0000", 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8820000000000007e-06 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710516+0000", 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.90030157399999999 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710523+0000", 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2790000000000004e-06 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713531+0000", 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030082760000001 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.452 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.452 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713539+0000", 2024-03-26T05:17:53.452 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2050000000000002e-06 2024-03-26T05:17:53.452 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.452 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.452 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:53.453 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:53.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:53.455 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141873 10.18 10.4c990018 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.455 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.809190+0000", 2024-03-26T05:17:53.455 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.587987253999998, 2024-03-26T05:17:53.455 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.90438094, 2024-03-26T05:17:53.455 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.455 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.455 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.455 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.455 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.455 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141873 2024-03-26T05:17:53.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.455 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.809190+0000", 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.809190+0000", 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.809191+0000", 2024-03-26T05:17:53.456 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3200000000000004e-07 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.809195+0000", 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9949999999999999e-06 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.809206+0000", 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0743e-05 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.809213+0000", 2024-03-26T05:17:53.457 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3810000000000001e-06 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.810236+0000", 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010223000000000001 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.810241+0000", 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5119999999999999e-06 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.458 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710544+0000", 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.90030308199999998 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710551+0000", 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1620000000000002e-06 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713564+0000", 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030123719999999 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.459 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713571+0000", 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5610000000000001e-06 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141891 10.18 10.e5b3c9d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.825300+0000", 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.571877063999999, 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8883019509999999, 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.460 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141891 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825300+0000", 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825300+0000", 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.461 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825302+0000", 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3149999999999999e-06 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825305+0000", 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2780000000000002e-06 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.462 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825314+0000", 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5180000000000004e-06 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825320+0000", 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8569999999999997e-06 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825825+0000", 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000504262 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.463 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.826868+0000", 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001043247 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710579+0000", 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.88371147800000005 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710587+0000", 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7980000000000006e-06 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.464 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713594+0000", 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030070179999999 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713602+0000", 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1799999999999996e-06 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141933 10.18 10.ccf33018 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.465 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.855386+0000", 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.541792010999998, 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8582478579999999, 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141933 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.466 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855386+0000", 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855386+0000", 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855387+0000", 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5650000000000001e-06 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.467 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855392+0000", 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9609999999999996e-06 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855408+0000", 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6242000000000001e-05 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855425+0000", 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6572e-05 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.468 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855532+0000", 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000106999 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855543+0000", 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0845999999999999e-05 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710612+0000", 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.85506965499999998 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.469 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710620+0000", 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1550000000000001e-06 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713625+0000", 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030055760000001 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713633+0000", 2024-03-26T05:17:53.470 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2870000000000004e-06 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142054 10.18 10.8fe3ef98 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.933106+0000", 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.464071247, 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7805604609999999, 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.471 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142054 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933106+0000", 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933106+0000", 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.472 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933108+0000", 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.257e-06 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933111+0000", 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.765e-06 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933164+0000", 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2379000000000001e-05 2024-03-26T05:17:53.473 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933278+0000", 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000114252 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933403+0000", 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012489399999999999 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933412+0000", 2024-03-26T05:17:53.474 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5000000000000005e-06 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710645+0000", 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.77723299800000001 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710652+0000", 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0910000000000004e-06 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.475 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713658+0000", 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.003005916 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713667+0000", 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4090000000000003e-06 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142075 10.18 10.efe5c358 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.476 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.945089+0000", 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.452088670999998, 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.768610056, 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142075 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945089+0000", 2024-03-26T05:17:53.477 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945089+0000", 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945090+0000", 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.04e-06 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.478 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945094+0000", 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1400000000000002e-06 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945106+0000", 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2123999999999999e-05 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945114+0000", 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8860000000000004e-06 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.479 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945149+0000", 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5429999999999998e-05 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945155+0000", 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2279999999999998e-06 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710679+0000", 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.76552449199999995 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.480 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710687+0000", 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3350000000000001e-06 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713691+0000", 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.003004556 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713699+0000", 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8250000000000005e-06 2024-03-26T05:17:53.481 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142081 10.18 10.809f8d38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.947216+0000", 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.449961840999997, 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.76651465, 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.482 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142081 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947216+0000", 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947216+0000", 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.483 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947216+0000", 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.37e-07 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947220+0000", 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8199999999999998e-06 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947230+0000", 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2969999999999992e-06 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.484 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947236+0000", 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1990000000000003e-06 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947679+0000", 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00044369400000000001 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947688+0000", 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2179999999999999e-06 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.485 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710712+0000", 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.76302396100000003 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710719+0000", 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2989999999999999e-06 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713723+0000", 2024-03-26T05:17:53.486 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030039209999999 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713730+0000", 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5039999999999996e-06 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142089 10.18 10.b56a2a38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.952307+0000", 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.444870733000002, 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7614548509999999, 2024-03-26T05:17:53.487 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142089 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952307+0000", 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.488 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952307+0000", 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952308+0000", 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.035e-06 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952312+0000", 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8619999999999996e-06 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.489 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952348+0000", 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6599e-05 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952357+0000", 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4440000000000008e-06 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952508+0000", 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015090000000000001 2024-03-26T05:17:53.490 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952517+0000", 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2560000000000008e-06 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710745+0000", 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.75822770399999995 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710751+0000", 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6209999999999998e-06 2024-03-26T05:17:53.491 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713754+0000", 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030027450000001 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713762+0000", 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6850000000000001e-06 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.492 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142118 10.18 10.c99f83d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.978316+0000", 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.418861137999997, 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.735477129, 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142118 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.493 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978316+0000", 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978316+0000", 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978318+0000", 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7060000000000001e-06 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.494 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978323+0000", 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9629999999999997e-06 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978338+0000", 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5274e-05 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978346+0000", 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6909999999999997e-06 2024-03-26T05:17:53.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978414+0000", 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8128999999999998e-05 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978428+0000", 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3898e-05 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710774+0000", 2024-03-26T05:17:53.496 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.73234595899999999 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710781+0000", 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.204e-06 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713785+0000", 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030040730000001 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.497 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713794+0000", 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2320000000000001e-06 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142151 10.18 10.ce0d3fd8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:52.011777+0000", 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.385400046000001, 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.702047256, 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.498 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142151 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011777+0000", 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011777+0000", 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.499 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011779+0000", 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.742e-06 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011784+0000", 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.028e-06 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011796+0000", 2024-03-26T05:17:53.500 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1688999999999999e-05 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011803+0000", 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.861e-06 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011860+0000", 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7661000000000002e-05 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011869+0000", 2024-03-26T05:17:53.501 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3420000000000008e-06 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710806+0000", 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.69893755000000002 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710813+0000", 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8090000000000004e-06 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.502 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713818+0000", 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.003004354 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713825+0000", 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2200000000000003e-06 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142154 10.18 10.78c53af8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:52.016807+0000", 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.380370747000001, 2024-03-26T05:17:53.503 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6970536140000001, 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142154 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016807+0000", 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.504 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016807+0000", 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016808+0000", 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.297e-06 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016813+0000", 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.724e-06 2024-03-26T05:17:53.505 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016826+0000", 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3387000000000001e-05 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016843+0000", 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6378000000000001e-05 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016885+0000", 2024-03-26T05:17:53.506 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2179999999999999e-05 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016892+0000", 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3289999999999996e-06 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710845+0000", 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.69395327799999995 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.507 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710854+0000", 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4749999999999993e-06 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713853+0000", 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0029995650000001 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713860+0000", 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0010000000000004e-06 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.508 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142155 10.18 10.ffb63458 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:52.016979+0000", 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 61.380198745000001, 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.697838637, 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142155 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:17:53.509 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.510 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:17:53.510 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016979+0000", 2024-03-26T05:17:53.510 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.510 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.510 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.510 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:17:53.510 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016979+0000", 2024-03-26T05:17:53.510 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016979+0000", 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3099999999999997e-07 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016983+0000", 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4920000000000002e-06 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:17:53.511 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016992+0000", 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4520000000000003e-06 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.017000+0000", 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5259999999999996e-06 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.017031+0000", 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.0621999999999999e-05 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.512 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.017034+0000", 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9519999999999996e-06 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710879+0000", 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.69384490899999995 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710887+0000", 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1359999999999999e-06 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.513 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.714807+0000", 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.003920022 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.714817+0000", 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0895e-05 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:17:53.514 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:17:53.555 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:53.602 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:53.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:53.703 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:53.741 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:53.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:53.843 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:53.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:53 smithi080 ceph-mon[29691]: pgmap v812: 168 pgs: 1 active+remapped+backfilling, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 9019/67152 objects degraded (13.431%); 4081/67152 objects misplaced (6.077%) 2024-03-26T05:17:53.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:53 smithi080 ceph-mon[29691]: 6.18 scrub ok 2024-03-26T05:17:53.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:53 smithi080 ceph-mon[29691]: 5.b scrub starts 2024-03-26T05:17:53.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:53 smithi080 ceph-mon[29691]: 5.b scrub ok 2024-03-26T05:17:53.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:53 smithi080 ceph-mon[29691]: 4.16 scrub starts 2024-03-26T05:17:53.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:53 smithi080 ceph-mon[40404]: pgmap v812: 168 pgs: 1 active+remapped+backfilling, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 9019/67152 objects degraded (13.431%); 4081/67152 objects misplaced (6.077%) 2024-03-26T05:17:53.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:53 smithi080 ceph-mon[40404]: 6.18 scrub ok 2024-03-26T05:17:53.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:53 smithi080 ceph-mon[40404]: 5.b scrub starts 2024-03-26T05:17:53.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:53 smithi080 ceph-mon[40404]: 5.b scrub ok 2024-03-26T05:17:53.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:53 smithi080 ceph-mon[40404]: 4.16 scrub starts 2024-03-26T05:17:53.878 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:53.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:53.979 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:54.012 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:54.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:54.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:53 smithi132 ceph-mon[33500]: pgmap v812: 168 pgs: 1 active+remapped+backfilling, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 9019/67152 objects degraded (13.431%); 4081/67152 objects misplaced (6.077%) 2024-03-26T05:17:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:53 smithi132 ceph-mon[33500]: 6.18 scrub ok 2024-03-26T05:17:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:53 smithi132 ceph-mon[33500]: 5.b scrub starts 2024-03-26T05:17:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:53 smithi132 ceph-mon[33500]: 5.b scrub ok 2024-03-26T05:17:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:53 smithi132 ceph-mon[33500]: 4.16 scrub starts 2024-03-26T05:17:54.114 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:54.140 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:54.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:54.182 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:17:54.241 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:54.268 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:54.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:54.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:17:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:17:54.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:54.394 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:54.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:54.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:54.496 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:54.520 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:54.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:54.621 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:54.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:54.646 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:54.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:54.747 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:54.788 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:54.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:54.890 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:54.924 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:54.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:55.026 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:55.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[29691]: 4.8 scrub starts 2024-03-26T05:17:55.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[29691]: 4.8 scrub ok 2024-03-26T05:17:55.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[29691]: 4.16 scrub ok 2024-03-26T05:17:55.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[29691]: osdmap e394: 8 total, 7 up, 8 in 2024-03-26T05:17:55.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[29691]: 4.0 scrub starts 2024-03-26T05:17:55.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[29691]: 4.0 scrub ok 2024-03-26T05:17:55.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[29691]: 4.1c scrub starts 2024-03-26T05:17:55.041 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[29691]: 4.1c scrub ok 2024-03-26T05:17:55.041 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[40404]: 4.8 scrub starts 2024-03-26T05:17:55.041 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[40404]: 4.8 scrub ok 2024-03-26T05:17:55.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[40404]: 4.16 scrub ok 2024-03-26T05:17:55.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[40404]: osdmap e394: 8 total, 7 up, 8 in 2024-03-26T05:17:55.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[40404]: 4.0 scrub starts 2024-03-26T05:17:55.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[40404]: 4.0 scrub ok 2024-03-26T05:17:55.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[40404]: 4.1c scrub starts 2024-03-26T05:17:55.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:54 smithi080 ceph-mon[40404]: 4.1c scrub ok 2024-03-26T05:17:55.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:54 smithi132 ceph-mon[33500]: 4.8 scrub starts 2024-03-26T05:17:55.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:54 smithi132 ceph-mon[33500]: 4.8 scrub ok 2024-03-26T05:17:55.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:54 smithi132 ceph-mon[33500]: 4.16 scrub ok 2024-03-26T05:17:55.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:54 smithi132 ceph-mon[33500]: osdmap e394: 8 total, 7 up, 8 in 2024-03-26T05:17:55.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:54 smithi132 ceph-mon[33500]: 4.0 scrub starts 2024-03-26T05:17:55.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:54 smithi132 ceph-mon[33500]: 4.0 scrub ok 2024-03-26T05:17:55.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:54 smithi132 ceph-mon[33500]: 4.1c scrub starts 2024-03-26T05:17:55.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:54 smithi132 ceph-mon[33500]: 4.1c scrub ok 2024-03-26T05:17:55.065 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T05:17:55.065 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:17:55.066 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-26T05:17:55.066 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-26T05:17:55.070 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:55.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:55.172 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:55.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:55.196 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:55.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:55.297 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:55.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:55.351 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:55.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:55.452 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:55.484 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:55.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:55.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:55.622 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:55.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:55.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:55.724 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:55.756 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:55.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:55.857 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:55.887 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:55.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:55.988 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:56.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:56.014 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:56.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:56.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:17:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:17:56.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:55 smithi132 ceph-mon[33500]: pgmap v814: 168 pgs: 1 active+remapped+backfilling, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 9019/67152 objects degraded (13.431%); 4081/67152 objects misplaced (6.077%) 2024-03-26T05:17:56.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:55 smithi132 ceph-mon[33500]: 6.8 scrub starts 2024-03-26T05:17:56.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:55 smithi132 ceph-mon[33500]: 6.8 scrub ok 2024-03-26T05:17:56.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:55 smithi132 ceph-mon[33500]: osdmap e395: 8 total, 7 up, 8 in 2024-03-26T05:17:56.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:55 smithi132 ceph-mon[33500]: 10.15 scrub starts 2024-03-26T05:17:56.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:55 smithi132 ceph-mon[33500]: 10.15 scrub ok 2024-03-26T05:17:56.115 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:56.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:55 smithi080 ceph-mon[40404]: pgmap v814: 168 pgs: 1 active+remapped+backfilling, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 9019/67152 objects degraded (13.431%); 4081/67152 objects misplaced (6.077%) 2024-03-26T05:17:56.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:55 smithi080 ceph-mon[40404]: 6.8 scrub starts 2024-03-26T05:17:56.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:55 smithi080 ceph-mon[40404]: 6.8 scrub ok 2024-03-26T05:17:56.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:55 smithi080 ceph-mon[40404]: osdmap e395: 8 total, 7 up, 8 in 2024-03-26T05:17:56.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:55 smithi080 ceph-mon[40404]: 10.15 scrub starts 2024-03-26T05:17:56.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:55 smithi080 ceph-mon[40404]: 10.15 scrub ok 2024-03-26T05:17:56.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:55 smithi080 ceph-mon[29691]: pgmap v814: 168 pgs: 1 active+remapped+backfilling, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 96 active+clean, 29 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 9019/67152 objects degraded (13.431%); 4081/67152 objects misplaced (6.077%) 2024-03-26T05:17:56.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:55 smithi080 ceph-mon[29691]: 6.8 scrub starts 2024-03-26T05:17:56.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:55 smithi080 ceph-mon[29691]: 6.8 scrub ok 2024-03-26T05:17:56.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:55 smithi080 ceph-mon[29691]: osdmap e395: 8 total, 7 up, 8 in 2024-03-26T05:17:56.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:55 smithi080 ceph-mon[29691]: 10.15 scrub starts 2024-03-26T05:17:56.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:55 smithi080 ceph-mon[29691]: 10.15 scrub ok 2024-03-26T05:17:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:56.144 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:56.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:56.245 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:56.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:56.278 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:56.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:56.379 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:56.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:56.421 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:56.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:56.522 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:56.564 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:56.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:56.666 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:56.711 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:56.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:56.813 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:56.850 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:56.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:56.952 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:56 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:17:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:56 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:17:57.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:56 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 9019/67152 objects degraded (13.431%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:57.046 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:57.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:57.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:56 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:17:57.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:56 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:17:57.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:56 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 9019/67152 objects degraded (13.431%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:56 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:17:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:56 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:17:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:56 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 9019/67152 objects degraded (13.431%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T05:17:57.148 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:57.187 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:57.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:57.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:57.289 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:17:57.327 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:57.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:57.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:17:57.429 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:57.454 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:57.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:57.555 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:57.592 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:57.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:57.694 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:57.738 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:57.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:57.840 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:57.869 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:57.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:57.971 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:57.995 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:57.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:58.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:57 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:58.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:57 smithi132 ceph-mon[33500]: pgmap v816: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 1.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 6918/51954 objects degraded (13.316%); 2792/51954 objects misplaced (5.374%); 6.5 MiB/s, 104 objects/s recovering 2024-03-26T05:17:58.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:57 smithi132 ceph-mon[33500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:58.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:57 smithi132 ceph-mon[33500]: 6.1f deep-scrub starts 2024-03-26T05:17:58.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:57 smithi132 ceph-mon[33500]: 6.1f deep-scrub ok 2024-03-26T05:17:58.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:58.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:57 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:58.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:57 smithi080 ceph-mon[40404]: pgmap v816: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 1.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 6918/51954 objects degraded (13.316%); 2792/51954 objects misplaced (5.374%); 6.5 MiB/s, 104 objects/s recovering 2024-03-26T05:17:58.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:57 smithi080 ceph-mon[40404]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:58.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:57 smithi080 ceph-mon[40404]: 6.1f deep-scrub starts 2024-03-26T05:17:58.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:57 smithi080 ceph-mon[40404]: 6.1f deep-scrub ok 2024-03-26T05:17:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:57 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:17:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:57 smithi080 ceph-mon[29691]: pgmap v816: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 1.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 6918/51954 objects degraded (13.316%); 2792/51954 objects misplaced (5.374%); 6.5 MiB/s, 104 objects/s recovering 2024-03-26T05:17:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:57 smithi080 ceph-mon[29691]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:17:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:57 smithi080 ceph-mon[29691]: 6.1f deep-scrub starts 2024-03-26T05:17:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:57 smithi080 ceph-mon[29691]: 6.1f deep-scrub ok 2024-03-26T05:17:58.139 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:58.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:58.241 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:58.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:58.282 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:58.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:58.383 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:58.424 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:58.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:58.526 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:58.557 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:58.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:58.658 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:58.692 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:58.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:58.794 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:58.820 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:58.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:58.922 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:58.960 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:58.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:58 smithi132 ceph-mon[33500]: 10.b scrub starts 2024-03-26T05:17:59.061 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:17:59.087 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:59.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:17:59.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:58 smithi080 ceph-mon[29691]: 10.b scrub starts 2024-03-26T05:17:59.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:58 smithi080 ceph-mon[40404]: 10.b scrub starts 2024-03-26T05:17:59.189 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:17:59.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:59.214 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:59.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:17:59.315 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:59.338 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:59.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:59.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:59.440 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:59.463 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:59.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:59.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:17:59.565 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:17:59.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:59.589 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:59.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:17:59.690 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:17:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:17:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:59.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:17:59.821 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:17:59.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:59.856 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:59.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:17:59.957 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:17:59.998 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:17:59.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:17:59.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:00.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:59 smithi132 ceph-mon[33500]: pgmap v817: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 1.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 6608/50490 objects degraded (13.088%); 2470/50490 objects misplaced (4.892%); 3.8 MiB/s, 60 objects/s recovering 2024-03-26T05:18:00.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:59 smithi132 ceph-mon[33500]: 10.b scrub ok 2024-03-26T05:18:00.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:59 smithi132 ceph-mon[33500]: osdmap e396: 8 total, 7 up, 8 in 2024-03-26T05:18:00.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:59 smithi132 ceph-mon[33500]: 1.4 deep-scrub starts 2024-03-26T05:18:00.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:59 smithi132 ceph-mon[33500]: 1.4 deep-scrub ok 2024-03-26T05:18:00.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:17:59 smithi132 ceph-mon[33500]: 6.16 deep-scrub starts 2024-03-26T05:18:00.092 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:59 smithi080 ceph-mon[40404]: pgmap v817: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 1.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 6608/50490 objects degraded (13.088%); 2470/50490 objects misplaced (4.892%); 3.8 MiB/s, 60 objects/s recovering 2024-03-26T05:18:00.092 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:59 smithi080 ceph-mon[40404]: 10.b scrub ok 2024-03-26T05:18:00.092 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:59 smithi080 ceph-mon[40404]: osdmap e396: 8 total, 7 up, 8 in 2024-03-26T05:18:00.092 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:59 smithi080 ceph-mon[40404]: 1.4 deep-scrub starts 2024-03-26T05:18:00.092 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:59 smithi080 ceph-mon[40404]: 1.4 deep-scrub ok 2024-03-26T05:18:00.092 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:17:59 smithi080 ceph-mon[40404]: 6.16 deep-scrub starts 2024-03-26T05:18:00.093 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:59 smithi080 ceph-mon[29691]: pgmap v817: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 1.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 6608/50490 objects degraded (13.088%); 2470/50490 objects misplaced (4.892%); 3.8 MiB/s, 60 objects/s recovering 2024-03-26T05:18:00.093 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:59 smithi080 ceph-mon[29691]: 10.b scrub ok 2024-03-26T05:18:00.093 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:59 smithi080 ceph-mon[29691]: osdmap e396: 8 total, 7 up, 8 in 2024-03-26T05:18:00.093 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:59 smithi080 ceph-mon[29691]: 1.4 deep-scrub starts 2024-03-26T05:18:00.094 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:59 smithi080 ceph-mon[29691]: 1.4 deep-scrub ok 2024-03-26T05:18:00.094 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:17:59 smithi080 ceph-mon[29691]: 6.16 deep-scrub starts 2024-03-26T05:18:00.100 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:00.147 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T05:18:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:00.166 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:00.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:00.268 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:00.298 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:00.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:00.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:00.400 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:00.436 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:00.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:00.538 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:00.588 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:00.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:00.690 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:00.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:00.730 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:00.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:00.831 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:00.868 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:00.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:00.969 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:01.001 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:01.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:01.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:00 smithi132 ceph-mon[33500]: 6.16 deep-scrub ok 2024-03-26T05:18:01.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:00 smithi132 ceph-mon[33500]: osdmap e397: 8 total, 7 up, 8 in 2024-03-26T05:18:01.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:00 smithi132 ceph-mon[33500]: pgmap v820: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 882 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 263 op/s; 5827/43017 objects degraded (13.546%); 2327/43017 objects misplaced (5.409%) 2024-03-26T05:18:01.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:00 smithi132 ceph-mon[33500]: 4.6 scrub starts 2024-03-26T05:18:01.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:00 smithi132 ceph-mon[33500]: 4.6 scrub ok 2024-03-26T05:18:01.103 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:01.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:00 smithi080 ceph-mon[40404]: 6.16 deep-scrub ok 2024-03-26T05:18:01.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:00 smithi080 ceph-mon[40404]: osdmap e397: 8 total, 7 up, 8 in 2024-03-26T05:18:01.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:00 smithi080 ceph-mon[40404]: pgmap v820: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 882 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 263 op/s; 5827/43017 objects degraded (13.546%); 2327/43017 objects misplaced (5.409%) 2024-03-26T05:18:01.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:00 smithi080 ceph-mon[40404]: 4.6 scrub starts 2024-03-26T05:18:01.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:00 smithi080 ceph-mon[40404]: 4.6 scrub ok 2024-03-26T05:18:01.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:00 smithi080 ceph-mon[29691]: 6.16 deep-scrub ok 2024-03-26T05:18:01.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:00 smithi080 ceph-mon[29691]: osdmap e397: 8 total, 7 up, 8 in 2024-03-26T05:18:01.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:00 smithi080 ceph-mon[29691]: pgmap v820: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 882 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 263 op/s; 5827/43017 objects degraded (13.546%); 2327/43017 objects misplaced (5.409%) 2024-03-26T05:18:01.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:00 smithi080 ceph-mon[29691]: 4.6 scrub starts 2024-03-26T05:18:01.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:00 smithi080 ceph-mon[29691]: 4.6 scrub ok 2024-03-26T05:18:01.133 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:01.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:01.235 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:01.286 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:01.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:01.387 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:01.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:01.426 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:01.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:01.527 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:01.559 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:01.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:01.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:01.661 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:01.703 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:01.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:01.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:01.805 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:01.838 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:01.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:01.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:01.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:01 smithi080 ceph-mon[29691]: 10.10 deep-scrub starts 2024-03-26T05:18:01.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:01 smithi080 ceph-mon[29691]: 4.7 scrub starts 2024-03-26T05:18:01.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:01 smithi080 ceph-mon[29691]: 4.7 scrub ok 2024-03-26T05:18:01.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:01 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 5827/43017 objects degraded (13.546%), 31 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:01.868 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:01 smithi080 ceph-mon[40404]: 10.10 deep-scrub starts 2024-03-26T05:18:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:01 smithi080 ceph-mon[40404]: 4.7 scrub starts 2024-03-26T05:18:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:01 smithi080 ceph-mon[40404]: 4.7 scrub ok 2024-03-26T05:18:01.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:01 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 5827/43017 objects degraded (13.546%), 31 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:01.940 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:01.980 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:01.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:02.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:01 smithi132 ceph-mon[33500]: 10.10 deep-scrub starts 2024-03-26T05:18:02.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:01 smithi132 ceph-mon[33500]: 4.7 scrub starts 2024-03-26T05:18:02.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:01 smithi132 ceph-mon[33500]: 4.7 scrub ok 2024-03-26T05:18:02.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:01 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 5827/43017 objects degraded (13.546%), 31 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:02.082 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:02.117 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:02.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:02.221 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:02.261 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:02.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:02.363 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:02.405 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:02.411 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:02.430 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:02.444 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:02.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:02.545 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:02.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:02.577 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:02.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:02.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:02.724 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:02.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:02.826 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:02.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:02 smithi080 ceph-mon[40404]: 10.f scrub starts 2024-03-26T05:18:02.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:02 smithi080 ceph-mon[40404]: 10.f scrub ok 2024-03-26T05:18:02.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:02 smithi080 ceph-mon[40404]: 6.3 deep-scrub starts 2024-03-26T05:18:02.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:02 smithi080 ceph-mon[40404]: 6.3 deep-scrub ok 2024-03-26T05:18:02.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:02 smithi080 ceph-mon[40404]: pgmap v821: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing+deep, 3 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 724 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 552 op/s; 4389/35433 objects degraded (12.387%); 1640/35433 objects misplaced (4.628%); 808 KiB/s, 12 objects/s recovering 2024-03-26T05:18:02.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:02 smithi080 ceph-mon[40404]: 6.0 scrub starts 2024-03-26T05:18:02.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:02 smithi080 ceph-mon[29691]: 10.f scrub starts 2024-03-26T05:18:02.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:02 smithi080 ceph-mon[29691]: 10.f scrub ok 2024-03-26T05:18:02.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:02 smithi080 ceph-mon[29691]: 6.3 deep-scrub starts 2024-03-26T05:18:02.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:02 smithi080 ceph-mon[29691]: 6.3 deep-scrub ok 2024-03-26T05:18:02.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:02 smithi080 ceph-mon[29691]: pgmap v821: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing+deep, 3 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 724 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 552 op/s; 4389/35433 objects degraded (12.387%); 1640/35433 objects misplaced (4.628%); 808 KiB/s, 12 objects/s recovering 2024-03-26T05:18:02.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:02 smithi080 ceph-mon[29691]: 6.0 scrub starts 2024-03-26T05:18:02.872 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:02.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:02.974 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:03.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:03.015 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:03.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:03.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:02 smithi132 ceph-mon[33500]: 10.f scrub starts 2024-03-26T05:18:03.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:02 smithi132 ceph-mon[33500]: 10.f scrub ok 2024-03-26T05:18:03.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:02 smithi132 ceph-mon[33500]: 6.3 deep-scrub starts 2024-03-26T05:18:03.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:02 smithi132 ceph-mon[33500]: 6.3 deep-scrub ok 2024-03-26T05:18:03.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:02 smithi132 ceph-mon[33500]: pgmap v821: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing+deep, 3 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 724 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 552 op/s; 4389/35433 objects degraded (12.387%); 1640/35433 objects misplaced (4.628%); 808 KiB/s, 12 objects/s recovering 2024-03-26T05:18:03.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:02 smithi132 ceph-mon[33500]: 6.0 scrub starts 2024-03-26T05:18:03.115 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:03.155 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:03.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:03.258 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:03.295 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:03.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:03.397 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:03.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:03.430 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:03.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:03.531 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:03.573 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:03.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:03.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:03.675 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:03.705 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:03.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:03.807 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:03.845 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:03.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:03.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:03.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:18:03.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:18:03.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[29691]: 10.10 deep-scrub ok 2024-03-26T05:18:03.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[29691]: 6.0 scrub ok 2024-03-26T05:18:03.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[29691]: 6.4 scrub starts 2024-03-26T05:18:03.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[29691]: 6.4 scrub ok 2024-03-26T05:18:03.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[29691]: 5.17 scrub starts 2024-03-26T05:18:03.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[29691]: 5.17 scrub ok 2024-03-26T05:18:03.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[29691]: 2.1 deep-scrub starts 2024-03-26T05:18:03.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[29691]: 2.1 deep-scrub ok 2024-03-26T05:18:03.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:18:03.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:18:03.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[40404]: 10.10 deep-scrub ok 2024-03-26T05:18:03.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[40404]: 6.0 scrub ok 2024-03-26T05:18:03.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[40404]: 6.4 scrub starts 2024-03-26T05:18:03.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[40404]: 6.4 scrub ok 2024-03-26T05:18:03.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[40404]: 5.17 scrub starts 2024-03-26T05:18:03.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[40404]: 5.17 scrub ok 2024-03-26T05:18:03.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[40404]: 2.1 deep-scrub starts 2024-03-26T05:18:03.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:03 smithi080 ceph-mon[40404]: 2.1 deep-scrub ok 2024-03-26T05:18:03.947 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:03.991 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:03.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:03 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:18:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:03 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:18:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:03 smithi132 ceph-mon[33500]: 10.10 deep-scrub ok 2024-03-26T05:18:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:03 smithi132 ceph-mon[33500]: 6.0 scrub ok 2024-03-26T05:18:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:03 smithi132 ceph-mon[33500]: 6.4 scrub starts 2024-03-26T05:18:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:03 smithi132 ceph-mon[33500]: 6.4 scrub ok 2024-03-26T05:18:04.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:03 smithi132 ceph-mon[33500]: 5.17 scrub starts 2024-03-26T05:18:04.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:03 smithi132 ceph-mon[33500]: 5.17 scrub ok 2024-03-26T05:18:04.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:03 smithi132 ceph-mon[33500]: 2.1 deep-scrub starts 2024-03-26T05:18:04.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:03 smithi132 ceph-mon[33500]: 2.1 deep-scrub ok 2024-03-26T05:18:04.092 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:04.123 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:04.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:04.224 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:04.264 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:04.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:04.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:04.320 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:18:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:18:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:18:04.366 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:04.400 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:04.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:04.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:04.501 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:04.545 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:04.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:04.646 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:04.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:04.694 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:04.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:04.795 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:04.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:04.844 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:04.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:04.945 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:04.980 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:04.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:05.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:04 smithi132 ceph-mon[33500]: pgmap v822: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing+deep, 3 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 724 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 4389/35433 objects degraded (12.387%); 1640/35433 objects misplaced (4.628%) 2024-03-26T05:18:05.082 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:05.104 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:04 smithi080 ceph-mon[29691]: pgmap v822: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing+deep, 3 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 724 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 4389/35433 objects degraded (12.387%); 1640/35433 objects misplaced (4.628%) 2024-03-26T05:18:05.104 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:04 smithi080 ceph-mon[40404]: pgmap v822: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped+scrubbing+deep, 3 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 724 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 4389/35433 objects degraded (12.387%); 1640/35433 objects misplaced (4.628%) 2024-03-26T05:18:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:05.122 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:05.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:05.223 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:05.255 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:05.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:05.357 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:05.395 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:05.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:05.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:05.497 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:05.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:05.627 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:05.665 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:05.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:05.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:05.767 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:05.808 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:05.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:05.910 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:05.947 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:05.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:05 smithi132 ceph-mon[33500]: 4.17 scrub starts 2024-03-26T05:18:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:05 smithi132 ceph-mon[33500]: 4.17 scrub ok 2024-03-26T05:18:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:05 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/831106800' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:18:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:18:06.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:18:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:18:06.049 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:06.080 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:05 smithi080 ceph-mon[40404]: 4.17 scrub starts 2024-03-26T05:18:06.080 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:05 smithi080 ceph-mon[40404]: 4.17 scrub ok 2024-03-26T05:18:06.081 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:05 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/831106800' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:18:06.081 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:18:06.081 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:05 smithi080 ceph-mon[29691]: 4.17 scrub starts 2024-03-26T05:18:06.081 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:05 smithi080 ceph-mon[29691]: 4.17 scrub ok 2024-03-26T05:18:06.081 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:05 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/831106800' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:18:06.081 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:18:06.109 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:06.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:06.177 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:18:06.211 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:06.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:06.246 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:06.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:06.347 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:06.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:06.384 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:06.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:06.485 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:06.523 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:06.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:06.624 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:06.665 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:06.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:06.766 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:06.809 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:06.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:06.910 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:06.949 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:06.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:06 smithi132 ceph-mon[33500]: 5.1b scrub starts 2024-03-26T05:18:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:06 smithi132 ceph-mon[33500]: 5.1b scrub ok 2024-03-26T05:18:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:06 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:18:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:06 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:06 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/831106800' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:18:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:06 smithi132 ceph-mon[33500]: osdmap e398: 8 total, 7 up, 8 in 2024-03-26T05:18:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:06 smithi132 ceph-mon[33500]: pgmap v824: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 375 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 2456/18651 objects degraded (13.168%); 1057/18651 objects misplaced (5.667%) 2024-03-26T05:18:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:06 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 4389/35433 objects degraded (12.387%), 30 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:06 smithi132 ceph-mon[33500]: osdmap e399: 8 total, 7 up, 8 in 2024-03-26T05:18:07.051 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:07.086 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:07.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:07.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[40404]: 5.1b scrub starts 2024-03-26T05:18:07.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[40404]: 5.1b scrub ok 2024-03-26T05:18:07.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:18:07.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:07.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/831106800' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:18:07.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[40404]: osdmap e398: 8 total, 7 up, 8 in 2024-03-26T05:18:07.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[40404]: pgmap v824: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 375 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 2456/18651 objects degraded (13.168%); 1057/18651 objects misplaced (5.667%) 2024-03-26T05:18:07.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 4389/35433 objects degraded (12.387%), 30 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:07.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[40404]: osdmap e399: 8 total, 7 up, 8 in 2024-03-26T05:18:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[29691]: 5.1b scrub starts 2024-03-26T05:18:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[29691]: 5.1b scrub ok 2024-03-26T05:18:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:18:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:07.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/831106800' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:18:07.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[29691]: osdmap e398: 8 total, 7 up, 8 in 2024-03-26T05:18:07.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[29691]: pgmap v824: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 375 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 2456/18651 objects degraded (13.168%); 1057/18651 objects misplaced (5.667%) 2024-03-26T05:18:07.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 4389/35433 objects degraded (12.387%), 30 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:07.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:06 smithi080 ceph-mon[29691]: osdmap e399: 8 total, 7 up, 8 in 2024-03-26T05:18:07.187 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:07.225 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:07.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:07.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:07.327 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:07.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:07.362 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:07.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:07.463 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:07.503 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:07.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:07.604 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:07.646 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:07.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:07.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:07.748 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:07.781 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:07.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:07.883 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:07.922 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:07.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:07.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:08.023 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:08.072 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:08.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:08.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:08.174 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:08.213 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:08.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:08.315 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:08.357 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:08.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:08.458 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:08.488 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:08.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:08.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:08.575 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:18:08.575 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:18:08.575 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:18:08.575 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:18:08.575 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:18:08.576 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:18:08.576 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:18:08.576 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:18:08.576 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:18:08.576 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:18:08.576 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:18:08.576 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:18:08.591 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:08.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:08.625 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:08.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:08.726 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:08.766 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:08.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:08.868 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:08.896 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:08.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:08.998 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:09.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:09.037 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:09.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:09.138 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:09.167 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:09.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:09.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:18:09.239 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:18:09.239 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-26T05:18:09.269 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:09.298 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:09.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:09.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:09.400 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:09.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:09.436 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:09.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:09.444 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:09 smithi080 ceph-mon[29691]: pgmap v826: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 336 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 260 op/s; 2067/16803 objects degraded (12.301%); 655/16803 objects misplaced (3.898%) 2024-03-26T05:18:09.444 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:09 smithi080 ceph-mon[40404]: pgmap v826: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 336 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 260 op/s; 2067/16803 objects degraded (12.301%); 655/16803 objects misplaced (3.898%) 2024-03-26T05:18:09.454 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Removed 111720 objects 2024-03-26T05:18:09.454 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Clean up completed and total clean up time :106.837 2024-03-26T05:18:09.538 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:09.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:09 smithi132 ceph-mon[33500]: pgmap v826: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+clean+remapped, 37 active+undersized, 97 active+clean, 29 active+undersized+degraded; 336 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 260 op/s; 2067/16803 objects degraded (12.301%); 655/16803 objects misplaced (3.898%) 2024-03-26T05:18:09.562 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:09.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:09.609 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:18:09.663 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:09.686 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:09.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:09.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:09.789 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:09.816 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:09.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:09.830 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:18:09.849 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:18:09.868 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:18:09.887 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:18:09.907 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:18:09.918 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:09.928 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:18:09.942 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:09.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:09.947 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:18:09.948 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:18:09.948 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:18:09.948 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:18:09.948 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:18:09.948 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:18:09.948 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:18:09.948 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:18:09.948 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:18:10.041 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:18:10.041 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:18:10.043 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:10.067 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:10.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:10.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:10.169 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:10.194 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:10.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:10.279 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:10 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3907177837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:18:10.280 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:10 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3907177837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:18:10.294 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:10.318 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:10.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:10.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:10.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:10.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:10.445 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:10.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:10.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:10 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3907177837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:18:10.545 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:10.570 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:10.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:10.671 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:10.695 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:10.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:10.797 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:10.823 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:10.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:10.924 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:10.948 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:10.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:10.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:10.999 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T05:18:11.051 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:11.081 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:11.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:11.182 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:11.206 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:11.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:11.307 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:11.330 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:11.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:11.359 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:11 smithi080 ceph-mon[40404]: pgmap v827: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+clean+remapped, 38 active+undersized, 97 active+clean, 28 active+undersized+degraded; 147 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 324 op/s; 1134/7716 objects degraded (14.697%); 456/7716 objects misplaced (5.910%) 2024-03-26T05:18:11.359 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:11 smithi080 ceph-mon[29691]: pgmap v827: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+clean+remapped, 38 active+undersized, 97 active+clean, 28 active+undersized+degraded; 147 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 324 op/s; 1134/7716 objects degraded (14.697%); 456/7716 objects misplaced (5.910%) 2024-03-26T05:18:11.433 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:11.456 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:11.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:11.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:11.491 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:18:11.525 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:18:11.538 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:11.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:11 smithi132 ceph-mon[33500]: pgmap v827: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+clean+remapped, 38 active+undersized, 97 active+clean, 28 active+undersized+degraded; 147 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 324 op/s; 1134/7716 objects degraded (14.697%); 456/7716 objects misplaced (5.910%) 2024-03-26T05:18:11.558 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:11.586 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:11.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:11.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:11.687 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:11.715 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:11.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:11.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:11.844 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:11.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:11.946 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:11.972 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:11.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:12.074 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:12.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:12.103 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:12.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:12.204 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:12.229 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:12.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:12.331 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:12.359 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:12.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:12.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:12 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 1134/7716 objects degraded (14.697%), 29 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:12.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:12 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 1134/7716 objects degraded (14.697%), 29 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:12.461 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:12.487 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:12.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:12.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:12 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 1134/7716 objects degraded (14.697%), 29 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:12.588 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:12.614 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:12.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:12.716 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:12.742 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:12.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:12.844 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:12.869 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:12.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:12.971 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:12.999 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:12.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:13.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:13.100 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:13.128 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:13.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:13.140 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:13.229 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:13.257 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:13.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:13.359 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:13.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:13 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:18:13.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:13 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:18:13.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:13 smithi080 ceph-mon[29691]: pgmap v828: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 40 active+undersized, 97 active+clean, 26 active+undersized+degraded; 3.9 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 614 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:13.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:13 smithi080 ceph-mon[29691]: osdmap e400: 8 total, 7 up, 8 in 2024-03-26T05:18:13.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:13 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:18:13.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:13 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:18:13.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:13 smithi080 ceph-mon[40404]: pgmap v828: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 40 active+undersized, 97 active+clean, 26 active+undersized+degraded; 3.9 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 614 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:13.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:13 smithi080 ceph-mon[40404]: osdmap e400: 8 total, 7 up, 8 in 2024-03-26T05:18:13.386 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:13.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:13.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:13.512 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:13.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:13.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:13.526 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:18:13.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:13 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:18:13.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:13 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:18:13.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:13 smithi132 ceph-mon[33500]: pgmap v828: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 40 active+undersized, 97 active+clean, 26 active+undersized+degraded; 3.9 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 614 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:13.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:13 smithi132 ceph-mon[33500]: osdmap e400: 8 total, 7 up, 8 in 2024-03-26T05:18:13.613 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:13.641 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:13.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:13.742 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:13.769 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:13.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:13.822 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-26T05:18:13.822 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-26T05:18:13.872 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:13.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:13.897 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:13.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:13.998 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:14.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:14.027 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:14.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:14.115 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:18:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:18:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:18:14.128 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:14.155 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:14.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:14.257 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:14.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:14.282 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:14.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:14.383 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:14.412 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:14.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:14.513 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:14.538 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:14.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:14.639 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:14.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:14.663 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:14.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:14.764 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:14.792 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:14.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:14.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:14.894 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:14.918 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:14.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:15.019 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:15.047 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:15.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:15.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:15.149 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:15.174 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:15.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:15.276 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:15.304 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:15.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:15.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:15.398 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:15 smithi080 ceph-mon[29691]: pgmap v830: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 40 active+undersized, 97 active+clean, 26 active+undersized+degraded; 3.9 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 746 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:15.399 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:15 smithi080 ceph-mon[40404]: pgmap v830: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 40 active+undersized, 97 active+clean, 26 active+undersized+degraded; 3.9 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 746 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:15.406 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:15.432 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:15.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:15.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:15.450 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:18:15.450 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:18:15.450 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:18:15.450 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:18:15.451 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:18:15.451 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:18:15.451 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:18:15.451 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:18:15.451 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:18:15.451 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:18:15.451 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:18:15.451 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:18:15.534 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:15.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:15 smithi132 ceph-mon[33500]: pgmap v830: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 40 active+undersized, 97 active+clean, 26 active+undersized+degraded; 3.9 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 746 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:15.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:15.563 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:15.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:15.664 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:15.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:15.689 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:15.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:15.790 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:15.814 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:15.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:15.915 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:15.939 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:15.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:16.041 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:16.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:18:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:18:16.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:16.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:16.170 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:16.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:16.195 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:16.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:16.208 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:18:16.297 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:16.325 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:16.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:16.426 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:16.452 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:16.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:16.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:16.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:16 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4069578201' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:18:16.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:16 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:18:16.550 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:16 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4069578201' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:18:16.550 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:16 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:18:16.550 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:16 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4069578201' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:18:16.550 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:16 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:18:16.555 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:16.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:16.579 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:16.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:16.680 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:16.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:16.710 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:16.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:16.811 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:16.839 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:16.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:16.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:16.897 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:16.941 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:16.969 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:16.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:17.071 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:17.097 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:17.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:17.123 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:18:17.123 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:18:17.199 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:17.225 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:17.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:17.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:17 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:17.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:17 smithi132 ceph-mon[33500]: pgmap v831: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 631 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:17.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:17 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:18:17.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:17 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:18:17.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:17 smithi132 ceph-mon[33500]: osdmap e401: 8 total, 7 up, 8 in 2024-03-26T05:18:17.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:17 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 141/846 objects degraded (16.667%), 24 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:17.310 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:17 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:17.310 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:17 smithi080 ceph-mon[29691]: pgmap v831: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 631 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:17.311 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:17 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:18:17.311 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:17 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:18:17.311 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:17 smithi080 ceph-mon[29691]: osdmap e401: 8 total, 7 up, 8 in 2024-03-26T05:18:17.311 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:17 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 141/846 objects degraded (16.667%), 24 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:17.311 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:17 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:17.311 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:17 smithi080 ceph-mon[40404]: pgmap v831: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 631 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:17.311 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:17 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:18:17.311 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:17 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:18:17.311 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:17 smithi080 ceph-mon[40404]: osdmap e401: 8 total, 7 up, 8 in 2024-03-26T05:18:17.312 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:17 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 141/846 objects degraded (16.667%), 24 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:17.327 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:17.353 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:17.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:17.455 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:17.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:17.480 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:17.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:17.581 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:17.607 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:17.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:17.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:17.709 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:17.733 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:17.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:17.835 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:17.859 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:17.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:17.961 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:17.986 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:17.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:18.088 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:18.114 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:18.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:18.216 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:18.241 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:18.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:18.343 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:18.368 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:18.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:18.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:18:18.415 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:18:18.416 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-26T05:18:18.469 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:18.497 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:18.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:18.599 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:18.625 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:18.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:18.726 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:18.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:18.753 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:18.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:18.854 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:18.877 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:18.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:18.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:18.979 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:19.003 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:19.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:19.105 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:19.129 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:19.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:19.231 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:19.258 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:19.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:19.360 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:19.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:19 smithi080 ceph-mon[29691]: pgmap v833: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 290 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:19.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:19 smithi080 ceph-mon[40404]: pgmap v833: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 290 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:19.383 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:19.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:19.485 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:19.508 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:19.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:19.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:19 smithi132 ceph-mon[33500]: pgmap v833: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 290 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:19.610 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:19.638 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:19.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:19.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:19.643 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T05:18:19.655 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:19.740 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:19.767 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:19.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:19.869 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:19.894 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:19.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:19.995 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:20.018 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:20.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:20.120 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:20.149 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:20.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:20.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:20.251 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:20.278 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:20.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:20.334 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:20 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:18:20.334 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:20 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:18:20.334 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:20 smithi080 ceph-mon[40404]: osdmap e402: 8 total, 7 up, 8 in 2024-03-26T05:18:20.335 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:20 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:18:20.335 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:20 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:18:20.335 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:20 smithi080 ceph-mon[29691]: osdmap e402: 8 total, 7 up, 8 in 2024-03-26T05:18:20.379 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:20.403 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:20.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:20.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:20.505 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:20.529 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:20.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:20.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:20 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:18:20.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:20 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:18:20.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:20 smithi132 ceph-mon[33500]: osdmap e402: 8 total, 7 up, 8 in 2024-03-26T05:18:20.631 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:20.660 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:20.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:20.762 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:20.787 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:20.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:20.888 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:20.916 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:20.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:21.018 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:21.042 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:21.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:21.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:21.092 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:18:21.144 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:21.172 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:21.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:21.274 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:21.302 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:21.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:21.357 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:21.404 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:21.428 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:21.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:21.529 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:21.556 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:21.556 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:21.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:21.557 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:21.658 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:21.686 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:21.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:21.787 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:21.802 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:21 smithi080 ceph-mon[40404]: pgmap v835: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:21.802 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:18:21.802 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:18:21.802 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:21 smithi080 ceph-mon[29691]: pgmap v835: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:21.802 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:18:21.802 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:18:21.811 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:21.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:21.913 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:21.941 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:21.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:21.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:22.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:21 smithi132 ceph-mon[33500]: pgmap v835: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:18:22.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:18:22.042 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:22.070 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:22.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:22.172 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:22.196 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:22.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:22.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:22.321 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:22.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:22.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:22.423 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:22.452 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:22.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:22.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:22.554 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:22.578 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:22.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:22.647 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:18:22.667 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:18:22.680 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:22.688 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:18:22.703 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:22.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:22.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:22.708 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:18:22.730 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:18:22.752 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:18:22.772 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:18:22.772 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:22 smithi080 ceph-mon[29691]: osdmap e403: 8 total, 7 up, 8 in 2024-03-26T05:18:22.772 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:22 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4123631816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:18:22.772 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:22 smithi080 ceph-mon[40404]: osdmap e403: 8 total, 7 up, 8 in 2024-03-26T05:18:22.772 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:22 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4123631816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:18:22.773 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:18:22.773 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:18:22.773 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:18:22.773 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:18:22.773 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:18:22.773 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:18:22.773 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:18:22.773 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:18:22.805 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:22.829 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:22.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:22.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:22.931 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:22.955 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:22.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:23.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:22 smithi132 ceph-mon[33500]: osdmap e403: 8 total, 7 up, 8 in 2024-03-26T05:18:23.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:22 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4123631816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:18:23.057 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:23.081 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:23.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:23.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:23.182 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:23.211 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:23.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:23.313 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:23.337 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:23.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:23.438 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:23.467 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:23.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:23.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:23.568 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:23.592 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:23.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:23.695 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:23.723 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:23.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:23.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:23.826 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:23 smithi080 ceph-mon[40404]: pgmap v837: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:23.826 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:23 smithi080 ceph-mon[29691]: pgmap v837: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:23.828 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:23.851 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:23.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:23.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:23.953 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:23.976 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:23.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:23.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:24.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:23 smithi132 ceph-mon[33500]: pgmap v837: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:24.078 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:24.101 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:24.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:24.115 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:18:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:18:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:18:24.203 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:24.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:24.233 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:24.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:24.334 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:24.361 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:24.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:24.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:24.463 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:24.490 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:24.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:24.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:24.592 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:24.618 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:24.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:24.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:24.686 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'unique_pool_3' removed 2024-03-26T05:18:24.720 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:24.746 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:24.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:24.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:24.848 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:24.873 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:24.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:24.975 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:24.999 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:25.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:25.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:25.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:24 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/652389934' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:18:25.009 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:24 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/652389934' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:18:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/652389934' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:18:25.101 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:25.127 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:25.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:25.228 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:25.253 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:25.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:25.355 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:25.381 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:25.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:25.482 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:25.507 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:25.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:25.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:25.609 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:25.633 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:25.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:25.735 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:25.759 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:25.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:25.861 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:25.884 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:25.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:25.987 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:26.011 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:26.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:26.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:25 smithi132 ceph-mon[33500]: pgmap v838: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 646 B/s rd, 0 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:26.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:25 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/652389934' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:18:26.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:25 smithi132 ceph-mon[33500]: osdmap e404: 8 total, 7 up, 8 in 2024-03-26T05:18:26.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:18:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:18:26.053 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:25 smithi080 ceph-mon[40404]: pgmap v838: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 646 B/s rd, 0 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:26.054 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:25 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/652389934' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:18:26.054 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:25 smithi080 ceph-mon[40404]: osdmap e404: 8 total, 7 up, 8 in 2024-03-26T05:18:26.054 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:25 smithi080 ceph-mon[29691]: pgmap v838: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 4 active+clean+remapped, 42 active+undersized, 97 active+clean, 24 active+undersized+degraded; 3.9 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 646 B/s rd, 0 op/s; 141/846 objects degraded (16.667%); 5/846 objects misplaced (0.591%) 2024-03-26T05:18:26.054 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:25 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/652389934' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:18:26.054 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:25 smithi080 ceph-mon[29691]: osdmap e404: 8 total, 7 up, 8 in 2024-03-26T05:18:26.111 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:26.135 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:26.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:26.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:26.237 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:26.265 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:26.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:26.367 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:26.390 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:26.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:26.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:26.492 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:26.516 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:26.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:26.618 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:26.641 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:26.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:26.743 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:26.766 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:26.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:26.867 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:26.891 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:26.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:26.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:26.992 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:27.016 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:27.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:27.118 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:27.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:27.142 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:27.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:27.243 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:27.268 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:27.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:27.370 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:27.395 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:27.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:27.496 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:27.519 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:27.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:27.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:27.621 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:27.661 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T05:18:27.662 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T05:18:27.662 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-26T05:18:27.662 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-26T05:18:27.673 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:27.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:27.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:27.774 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:27.798 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:27.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:27.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:27.899 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:27.923 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:27.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:27.960 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:27 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:27.960 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:27 smithi080 ceph-mon[29691]: pgmap v840: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:27.960 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:27 smithi080 ceph-mon[29691]: osdmap e405: 8 total, 7 up, 8 in 2024-03-26T05:18:27.960 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:27 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 90/693 objects degraded (12.987%), 21 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:27.960 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:27 smithi080 ceph-mon[29691]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T05:18:27.960 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:27 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:27.961 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:27 smithi080 ceph-mon[40404]: pgmap v840: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:27.961 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:27 smithi080 ceph-mon[40404]: osdmap e405: 8 total, 7 up, 8 in 2024-03-26T05:18:27.961 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:27 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 90/693 objects degraded (12.987%), 21 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:27.961 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:27 smithi080 ceph-mon[40404]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T05:18:28.024 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:28.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:27 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:28.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:27 smithi132 ceph-mon[33500]: pgmap v840: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:28.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:27 smithi132 ceph-mon[33500]: osdmap e405: 8 total, 7 up, 8 in 2024-03-26T05:18:28.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:27 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 90/693 objects degraded (12.987%), 21 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:28.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:27 smithi132 ceph-mon[33500]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T05:18:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:28.054 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:28.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:28.155 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:28.179 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:28.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:28.281 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:28.308 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:28.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:28.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:28.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:28.433 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:28.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:28.535 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:28.559 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:28.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:28.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:28.569 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:18:28.569 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:18:28.661 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:28.689 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:28.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:28.694 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:18:28.694 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:18:28.694 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:18:28.694 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:18:28.694 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.694 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15093.0:142102 8.a 8:5d371889:::benchmark_data_smithi080_109213_object27585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-26T05:18:28.694 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:08:54.540155+0000", 2024-03-26T05:18:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 572.86068933499996, 2024-03-26T05:18:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.032435765, 2024-03-26T05:18:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:18:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15093", 2024-03-26T05:18:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3240786606", 2024-03-26T05:18:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142102 2024-03-26T05:18:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540155+0000", 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540155+0000", 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540157+0000", 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.505e-06 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540162+0000", 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4240000000000001e-06 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540173+0000", 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1156e-05 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540181+0000", 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2230000000000001e-06 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540234+0000", 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2961000000000003e-05 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540370+0000", 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000136611 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.540527+0000", 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015644199999999999 2024-03-26T05:18:28.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.549579+0000", 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0090522169999999996 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.550582+0000", 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010026220000000001 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.555385+0000", 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0048038550000000001 2024-03-26T05:18:28.699 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.576371+0000", 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.020985875000000001 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:54.576390+0000", 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8377e-05 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.572512+0000", 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99612238600000003 2024-03-26T05:18:28.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:08:55.572591+0000", 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9110999999999995e-05 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:165606 10.19 10:9cdc6bfe:::benchmark_data_smithi080_161990_object53883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:17.696202+0000", 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 69.704642703000005, 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0290944799999999, 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.701 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 165606 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696202+0000", 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.702 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696202+0000", 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696203+0000", 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.012e-06 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696208+0000", 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8380000000000001e-06 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.703 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696222+0000", 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3747000000000001e-05 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696230+0000", 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9340000000000006e-06 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696291+0000", 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1270000000000001e-05 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696301+0000", 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0356999999999999e-05 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.720378+0000", 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.024076612 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.720507+0000", 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012966400000000001 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.720862+0000", 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00035415799999999998 2024-03-26T05:18:28.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.724880+0000", 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.004018527 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.724925+0000", 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5166000000000001e-05 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.706 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.725126+0000", 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020027700000000001 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.725181+0000", 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5071999999999997e-05 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.725296+0000", 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115846 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.707 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:165608 10.19 10:9cef172c:::benchmark_data_smithi080_161990_object53885:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:17.698646+0000", 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 69.702198476999996, 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0308346859999999, 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 165608 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698646+0000", 2024-03-26T05:18:28.708 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698646+0000", 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698647+0000", 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1459999999999999e-06 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698652+0000", 2024-03-26T05:18:28.709 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9080000000000003e-06 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698663+0000", 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0736e-05 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698670+0000", 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1709999999999996e-06 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.710 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698744+0000", 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3672999999999997e-05 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698754+0000", 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.679e-06 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.723226+0000", 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0244719899999999 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.723356+0000", 2024-03-26T05:18:28.711 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013032199999999999 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.723607+0000", 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025147 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729207+0000", 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055995130000000004 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729272+0000", 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5174999999999999e-05 2024-03-26T05:18:28.712 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729342+0000", 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9556999999999998e-05 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729380+0000", 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8303999999999998e-05 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.713 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729481+0000", 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000101042 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:165613 10.19 10:9fafadff:::benchmark_data_smithi080_161990_object53890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:17.700875+0000", 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 69.699969396, 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0288853499999999, 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.714 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 165613 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.700875+0000", 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.700875+0000", 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.715 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.700877+0000", 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2389999999999999e-06 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.700882+0000", 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6540000000000004e-06 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.700895+0000", 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2431e-05 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.716 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.700904+0000", 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7329999999999998e-06 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.700966+0000", 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1130000000000006e-05 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.700982+0000", 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6776999999999999e-05 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.717 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.725342+0000", 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0243595940000001 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.725454+0000", 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000112592 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.725700+0000", 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00024530500000000002 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.718 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729496+0000", 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0037959090000000001 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729533+0000", 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7636999999999999e-05 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729583+0000", 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9441000000000003e-05 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729648+0000", 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5522e-05 2024-03-26T05:18:28.719 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729761+0000", 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000112386 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:165622 10.19 10:9b0e1fb1:::benchmark_data_smithi080_161990_object53899:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:17.710689+0000", 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 69.690155265000001, 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0234221569999999, 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.720 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 165622 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.710689+0000", 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.710689+0000", 2024-03-26T05:18:28.721 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.710691+0000", 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6619999999999999e-06 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.710697+0000", 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9220000000000002e-06 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.710714+0000", 2024-03-26T05:18:28.722 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7408e-05 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.710723+0000", 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8519999999999993e-06 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.710891+0000", 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016743699999999999 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.710899+0000", 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6740000000000003e-06 2024-03-26T05:18:28.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.727284+0000", 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.016385042 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.727414+0000", 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012960000000000001 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.727693+0000", 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00027926600000000002 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.724 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729545+0000", 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001851381 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.733826+0000", 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0042812909999999996 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.733917+0000", 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0546999999999999e-05 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.725 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.733996+0000", 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9802000000000006e-05 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.734112+0000", 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011527300000000001 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:165644 10.19 10:9f5a1173:::benchmark_data_smithi080_161990_object53921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:17.726357+0000", 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 69.674487958, 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.027297592, 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 165644 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.726357+0000", 2024-03-26T05:18:28.727 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.726357+0000", 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.726358+0000", 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6199999999999996e-07 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.726362+0000", 2024-03-26T05:18:28.728 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5639999999999998e-06 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.726372+0000", 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4339999999999998e-06 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.726379+0000", 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8320000000000006e-06 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.726436+0000", 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6817000000000002e-05 2024-03-26T05:18:28.729 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.726445+0000", 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3259999999999996e-06 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729800+0000", 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0033549260000001 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729954+0000", 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015399500000000001 2024-03-26T05:18:28.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.730205+0000", 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025187199999999999 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.740842+0000", 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010636252000000001 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.740894+0000", 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2045000000000001e-05 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.731 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.753492+0000", 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.012598349999999999 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.753550+0000", 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8077e-05 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.753654+0000", 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010424 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.732 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:165668 10.19 10:9dd2e1dd:::benchmark_data_smithi080_161990_object53945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:17.746379+0000", 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 69.654465427999995, 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.012379154, 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 165668 2024-03-26T05:18:28.733 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.746379+0000", 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.746379+0000", 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.734 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.746381+0000", 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.359e-06 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.746387+0000", 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1029999999999998e-06 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.746400+0000", 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2778e-05 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.735 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.746407+0000", 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5719999999999997e-06 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.746467+0000", 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9463999999999999e-05 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.746476+0000", 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4409999999999999e-06 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.736 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.731754+0000", 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98527763000000002 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.731885+0000", 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013168199999999999 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.732141+0000", 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025546899999999998 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.737 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.740848+0000", 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0087073830000000008 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.740941+0000", 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.3202000000000007e-05 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.758590+0000", 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.017648206999999999 2024-03-26T05:18:28.738 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.758650+0000", 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0752999999999999e-05 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.758758+0000", 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000108111 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.739 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:165671 10.19 10:989d8adb:::benchmark_data_smithi080_161990_object53948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T05:18:28.740 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:17.747885+0000", 2024-03-26T05:18:28.740 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 69.652959468000006, 2024-03-26T05:18:28.740 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.011060412, 2024-03-26T05:18:28.740 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.740 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.740 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.740 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.740 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.740 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 165671 2024-03-26T05:18:28.740 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.740 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.747885+0000", 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.747885+0000", 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.747886+0000", 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1519999999999999e-06 2024-03-26T05:18:28.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.747891+0000", 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7180000000000004e-06 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.747907+0000", 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5937999999999999e-05 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.747915+0000", 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9750000000000007e-06 2024-03-26T05:18:28.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.747972+0000", 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7182999999999998e-05 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.747981+0000", 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1730000000000001e-06 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.734150+0000", 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98616879099999999 2024-03-26T05:18:28.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.734274+0000", 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012386799999999999 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.734554+0000", 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000280394 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.740854+0000", 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0062991710000000001 2024-03-26T05:18:28.744 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.751222+0000", 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010367996000000001 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.758799+0000", 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0075778340000000003 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.758853+0000", 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3952000000000001e-05 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.758946+0000", 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2267000000000003e-05 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:165677 10.19 10:9ba95010:::benchmark_data_smithi080_161990_object53954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:17.751995+0000", 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 69.648849745999996, 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0071160459999999, 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 165677 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.751995+0000", 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.751995+0000", 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.747 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.751996+0000", 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3099999999999996e-07 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.752001+0000", 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8590000000000004e-06 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.752010+0000", 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.696e-06 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.748 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.752018+0000", 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6580000000000002e-06 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.752076+0000", 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8439000000000001e-05 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.752086+0000", 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0850000000000003e-06 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.736161+0000", 2024-03-26T05:18:28.749 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98407547100000003 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.736273+0000", 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000111808 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.736540+0000", 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00026745099999999998 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.750 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.740905+0000", 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0043645979999999999 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.751261+0000", 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010356512999999999 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.758983+0000", 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.007721678 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.751 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.759028+0000", 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4518999999999998e-05 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.759111+0000", 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3440000000000001e-05 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168339 10.13 10:ccc6060e:::benchmark_data_smithi080_161990_object56616:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.683443+0000", 2024-03-26T05:18:28.752 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 66.717401784000003, 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.401619631, 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168339 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683443+0000", 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.753 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683443+0000", 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683444+0000", 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.305e-06 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683450+0000", 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9100000000000002e-06 2024-03-26T05:18:28.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683461+0000", 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0479000000000001e-05 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683469+0000", 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3550000000000005e-06 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.911908+0000", 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.228439168 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.756 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.756 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.911916+0000", 2024-03-26T05:18:28.756 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8699999999999992e-06 2024-03-26T05:18:28.756 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.756 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.756 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.756 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.084959+0000", 2024-03-26T05:18:28.756 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1730434750000001 2024-03-26T05:18:28.756 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.756 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.756 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.756 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.085063+0000", 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103069 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168349 10.13 10:cb8495a4:::benchmark_data_smithi080_161990_object56626:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.683484+0000", 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 66.717360907, 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4120273919999999, 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.757 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168349 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683484+0000", 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683484+0000", 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.758 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683484+0000", 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8100000000000003e-07 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683487+0000", 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6869999999999999e-06 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683494+0000", 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8519999999999998e-06 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.759 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683499+0000", 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5809999999999996e-06 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.911937+0000", 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.22843730400000001 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.911942+0000", 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4790000000000004e-06 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.760 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.085360+0000", 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.173417366 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for missing object", 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.085466+0000", 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000106228 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095470+0000", 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010004503999999999 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.761 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095511+0000", 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.091e-05 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168383 10.13 10:ca51ce05:::benchmark_data_smithi080_161990_object56660:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.925543+0000", 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 66.475301787999996, 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.165529024, 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.762 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168383 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925543+0000", 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.763 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925543+0000", 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925544+0000", 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.455e-06 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925550+0000", 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0410000000000003e-06 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.764 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925568+0000", 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.73e-05 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925580+0000", 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1902999999999999e-05 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925666+0000", 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6580000000000001e-05 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.765 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925686+0000", 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0253000000000001e-05 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.085520+0000", 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1598332060000001 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.085623+0000", 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103026 2024-03-26T05:18:28.766 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.085887+0000", 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00026435400000000001 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.090700+0000", 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0048128499999999996 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.090754+0000", 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4143e-05 2024-03-26T05:18:28.767 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.090969+0000", 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000215305 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091008+0000", 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8424999999999999e-05 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091072+0000", 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4183000000000005e-05 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.768 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168409 10.13 10:cf3fad42:::benchmark_data_smithi080_161990_object56686:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.941003+0000", 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 66.459842123000001, 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1502961890000001, 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168409 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.769 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.941003+0000", 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.941003+0000", 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.941004+0000", 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5319999999999999e-06 2024-03-26T05:18:28.770 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.941010+0000", 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6239999999999999e-06 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.941023+0000", 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3076e-05 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.941031+0000", 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1259999999999998e-06 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.771 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.772 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.772 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.943036+0000", 2024-03-26T05:18:28.772 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0020049070000000002 2024-03-26T05:18:28.772 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.772 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.772 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.772 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.943045+0000", 2024-03-26T05:18:28.772 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8729999999999996e-06 2024-03-26T05:18:28.772 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.772 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.772 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.772 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.086959+0000", 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.143914691 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.087044+0000", 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4965999999999999e-05 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.087270+0000", 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000225629 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.773 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.090707+0000", 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0034370410000000001 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.090899+0000", 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019213499999999999 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091138+0000", 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00023836300000000001 2024-03-26T05:18:28.774 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091191+0000", 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3118999999999997e-05 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091299+0000", 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000108107 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.775 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168440 10.13 10:cd651acd:::benchmark_data_smithi080_161990_object56717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.966300+0000", 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 66.434544755999994, 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.128878874, 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168440 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966300+0000", 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.776 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966300+0000", 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966301+0000", 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.373e-06 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966307+0000", 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0569999999999998e-06 2024-03-26T05:18:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966323+0000", 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6018000000000001e-05 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966333+0000", 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7189999999999996e-06 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966417+0000", 2024-03-26T05:18:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3811000000000001e-05 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966426+0000", 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5130000000000002e-06 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.089436+0000", 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.12301001 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.089524+0000", 2024-03-26T05:18:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7381999999999998e-05 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.089697+0000", 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017316 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091085+0000", 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001387934 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.094840+0000", 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.003755525 2024-03-26T05:18:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095034+0000", 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000193921 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095081+0000", 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6910000000000003e-05 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095179+0000", 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7540999999999993e-05 2024-03-26T05:18:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168459 10.13 10:cd1968e0:::benchmark_data_smithi080_161990_object56736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.977540+0000", 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 66.423304674999997, 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1180914230000001, 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168459 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977540+0000", 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977540+0000", 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977541+0000", 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1379999999999999e-06 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977545+0000", 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7500000000000001e-06 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977556+0000", 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1479e-05 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977565+0000", 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0879999999999995e-06 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977626+0000", 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1005000000000003e-05 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977635+0000", 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.713e-06 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091346+0000", 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1137104769999999 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091467+0000", 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000120827 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091660+0000", 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019336900000000001 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.094855+0000", 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00319528 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095531+0000", 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000676251 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095553+0000", 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1467999999999998e-05 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095579+0000", 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.614e-05 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095631+0000", 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2438000000000002e-05 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168464 10.13 10:cb7a0ce6:::benchmark_data_smithi080_161990_object56741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.981939+0000", 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 66.418905714000005, 2024-03-26T05:18:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.125574021, 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168464 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.981939+0000", 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.981939+0000", 2024-03-26T05:18:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.981940+0000", 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.16e-07 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.981945+0000", 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2379999999999997e-06 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.981957+0000", 2024-03-26T05:18:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1384e-05 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.981964+0000", 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5959999999999998e-06 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.982024+0000", 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0302e-05 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.982033+0000", 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0669999999999992e-06 2024-03-26T05:18:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.093107+0000", 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.111074559 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.093219+0000", 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011222000000000001 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.093439+0000", 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000219483 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095654+0000", 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022155450000000002 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101179+0000", 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055248880000000004 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.107325+0000", 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0061455609999999999 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.107395+0000", 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0432999999999995e-05 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.107513+0000", 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117829 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168500 10.13 10:ca1c5b2d:::benchmark_data_smithi080_161990_object56777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:21.010741+0000", 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 66.390103851999996, 2024-03-26T05:18:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1041035909999999, 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168500 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010741+0000", 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.795 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010741+0000", 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010742+0000", 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4780000000000001e-06 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010747+0000", 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7729999999999999e-06 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.796 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010763+0000", 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5953000000000001e-05 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010771+0000", 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6389999999999992e-06 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010830+0000", 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9478999999999998e-05 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010848+0000", 2024-03-26T05:18:28.797 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7520000000000002e-05 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095678+0000", 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.084830381 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095804+0000", 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000126074 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:18:28.798 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.096062+0000", 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025742700000000001 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101217+0000", 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051549079999999997 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101228+0000", 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1559999999999999e-05 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.799 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.114439+0000", 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.013210629 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.114719+0000", 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00028040600000000003 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.114844+0000", 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012536399999999999 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.800 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168515 10.13 10:cf8059c6:::benchmark_data_smithi080_161990_object56792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:21.022607+0000", 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 66.378237463999994, 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0925560329999999, 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168515 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.801 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022607+0000", 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022607+0000", 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022608+0000", 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1999999999999999e-06 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.802 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022613+0000", 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7389999999999999e-06 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022624+0000", 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0397000000000001e-05 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022631+0000", 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9249999999999998e-06 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.803 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022918+0000", 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00028781 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022926+0000", 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0430000000000008e-06 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.097531+0000", 2024-03-26T05:18:28.804 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0746048880000001 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.097654+0000", 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012278899999999999 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.097906+0000", 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025170699999999999 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101233+0000", 2024-03-26T05:18:28.805 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0033272610000000002 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101263+0000", 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.9913999999999999e-05 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.114892+0000", 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.013629535999999999 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.806 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115069+0000", 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000176121 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115163+0000", 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4703000000000005e-05 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168578 10.13 10:c8138cee:::benchmark_data_smithi080_161990_object56855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:21.072743+0000", 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 66.328101946999993, 2024-03-26T05:18:28.807 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0427139839999999, 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168578 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072743+0000", 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.808 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072743+0000", 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072744+0000", 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1909999999999999e-06 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072749+0000", 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8770000000000001e-06 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.809 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072762+0000", 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2911000000000001e-05 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072768+0000", 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.083e-06 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072865+0000", 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6766000000000002e-05 2024-03-26T05:18:28.810 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072874+0000", 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7860000000000008e-06 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.099335+0000", 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0264605760000001 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.099469+0000", 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013444499999999999 2024-03-26T05:18:28.811 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.099684+0000", 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021462799999999999 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.103173+0000", 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.003488545 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.107277+0000", 2024-03-26T05:18:28.812 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0041039509999999998 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115206+0000", 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0079292340000000003 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115365+0000", 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000159073 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.813 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115457+0000", 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1917999999999996e-05 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168588 10.13 10:c9ff2f28:::benchmark_data_smithi080_161990_object56865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:21.080490+0000", 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 66.32035501, 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.035269016, 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:18:28.814 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168588 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080490+0000", 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080490+0000", 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:18:28.815 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080491+0000", 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.562e-06 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080497+0000", 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8209999999999998e-06 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080510+0000", 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3188e-05 2024-03-26T05:18:28.816 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080518+0000", 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9039999999999992e-06 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080581+0000", 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2546999999999996e-05 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080591+0000", 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8689999999999999e-06 2024-03-26T05:18:28.817 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101289+0000", 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.020698133 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101398+0000", 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000109347 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101607+0000", 2024-03-26T05:18:28.818 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020852800000000001 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.107256+0000", 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0056498429999999999 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.107283+0000", 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6378000000000001e-05 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115499+0000", 2024-03-26T05:18:28.819 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0082165060000000002 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115666+0000", 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016688900000000001 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115759+0000", 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2501000000000001e-05 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:18:28.820 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:18:28.821 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:28.846 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:28.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:28.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:28 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:18:28.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:28 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:18:28.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:28 smithi080 ceph-mon[40404]: osdmap e406: 8 total, 7 up, 8 in 2024-03-26T05:18:28.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:28 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:18:28.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:28 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:18:28.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:28 smithi080 ceph-mon[29691]: osdmap e406: 8 total, 7 up, 8 in 2024-03-26T05:18:28.947 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:28.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:28.972 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:28.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:29.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:28 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:18:29.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:28 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:18:29.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:28 smithi132 ceph-mon[33500]: osdmap e406: 8 total, 7 up, 8 in 2024-03-26T05:18:29.072 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:29.096 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:29.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:29.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:29.198 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:29.226 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:29.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:29.307 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T05:18:29.328 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:29.372 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:29.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:29.474 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:29.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:29.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:29.600 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:29.628 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:29.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:29.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:29.730 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:29.757 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:29.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:29.859 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:29.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:29 smithi080 ceph-mon[29691]: pgmap v843: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:29.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:29 smithi080 ceph-mon[40404]: pgmap v843: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:29.883 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:29.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:29.961 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:29.985 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:29.999 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:30.013 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:30.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:30.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:29 smithi132 ceph-mon[33500]: pgmap v843: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:30.041 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:18:30.041 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:18:30.041 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:18:30.041 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:18:30.041 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:18:30.041 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:18:30.042 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:18:30.042 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:18:30.042 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:18:30.042 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:18:30.042 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:18:30.042 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:18:30.114 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:30.138 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:30.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:30.240 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:30.267 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:30.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:30.369 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:30.397 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:30.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:30.499 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:30.523 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:30.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:30.570 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:18:30.626 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:30.654 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:30.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:30.708 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:18:30.756 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:30.779 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:30.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:30.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:30 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4080882089' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:18:30.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:30 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:18:30.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:30 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4080882089' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:18:30.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:30 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:18:30.881 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:30.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:30.906 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:30.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:31.007 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:31.036 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:31.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:31.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:30 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4080882089' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:18:31.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:30 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:18:31.099 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:18:31.100 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:18:31.100 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-26T05:18:31.137 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:31.164 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:31.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:31.266 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:31.295 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:31.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:31.396 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:31.419 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:31.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:31.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:31.521 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:31.550 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:31.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:31.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:31.652 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:31.677 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:31.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:31.779 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:31.802 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:31.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:31.904 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:31.928 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:31.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:32.029 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:32.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:31 smithi132 ceph-mon[33500]: pgmap v844: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:32.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:31 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:18:32.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:31 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:18:32.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:31 smithi132 ceph-mon[33500]: osdmap e407: 8 total, 7 up, 8 in 2024-03-26T05:18:32.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:31 smithi132 ceph-mon[33500]: 4.b scrub starts 2024-03-26T05:18:32.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:31 smithi132 ceph-mon[33500]: 4.b scrub ok 2024-03-26T05:18:32.054 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:32.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:32.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:32.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:31 smithi080 ceph-mon[29691]: pgmap v844: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:32.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:31 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:18:32.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:31 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:18:32.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:31 smithi080 ceph-mon[29691]: osdmap e407: 8 total, 7 up, 8 in 2024-03-26T05:18:32.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:31 smithi080 ceph-mon[29691]: 4.b scrub starts 2024-03-26T05:18:32.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:31 smithi080 ceph-mon[29691]: 4.b scrub ok 2024-03-26T05:18:32.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:31 smithi080 ceph-mon[40404]: pgmap v844: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:32.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:31 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:18:32.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:31 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:18:32.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:31 smithi080 ceph-mon[40404]: osdmap e407: 8 total, 7 up, 8 in 2024-03-26T05:18:32.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:31 smithi080 ceph-mon[40404]: 4.b scrub starts 2024-03-26T05:18:32.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:31 smithi080 ceph-mon[40404]: 4.b scrub ok 2024-03-26T05:18:32.155 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:32.183 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:32.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:32.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:32.285 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:32.313 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:32.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:32.414 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:32.439 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:32.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:32.540 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:32.568 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:32.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:32.670 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:32.693 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:32.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:32.796 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:32.823 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:32.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:32.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:32.925 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:32.952 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:32.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:33.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:32 smithi132 ceph-mon[33500]: 1.9 scrub starts 2024-03-26T05:18:33.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:32 smithi132 ceph-mon[33500]: 1.9 scrub ok 2024-03-26T05:18:33.050 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:32 smithi080 ceph-mon[40404]: 1.9 scrub starts 2024-03-26T05:18:33.050 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:32 smithi080 ceph-mon[40404]: 1.9 scrub ok 2024-03-26T05:18:33.051 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:32 smithi080 ceph-mon[29691]: 1.9 scrub starts 2024-03-26T05:18:33.051 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:32 smithi080 ceph-mon[29691]: 1.9 scrub ok 2024-03-26T05:18:33.053 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:33.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:33.081 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:33.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:33.182 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:33.210 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:33.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:33.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:33.312 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:33.337 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:33.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:33.438 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:33.462 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:33.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:33.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:33.564 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:33.592 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:33.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:33.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:33.693 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:33.718 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:33.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:33.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:33.820 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:33.847 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:33.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:33.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:33.898 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:33.949 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:33.972 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:33.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:34.022 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:33 smithi080 ceph-mon[40404]: pgmap v846: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:34.022 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:33 smithi080 ceph-mon[40404]: 3.c scrub starts 2024-03-26T05:18:34.022 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:33 smithi080 ceph-mon[40404]: 3.c scrub ok 2024-03-26T05:18:34.023 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:33 smithi080 ceph-mon[29691]: pgmap v846: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:34.023 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:33 smithi080 ceph-mon[29691]: 3.c scrub starts 2024-03-26T05:18:34.023 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:33 smithi080 ceph-mon[29691]: 3.c scrub ok 2024-03-26T05:18:34.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:33 smithi132 ceph-mon[33500]: pgmap v846: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:34.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:33 smithi132 ceph-mon[33500]: 3.c scrub starts 2024-03-26T05:18:34.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:33 smithi132 ceph-mon[33500]: 3.c scrub ok 2024-03-26T05:18:34.074 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:34.098 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:34.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:34.199 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:34.223 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:34.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:34.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:34.325 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:34.353 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:34.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:34.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:34.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:18:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:18:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:18:34.456 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:34.484 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:34.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:34.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:34.585 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:34.612 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:34.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:34.714 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:34.741 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:34.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:34.842 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:34.868 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:34.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:34.970 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:34.997 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:34.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:35.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:34 smithi132 ceph-mon[33500]: pgmap v847: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 685 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:35.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:34 smithi132 ceph-mon[33500]: 3.1f scrub starts 2024-03-26T05:18:35.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:34 smithi132 ceph-mon[33500]: 3.1f scrub ok 2024-03-26T05:18:35.083 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:18:35.099 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:35.109 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:34 smithi080 ceph-mon[40404]: pgmap v847: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 685 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:35.110 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:34 smithi080 ceph-mon[40404]: 3.1f scrub starts 2024-03-26T05:18:35.110 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:34 smithi080 ceph-mon[40404]: 3.1f scrub ok 2024-03-26T05:18:35.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:34 smithi080 ceph-mon[29691]: pgmap v847: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 685 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:35.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:34 smithi080 ceph-mon[29691]: 3.1f scrub starts 2024-03-26T05:18:35.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:34 smithi080 ceph-mon[29691]: 3.1f scrub ok 2024-03-26T05:18:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:35.127 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:35.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:35.228 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:35.251 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:35.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:35.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:35.353 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:35.381 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:35.382 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:35.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:35.483 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:35.506 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:35.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:35.608 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:35.631 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:35.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:35.733 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:35.761 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:35.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:35.863 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:35.891 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:35.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:35.914 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:35 smithi080 ceph-mon[29691]: 3.5 scrub starts 2024-03-26T05:18:35.915 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:35 smithi080 ceph-mon[29691]: 3.5 scrub ok 2024-03-26T05:18:35.915 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:18:35.915 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:35 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:18:35.915 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:35 smithi080 ceph-mon[40404]: 3.5 scrub starts 2024-03-26T05:18:35.915 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:35 smithi080 ceph-mon[40404]: 3.5 scrub ok 2024-03-26T05:18:35.915 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:18:35.915 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:35 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:18:35.992 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:36.020 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:36.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:36.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:36.038 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:18:36.038 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:18:36.039 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 0 2024-03-26T05:18:36.039 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:18:36.122 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:36.149 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:36.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:36.251 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:36.279 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:36.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:36.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:35 smithi132 ceph-mon[33500]: 3.5 scrub starts 2024-03-26T05:18:36.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:35 smithi132 ceph-mon[33500]: 3.5 scrub ok 2024-03-26T05:18:36.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:18:36.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:35 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:18:36.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:18:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:18:36.381 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:36.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:36.407 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:36.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:36.508 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:36.535 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:36.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:36.623 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'unique_pool_4' created 2024-03-26T05:18:36.637 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:36.666 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:36.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:36.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:36.767 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:36.794 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:36.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:36.895 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:36.921 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:36.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:37.023 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:37.048 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:37.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:37.149 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:37.172 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:37.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:37.274 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:36 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/887546897' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T05:18:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:36 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T05:18:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:36 smithi132 ceph-mon[33500]: pgmap v848: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:36 smithi132 ceph-mon[33500]: 5.16 scrub starts 2024-03-26T05:18:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:36 smithi132 ceph-mon[33500]: 5.16 scrub ok 2024-03-26T05:18:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:36 smithi132 ceph-mon[33500]: 5.3 scrub starts 2024-03-26T05:18:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:36 smithi132 ceph-mon[33500]: 5.3 scrub ok 2024-03-26T05:18:37.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:36 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-26T05:18:37.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:36 smithi132 ceph-mon[33500]: osdmap e408: 8 total, 7 up, 8 in 2024-03-26T05:18:37.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:36 smithi132 ceph-mon[33500]: osdmap e409: 8 total, 7 up, 8 in 2024-03-26T05:18:37.294 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/887546897' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T05:18:37.294 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T05:18:37.294 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:37.294 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[40404]: pgmap v848: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:37.295 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[40404]: 5.16 scrub starts 2024-03-26T05:18:37.295 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[40404]: 5.16 scrub ok 2024-03-26T05:18:37.295 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[40404]: 5.3 scrub starts 2024-03-26T05:18:37.295 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[40404]: 5.3 scrub ok 2024-03-26T05:18:37.295 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-26T05:18:37.295 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[40404]: osdmap e408: 8 total, 7 up, 8 in 2024-03-26T05:18:37.295 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[40404]: osdmap e409: 8 total, 7 up, 8 in 2024-03-26T05:18:37.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/887546897' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T05:18:37.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T05:18:37.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:37.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[29691]: pgmap v848: 142 pgs: 1 active+clean+remapped, 37 active+undersized, 83 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:37.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[29691]: 5.16 scrub starts 2024-03-26T05:18:37.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[29691]: 5.16 scrub ok 2024-03-26T05:18:37.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[29691]: 5.3 scrub starts 2024-03-26T05:18:37.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[29691]: 5.3 scrub ok 2024-03-26T05:18:37.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-26T05:18:37.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[29691]: osdmap e408: 8 total, 7 up, 8 in 2024-03-26T05:18:37.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:36 smithi080 ceph-mon[29691]: osdmap e409: 8 total, 7 up, 8 in 2024-03-26T05:18:37.297 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:37.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:37.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:37.399 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:37.423 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:37.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:37.524 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:37.548 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:37.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:37.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:37.650 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:37.682 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:37.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:37.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:37.784 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:37.812 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:37.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:37.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:37.914 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:37.942 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:37.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:37.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:38.044 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:38.072 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:38.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:38.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:37 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:18:38.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:37 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:18:38.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:37 smithi080 ceph-mon[29691]: 3.7 scrub starts 2024-03-26T05:18:38.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:37 smithi080 ceph-mon[29691]: 3.7 scrub ok 2024-03-26T05:18:38.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:37 smithi080 ceph-mon[29691]: 5.7 scrub starts 2024-03-26T05:18:38.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:37 smithi080 ceph-mon[29691]: osdmap e410: 8 total, 7 up, 8 in 2024-03-26T05:18:38.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:37 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:18:38.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:37 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:18:38.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:37 smithi080 ceph-mon[40404]: 3.7 scrub starts 2024-03-26T05:18:38.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:37 smithi080 ceph-mon[40404]: 3.7 scrub ok 2024-03-26T05:18:38.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:37 smithi080 ceph-mon[40404]: 5.7 scrub starts 2024-03-26T05:18:38.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:37 smithi080 ceph-mon[40404]: osdmap e410: 8 total, 7 up, 8 in 2024-03-26T05:18:38.173 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:38.200 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:38.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:38.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:38.237 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T05:18:38.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:37 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:18:38.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:37 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:18:38.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:37 smithi132 ceph-mon[33500]: 3.7 scrub starts 2024-03-26T05:18:38.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:37 smithi132 ceph-mon[33500]: 3.7 scrub ok 2024-03-26T05:18:38.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:37 smithi132 ceph-mon[33500]: 5.7 scrub starts 2024-03-26T05:18:38.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:37 smithi132 ceph-mon[33500]: osdmap e410: 8 total, 7 up, 8 in 2024-03-26T05:18:38.303 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:38.327 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:38.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:38.428 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:38.455 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:38.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:38.556 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:38.580 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:38.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:38.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:38.682 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:38.707 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:38.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:38.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:38.809 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:38.837 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:38.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:38.938 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:38.966 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:38.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:39.037 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:18:39.056 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:18:39.068 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:39.075 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:18:39.092 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:39.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:39.095 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:18:39.116 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:18:39.136 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:18:39.157 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:18:39.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:38 smithi080 ceph-mon[40404]: 5.7 scrub ok 2024-03-26T05:18:39.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:38 smithi080 ceph-mon[40404]: pgmap v852: 158 pgs: 1 active+clean+remapped, 15 unknown, 37 active+undersized, 84 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:39.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:38 smithi080 ceph-mon[40404]: 6.19 scrub starts 2024-03-26T05:18:39.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:38 smithi080 ceph-mon[40404]: 6.10 scrub starts 2024-03-26T05:18:39.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:38 smithi080 ceph-mon[40404]: 6.10 scrub ok 2024-03-26T05:18:39.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:38 smithi080 ceph-mon[29691]: 5.7 scrub ok 2024-03-26T05:18:39.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:38 smithi080 ceph-mon[29691]: pgmap v852: 158 pgs: 1 active+clean+remapped, 15 unknown, 37 active+undersized, 84 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:39.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:38 smithi080 ceph-mon[29691]: 6.19 scrub starts 2024-03-26T05:18:39.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:38 smithi080 ceph-mon[29691]: 6.10 scrub starts 2024-03-26T05:18:39.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:38 smithi080 ceph-mon[29691]: 6.10 scrub ok 2024-03-26T05:18:39.158 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:18:39.158 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:18:39.158 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:18:39.159 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:18:39.159 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:18:39.159 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:18:39.159 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:18:39.159 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:18:39.194 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:39.221 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:39.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:38 smithi132 ceph-mon[33500]: 5.7 scrub ok 2024-03-26T05:18:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:38 smithi132 ceph-mon[33500]: pgmap v852: 158 pgs: 1 active+clean+remapped, 15 unknown, 37 active+undersized, 84 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:38 smithi132 ceph-mon[33500]: 6.19 scrub starts 2024-03-26T05:18:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:38 smithi132 ceph-mon[33500]: 6.10 scrub starts 2024-03-26T05:18:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:38 smithi132 ceph-mon[33500]: 6.10 scrub ok 2024-03-26T05:18:39.323 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:39.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:39.351 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:39.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:39.452 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:39.479 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:39.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:39.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:39.581 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:39.590 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-26T05:18:39.608 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:39.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:39.710 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:39.736 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:39.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:39.838 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:39.862 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:39.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:39.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:39.964 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:39.988 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:39.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:40.089 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:40.117 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:40.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:40.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:40.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:39 smithi080 ceph-mon[40404]: 6.19 scrub ok 2024-03-26T05:18:40.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:39 smithi080 ceph-mon[40404]: 6.11 scrub starts 2024-03-26T05:18:40.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:39 smithi080 ceph-mon[40404]: 6.11 scrub ok 2024-03-26T05:18:40.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:39 smithi080 ceph-mon[40404]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:18:40.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:39 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/26774792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:18:40.179 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:39 smithi080 ceph-mon[29691]: 6.19 scrub ok 2024-03-26T05:18:40.179 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:39 smithi080 ceph-mon[29691]: 6.11 scrub starts 2024-03-26T05:18:40.179 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:39 smithi080 ceph-mon[29691]: 6.11 scrub ok 2024-03-26T05:18:40.179 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:39 smithi080 ceph-mon[29691]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:18:40.179 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:39 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/26774792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:18:40.202 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:18:40.203 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:18:40.219 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:40.246 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:40.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:39 smithi132 ceph-mon[33500]: 6.19 scrub ok 2024-03-26T05:18:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:39 smithi132 ceph-mon[33500]: 6.11 scrub starts 2024-03-26T05:18:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:39 smithi132 ceph-mon[33500]: 6.11 scrub ok 2024-03-26T05:18:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:39 smithi132 ceph-mon[33500]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:18:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:39 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/26774792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:18:40.303 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:18:40.303 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:18:40.303 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:18:40.303 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:18:40.303 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:18:40.303 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:18:40.303 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:18:40.304 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:18:40.304 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:18:40.304 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:18:40.304 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:18:40.304 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:18:40.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:40.373 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:40.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:40.474 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:40.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:40.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:40.600 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:40.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:40.625 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:40.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:40.726 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:40.755 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:40.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:40.856 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:40.880 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:40.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:18:40.893 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:18:40.894 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-26T05:18:40.967 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:40.982 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:41.011 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:41.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:41.015 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:41.021 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:41.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:41.134 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[40404]: 1.6 scrub starts 2024-03-26T05:18:41.134 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[40404]: 1.6 scrub ok 2024-03-26T05:18:41.134 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[40404]: 5.15 scrub starts 2024-03-26T05:18:41.134 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[40404]: 5.15 scrub ok 2024-03-26T05:18:41.134 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[40404]: pgmap v853: 158 pgs: 1 active+clean+remapped, 9 unknown, 37 active+undersized, 90 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:41.134 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[40404]: 3.17 scrub starts 2024-03-26T05:18:41.134 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[40404]: 3.17 scrub ok 2024-03-26T05:18:41.135 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[29691]: 1.6 scrub starts 2024-03-26T05:18:41.135 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[29691]: 1.6 scrub ok 2024-03-26T05:18:41.135 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[29691]: 5.15 scrub starts 2024-03-26T05:18:41.135 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[29691]: 5.15 scrub ok 2024-03-26T05:18:41.135 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[29691]: pgmap v853: 158 pgs: 1 active+clean+remapped, 9 unknown, 37 active+undersized, 90 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:41.135 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[29691]: 3.17 scrub starts 2024-03-26T05:18:41.135 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:40 smithi080 ceph-mon[29691]: 3.17 scrub ok 2024-03-26T05:18:41.138 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:41.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:41.242 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:41.271 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:41.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:40 smithi132 ceph-mon[33500]: 1.6 scrub starts 2024-03-26T05:18:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:40 smithi132 ceph-mon[33500]: 1.6 scrub ok 2024-03-26T05:18:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:40 smithi132 ceph-mon[33500]: 5.15 scrub starts 2024-03-26T05:18:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:40 smithi132 ceph-mon[33500]: 5.15 scrub ok 2024-03-26T05:18:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:40 smithi132 ceph-mon[33500]: pgmap v853: 158 pgs: 1 active+clean+remapped, 9 unknown, 37 active+undersized, 90 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:40 smithi132 ceph-mon[33500]: 3.17 scrub starts 2024-03-26T05:18:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:40 smithi132 ceph-mon[33500]: 3.17 scrub ok 2024-03-26T05:18:41.373 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:41.398 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:41.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:41.499 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:41.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:41.524 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:41.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:41.625 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:41.649 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:41.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:41.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:41.750 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:41.780 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:41.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:41.882 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:41.907 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:41.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:42.008 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:42.033 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:42.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:42.108 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:41 smithi080 ceph-mon[40404]: 2.2 scrub starts 2024-03-26T05:18:42.108 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:41 smithi080 ceph-mon[40404]: 2.2 scrub ok 2024-03-26T05:18:42.108 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:41 smithi080 ceph-mon[40404]: osdmap e411: 8 total, 7 up, 8 in 2024-03-26T05:18:42.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:41 smithi080 ceph-mon[29691]: 2.2 scrub starts 2024-03-26T05:18:42.109 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:41 smithi080 ceph-mon[29691]: 2.2 scrub ok 2024-03-26T05:18:42.109 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:41 smithi080 ceph-mon[29691]: osdmap e411: 8 total, 7 up, 8 in 2024-03-26T05:18:42.135 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:42.159 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:42.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:42.204 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:18:42.261 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:42.289 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:42.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:41 smithi132 ceph-mon[33500]: 2.2 scrub starts 2024-03-26T05:18:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:41 smithi132 ceph-mon[33500]: 2.2 scrub ok 2024-03-26T05:18:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:41 smithi132 ceph-mon[33500]: osdmap e411: 8 total, 7 up, 8 in 2024-03-26T05:18:42.390 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:42.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:42.415 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:42.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:42.516 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:42.544 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:42.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:42.645 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:42.669 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:42.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:42.771 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:42.795 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:42.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:42.896 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:42.912 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:42.925 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:42.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:43.027 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:43.052 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:43.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:43.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:42 smithi080 ceph-mon[29691]: 6.5 scrub starts 2024-03-26T05:18:43.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:42 smithi080 ceph-mon[29691]: 6.5 scrub ok 2024-03-26T05:18:43.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:42 smithi080 ceph-mon[29691]: pgmap v855: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 926 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:43.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:42 smithi080 ceph-mon[40404]: 6.5 scrub starts 2024-03-26T05:18:43.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:42 smithi080 ceph-mon[40404]: 6.5 scrub ok 2024-03-26T05:18:43.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:42 smithi080 ceph-mon[40404]: pgmap v855: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 926 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:43.153 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:43.178 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:43.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:43.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:43.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:42 smithi132 ceph-mon[33500]: 6.5 scrub starts 2024-03-26T05:18:43.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:42 smithi132 ceph-mon[33500]: 6.5 scrub ok 2024-03-26T05:18:43.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:42 smithi132 ceph-mon[33500]: pgmap v855: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 926 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:43.309 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:43.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:43.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:43.462 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:43.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:43.564 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:43.589 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:43.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:43.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:43.690 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:43.714 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:43.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:43.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:43.844 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:43.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:43.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:43.946 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:43.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:43.971 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:43.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:44.072 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:44.101 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:44.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:44.115 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:18:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:18:44.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:43 smithi080 ceph-mon[40404]: 4.3 scrub starts 2024-03-26T05:18:44.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:43 smithi080 ceph-mon[40404]: 4.3 scrub ok 2024-03-26T05:18:44.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:43 smithi080 ceph-mon[29691]: 4.3 scrub starts 2024-03-26T05:18:44.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:43 smithi080 ceph-mon[29691]: 4.3 scrub ok 2024-03-26T05:18:44.203 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:44.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:44.228 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:44.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:44.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:43 smithi132 ceph-mon[33500]: 4.3 scrub starts 2024-03-26T05:18:44.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:43 smithi132 ceph-mon[33500]: 4.3 scrub ok 2024-03-26T05:18:44.328 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:44.352 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:44.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:44.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:44.454 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:44.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:44.481 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:44.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:44.582 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:44.606 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:44.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:44.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:44.708 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:44.733 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:44.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:44.834 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:44.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:44.859 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:44.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:44.916 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:18:44.961 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:44.984 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:44.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:45.086 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:45.110 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:45.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[29691]: 4.4 scrub starts 2024-03-26T05:18:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[29691]: 4.4 scrub ok 2024-03-26T05:18:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[29691]: 6.a scrub starts 2024-03-26T05:18:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[29691]: 6.a scrub ok 2024-03-26T05:18:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[29691]: pgmap v856: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 685 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4211370879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:18:45.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:18:45.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[40404]: 4.4 scrub starts 2024-03-26T05:18:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[40404]: 4.4 scrub ok 2024-03-26T05:18:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[40404]: 6.a scrub starts 2024-03-26T05:18:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[40404]: 6.a scrub ok 2024-03-26T05:18:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[40404]: pgmap v856: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 685 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4211370879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:18:45.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:44 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:18:45.211 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:45.234 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:45.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:45.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:44 smithi132 ceph-mon[33500]: 4.4 scrub starts 2024-03-26T05:18:45.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:44 smithi132 ceph-mon[33500]: 4.4 scrub ok 2024-03-26T05:18:45.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:44 smithi132 ceph-mon[33500]: 6.a scrub starts 2024-03-26T05:18:45.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:44 smithi132 ceph-mon[33500]: 6.a scrub ok 2024-03-26T05:18:45.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:44 smithi132 ceph-mon[33500]: pgmap v856: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 685 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:45.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:44 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4211370879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:18:45.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:44 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:18:45.334 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:45.336 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:45.365 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:45.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:45.466 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:45.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:45.494 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:45.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:45.595 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:45.619 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:45.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:45.721 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:45.749 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:45.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:45.850 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:45.878 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:45.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:45.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:45.981 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:46.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:46.009 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:46.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:46.110 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:46.137 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:46.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:46.239 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:46.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:46.264 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:46.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:46.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:45 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:18:46.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:45 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:18:46.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:45 smithi132 ceph-mon[33500]: osdmap e412: 8 total, 7 up, 8 in 2024-03-26T05:18:46.292 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:18:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:18:46.353 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:45 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:18:46.353 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:45 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:18:46.353 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:45 smithi080 ceph-mon[29691]: osdmap e412: 8 total, 7 up, 8 in 2024-03-26T05:18:46.354 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:45 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:18:46.354 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:45 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:18:46.354 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:45 smithi080 ceph-mon[40404]: osdmap e412: 8 total, 7 up, 8 in 2024-03-26T05:18:46.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:46.393 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:46.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:46.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:46.495 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:46.523 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:46.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:46.624 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:46.649 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:46.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:46.750 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:46.777 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:46.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:46.879 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:46.904 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:46.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:47.006 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:47.031 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:47.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:47.133 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:47.157 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:47.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:47.259 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:47.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:47.386 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:47.411 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:47.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:47.513 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:47.538 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:47.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:47.640 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:47.665 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:47.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:47.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:47.736 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:18:47.767 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:47.807 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:47.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:47.908 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:47.925 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:47.925 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[40404]: pgmap v858: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[40404]: 3.d deep-scrub starts 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[40404]: 3.d deep-scrub ok 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[40404]: 4.12 scrub starts 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[40404]: 4.12 scrub ok 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[40404]: osdmap e413: 8 total, 7 up, 8 in 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[40404]: 3.2 scrub starts 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[40404]: 3.2 scrub ok 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[40404]: 4.1f deep-scrub starts 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[40404]: 4.1f deep-scrub ok 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[29691]: pgmap v858: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[29691]: 3.d deep-scrub starts 2024-03-26T05:18:47.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[29691]: 3.d deep-scrub ok 2024-03-26T05:18:47.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[29691]: 4.12 scrub starts 2024-03-26T05:18:47.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[29691]: 4.12 scrub ok 2024-03-26T05:18:47.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[29691]: osdmap e413: 8 total, 7 up, 8 in 2024-03-26T05:18:47.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[29691]: 3.2 scrub starts 2024-03-26T05:18:47.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[29691]: 3.2 scrub ok 2024-03-26T05:18:47.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[29691]: 4.1f deep-scrub starts 2024-03-26T05:18:47.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:47 smithi080 ceph-mon[29691]: 4.1f deep-scrub ok 2024-03-26T05:18:47.933 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:47.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:48.035 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:47 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:47 smithi132 ceph-mon[33500]: pgmap v858: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:47 smithi132 ceph-mon[33500]: 3.d deep-scrub starts 2024-03-26T05:18:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:47 smithi132 ceph-mon[33500]: 3.d deep-scrub ok 2024-03-26T05:18:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:47 smithi132 ceph-mon[33500]: 4.12 scrub starts 2024-03-26T05:18:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:47 smithi132 ceph-mon[33500]: 4.12 scrub ok 2024-03-26T05:18:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:47 smithi132 ceph-mon[33500]: osdmap e413: 8 total, 7 up, 8 in 2024-03-26T05:18:48.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:47 smithi132 ceph-mon[33500]: 3.2 scrub starts 2024-03-26T05:18:48.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:47 smithi132 ceph-mon[33500]: 3.2 scrub ok 2024-03-26T05:18:48.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:47 smithi132 ceph-mon[33500]: 4.1f deep-scrub starts 2024-03-26T05:18:48.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:47 smithi132 ceph-mon[33500]: 4.1f deep-scrub ok 2024-03-26T05:18:48.061 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:48.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:48.141 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:18:48.162 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:48.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:48.187 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:48.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:48.288 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:48.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:48.312 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:48.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:48.413 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:48.441 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:48.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:48.543 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:48.577 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:48.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:48.678 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:48.705 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:48.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:48.715 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:48 smithi080 ceph-mon[40404]: 3.18 deep-scrub starts 2024-03-26T05:18:48.715 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:48 smithi080 ceph-mon[40404]: 3.18 deep-scrub ok 2024-03-26T05:18:48.715 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:48 smithi080 ceph-mon[29691]: 3.18 deep-scrub starts 2024-03-26T05:18:48.715 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:48 smithi080 ceph-mon[29691]: 3.18 deep-scrub ok 2024-03-26T05:18:48.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:48.834 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:48.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:48.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:48.838 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:18:48.838 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:18:48.936 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:48.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:48.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:49.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:48 smithi132 ceph-mon[33500]: 3.18 deep-scrub starts 2024-03-26T05:18:49.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:48 smithi132 ceph-mon[33500]: 3.18 deep-scrub ok 2024-03-26T05:18:49.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:49.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:49.091 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:49.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:49.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:49.216 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:49.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:49.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:49.318 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:49.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:49.343 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:49.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:49.444 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:49.470 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:49.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:49.547 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T05:18:49.572 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:49.614 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:49.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:49.680 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-26T05:18:49.715 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:49.740 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:49.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:49.842 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:49.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:49 smithi080 ceph-mon[29691]: pgmap v860: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 791 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:49.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:49 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3387860328' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:18:49.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:49 smithi080 ceph-mon[40404]: pgmap v860: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 791 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:49.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:49 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3387860328' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:18:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:49.866 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:49.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:49.967 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:49.991 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:49.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:49 smithi132 ceph-mon[33500]: pgmap v860: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 791 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:49 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3387860328' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:18:50.093 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:50.119 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:50.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:50.221 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:50.244 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:50.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:50.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:50.346 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:50.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:50.372 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:50.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:50.473 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:50.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:50.498 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:50.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:50.599 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:50.624 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:50.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:50.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:50.726 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:50.751 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:50.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:50.853 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:50.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:50.877 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:50.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:50.978 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:51.003 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:51.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:50 smithi132 ceph-mon[33500]: 6.d scrub starts 2024-03-26T05:18:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:50 smithi132 ceph-mon[33500]: 6.d scrub ok 2024-03-26T05:18:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:50 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3387860328' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:18:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:50 smithi132 ceph-mon[33500]: osdmap e414: 8 total, 7 up, 8 in 2024-03-26T05:18:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:18:51.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:18:51.104 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:51.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[29691]: 6.d scrub starts 2024-03-26T05:18:51.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[29691]: 6.d scrub ok 2024-03-26T05:18:51.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3387860328' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:18:51.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[29691]: osdmap e414: 8 total, 7 up, 8 in 2024-03-26T05:18:51.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:51.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:18:51.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:18:51.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[40404]: 6.d scrub starts 2024-03-26T05:18:51.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[40404]: 6.d scrub ok 2024-03-26T05:18:51.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3387860328' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:18:51.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[40404]: osdmap e414: 8 total, 7 up, 8 in 2024-03-26T05:18:51.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:51.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:18:51.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:18:51.127 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:51.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:18:51.191 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:18:51.191 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-26T05:18:51.229 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:51.257 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:51.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:51.358 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:51.383 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:51.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:51.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:51.485 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:51.509 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:51.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:51.600 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:51.611 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:51.641 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:51.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:51.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:51.655 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:51.663 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:51.664 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:18:51.689 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:18:51.709 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:18:51.728 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:18:51.743 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:51.749 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:18:51.766 DEBUG:teuthology.orchestra.run.smithi080:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-26T05:18:51.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:51.767 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:51.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:51.769 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:18:51.770 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T05:18:51.770 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T05:18:51.789 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:18:51.790 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:18:51.790 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:18:51.790 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:18:51.790 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:18:51.790 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:18:51.790 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:18:51.790 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:18:51.790 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: pgmap v862: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: 3.1c scrub starts 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: 3.1c scrub ok 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: 4.a scrub starts 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: 4.a scrub ok 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: 3.12 scrub starts 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: 3.12 scrub ok 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2406448533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:18:51.791 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[40404]: osdmap e415: 8 total, 7 up, 8 in 2024-03-26T05:18:51.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: pgmap v862: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:51.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: 3.1c scrub starts 2024-03-26T05:18:51.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: 3.1c scrub ok 2024-03-26T05:18:51.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:18:51.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: 4.a scrub starts 2024-03-26T05:18:51.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: 4.a scrub ok 2024-03-26T05:18:51.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:51.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:51.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:18:51.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: 3.12 scrub starts 2024-03-26T05:18:51.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: 3.12 scrub ok 2024-03-26T05:18:51.792 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2406448533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:18:51.793 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:51 smithi080 ceph-mon[29691]: osdmap e415: 8 total, 7 up, 8 in 2024-03-26T05:18:51.868 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:51.892 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:51.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:51.959 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:hints = 1 2024-03-26T05:18:51.959 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T05:18:51.960 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Object prefix: benchmark_data_smithi080_189724 2024-03-26T05:18:51.960 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:18:51.960 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 0 0 0 0 0 0 - 0 2024-03-26T05:18:51.994 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:52.038 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:52.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:52.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: pgmap v862: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: 3.1c scrub starts 2024-03-26T05:18:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: 3.1c scrub ok 2024-03-26T05:18:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:18:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: 4.a scrub starts 2024-03-26T05:18:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: 4.a scrub ok 2024-03-26T05:18:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:18:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: 3.12 scrub starts 2024-03-26T05:18:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: 3.12 scrub ok 2024-03-26T05:18:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2406448533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:18:52.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:51 smithi132 ceph-mon[33500]: osdmap e415: 8 total, 7 up, 8 in 2024-03-26T05:18:52.140 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:52.180 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:52.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:52.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:52.322 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:52.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:52.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:52.424 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:52.465 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:52.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:52.567 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:52.599 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:52.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:52.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:52.701 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:52.733 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:52.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:52.834 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:52.868 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:52.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:52.959 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 1 16 3492 3476 217.237 217.25 0.00486979 0.00458729 2024-03-26T05:18:52.970 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:53.004 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:53.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:53.015 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:52 smithi080 ceph-mon[29691]: 5.8 deep-scrub starts 2024-03-26T05:18:53.015 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:52 smithi080 ceph-mon[29691]: 5.8 deep-scrub ok 2024-03-26T05:18:53.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:52 smithi080 ceph-mon[40404]: 5.8 deep-scrub starts 2024-03-26T05:18:53.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:52 smithi080 ceph-mon[40404]: 5.8 deep-scrub ok 2024-03-26T05:18:53.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:52 smithi132 ceph-mon[33500]: 5.8 deep-scrub starts 2024-03-26T05:18:53.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:52 smithi132 ceph-mon[33500]: 5.8 deep-scrub ok 2024-03-26T05:18:53.106 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:53.147 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:53.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:53.249 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:53.288 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:53.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:53.390 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:53.431 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:53.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:53.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:53.533 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:53.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:53.563 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:53.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:53.664 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:53.706 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:53.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:53.808 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:53.843 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:53.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:53.944 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:53.959 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 2 16 6514 6498 203.047 188.875 0.00554643 0.00491297 2024-03-26T05:18:53.986 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:53.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:53.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:54.023 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:53 smithi080 ceph-mon[40404]: pgmap v864: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:54.023 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:53 smithi080 ceph-mon[40404]: 6.13 deep-scrub starts 2024-03-26T05:18:54.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:53 smithi080 ceph-mon[40404]: 6.13 deep-scrub ok 2024-03-26T05:18:54.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:53 smithi080 ceph-mon[29691]: pgmap v864: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:54.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:53 smithi080 ceph-mon[29691]: 6.13 deep-scrub starts 2024-03-26T05:18:54.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:53 smithi080 ceph-mon[29691]: 6.13 deep-scrub ok 2024-03-26T05:18:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:53 smithi132 ceph-mon[33500]: pgmap v864: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:53 smithi132 ceph-mon[33500]: 6.13 deep-scrub starts 2024-03-26T05:18:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:53 smithi132 ceph-mon[33500]: 6.13 deep-scrub ok 2024-03-26T05:18:54.087 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:54.127 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:54.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:54.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:54.228 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:54.266 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:54.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:54.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:18:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:18:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:18:54.368 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:54.402 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:54.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:54.504 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:54.548 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:54.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:54.649 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:18:54.668 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:18:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:54.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:54.793 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:54.830 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:54.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:54.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:54.932 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:54.960 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 3 16 9650 9634 200.692 196 0.00630516 0.00497455 2024-03-26T05:18:54.964 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:54.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:55.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:54 smithi132 ceph-mon[33500]: pgmap v865: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 685 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:55.067 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:55.095 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:54 smithi080 ceph-mon[40404]: pgmap v865: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 685 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:55.095 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:54 smithi080 ceph-mon[29691]: pgmap v865: 158 pgs: 1 active+clean+remapped, 41 active+undersized, 95 active+clean, 21 active+undersized+degraded; 687 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 685 B/s rd, 0 op/s; 90/693 objects degraded (12.987%); 5/693 objects misplaced (0.722%) 2024-03-26T05:18:55.097 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:55.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:55.199 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:55.228 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:55.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:55.330 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:55.362 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:55.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:55.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:55.464 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:55.505 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:55.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:55.607 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:55.649 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:55.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:55.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:55.751 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:55.782 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:55.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:55.884 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:55.912 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:55.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:55.960 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 4 16 12116 12100 189.047 154.125 0.00372622 0.00528 2024-03-26T05:18:56.014 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:56.042 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:56.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:56.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:56.144 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:56.149 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:18:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:18:56.175 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:56.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:56.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:56.277 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:56.301 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:56.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:56.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:56.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:56 smithi080 ceph-mon[29691]: 5.12 deep-scrub starts 2024-03-26T05:18:56.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:56 smithi080 ceph-mon[29691]: 5.12 deep-scrub ok 2024-03-26T05:18:56.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:56 smithi080 ceph-mon[40404]: 5.12 deep-scrub starts 2024-03-26T05:18:56.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:56 smithi080 ceph-mon[40404]: 5.12 deep-scrub ok 2024-03-26T05:18:56.403 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:18:56.434 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:56.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:18:56.536 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:56.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:56 smithi132 ceph-mon[33500]: 5.12 deep-scrub starts 2024-03-26T05:18:56.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:56 smithi132 ceph-mon[33500]: 5.12 deep-scrub ok 2024-03-26T05:18:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:56.578 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:56.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:56.669 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:18:56.680 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:56.761 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:56.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:56.863 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:56.951 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:56.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:56.960 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 5 16 14257 14241 177.998 133.812 0.00273503 0.0056113 2024-03-26T05:18:57.053 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:57.080 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:57.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:57.182 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:57.227 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:57.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:57.303 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:57 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:57.303 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:57 smithi080 ceph-mon[40404]: pgmap v866: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 642 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 3053/31473 objects degraded (9.700%); 5/31473 objects misplaced (0.016%) 2024-03-26T05:18:57.303 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:57 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 3053/31473 objects degraded (9.700%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:57.303 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:57 smithi080 ceph-mon[40404]: 5.d scrub starts 2024-03-26T05:18:57.303 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:57 smithi080 ceph-mon[40404]: 5.d scrub ok 2024-03-26T05:18:57.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:57 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:57.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:57 smithi080 ceph-mon[29691]: pgmap v866: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 642 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 3053/31473 objects degraded (9.700%); 5/31473 objects misplaced (0.016%) 2024-03-26T05:18:57.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:57 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 3053/31473 objects degraded (9.700%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:57.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:57 smithi080 ceph-mon[29691]: 5.d scrub starts 2024-03-26T05:18:57.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:57 smithi080 ceph-mon[29691]: 5.d scrub ok 2024-03-26T05:18:57.328 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:57.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:57 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:18:57.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:57 smithi132 ceph-mon[33500]: pgmap v866: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 642 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 3053/31473 objects degraded (9.700%); 5/31473 objects misplaced (0.016%) 2024-03-26T05:18:57.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:57 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 3053/31473 objects degraded (9.700%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:18:57.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:57 smithi132 ceph-mon[33500]: 5.d scrub starts 2024-03-26T05:18:57.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:57 smithi132 ceph-mon[33500]: 5.d scrub ok 2024-03-26T05:18:57.571 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:57.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:57.673 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:57.715 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:57.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:57.816 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:57.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:57.850 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:57.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:57.952 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:57.960 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 6 16 16623 16607 172.969 147.875 0.00405538 0.00577125 2024-03-26T05:18:57.992 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:57.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:58.094 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:18:58.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:58.135 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:58.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:18:58.236 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:58.278 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:58.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:58.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:58.380 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:58.409 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:18:58.410 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:18:58.410 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[40404]: 4.10 scrub starts 2024-03-26T05:18:58.410 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[40404]: 4.10 scrub ok 2024-03-26T05:18:58.410 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[40404]: 4.1e scrub starts 2024-03-26T05:18:58.410 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[40404]: 4.1e scrub ok 2024-03-26T05:18:58.410 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[40404]: osdmap e416: 8 total, 7 up, 8 in 2024-03-26T05:18:58.410 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:18:58.411 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:18:58.411 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[29691]: 4.10 scrub starts 2024-03-26T05:18:58.411 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[29691]: 4.10 scrub ok 2024-03-26T05:18:58.411 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[29691]: 4.1e scrub starts 2024-03-26T05:18:58.411 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[29691]: 4.1e scrub ok 2024-03-26T05:18:58.411 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:58 smithi080 ceph-mon[29691]: osdmap e416: 8 total, 7 up, 8 in 2024-03-26T05:18:58.411 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:18:58.411 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:18:58.411 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:18:58.411 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:18:58.411 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:18:58.412 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:18:58.412 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:18:58.412 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:18:58.412 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:18:58.412 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:18:58.412 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:18:58.412 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:18:58.438 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:58.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:58.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:58.540 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:58.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:58 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:18:58.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:58 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:18:58.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:58 smithi132 ceph-mon[33500]: 4.10 scrub starts 2024-03-26T05:18:58.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:58 smithi132 ceph-mon[33500]: 4.10 scrub ok 2024-03-26T05:18:58.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:58 smithi132 ceph-mon[33500]: 4.1e scrub starts 2024-03-26T05:18:58.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:58 smithi132 ceph-mon[33500]: 4.1e scrub ok 2024-03-26T05:18:58.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:58 smithi132 ceph-mon[33500]: osdmap e416: 8 total, 7 up, 8 in 2024-03-26T05:18:58.581 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:58.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:58.683 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:58.724 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:58.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:58.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:58.826 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:18:58.858 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:58.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:18:58.960 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:58.961 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 7 16 18867 18851 168.293 140.25 0.00409215 0.00593608 2024-03-26T05:18:59.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:59.001 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:59.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:59.102 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:59.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:59.148 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:59.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:59.185 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:18:59.248 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:59.281 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:59.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:59.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[29691]: 3.16 scrub starts 2024-03-26T05:18:59.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[29691]: 3.16 scrub ok 2024-03-26T05:18:59.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[29691]: pgmap v868: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 696 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 3053/34071 objects degraded (8.961%); 5/34071 objects misplaced (0.015%) 2024-03-26T05:18:59.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:18:59.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:18:59.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3715240981' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:18:59.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:18:59.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[29691]: 1.0 scrub starts 2024-03-26T05:18:59.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[29691]: 1.0 scrub ok 2024-03-26T05:18:59.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[40404]: 3.16 scrub starts 2024-03-26T05:18:59.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[40404]: 3.16 scrub ok 2024-03-26T05:18:59.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[40404]: pgmap v868: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 696 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 3053/34071 objects degraded (8.961%); 5/34071 objects misplaced (0.015%) 2024-03-26T05:18:59.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:18:59.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:18:59.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3715240981' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:18:59.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:18:59.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[40404]: 1.0 scrub starts 2024-03-26T05:18:59.368 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:18:59 smithi080 ceph-mon[40404]: 1.0 scrub ok 2024-03-26T05:18:59.382 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:18:59.413 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:59.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:59.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:18:59.515 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:18:59.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:59 smithi132 ceph-mon[33500]: 3.16 scrub starts 2024-03-26T05:18:59.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:59 smithi132 ceph-mon[33500]: 3.16 scrub ok 2024-03-26T05:18:59.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:59 smithi132 ceph-mon[33500]: pgmap v868: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 696 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 3053/34071 objects degraded (8.961%); 5/34071 objects misplaced (0.015%) 2024-03-26T05:18:59.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:59 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:18:59.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:59 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:18:59.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:59 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3715240981' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:18:59.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:59 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:18:59.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:59 smithi132 ceph-mon[33500]: 1.0 scrub starts 2024-03-26T05:18:59.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:18:59 smithi132 ceph-mon[33500]: 1.0 scrub ok 2024-03-26T05:18:59.544 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:18:59.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:59.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:18:59.646 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:59.687 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:59.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:18:59.788 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:18:59.826 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:59.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:18:59.928 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:18:59.960 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 8 16 20966 20950 163.654 131.188 0.00808767 0.00610385 2024-03-26T05:18:59.968 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:18:59.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:18:59.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:00.070 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:00.094 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:00.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:00.196 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:00.228 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:00.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:00.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:00.330 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:00.361 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:00.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:00.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:00.463 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:00.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:00.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:00.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:00 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:19:00.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:00 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:19:00.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:00 smithi132 ceph-mon[33500]: osdmap e417: 8 total, 7 up, 8 in 2024-03-26T05:19:00.601 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:00.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:00 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:19:00.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:00 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:19:00.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:00 smithi080 ceph-mon[29691]: osdmap e417: 8 total, 7 up, 8 in 2024-03-26T05:19:00.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:00 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:19:00.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:00 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:19:00.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:00 smithi080 ceph-mon[40404]: osdmap e417: 8 total, 7 up, 8 in 2024-03-26T05:19:00.625 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:00.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:00.726 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:00.758 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:00.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:00.860 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:00.890 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:00.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:00.960 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 9 16 23091 23075 160.226 132.812 0.0070298 0.00623507 2024-03-26T05:19:00.992 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:01.026 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:01.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:01.128 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:01.157 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:01.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:01.259 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:01.300 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:01.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:01.401 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:01.434 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:01.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:01.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:01.536 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:01.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:01 smithi132 ceph-mon[33500]: pgmap v870: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 963 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 120 MiB/s wr, 1.92k op/s; 3053/46881 objects degraded (6.512%); 5/46881 objects misplaced (0.011%) 2024-03-26T05:19:01.558 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:01.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:01.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:01.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:01 smithi080 ceph-mon[29691]: pgmap v870: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 963 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 120 MiB/s wr, 1.92k op/s; 3053/46881 objects degraded (6.512%); 5/46881 objects misplaced (0.011%) 2024-03-26T05:19:01.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:01 smithi080 ceph-mon[40404]: pgmap v870: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 963 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 120 MiB/s wr, 1.92k op/s; 3053/46881 objects degraded (6.512%); 5/46881 objects misplaced (0.011%) 2024-03-26T05:19:01.661 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:01.691 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:01.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:01.793 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:01.831 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:01.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:01.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:01.933 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:01.960 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 10 16 25045 25029 156.415 122.125 0.00517088 0.00638706 2024-03-26T05:19:01.964 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:01.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:02.066 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:02.099 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:02.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:02.201 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:02.233 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:02.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:02.335 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:02.377 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:02.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:02.437 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:02 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 3053/46881 objects degraded (6.512%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:02.437 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:02 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 3053/46881 objects degraded (6.512%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:02.479 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:02.505 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:02.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:02.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:02 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 3053/46881 objects degraded (6.512%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:02.607 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:02.637 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:02.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:02.739 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:02.773 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:02.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:02.875 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:02.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:02.906 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:02.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:02.960 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 11 16 26666 26650 151.404 101.312 0.00184824 0.00655931 2024-03-26T05:19:03.007 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:03.033 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:03.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:03.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:03.135 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:03.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:03.167 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:03.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:03.268 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:03.308 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:03.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:03.410 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:03.442 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:03.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:03.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:03.544 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:03.574 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:03.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:03.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:03.676 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:03.717 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:03.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:03.819 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:03.844 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:03.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:03.946 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:03.961 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 12 16 28328 28312 147.437 103.875 0.00398779 0.00677644 2024-03-26T05:19:03.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:03.985 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:03.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:04.023 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:03 smithi080 ceph-mon[29691]: pgmap v871: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 174 MiB/s wr, 2.78k op/s; 5778/67386 objects degraded (8.574%); 5/67386 objects misplaced (0.007%) 2024-03-26T05:19:04.023 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:03 smithi080 ceph-mon[40404]: pgmap v871: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 174 MiB/s wr, 2.78k op/s; 5778/67386 objects degraded (8.574%); 5/67386 objects misplaced (0.007%) 2024-03-26T05:19:04.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:03 smithi132 ceph-mon[33500]: pgmap v871: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 174 MiB/s wr, 2.78k op/s; 5778/67386 objects degraded (8.574%); 5/67386 objects misplaced (0.007%) 2024-03-26T05:19:04.086 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:04.126 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:04.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:04.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:04.228 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:04.260 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:04.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:04.362 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:04.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:19:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:19:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:19:04.401 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:04.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:04.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:04.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:04.531 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:04.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:04.633 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:04.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:04.682 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:04.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:04.783 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:04.807 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:04.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:04.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:04.909 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:04.943 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:04.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:04.961 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 13 16 30192 30176 145.056 116.5 0.00899976 0.00688857 2024-03-26T05:19:05.045 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:05.069 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:05.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:05.170 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:05.209 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:05.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:05.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:05.311 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:05.349 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:05.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:05.450 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:05.481 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:05.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:05.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:05.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:05.614 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:05.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:05.716 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:05.741 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:05.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:05.843 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:05.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:05.868 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:05.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:05.961 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 14 16 31570 31554 140.847 86.125 0.00474603 0.00709446 2024-03-26T05:19:05.969 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:05.991 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:05.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:05 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:19:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:05 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:19:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:05 smithi132 ceph-mon[33500]: pgmap v872: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 93 MiB/s wr, 1.50k op/s; 5778/67386 objects degraded (8.574%); 5/67386 objects misplaced (0.007%) 2024-03-26T05:19:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:05 smithi132 ceph-mon[33500]: osdmap e418: 8 total, 7 up, 8 in 2024-03-26T05:19:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:19:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:05 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:19:06.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:19:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:19:06.093 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:05 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:19:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:05 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:19:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:05 smithi080 ceph-mon[40404]: pgmap v872: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 93 MiB/s wr, 1.50k op/s; 5778/67386 objects degraded (8.574%); 5/67386 objects misplaced (0.007%) 2024-03-26T05:19:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:05 smithi080 ceph-mon[40404]: osdmap e418: 8 total, 7 up, 8 in 2024-03-26T05:19:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:19:06.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:05 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:19:06.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:05 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:19:06.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:05 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:19:06.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:05 smithi080 ceph-mon[29691]: pgmap v872: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 93 MiB/s wr, 1.50k op/s; 5778/67386 objects degraded (8.574%); 5/67386 objects misplaced (0.007%) 2024-03-26T05:19:06.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:05 smithi080 ceph-mon[29691]: osdmap e418: 8 total, 7 up, 8 in 2024-03-26T05:19:06.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:19:06.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:05 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:19:06.131 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:06.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:06.233 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:06.257 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:06.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:06.359 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:06.393 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:06.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:06.495 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:06.527 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:06.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:06.629 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:06.658 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:06.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:06.760 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:06.797 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:06.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:06.900 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:06.934 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:06 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 5778/67386 objects degraded (8.574%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:06.934 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:06 smithi080 ceph-mon[40404]: osdmap e419: 8 total, 7 up, 8 in 2024-03-26T05:19:06.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:06 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 5778/67386 objects degraded (8.574%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:06.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:06 smithi080 ceph-mon[29691]: osdmap e419: 8 total, 7 up, 8 in 2024-03-26T05:19:06.941 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:06.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:06.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:06.961 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 15 16 33159 33143 138.077 99.3125 0.00426037 0.00723561 2024-03-26T05:19:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:06 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 5778/67386 objects degraded (8.574%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:06 smithi132 ceph-mon[33500]: osdmap e419: 8 total, 7 up, 8 in 2024-03-26T05:19:07.043 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:07.074 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:07.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:07.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:07.176 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:07.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:07.259 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:07.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:07.294 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:19:07.341 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:19:07.341 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-26T05:19:07.360 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:07.393 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:07.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:07.495 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:07.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:07.542 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:07.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:07.643 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:07.694 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:07.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:07.796 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:07.842 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:07.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:07.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:07 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:07.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:07 smithi080 ceph-mon[29691]: pgmap v874: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 151 MiB/s wr, 2.42k op/s; 7891/92136 objects degraded (8.565%); 5/92136 objects misplaced (0.005%) 2024-03-26T05:19:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:07 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:07.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:07 smithi080 ceph-mon[40404]: pgmap v874: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 151 MiB/s wr, 2.42k op/s; 7891/92136 objects degraded (8.565%); 5/92136 objects misplaced (0.005%) 2024-03-26T05:19:07.931 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:19:07.932 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:19:07.932 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:19:07.932 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:19:07.932 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.932 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:176711 10.18 10:19868206:::benchmark_data_smithi080_161990_object64988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-03-26T05:19:07.932 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:28.738372+0000", 2024-03-26T05:19:07.932 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 99.198472449999997, 2024-03-26T05:19:07.932 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.021871773, 2024-03-26T05:19:07.932 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:07.932 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:19:07.933 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:07.933 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:07.933 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:07.933 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 176711 2024-03-26T05:19:07.933 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.933 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:07.933 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.933 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:07.933 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.738372+0000", 2024-03-26T05:19:07.933 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.933 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.934 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.934 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:07.934 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.738372+0000", 2024-03-26T05:19:07.935 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.935 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.935 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.935 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.738374+0000", 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1990000000000001e-06 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.738379+0000", 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3730000000000001e-06 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.738387+0000", 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4190000000000002e-06 2024-03-26T05:19:07.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.937 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.937 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:07.937 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.738394+0000", 2024-03-26T05:19:07.937 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4590000000000003e-06 2024-03-26T05:19:07.937 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.937 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.937 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.938 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.738567+0000", 2024-03-26T05:19:07.938 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017306600000000001 2024-03-26T05:19:07.938 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.938 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.938 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:07.938 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.739063+0000", 2024-03-26T05:19:07.938 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00049592199999999999 2024-03-26T05:19:07.938 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.938 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.938 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,1,3,7", 2024-03-26T05:19:07.938 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.739437+0000", 2024-03-26T05:19:07.939 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00037435099999999998 2024-03-26T05:19:07.939 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.939 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.939 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.939 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.747933+0000", 2024-03-26T05:19:07.939 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0084960190000000005 2024-03-26T05:19:07.939 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.939 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.939 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.940 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.747975+0000", 2024-03-26T05:19:07.940 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1248000000000002e-05 2024-03-26T05:19:07.940 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.940 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.940 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:07.940 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.748990+0000", 2024-03-26T05:19:07.940 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001015625 2024-03-26T05:19:07.940 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.940 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.940 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.940 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.782818+0000", 2024-03-26T05:19:07.940 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.033827529000000002 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.782839+0000", 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1239000000000001e-05 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.760167+0000", 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97732843899999999 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:07.941 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.760244+0000", 2024-03-26T05:19:07.942 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6885000000000007e-05 2024-03-26T05:19:07.942 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.942 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:07.942 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.942 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.942 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.942 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:176789 10.18 10:19d5bf53:::benchmark_data_smithi080_161990_object65066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:07.942 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:28.833685+0000", 2024-03-26T05:19:07.942 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 99.103160243999994, 2024-03-26T05:19:07.942 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93573055299999996, 2024-03-26T05:19:07.942 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:07.942 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 176789 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.833685+0000", 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.833685+0000", 2024-03-26T05:19:07.943 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.833686+0000", 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.415e-06 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.833692+0000", 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.75e-06 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:07.944 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.833709+0000", 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6918e-05 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.833717+0000", 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7710000000000006e-06 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.835096+0000", 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0013798619999999999 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:07.945 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.835106+0000", 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.3880000000000006e-06 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.760707+0000", 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.92560083800000004 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.760829+0000", 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000122719 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.946 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:07.947 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.761176+0000", 2024-03-26T05:19:07.947 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00034621100000000002 2024-03-26T05:19:07.947 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.947 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.947 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:07.947 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.767122+0000", 2024-03-26T05:19:07.947 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.005946588 2024-03-26T05:19:07.947 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.947 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.947 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.769283+0000", 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0021606849999999999 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.769324+0000", 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1213e-05 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.769372+0000", 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8149000000000003e-05 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.948 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.769415+0000", 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3046000000000001e-05 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:176842 10.18 10:1b57f3cb:::benchmark_data_smithi080_161990_object65119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:28.885793+0000", 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 99.051052236999993, 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.88769193599999996, 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:07.949 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 176842 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.885793+0000", 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:07.950 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.885793+0000", 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.885794+0000", 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.646e-06 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.885805+0000", 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0515e-05 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.951 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.952 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:07.952 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.885818+0000", 2024-03-26T05:19:07.952 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3633e-05 2024-03-26T05:19:07.952 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.952 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.885828+0000", 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4739999999999994e-06 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.885987+0000", 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015886399999999999 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.885995+0000", 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4970000000000001e-06 2024-03-26T05:19:07.953 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.954 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.954 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.954 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.762251+0000", 2024-03-26T05:19:07.954 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.87625575499999997 2024-03-26T05:19:07.954 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.954 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.954 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:07.954 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.762332+0000", 2024-03-26T05:19:07.954 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1093000000000004e-05 2024-03-26T05:19:07.954 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.954 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.954 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:07.955 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.762510+0000", 2024-03-26T05:19:07.955 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017830999999999999 2024-03-26T05:19:07.955 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.955 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.955 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.955 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.769457+0000", 2024-03-26T05:19:07.955 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00694657 2024-03-26T05:19:07.955 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.955 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.955 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.955 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.769543+0000", 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5501999999999996e-05 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773389+0000", 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0038466799999999999 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773439+0000", 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0138e-05 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.956 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773485+0000", 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5259000000000002e-05 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:176878 10.18 10:195f8522:::benchmark_data_smithi080_161990_object65155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:28.914291+0000", 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 99.022554173000003, 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.85931650699999995, 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:07.957 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:07.958 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:07.958 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:07.958 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 176878 2024-03-26T05:19:07.958 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.958 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:07.958 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.958 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:07.958 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.914291+0000", 2024-03-26T05:19:07.958 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.958 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.958 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.958 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.914291+0000", 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.914292+0000", 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9399999999999993e-07 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.914296+0000", 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5000000000000001e-06 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.959 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.914308+0000", 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1909999999999999e-05 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.914316+0000", 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6660000000000008e-06 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.918460+0000", 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0041443519999999996 2024-03-26T05:19:07.960 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.918470+0000", 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0054000000000001e-05 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.763693+0000", 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.84522287699999998 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.763804+0000", 2024-03-26T05:19:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000111159 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.764001+0000", 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019629599999999999 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.769492+0000", 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0054911869999999998 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.769583+0000", 2024-03-26T05:19:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1067000000000001e-05 2024-03-26T05:19:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773536+0000", 2024-03-26T05:19:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0039529650000000001 2024-03-26T05:19:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773573+0000", 2024-03-26T05:19:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7562999999999999e-05 2024-03-26T05:19:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.964 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:07.965 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.965 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:07.965 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773607+0000", 2024-03-26T05:19:07.965 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3917000000000001e-05 2024-03-26T05:19:07.965 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.965 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:07.965 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.965 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.965 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:176904 10.18 10:1806a5a6:::benchmark_data_smithi080_161990_object65181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:28.937320+0000", 2024-03-26T05:19:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 98.999525133999995, 2024-03-26T05:19:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.83640434500000005, 2024-03-26T05:19:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 176904 2024-03-26T05:19:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.937320+0000", 2024-03-26T05:19:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.937320+0000", 2024-03-26T05:19:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.937321+0000", 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6670000000000001e-06 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.937328+0000", 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0639999999999999e-06 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.937345+0000", 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.713e-05 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.937354+0000", 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.628e-06 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.937419+0000", 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4703999999999996e-05 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.937432+0000", 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2815e-05 2024-03-26T05:19:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.767183+0000", 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.82975140300000005 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.767311+0000", 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012781700000000001 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.767657+0000", 2024-03-26T05:19:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00034599000000000001 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773525+0000", 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0058677620000000003 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773613+0000", 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7999999999999998e-05 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773652+0000", 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8735000000000001e-05 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773692+0000", 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0763999999999998e-05 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773724+0000", 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.1866000000000003e-05 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:176935 10.18 10:18817632:::benchmark_data_smithi080_161990_object65212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:28.968119+0000", 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 98.968726028000006, 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.81791180900000005, 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 176935 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.968119+0000", 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.968119+0000", 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.968120+0000", 2024-03-26T05:19:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2559999999999999e-06 2024-03-26T05:19:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.968125+0000", 2024-03-26T05:19:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2009999999999998e-06 2024-03-26T05:19:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.968137+0000", 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2106000000000001e-05 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.968145+0000", 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8550000000000002e-06 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.968208+0000", 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2390000000000004e-05 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:28.968217+0000", 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2569999999999996e-06 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.769610+0000", 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.80139323200000001 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.769726+0000", 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115705 2024-03-26T05:19:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.769969+0000", 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00024343600000000001 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773617+0000", 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036480140000000002 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.777539+0000", 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0039220560000000002 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.785675+0000", 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0081351070000000008 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.785975+0000", 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00030031900000000001 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.786031+0000", 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5875000000000003e-05 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:177024 10.18 10:1c10eef8:::benchmark_data_smithi080_161990_object65301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:29.046963+0000", 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 98.889881443999997, 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.74765746899999996, 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 177024 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.046963+0000", 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.046963+0000", 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.046965+0000", 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3599999999999999e-06 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.046970+0000", 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1590000000000001e-06 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.046982+0000", 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.171e-05 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.046990+0000", 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4160000000000004e-06 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.047053+0000", 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2695999999999997e-05 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.047062+0000", 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6670000000000008e-06 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.771633+0000", 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.72457052 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.771738+0000", 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000104959 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.984 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.771990+0000", 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025213200000000002 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.777470+0000", 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0054794409999999998 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.777587+0000", 2024-03-26T05:19:07.985 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011779 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.794523+0000", 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016935479 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.794577+0000", 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4206999999999998e-05 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.986 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.794621+0000", 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3933000000000001e-05 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:177175 10.18 10:199aa712:::benchmark_data_smithi080_161990_object65452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:29.200181+0000", 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 98.736663892999999, 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.59454604300000002, 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:07.987 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 177175 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.200181+0000", 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.200181+0000", 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.988 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.200183+0000", 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4950000000000001e-06 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.200188+0000", 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1950000000000002e-06 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:07.989 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.200203+0000", 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5376999999999999e-05 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.200211+0000", 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9310000000000008e-06 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.200822+0000", 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00061128099999999998 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.990 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.200832+0000", 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7890000000000006e-06 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773751+0000", 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.572918763 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.773867+0000", 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011574900000000001 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.774122+0000", 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025565700000000001 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.781541+0000", 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0074186590000000002 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.786105+0000", 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.004564472 2024-03-26T05:19:07.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.794654+0000", 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0085489069999999997 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.794695+0000", 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0518000000000001e-05 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.794727+0000", 2024-03-26T05:19:07.993 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2249999999999998e-05 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:177183 10.18 10:19f4b354:::benchmark_data_smithi080_161990_object65460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:29.206102+0000", 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 98.730742653999997, 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.58922412000000002, 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:07.994 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 177183 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.206102+0000", 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.206102+0000", 2024-03-26T05:19:07.995 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.206103+0000", 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.192e-06 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.206108+0000", 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8579999999999999e-06 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.996 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.206120+0000", 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.188e-05 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.206128+0000", 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7309999999999993e-06 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.208312+0000", 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0021841400000000002 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.998 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:07.998 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.208320+0000", 2024-03-26T05:19:07.998 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.841e-06 2024-03-26T05:19:07.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.998 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.998 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:07.998 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.775664+0000", 2024-03-26T05:19:07.998 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.56734412700000003 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.775764+0000", 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010042700000000001 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.776000+0000", 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00023602200000000001 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:07.999 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.786091+0000", 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010090254 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.788264+0000", 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0021729039999999998 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795240+0000", 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0069764340000000001 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.000 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795288+0000", 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8374999999999997e-05 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795326+0000", 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7935000000000002e-05 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.001 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:177203 10.18 10:194a4b16:::benchmark_data_smithi080_161990_object65480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:29.224880+0000", 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 98.71196501, 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.57056609700000005, 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 177203 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:08.002 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.224880+0000", 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.224880+0000", 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.224881+0000", 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.237e-06 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.003 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.224886+0000", 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0610000000000003e-06 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.224899+0000", 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2508e-05 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.224910+0000", 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1251e-05 2024-03-26T05:19:08.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.005 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.005 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.225055+0000", 2024-03-26T05:19:08.005 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000145198 2024-03-26T05:19:08.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.005 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:08.005 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.225066+0000", 2024-03-26T05:19:08.005 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0543000000000001e-05 2024-03-26T05:19:08.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.777616+0000", 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.55255030299999996 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.777717+0000", 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000101445 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.777951+0000", 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000233921 2024-03-26T05:19:08.006 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.788270+0000", 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010318572 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.792717+0000", 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0044467300000000003 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.007 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795364+0000", 2024-03-26T05:19:08.008 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 16 16 34890 34874 136.209 108.188 0.00920765 0.00733574 2024-03-26T05:19:08.009 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002647375 2024-03-26T05:19:08.009 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.009 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.009 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:08.009 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795411+0000", 2024-03-26T05:19:08.009 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6489999999999997e-05 2024-03-26T05:19:08.009 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.009 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.009 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:08.009 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795446+0000", 2024-03-26T05:19:08.009 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5463000000000002e-05 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:177444 10.18 10:1c05209f:::benchmark_data_smithi080_161990_object65721:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:29.436232+0000", 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 98.500612481999994, 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.35933168199999999, 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:08.010 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 177444 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.436232+0000", 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.436232+0000", 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.011 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.436234+0000", 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.234e-06 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.436238+0000", 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3560000000000003e-06 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.436251+0000", 2024-03-26T05:19:08.012 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3067999999999999e-05 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.436257+0000", 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7169999999999996e-06 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.436324+0000", 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7113999999999995e-05 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:08.013 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.436334+0000", 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7529999999999996e-06 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.779355+0000", 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.34302103499999997 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.779456+0000", 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010163 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.014 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.779660+0000", 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020385600000000001 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.788275+0000", 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.008615279 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.792771+0000", 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0044959580000000004 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.015 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795483+0000", 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027112759999999999 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795529+0000", 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6567999999999999e-05 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795564+0000", 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4838000000000001e-05 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.016 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:177485 10.18 10:1ed21f93:::benchmark_data_smithi080_161990_object65762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:29.477016+0000", 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 98.459828822000006, 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.31865934299999998, 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 177485 2024-03-26T05:19:08.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.018 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:08.018 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.018 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:08.018 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.477016+0000", 2024-03-26T05:19:08.018 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.018 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.018 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:08.018 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.477016+0000", 2024-03-26T05:19:08.018 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.477018+0000", 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5290000000000001e-06 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.477024+0000", 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9009999999999999e-06 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.477042+0000", 2024-03-26T05:19:08.019 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8094999999999999e-05 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.477052+0000", 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0380000000000001e-05 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.477110+0000", 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8118999999999997e-05 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.477119+0000", 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8789999999999992e-06 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.781579+0000", 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.30445952700000001 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.781700+0000", 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000121852 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.781939+0000", 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00023904199999999999 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.788281+0000", 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0063417430000000004 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.792812+0000", 2024-03-26T05:19:08.022 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0045307969999999996 2024-03-26T05:19:08.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.023 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.023 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.023 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795597+0000", 2024-03-26T05:19:08.023 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002785353 2024-03-26T05:19:08.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.023 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.023 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:08.023 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795641+0000", 2024-03-26T05:19:08.023 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3463e-05 2024-03-26T05:19:08.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.023 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795675+0000", 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4663000000000003e-05 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:177486 10.18 10:1bfe9341:::benchmark_data_smithi080_161990_object65763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:29.478346+0000", 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 98.458499165999996, 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.31902899200000001, 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:08.024 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 177486 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.478346+0000", 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:08.025 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.478346+0000", 2024-03-26T05:19:08.026 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.026 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.026 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:08.026 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.478346+0000", 2024-03-26T05:19:08.026 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3399999999999998e-07 2024-03-26T05:19:08.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.026 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.026 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:08.026 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.478350+0000", 2024-03-26T05:19:08.026 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7610000000000001e-06 2024-03-26T05:19:08.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.478362+0000", 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1786e-05 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.478368+0000", 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7030000000000003e-06 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.027 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.478441+0000", 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3225000000000004e-05 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.478478+0000", 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7011999999999997e-05 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.783515+0000", 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.30503739800000002 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.783623+0000", 2024-03-26T05:19:08.028 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000107452 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.783916+0000", 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000293584 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.788287+0000", 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0043700950000000001 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.794486+0000", 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0061996710000000003 2024-03-26T05:19:08.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.797267+0000", 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027809509999999998 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.797319+0000", 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2012999999999998e-05 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.797375+0000", 2024-03-26T05:19:08.030 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5606999999999998e-05 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:177489 10.18 10:180affb4:::benchmark_data_smithi080_161990_object65766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:29.480301+0000", 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 98.456544065000003, 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.31718893599999998, 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:08.031 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 177489 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.480301+0000", 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.480301+0000", 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.480302+0000", 2024-03-26T05:19:08.032 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.001e-06 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.480307+0000", 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9679999999999997e-06 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.480318+0000", 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1586000000000001e-05 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:08.033 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.480326+0000", 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0399999999999993e-06 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.480385+0000", 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8566000000000001e-05 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.480393+0000", 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3979999999999999e-06 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.034 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.786168+0000", 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.30577469800000001 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.786284+0000", 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000116051 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.786571+0000", 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00028679399999999999 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:08.035 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.792569+0000", 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0059976329999999996 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795771+0000", 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.003202089 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.797414+0000", 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001643208 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.036 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.797456+0000", 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1788e-05 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.797490+0000", 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4116000000000003e-05 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:177538 10.18 10:1c928827:::benchmark_data_smithi080_161990_object65815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:29.559160+0000", 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 98.377684893999998, 2024-03-26T05:19:08.037 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.244066855, 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 177538 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.559160+0000", 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.559160+0000", 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.559161+0000", 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1930000000000001e-06 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.559166+0000", 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.437e-06 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.559176+0000", 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0767e-05 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.559184+0000", 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5920000000000003e-06 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.559245+0000", 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1005999999999998e-05 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.559253+0000", 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9000000000000006e-06 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.788321+0000", 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.22906831999999999 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.788496+0000", 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017454699999999999 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,3", 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.788753+0000", 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025683699999999999 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.794742+0000", 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0059891509999999998 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.795800+0000", 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010578740000000001 2024-03-26T05:19:08.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.803133+0000", 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.007333483 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.803184+0000", 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0628999999999999e-05 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:29.803227+0000", 2024-03-26T05:19:08.043 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3118999999999998e-05 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:26636 11.2 11:48284595:::benchmark_data_smithi080_189724_object26635:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:02.894143+0000", 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 5.0427023179999999, 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.146436173, 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:19:08.044 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 26636 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.894143+0000", 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.894143+0000", 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.894144+0000", 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.547e-06 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.894167+0000", 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3229000000000001e-05 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.894180+0000", 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2622999999999999e-05 2024-03-26T05:19:08.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.894190+0000", 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0434999999999999e-05 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.894254+0000", 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3695000000000006e-05 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.894392+0000", 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000137935 2024-03-26T05:19:08.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.894562+0000", 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016992900000000001 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.040340+0000", 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.145778142 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.040438+0000", 2024-03-26T05:19:08.048 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7819000000000007e-05 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.040481+0000", 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3288999999999998e-05 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.040534+0000", 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3165999999999999e-05 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.040579+0000", 2024-03-26T05:19:08.049 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4363999999999997e-05 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:26638 11.d 11:b86d3efd:::benchmark_data_smithi080_189724_object26637:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:02.896037+0000", 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 5.0408077499999999, 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14451043, 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:19:08.050 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 26638 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.896037+0000", 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.896037+0000", 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.051 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.896039+0000", 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.545e-06 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.896062+0000", 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3187999999999999e-05 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.896074+0000", 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2395000000000001e-05 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.052 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.896083+0000", 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5709999999999998e-06 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.896150+0000", 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7372000000000005e-05 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.896282+0000", 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013159200000000001 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.053 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 6", 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.896442+0000", 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015983300000000001 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.898064+0000", 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001622365 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.040382+0000", 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.142317677 2024-03-26T05:19:08.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.040447+0000", 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5440999999999999e-05 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.040548+0000", 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000100451 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15930.0:26640 11.c e417/408 11:3b8ebe4c:::benchmark_data_smithi080_189724_object26639:head v 417'1644, mlcod=417'1642)", 2024-03-26T05:19:08.055 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:02.897851+0000", 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 5.0389939940000001, 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14633225699999999, 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.897851+0000", 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.897851+0000", 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.056 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.897852+0000", 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6800000000000009e-07 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:02.899306+0000", 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014541910000000001 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.040338+0000", 2024-03-26T05:19:08.057 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.14103192000000001 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.040359+0000", 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0533000000000001e-05 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.040420+0000", 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1488000000000005e-05 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.058 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.040450+0000", 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.9977000000000001e-05 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.044089+0000", 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036394420000000001 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:03.044183+0000", 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.3738000000000003e-05 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.059 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:31418 11.2 11:4505c645:::benchmark_data_smithi080_189724_object31417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:05.724015+0000", 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 2.212829444, 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.19924818899999999, 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31418 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:08.060 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.724015+0000", 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.724015+0000", 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.724017+0000", 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.688e-06 2024-03-26T05:19:08.061 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.062 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.062 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:08.062 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.724044+0000", 2024-03-26T05:19:08.062 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.7008999999999998e-05 2024-03-26T05:19:08.062 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.062 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.062 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:08.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:07 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:08.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:07 smithi132 ceph-mon[33500]: pgmap v874: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 151 MiB/s wr, 2.42k op/s; 7891/92136 objects degraded (8.565%); 5/92136 objects misplaced (0.005%) 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.724063+0000", 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8442999999999999e-05 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.724072+0000", 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4299999999999995e-06 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.724411+0000", 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00033873399999999999 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.064 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.724580+0000", 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016921300000000001 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.724738+0000", 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015836100000000001 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.726450+0000", 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00171182 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.729260+0000", 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002810206 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923155+0000", 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.19389437600000001 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923219+0000", 2024-03-26T05:19:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4623999999999997e-05 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923264+0000", 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4285e-05 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:31439 11.2 11:480ae1e0:::benchmark_data_smithi080_189724_object31438:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:05.733649+0000", 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 2.2031957090000001, 2024-03-26T05:19:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.19473823400000001, 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31439 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.733649+0000", 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.733649+0000", 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.733651+0000", 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5319999999999999e-06 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.733676+0000", 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5357000000000001e-05 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.733696+0000", 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9952000000000001e-05 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.733706+0000", 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.533e-06 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.733767+0000", 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1198999999999995e-05 2024-03-26T05:19:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.733919+0000", 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000151893 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.734069+0000", 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015022500000000001 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.735416+0000", 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0013466979999999999 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.735713+0000", 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00029736600000000003 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.928286+0000", 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.192572674 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.928342+0000", 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6579000000000002e-05 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:08.073 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:08.073 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.928387+0000", 2024-03-26T05:19:08.073 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5225999999999997e-05 2024-03-26T05:19:08.073 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.073 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.073 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.073 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:08.073 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:08.073 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:19:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:08.074 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:08.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:08.175 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:08.209 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:08.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:08.310 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:08.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:08.342 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:08.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:08.443 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:08.473 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:08.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:08.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:08.575 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:08.606 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:08.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:08.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:08.709 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:08.749 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:08.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:08.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:08.851 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:08.890 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:08.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:08.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:08.962 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 17 15 36770 36755 135.111 117.562 0.020545 0.00739667 2024-03-26T05:19:08.992 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:09.020 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:09.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:09.122 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:09.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:09.160 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:09.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:09.175 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:19:09.261 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:09.292 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:09.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:09.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:09.394 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:09.426 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:09.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:09.527 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:09.571 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:09.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:09.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:09.705 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:09.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:09.806 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:09.837 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:09.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:09.939 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:09.962 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 18 16 38580 38564 133.882 113.062 0.00912194 0.00746292 2024-03-26T05:19:09.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:09 smithi080 ceph-mon[40404]: pgmap v876: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s; 7891/93723 objects degraded (8.419%); 5/93723 objects misplaced (0.005%) 2024-03-26T05:19:10.000 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:09 smithi080 ceph-mon[29691]: pgmap v876: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s; 7891/93723 objects degraded (8.419%); 5/93723 objects misplaced (0.005%) 2024-03-26T05:19:10.000 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:10.001 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T05:19:10.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:10.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:09 smithi132 ceph-mon[33500]: pgmap v876: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s; 7891/93723 objects degraded (8.419%); 5/93723 objects misplaced (0.005%) 2024-03-26T05:19:10.103 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:10.150 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:10.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:10.252 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:10.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:10.307 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:10.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:10.408 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:10.439 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:10.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:10.541 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:10.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:10.583 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:10.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:10.642 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:10.684 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:10.713 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:10.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:10.814 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:10.857 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:10.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:10.959 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:10.962 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 19 16 40188 40172 132.125 100.5 0.0157096 0.00756355 2024-03-26T05:19:10.991 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:10.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:11.093 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:11.122 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:11.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:11.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:11.224 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:11.263 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:11.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:11.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:11.364 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:11.405 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:11.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:11.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:11.507 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:11.551 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:11.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:11.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:11.653 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:11.694 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:11.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:11.796 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:11.837 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:11.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:11.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:11.939 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:11.962 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:19:11.973845+0000 min lat: 0.0012195 max lat: 0.2094 avg lat: 0.00763508 2024-03-26T05:19:11.963 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:19:11.963 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 20 15 41900 41885 130.87 107.062 0.0070243 0.00763508 2024-03-26T05:19:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:11.979 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:11.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:12.003 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:11 smithi080 ceph-mon[29691]: pgmap v877: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 93 MiB/s wr, 1.49k op/s; 7891/103041 objects degraded (7.658%); 5/103041 objects misplaced (0.005%) 2024-03-26T05:19:12.003 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:11 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 7891/103041 objects degraded (7.658%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:12.003 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:11 smithi080 ceph-mon[40404]: pgmap v877: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 93 MiB/s wr, 1.49k op/s; 7891/103041 objects degraded (7.658%); 5/103041 objects misplaced (0.005%) 2024-03-26T05:19:12.003 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:11 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 7891/103041 objects degraded (7.658%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:11 smithi132 ceph-mon[33500]: pgmap v877: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 93 MiB/s wr, 1.49k op/s; 7891/103041 objects degraded (7.658%); 5/103041 objects misplaced (0.005%) 2024-03-26T05:19:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:11 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 7891/103041 objects degraded (7.658%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:12.068 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:12.072 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:12.080 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:12.108 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:12.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:12.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:12.209 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:12.250 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:12.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:12.352 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:12.381 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:12.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:12.483 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:12.521 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:12.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:12.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:12.623 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:12.653 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:12.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:12.755 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:12.785 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:12.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:12.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:12.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:12 smithi080 ceph-mon[29691]: pgmap v878: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.17k op/s; 10010/119403 objects degraded (8.383%); 5/119403 objects misplaced (0.004%) 2024-03-26T05:19:12.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:12 smithi080 ceph-mon[40404]: pgmap v878: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.17k op/s; 10010/119403 objects degraded (8.383%); 5/119403 objects misplaced (0.004%) 2024-03-26T05:19:12.887 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:12.923 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:19:12.924 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:12.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:12.962 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 21 16 43485 43469 129.352 99 0.0107606 0.00772478 2024-03-26T05:19:12.968 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:19:13.009 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:19:13.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:13.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:12 smithi132 ceph-mon[33500]: pgmap v878: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.17k op/s; 10010/119403 objects degraded (8.383%); 5/119403 objects misplaced (0.004%) 2024-03-26T05:19:13.053 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:19:13.054 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:13.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:13.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:13.093 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:19:13.126 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:19:13.156 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:13.180 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:19:13.180 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:19:13.180 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:19:13.180 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:19:13.180 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:19:13.180 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:19:13.180 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:19:13.180 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:19:13.180 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:19:13.205 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:13.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:13.307 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:13.338 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:13.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:13.440 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:13.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:13.482 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:13.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:13.583 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:13.628 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:13.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:13.729 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:13.768 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:13.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:13.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:13.870 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:13.895 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:13.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:13.963 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 22 16 45015 44999 127.819 95.625 0.00811576 0.00781916 2024-03-26T05:19:13.996 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:14.027 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:13 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:19:14.027 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:13 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:19:14.027 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:13 smithi080 ceph-mon[29691]: osdmap e420: 8 total, 7 up, 8 in 2024-03-26T05:19:14.027 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:13 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3386461905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:19:14.027 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:13 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:19:14.027 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:13 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:19:14.027 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:13 smithi080 ceph-mon[40404]: osdmap e420: 8 total, 7 up, 8 in 2024-03-26T05:19:14.027 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:13 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3386461905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:19:14.037 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:14.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:14.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:13 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:19:14.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:13 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:19:14.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:13 smithi132 ceph-mon[33500]: osdmap e420: 8 total, 7 up, 8 in 2024-03-26T05:19:14.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:13 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3386461905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:19:14.138 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:14.178 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:14.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:14.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:14.309 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:14.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:14.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:19:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:19:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:19:14.412 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:14.451 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:14.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:14.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:14.554 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:14.589 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:14.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:14.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:14.691 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:14.729 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:14.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:14.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:14.831 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:14.872 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:14.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:14.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:14.963 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 23 16 46676 46660 126.775 103.812 0.0202405 0.00788178 2024-03-26T05:19:14.974 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:15.014 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:15.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:15.017 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:14 smithi080 ceph-mon[29691]: pgmap v880: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s; 10010/119403 objects degraded (8.383%); 5/119403 objects misplaced (0.004%) 2024-03-26T05:19:15.018 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:14 smithi080 ceph-mon[40404]: pgmap v880: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s; 10010/119403 objects degraded (8.383%); 5/119403 objects misplaced (0.004%) 2024-03-26T05:19:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:14 smithi132 ceph-mon[33500]: pgmap v880: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s; 10010/119403 objects degraded (8.383%); 5/119403 objects misplaced (0.004%) 2024-03-26T05:19:15.041 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:19:15.041 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:19:15.041 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:19:15.042 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:19:15.042 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:19:15.042 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:19:15.042 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:19:15.042 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:19:15.042 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:19:15.042 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:19:15.042 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:19:15.042 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:19:15.117 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:15.159 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:15.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:15.261 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:15.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:15.303 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:15.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:15.404 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:15.445 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:15.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:15.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:15.547 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:15.587 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:15.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:15.689 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:15.717 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:15.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:15.819 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:15.849 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:15.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:15.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:15.952 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:15.962 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 24 16 48323 48307 125.781 102.938 0.0158944 0.00794364 2024-03-26T05:19:15.998 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:15.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:16.008 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:19:16.008 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:19:16.008 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-26T05:19:16.100 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:16.150 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:16.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:16.251 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:16.290 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:16.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:19:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:19:16.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:16.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:16.392 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:16.426 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:16.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:16.527 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:16.570 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:16.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:16.671 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:16.737 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:16.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:16.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:16.842 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:16.885 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:16.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:16.963 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 25 16 49852 49836 124.572 95.5625 0.0106032 0.00802224 2024-03-26T05:19:16.986 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:17.020 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:17.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:17.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:19:17.108 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:19:17.122 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:17.149 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:17.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:17.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:17.251 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:17.281 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:17.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:17.362 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:17 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:17.362 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:17 smithi080 ceph-mon[40404]: pgmap v881: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.77k op/s; 12058/142356 objects degraded (8.470%); 5/142356 objects misplaced (0.004%) 2024-03-26T05:19:17.362 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:17 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 12058/142356 objects degraded (8.470%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:17.363 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:17 smithi080 ceph-mon[40404]: osdmap e421: 8 total, 7 up, 8 in 2024-03-26T05:19:17.363 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:17 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:17.363 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:17 smithi080 ceph-mon[29691]: pgmap v881: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.77k op/s; 12058/142356 objects degraded (8.470%); 5/142356 objects misplaced (0.004%) 2024-03-26T05:19:17.363 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:17 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 12058/142356 objects degraded (8.470%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:17.363 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:17 smithi080 ceph-mon[29691]: osdmap e421: 8 total, 7 up, 8 in 2024-03-26T05:19:17.382 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:17.414 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:17.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:17.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:17.515 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:17.535 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:17 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:17.535 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:17 smithi132 ceph-mon[33500]: pgmap v881: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.77k op/s; 12058/142356 objects degraded (8.470%); 5/142356 objects misplaced (0.004%) 2024-03-26T05:19:17.535 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:17 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 12058/142356 objects degraded (8.470%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:17.536 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:17 smithi132 ceph-mon[33500]: osdmap e421: 8 total, 7 up, 8 in 2024-03-26T05:19:17.557 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:17.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:17.653 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:17.659 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:17.708 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:17.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:17.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:17.812 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:17.842 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:17.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:17.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:17.963 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 26 16 51501 51485 123.745 103.062 0.00692255 0.00807632 2024-03-26T05:19:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:17.975 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:17.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:18.076 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:18.114 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:18.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:18.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:18.217 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:18.248 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:18.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:18.350 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:18.418 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:18.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:18.518 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:18.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:18.560 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:18.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:18.661 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:18.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:18.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:18.795 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:18.825 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:18.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:18.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:18.927 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:18.956 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:18.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:18.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:18.963 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 27 16 52963 52947 122.546 91.375 0.0190519 0.00815496 2024-03-26T05:19:19.058 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:19.107 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:19.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:19.109 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:19:19.210 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:19.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:19 smithi080 ceph-mon[29691]: pgmap v883: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 12058/143808 objects degraded (8.385%); 5/143808 objects misplaced (0.003%) 2024-03-26T05:19:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:19.273 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:19.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:19.374 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:19.394 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:19:19.394 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:19:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:19.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:19.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:19 smithi080 ceph-mon[40404]: pgmap v883: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 12058/143808 objects degraded (8.385%); 5/143808 objects misplaced (0.003%) 2024-03-26T05:19:19.505 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:19.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:19 smithi132 ceph-mon[33500]: pgmap v883: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 12058/143808 objects degraded (8.385%); 5/143808 objects misplaced (0.003%) 2024-03-26T05:19:19.547 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:19.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:19.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:19.648 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:19.677 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:19.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:19.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:19.809 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:19.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:19.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:19.911 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:19.942 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:19.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:19.963 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 28 16 54364 54348 121.296 87.5625 0.00390337 0.008239 2024-03-26T05:19:20.044 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:20.070 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:20.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:20.172 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:20.207 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:20.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:20.253 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:19:20.309 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:20.355 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:20.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:20.457 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:20.500 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:20.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:20.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:20.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:20 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1501318152' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:19:20.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:20 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:19:20.603 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:20.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:20 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1501318152' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:19:20.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:20 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:19:20.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:20 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1501318152' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:19:20.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:20 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:19:20.646 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:20.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:20.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:20.748 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:20.796 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:20.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:20.897 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:20.940 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:20.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:20.963 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 29 16 55827 55811 120.266 91.4375 0.00978897 0.00831008 2024-03-26T05:19:21.042 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:21.073 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:21.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:21.175 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:21.218 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:21.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:21.251 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T05:19:21.320 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:21.327 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:21 smithi080 ceph-mon[40404]: pgmap v884: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s; 12058/152352 objects degraded (7.915%); 5/152352 objects misplaced (0.003%) 2024-03-26T05:19:21.327 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:21 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:19:21.328 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:21 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:19:21.328 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:21 smithi080 ceph-mon[40404]: osdmap e422: 8 total, 7 up, 8 in 2024-03-26T05:19:21.328 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:19:21.328 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:21 smithi080 ceph-mon[29691]: pgmap v884: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s; 12058/152352 objects degraded (7.915%); 5/152352 objects misplaced (0.003%) 2024-03-26T05:19:21.328 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:21 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:19:21.328 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:21 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:19:21.328 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:21 smithi080 ceph-mon[29691]: osdmap e422: 8 total, 7 up, 8 in 2024-03-26T05:19:21.328 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:19:21.348 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:21.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:21.450 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:21.480 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:21.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:21.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:21.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:21 smithi132 ceph-mon[33500]: pgmap v884: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s; 12058/152352 objects degraded (7.915%); 5/152352 objects misplaced (0.003%) 2024-03-26T05:19:21.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:21 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:19:21.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:21 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:19:21.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:21 smithi132 ceph-mon[33500]: osdmap e422: 8 total, 7 up, 8 in 2024-03-26T05:19:21.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:19:21.582 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:21.632 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:21.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:21.734 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:21.774 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:21.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:21.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:21.876 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:21.906 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:21.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:21.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:21.963 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 30 16 57218 57202 119.155 86.9375 0.0083785 0.00838719 2024-03-26T05:19:22.008 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:22.036 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:22.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:22.138 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:22.180 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:22.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:22.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:22.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:22.321 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:22.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:22.422 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:22.463 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:22.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:22.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:22.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:22 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 12058/152352 objects degraded (7.915%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:22.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:22 smithi080 ceph-mon[40404]: osdmap e423: 8 total, 7 up, 8 in 2024-03-26T05:19:22.501 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:22 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 12058/152352 objects degraded (7.915%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:22.501 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:22 smithi080 ceph-mon[29691]: osdmap e423: 8 total, 7 up, 8 in 2024-03-26T05:19:22.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:22 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 12058/152352 objects degraded (7.915%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:22.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:22 smithi132 ceph-mon[33500]: osdmap e423: 8 total, 7 up, 8 in 2024-03-26T05:19:22.565 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:22.634 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:22.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:22.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:22.737 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:22.796 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:22.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:22.875 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:22.883 INFO:teuthology.orchestra.run.smithi080.stderr:no valid command found; 10 closest matches: 2024-03-26T05:19:22.883 INFO:teuthology.orchestra.run.smithi080.stderr:0 2024-03-26T05:19:22.883 INFO:teuthology.orchestra.run.smithi080.stderr:1 2024-03-26T05:19:22.883 INFO:teuthology.orchestra.run.smithi080.stderr:2 2024-03-26T05:19:22.883 INFO:teuthology.orchestra.run.smithi080.stderr:abort 2024-03-26T05:19:22.883 INFO:teuthology.orchestra.run.smithi080.stderr:assert 2024-03-26T05:19:22.883 INFO:teuthology.orchestra.run.smithi080.stderr:bench [] [] [] [] 2024-03-26T05:19:22.884 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs debug_inject_read_zeros 2024-03-26T05:19:22.884 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs files list 2024-03-26T05:19:22.884 INFO:teuthology.orchestra.run.smithi080.stderr:bluefs stats 2024-03-26T05:19:22.884 INFO:teuthology.orchestra.run.smithi080.stderr:bluestore allocator dump block 2024-03-26T05:19:22.884 INFO:teuthology.orchestra.run.smithi080.stderr:Error EINVAL: invalid command 2024-03-26T05:19:22.884 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:22.897 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:22.930 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:22.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:22.964 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 31 16 58607 58591 118.111 86.8125 0.0152543 0.00846245 2024-03-26T05:19:23.032 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:23.062 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:23.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:23.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:23.164 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:23.211 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:23.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:23.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:23.313 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:23.353 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:23.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:23.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:23.455 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:23.483 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:23.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:23.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:23.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:23.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:23.630 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:23.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:19:23.694 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T05:19:23.731 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:23.772 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:23.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:23.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:23.874 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:23.904 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:23 smithi080 ceph-mon[29691]: pgmap v887: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.34k op/s; 13967/166497 objects degraded (8.389%); 5/166497 objects misplaced (0.003%) 2024-03-26T05:19:23.905 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:23 smithi080 ceph-mon[40404]: pgmap v887: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.34k op/s; 13967/166497 objects degraded (8.389%); 5/166497 objects misplaced (0.003%) 2024-03-26T05:19:23.918 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:23.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:23.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:23.964 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 32 16 60066 60050 117.27 91.1875 0.0119036 0.00852259 2024-03-26T05:19:24.020 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:24.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:23 smithi132 ceph-mon[33500]: pgmap v887: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.34k op/s; 13967/166497 objects degraded (8.389%); 5/166497 objects misplaced (0.003%) 2024-03-26T05:19:24.050 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:24.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:24.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:24.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:24.183 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:24.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:24.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:24.256 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:19:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:19:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:19:24.285 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:24.333 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:24.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:24.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:24.435 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:24.467 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:24.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:24.552 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:19:24.569 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:24.639 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:24.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:24.740 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:24.765 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:24.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:24.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:24.868 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:24.916 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:24.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:24.964 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 33 16 61300 61284 116.05 77.125 0.012316 0.00861144 2024-03-26T05:19:25.018 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:25.048 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:25.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:25.150 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:25.179 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:25.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:25.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:25.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:25.308 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:25.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:25.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:25.410 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:25.444 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:25.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:25.544 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:25.603 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:25.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:25.704 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:25.749 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:25.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:25.850 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:25.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:25 smithi080 ceph-mon[40404]: pgmap v888: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 684 B/s rd, 67 MiB/s wr, 1.08k op/s; 13967/166497 objects degraded (8.389%); 5/166497 objects misplaced (0.003%) 2024-03-26T05:19:25.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:25 smithi080 ceph-mon[29691]: pgmap v888: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 684 B/s rd, 67 MiB/s wr, 1.08k op/s; 13967/166497 objects degraded (8.389%); 5/166497 objects misplaced (0.003%) 2024-03-26T05:19:25.879 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:25.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:25.964 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 34 16 62614 62598 115.053 82.125 0.00607489 0.00868689 2024-03-26T05:19:25.981 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:26.009 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:26.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:26.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:25 smithi132 ceph-mon[33500]: pgmap v888: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 684 B/s rd, 67 MiB/s wr, 1.08k op/s; 13967/166497 objects degraded (8.389%); 5/166497 objects misplaced (0.003%) 2024-03-26T05:19:26.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:19:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:19:26.112 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:26.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:26.138 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:26.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:26.239 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:26.267 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:26.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:26.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:26.330 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:26.369 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:26.396 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:26.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:26.497 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:26.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:26.529 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:26.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:26.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:26.659 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:26.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:26.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:26.761 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:26.770 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:19:26.770 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:19:26.770 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:19:26.816 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:26 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2645818254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:19:26.816 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:26 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 13967/166497 objects degraded (8.389%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:26.816 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:19:26.817 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:26 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2645818254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:19:26.817 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:26 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 13967/166497 objects degraded (8.389%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:26.823 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:26.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:26.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:26.854 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:19:26.901 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:19:26.925 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:26.944 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:19:26.944 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:19:26.944 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:19:26.944 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:19:26.944 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:19:26.944 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:19:26.945 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:19:26.945 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:19:26.945 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:19:26.952 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:26.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:26.967 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 35 16 63788 63772 113.853 73.375 0.0176698 0.00877754 2024-03-26T05:19:27.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:26 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2645818254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:19:27.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:26 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 13967/166497 objects degraded (8.389%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:27.054 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:27.089 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:19:27.089 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:19:27.089 INFO:teuthology.orchestra.run.smithi080.stdout: "complaint_time": 30, 2024-03-26T05:19:27.089 INFO:teuthology.orchestra.run.smithi080.stdout: "num_blocked_ops": 0 2024-03-26T05:19:27.089 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:19:27.098 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:27.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:27.200 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:27.242 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:27.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:27.343 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:27.375 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:27.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:27.477 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:27.518 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:27.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:27.619 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:27.661 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:27.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:27.762 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:27.786 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:27.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:27.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:27.887 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:27.926 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:27.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:27.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:27.967 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 36 16 65091 65075 112.952 81.4375 0.0076793 0.00884806 2024-03-26T05:19:28.028 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:28.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:27 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:28.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:27 smithi132 ceph-mon[33500]: pgmap v889: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.75k op/s; 15708/185751 objects degraded (8.456%); 5/185751 objects misplaced (0.003%) 2024-03-26T05:19:28.058 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:28.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:28.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:28.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:27 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:28.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:27 smithi080 ceph-mon[40404]: pgmap v889: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.75k op/s; 15708/185751 objects degraded (8.456%); 5/185751 objects misplaced (0.003%) 2024-03-26T05:19:28.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:27 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:28.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:27 smithi080 ceph-mon[29691]: pgmap v889: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.75k op/s; 15708/185751 objects degraded (8.456%); 5/185751 objects misplaced (0.003%) 2024-03-26T05:19:28.159 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:28.189 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:28.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:28.291 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:28.315 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:28.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:28.417 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:28.457 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:28.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:28.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:28.559 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:28.591 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:28.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:28.692 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:28.695 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T05:19:28.695 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:19:28.695 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-03-26T05:19:28.695 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:19:28.741 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:28.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:28.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:28.843 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:28.873 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:28.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:28.968 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 37 16 66352 66336 112.028 78.8125 0.00318512 0.00891977 2024-03-26T05:19:28.975 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:29.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:29.023 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:29.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:29.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:28 smithi132 ceph-mon[33500]: pgmap v890: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s; 15708/186954 objects degraded (8.402%); 5/186954 objects misplaced (0.003%) 2024-03-26T05:19:29.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:28 smithi080 ceph-mon[40404]: pgmap v890: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s; 15708/186954 objects degraded (8.402%); 5/186954 objects misplaced (0.003%) 2024-03-26T05:19:29.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:28 smithi080 ceph-mon[29691]: pgmap v890: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s; 15708/186954 objects degraded (8.402%); 5/186954 objects misplaced (0.003%) 2024-03-26T05:19:29.191 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:29.231 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:29.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:29.333 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:29.358 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:29.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:29.459 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:29.503 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:29.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:29.604 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:29.643 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:29.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:29.744 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:29.780 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:29.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:29.882 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:29.908 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:29.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:29.968 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 38 16 67607 67591 111.144 78.4375 0.00786382 0.00899256 2024-03-26T05:19:30.010 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:30.052 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:30.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:30.153 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:30.192 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:30.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:30.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:30.294 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:30.323 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:30.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:30.425 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:30.456 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:30.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:30.558 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:30.588 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:30.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:30.689 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:30.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:30.731 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:30.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:30.832 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:30.872 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:30.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:30.968 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 39 16 68881 68865 110.336 79.625 0.00969576 0.00905801 2024-03-26T05:19:30.973 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:31.007 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:31.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:31.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:31.109 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:31.138 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:31.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:31.242 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:31.277 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:31.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:31.379 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:31.420 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:31.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:31.522 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:31.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:31 smithi132 ceph-mon[33500]: pgmap v891: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 88 MiB/s wr, 1.40k op/s; 15708/194064 objects degraded (8.094%); 5/194064 objects misplaced (0.003%) 2024-03-26T05:19:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:31.554 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:31.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:31.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:31 smithi080 ceph-mon[29691]: pgmap v891: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 88 MiB/s wr, 1.40k op/s; 15708/194064 objects degraded (8.094%); 5/194064 objects misplaced (0.003%) 2024-03-26T05:19:31.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:31 smithi080 ceph-mon[40404]: pgmap v891: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 88 MiB/s wr, 1.40k op/s; 15708/194064 objects degraded (8.094%); 5/194064 objects misplaced (0.003%) 2024-03-26T05:19:31.655 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:31.687 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:31.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:31.789 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:31.813 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:31.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:31.914 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:31.955 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:31.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:31.968 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:19:31.979476+0000 min lat: 0.0012195 max lat: 0.2094 avg lat: 0.00911741 2024-03-26T05:19:31.968 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:19:31.968 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 40 16 70185 70169 109.615 81.5 0.0114754 0.00911741 2024-03-26T05:19:32.058 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:32.088 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:32.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:32.190 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:32.220 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:32.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:32.322 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:32.352 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:32.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:32.454 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:32.497 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:32.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:32.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:32 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 15708/194064 objects degraded (8.094%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:32.598 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:32.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:32 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 15708/194064 objects degraded (8.094%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:32.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:32 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 15708/194064 objects degraded (8.094%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:32.637 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:32.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:32.739 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:32.771 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:32.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:32.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:32.873 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:32.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:32.898 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:32.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:32.968 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 41 16 71361 71345 108.734 73.5 0.0109603 0.00919219 2024-03-26T05:19:32.999 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:33.053 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:33.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:33.154 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:33.182 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:33.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:33.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:33.284 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:33.321 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:33.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:33.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:33.423 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:33.454 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:33.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:33.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:33.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:33 smithi132 ceph-mon[33500]: pgmap v892: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 976 B/s rd, 81 MiB/s wr, 1.29k op/s; 17252/205782 objects degraded (8.384%); 5/205782 objects misplaced (0.002%) 2024-03-26T05:19:33.556 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:33.599 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:33.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:33.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:33 smithi080 ceph-mon[40404]: pgmap v892: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 976 B/s rd, 81 MiB/s wr, 1.29k op/s; 17252/205782 objects degraded (8.384%); 5/205782 objects misplaced (0.002%) 2024-03-26T05:19:33.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:33 smithi080 ceph-mon[29691]: pgmap v892: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 976 B/s rd, 81 MiB/s wr, 1.29k op/s; 17252/205782 objects degraded (8.384%); 5/205782 objects misplaced (0.002%) 2024-03-26T05:19:33.700 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:33.730 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:33.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:33.832 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:33.870 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:33.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:33.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:33.968 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 42 16 72553 72537 107.919 74.5 0.00281729 0.00924937 2024-03-26T05:19:33.972 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:33.996 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:33.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:34.098 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:34.127 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:34.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:34.228 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:34.255 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:34.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:34.356 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:34.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:19:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:19:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:19:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:34.392 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:34.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:34.493 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:34.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:34.523 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:34.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:34.625 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:34.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:34.650 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:34.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:34.751 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:34.773 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:34.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:34.875 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:34.907 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:34.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:34.968 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 43 16 73614 73598 106.952 66.3125 0.00349759 0.00934201 2024-03-26T05:19:35.009 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:35.054 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:35.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:35.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:35.156 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:35.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:35.181 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:35.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:35.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:35.321 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:35.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:35.423 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:35.452 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:35.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:35.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:35 smithi132 ceph-mon[33500]: pgmap v893: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s; 17252/205782 objects degraded (8.384%); 5/205782 objects misplaced (0.002%) 2024-03-26T05:19:35.553 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:35.556 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:35 smithi080 ceph-mon[40404]: pgmap v893: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s; 17252/205782 objects degraded (8.384%); 5/205782 objects misplaced (0.002%) 2024-03-26T05:19:35.556 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:35 smithi080 ceph-mon[29691]: pgmap v893: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s; 17252/205782 objects degraded (8.384%); 5/205782 objects misplaced (0.002%) 2024-03-26T05:19:35.593 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:35.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:35.695 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:35.735 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:19:35.736 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:19:35.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:35.755 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:35.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:35.855 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:35.886 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:35.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:35.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:35.968 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 44 16 74785 74769 106.184 73.1875 0.0106498 0.00941336 2024-03-26T05:19:35.988 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:36.033 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:36.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:36.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:36.136 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:36.162 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:36.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:36.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:36.264 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:36.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:19:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:19:36.301 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:36.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:36.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:36.403 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:36.442 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:36.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:36.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:36.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:36 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:19:36.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:36 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:19:36.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:36 smithi132 ceph-mon[33500]: osdmap e424: 8 total, 7 up, 8 in 2024-03-26T05:19:36.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:36 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:19:36.544 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:36.581 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:36.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:36.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:36 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:19:36.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:36 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:19:36.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:36 smithi080 ceph-mon[29691]: osdmap e424: 8 total, 7 up, 8 in 2024-03-26T05:19:36.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:36 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:19:36.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:36 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:19:36.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:36 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:19:36.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:36 smithi080 ceph-mon[40404]: osdmap e424: 8 total, 7 up, 8 in 2024-03-26T05:19:36.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:36 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:19:36.682 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:36.720 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:36.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:36.821 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:36.859 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:36.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:36.961 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:36.968 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 45 16 75986 75970 105.492 75.0625 0.02584 0.00947468 2024-03-26T05:19:37.007 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:37.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:37.109 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:37.138 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:37.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:37.240 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:37.268 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:37.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:37.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:37.399 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:37.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:37.501 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:37.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:37 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:37.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:37 smithi132 ceph-mon[33500]: pgmap v895: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s; 18707/222618 objects degraded (8.403%); 5/222618 objects misplaced (0.002%) 2024-03-26T05:19:37.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:37 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 18707/222618 objects degraded (8.403%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:37.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:37 smithi132 ceph-mon[33500]: osdmap e425: 8 total, 7 up, 8 in 2024-03-26T05:19:37.544 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:37.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:37.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:37 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:37.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:37 smithi080 ceph-mon[29691]: pgmap v895: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s; 18707/222618 objects degraded (8.403%); 5/222618 objects misplaced (0.002%) 2024-03-26T05:19:37.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:37 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 18707/222618 objects degraded (8.403%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:37.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:37 smithi080 ceph-mon[29691]: osdmap e425: 8 total, 7 up, 8 in 2024-03-26T05:19:37.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:37 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:37.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:37 smithi080 ceph-mon[40404]: pgmap v895: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s; 18707/222618 objects degraded (8.403%); 5/222618 objects misplaced (0.002%) 2024-03-26T05:19:37.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:37 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 18707/222618 objects degraded (8.403%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:37.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:37 smithi080 ceph-mon[40404]: osdmap e425: 8 total, 7 up, 8 in 2024-03-26T05:19:37.646 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:37.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:37.676 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:37.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:37.737 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:19:37.777 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:37.837 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:37.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:37.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:37.939 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:37.970 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 46 16 77168 77152 104.803 73.875 0.0158356 0.00953669 2024-03-26T05:19:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:37.995 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:37.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:38.096 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:38.123 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:38.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:38.225 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:38.261 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:38.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:38.363 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:38.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:38 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:19:38.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:38 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:19:38.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:38 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:19:38.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:38 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:19:38.398 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:38.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:38.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:38.500 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:38.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:38 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:19:38.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:38 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:19:38.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:38.542 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:38.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:38.577 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T05:19:38.643 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:38.671 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:38.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:38.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:38.773 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:38.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:38.824 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:38.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:38.925 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:38.969 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 47 16 78345 78329 104.138 73.5625 0.0118426 0.00959757 2024-03-26T05:19:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:38.981 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:38.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:39.039 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:39.082 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:39.108 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:39.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:39.210 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:39.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:39.240 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:39.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:39.341 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:39.392 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:39.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:39.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:39.494 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:39.497 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:39 smithi080 ceph-mon[29691]: pgmap v897: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 18707/223710 objects degraded (8.362%); 5/223710 objects misplaced (0.002%) 2024-03-26T05:19:39.497 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:39 smithi080 ceph-mon[29691]: osdmap e426: 8 total, 7 up, 8 in 2024-03-26T05:19:39.498 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:39 smithi080 ceph-mon[40404]: pgmap v897: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 18707/223710 objects degraded (8.362%); 5/223710 objects misplaced (0.002%) 2024-03-26T05:19:39.498 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:39 smithi080 ceph-mon[40404]: osdmap e426: 8 total, 7 up, 8 in 2024-03-26T05:19:39.524 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:39.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:39.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:39 smithi132 ceph-mon[33500]: pgmap v897: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 18707/223710 objects degraded (8.362%); 5/223710 objects misplaced (0.002%) 2024-03-26T05:19:39.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:39 smithi132 ceph-mon[33500]: osdmap e426: 8 total, 7 up, 8 in 2024-03-26T05:19:39.625 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:39.671 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:39.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:39.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:39.773 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:39.801 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:39.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:39.902 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:39.947 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:39.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:39.969 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 48 16 79488 79472 103.457 71.4375 0.00926135 0.00966054 2024-03-26T05:19:40.049 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:40.077 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:40.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:40.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:40.180 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:40.224 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:40.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:40.287 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:40.325 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:40.372 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:40.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:40.474 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:40.512 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:40.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:40.614 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:40.646 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:40.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:40.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:40.748 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:40.791 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:40.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:40.795 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:40.894 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:40.945 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:40.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:40.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:40.971 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 49 16 80656 80640 102.833 73 0.0119977 0.00972002 2024-03-26T05:19:41.047 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:41.098 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:41.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:41.199 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:41.227 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:41.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:41.327 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:41 smithi080 ceph-mon[40404]: pgmap v899: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.36k op/s; 18707/230268 objects degraded (8.124%); 5/230268 objects misplaced (0.002%) 2024-03-26T05:19:41.327 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:41 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/418211869' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:19:41.327 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:19:41.328 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:41 smithi080 ceph-mon[29691]: pgmap v899: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.36k op/s; 18707/230268 objects degraded (8.124%); 5/230268 objects misplaced (0.002%) 2024-03-26T05:19:41.328 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:41 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/418211869' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:19:41.329 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:41.358 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:41.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:41.459 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:41.488 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:41.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:41.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:41.590 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:41.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:41.633 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:41.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:41.734 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:41.777 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:41.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:41.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:41 smithi132 ceph-mon[33500]: pgmap v899: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.36k op/s; 18707/230268 objects degraded (8.124%); 5/230268 objects misplaced (0.002%) 2024-03-26T05:19:41.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:41 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/418211869' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:19:41.878 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:41.920 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:41.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:41.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:41.971 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 50 16 81764 81748 102.161 69.25 0.00779835 0.00978267 2024-03-26T05:19:42.023 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:42.065 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:42.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:42.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:42.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:42.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:42.209 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:42.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:42.310 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:42.350 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:42.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:42.453 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:42.483 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:42.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:42.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:42.544 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:42 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:19:42.544 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:42 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/418211869' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:19:42.544 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:42 smithi080 ceph-mon[40404]: osdmap e427: 8 total, 7 up, 8 in 2024-03-26T05:19:42.544 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:42 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 18707/230268 objects degraded (8.124%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:42.544 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:42 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:19:42.545 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:42 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/418211869' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:19:42.545 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:42 smithi080 ceph-mon[29691]: osdmap e427: 8 total, 7 up, 8 in 2024-03-26T05:19:42.545 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:42 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 18707/230268 objects degraded (8.124%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:42.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:42.628 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:42.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:42.729 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:42.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:42.762 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:42.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:42.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:42 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:19:42.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:42 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/418211869' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:19:42.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:42 smithi132 ceph-mon[33500]: osdmap e427: 8 total, 7 up, 8 in 2024-03-26T05:19:42.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:42 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 18707/230268 objects degraded (8.124%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:42.863 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:42.894 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:42.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:42.971 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 51 16 82817 82801 101.449 65.8125 0.010958 0.00985234 2024-03-26T05:19:42.996 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:43.043 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:43.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:43.145 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:43.163 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:19:43.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:43.173 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:43.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:43.175 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:19:43.274 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:43.334 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:43.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:43.435 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:43.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:43.483 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:43.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:43.584 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:43.616 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:43.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:43.718 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:43.757 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:43.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:43.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:43.859 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:43.882 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[29691]: pgmap v901: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s; 20217/241389 objects degraded (8.375%); 5/241389 objects misplaced (0.002%) 2024-03-26T05:19:43.882 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[29691]: 11.5 scrub starts 2024-03-26T05:19:43.883 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[29691]: 11.5 scrub ok 2024-03-26T05:19:43.883 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[29691]: 3.1e scrub starts 2024-03-26T05:19:43.883 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[29691]: 3.1e scrub ok 2024-03-26T05:19:43.883 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[29691]: 6.18 scrub starts 2024-03-26T05:19:43.883 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[29691]: 6.18 scrub ok 2024-03-26T05:19:43.883 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[40404]: pgmap v901: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s; 20217/241389 objects degraded (8.375%); 5/241389 objects misplaced (0.002%) 2024-03-26T05:19:43.883 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[40404]: 11.5 scrub starts 2024-03-26T05:19:43.883 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[40404]: 11.5 scrub ok 2024-03-26T05:19:43.883 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[40404]: 3.1e scrub starts 2024-03-26T05:19:43.883 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[40404]: 3.1e scrub ok 2024-03-26T05:19:43.883 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[40404]: 6.18 scrub starts 2024-03-26T05:19:43.883 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:43 smithi080 ceph-mon[40404]: 6.18 scrub ok 2024-03-26T05:19:43.900 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:43.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:43.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:43.971 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 52 16 83835 83819 100.721 63.625 0.0114347 0.00992336 2024-03-26T05:19:44.003 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:44.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:43 smithi132 ceph-mon[33500]: pgmap v901: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s; 20217/241389 objects degraded (8.375%); 5/241389 objects misplaced (0.002%) 2024-03-26T05:19:44.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:43 smithi132 ceph-mon[33500]: 11.5 scrub starts 2024-03-26T05:19:44.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:43 smithi132 ceph-mon[33500]: 11.5 scrub ok 2024-03-26T05:19:44.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:43 smithi132 ceph-mon[33500]: 3.1e scrub starts 2024-03-26T05:19:44.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:43 smithi132 ceph-mon[33500]: 3.1e scrub ok 2024-03-26T05:19:44.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:43 smithi132 ceph-mon[33500]: 6.18 scrub starts 2024-03-26T05:19:44.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:43 smithi132 ceph-mon[33500]: 6.18 scrub ok 2024-03-26T05:19:44.043 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:44.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:44.146 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:44.173 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:19:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:19:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:19:44.178 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:44.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:44.280 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:44.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:44.324 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:44.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:44.426 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:44.476 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:44.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:44.577 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:44.616 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:44.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:44.718 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:44.759 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:44.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:44.839 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:44 smithi080 ceph-mon[29691]: from='client.15981 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:19:44.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:44 smithi080 ceph-mon[29691]: 5.4 scrub starts 2024-03-26T05:19:44.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:44 smithi080 ceph-mon[29691]: 5.4 scrub ok 2024-03-26T05:19:44.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:44 smithi080 ceph-mon[29691]: 1.7 scrub starts 2024-03-26T05:19:44.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:44 smithi080 ceph-mon[29691]: 1.7 scrub ok 2024-03-26T05:19:44.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:44 smithi080 ceph-mon[40404]: from='client.15981 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:19:44.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:44 smithi080 ceph-mon[40404]: 5.4 scrub starts 2024-03-26T05:19:44.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:44 smithi080 ceph-mon[40404]: 5.4 scrub ok 2024-03-26T05:19:44.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:44 smithi080 ceph-mon[40404]: 1.7 scrub starts 2024-03-26T05:19:44.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:44 smithi080 ceph-mon[40404]: 1.7 scrub ok 2024-03-26T05:19:44.862 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:44.913 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:44.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:44.971 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 53 16 84901 84885 100.078 66.625 0.0108086 0.00998716 2024-03-26T05:19:45.015 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:45.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:44 smithi132 ceph-mon[33500]: from='client.15981 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:19:45.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:44 smithi132 ceph-mon[33500]: 5.4 scrub starts 2024-03-26T05:19:45.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:44 smithi132 ceph-mon[33500]: 5.4 scrub ok 2024-03-26T05:19:45.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:44 smithi132 ceph-mon[33500]: 1.7 scrub starts 2024-03-26T05:19:45.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:44 smithi132 ceph-mon[33500]: 1.7 scrub ok 2024-03-26T05:19:45.044 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:45.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:45.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:45.146 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:45.169 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":901,"stamp":"2024-03-26T05:19:42.180807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5258788010,"num_objects":80463,"num_object_clones":0,"num_object_copies":241389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20217,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":80463,"num_whiteouts":0,"num_read":1778,"num_read_kb":1516,"num_write":81179,"num_write_kb":5135333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":324828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79031,"ondisk_log_size":79031,"up":411,"acting":412,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21729468,"kb_used_data":16801576,"kb_used_omap":15,"kb_used_meta":4927792,"kb_avail":728231748,"statfs":{"total":767960285184,"available":745709309952,"internally_reserved":0,"allocated":17204813824,"data_stored":17198083066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":5046059790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,9,10,6,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":410058752,"num_objects":6257,"num_object_clones":0,"num_object_copies":18771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1510,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6257,"num_write_kb":400448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003231"},"pg_stats":[{"pgid":"11.3","version":"426'4917","reported_seq":4963,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:39.919007+0000","last_change":"2024-03-26T05:18:36.688678+0000","last_active":"2024-03-26T05:19:39.919007+0000","last_peered":"2024-03-26T05:19:39.919007+0000","last_clean":"2024-03-26T05:19:39.919007+0000","last_became_active":"2024-03-26T05:18:36.688371+0000","last_became_peered":"2024-03-26T05:18:36.688371+0000","last_unstale":"2024-03-26T05:19:39.919007+0000","last_undegraded":"2024-03-26T05:19:39.919007+0000","last_fullsized":"2024-03-26T05:19:39.919007+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4917,"log_dups_size":0,"ondisk_log_size":4917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:00.496923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322240512,"num_objects":4917,"num_object_clones":0,"num_object_copies":14751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4917,"num_write_kb":314688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"254'20","reported_seq":619,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.304740+0000","last_change":"2024-03-26T05:15:48.638251+0000","last_active":"2024-03-26T05:19:38.304740+0000","last_peered":"2024-03-26T05:19:38.304740+0000","last_clean":"2024-03-26T05:10:30.608013+0000","last_became_active":"2024-03-26T05:15:48.638251+0000","last_became_peered":"2024-03-26T05:15:48.638251+0000","last_unstale":"2024-03-26T05:19:38.304740+0000","last_undegraded":"2024-03-26T05:15:48.630289+0000","last_fullsized":"2024-03-26T05:15:48.630171+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'20","last_scrub_stamp":"2024-03-26T05:14:56.989928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:56.989928+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":803,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.316072+0000","last_change":"2024-03-26T05:17:34.941862+0000","last_active":"2024-03-26T05:19:38.316072+0000","last_peered":"2024-03-26T05:19:38.316072+0000","last_clean":"2024-03-26T05:19:38.316072+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:19:38.316072+0000","last_undegraded":"2024-03-26T05:19:38.316072+0000","last_fullsized":"2024-03-26T05:19:38.316072+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:34.941337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:17:34.941337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"336'13","reported_seq":1062,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.317195+0000","last_change":"2024-03-26T05:18:56.306154+0000","last_active":"2024-03-26T05:19:38.317195+0000","last_peered":"2024-03-26T05:19:38.317195+0000","last_clean":"2024-03-26T05:19:38.317195+0000","last_became_active":"2024-03-26T05:15:32.632283+0000","last_became_peered":"2024-03-26T05:15:32.632283+0000","last_unstale":"2024-03-26T05:19:38.317195+0000","last_undegraded":"2024-03-26T05:19:38.317195+0000","last_fullsized":"2024-03-26T05:19:38.317195+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'13","last_scrub_stamp":"2024-03-26T05:18:56.306025+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:18:56.306025+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:01.528725+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":177,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.302799+0000","last_change":"2024-03-26T05:15:48.650651+0000","last_active":"2024-03-26T05:19:38.302799+0000","last_peered":"2024-03-26T05:19:38.302799+0000","last_clean":"2024-03-26T05:00:27.943471+0000","last_became_active":"2024-03-26T05:15:48.650651+0000","last_became_peered":"2024-03-26T05:15:48.650651+0000","last_unstale":"2024-03-26T05:19:38.302799+0000","last_undegraded":"2024-03-26T05:19:38.302799+0000","last_fullsized":"2024-03-26T05:15:48.641490+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:54.067506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:54.067506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"88'128","reported_seq":970,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.303259+0000","last_change":"2024-03-26T05:18:31.557638+0000","last_active":"2024-03-26T05:19:38.303259+0000","last_peered":"2024-03-26T05:19:38.303259+0000","last_clean":"2024-03-26T05:19:38.303259+0000","last_became_active":"2024-03-26T05:15:33.616505+0000","last_became_peered":"2024-03-26T05:15:33.616505+0000","last_unstale":"2024-03-26T05:19:38.303259+0000","last_undegraded":"2024-03-26T05:19:38.303259+0000","last_fullsized":"2024-03-26T05:19:38.303259+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:18:31.557501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:18:31.557501+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:54.937137+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"426'4922","reported_seq":4967,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:40.908340+0000","last_change":"2024-03-26T05:18:36.692831+0000","last_active":"2024-03-26T05:19:40.908340+0000","last_peered":"2024-03-26T05:19:40.908340+0000","last_clean":"2024-03-26T05:19:40.908340+0000","last_became_active":"2024-03-26T05:18:36.692550+0000","last_became_peered":"2024-03-26T05:18:36.692550+0000","last_unstale":"2024-03-26T05:19:40.908340+0000","last_undegraded":"2024-03-26T05:19:40.908340+0000","last_fullsized":"2024-03-26T05:19:40.908340+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4922,"log_dups_size":0,"ondisk_log_size":4922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:53.403034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322568192,"num_objects":4922,"num_object_clones":0,"num_object_copies":14766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4922,"num_write_kb":315008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"254'31","reported_seq":638,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.317388+0000","last_change":"2024-03-26T05:15:48.650607+0000","last_active":"2024-03-26T05:19:38.317388+0000","last_peered":"2024-03-26T05:19:38.317388+0000","last_clean":"2024-03-26T05:10:30.646940+0000","last_became_active":"2024-03-26T05:15:48.650607+0000","last_became_peered":"2024-03-26T05:15:48.650607+0000","last_unstale":"2024-03-26T05:19:38.317388+0000","last_undegraded":"2024-03-26T05:15:48.634034+0000","last_fullsized":"2024-03-26T05:15:48.633929+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'31","last_scrub_stamp":"2024-03-26T05:14:50.944513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:50.944513+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:39.002500+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":826,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.307655+0000","last_change":"2024-03-26T05:17:56.069026+0000","last_active":"2024-03-26T05:19:38.307655+0000","last_peered":"2024-03-26T05:19:38.307655+0000","last_clean":"2024-03-26T05:19:38.307655+0000","last_became_active":"2024-03-26T05:06:06.521150+0000","last_became_peered":"2024-03-26T05:06:06.521150+0000","last_unstale":"2024-03-26T05:19:38.307655+0000","last_undegraded":"2024-03-26T05:19:38.307655+0000","last_fullsized":"2024-03-26T05:19:38.307655+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:56.068870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:17:56.068870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.900506+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":784,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.309732+0000","last_change":"2024-03-26T05:15:48.649219+0000","last_active":"2024-03-26T05:19:38.309732+0000","last_peered":"2024-03-26T05:19:38.309732+0000","last_clean":"2024-03-26T05:15:46.609940+0000","last_became_active":"2024-03-26T05:15:48.649219+0000","last_became_peered":"2024-03-26T05:15:48.649219+0000","last_unstale":"2024-03-26T05:19:38.309732+0000","last_undegraded":"2024-03-26T05:19:38.309732+0000","last_fullsized":"2024-03-26T05:15:48.637386+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:21.059907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:14:21.059907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":780,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.317569+0000","last_change":"2024-03-26T05:15:48.640264+0000","last_active":"2024-03-26T05:19:38.317569+0000","last_peered":"2024-03-26T05:19:38.317569+0000","last_clean":"2024-03-26T05:15:46.627568+0000","last_became_active":"2024-03-26T05:15:48.640264+0000","last_became_peered":"2024-03-26T05:15:48.640264+0000","last_unstale":"2024-03-26T05:19:38.317569+0000","last_undegraded":"2024-03-26T05:19:38.317569+0000","last_fullsized":"2024-03-26T05:15:48.632563+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:19.269035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:19.269035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:39.002475+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"88'128","reported_seq":594,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.300155+0000","last_change":"2024-03-26T05:16:46.320882+0000","last_active":"2024-03-26T05:19:38.300155+0000","last_peered":"2024-03-26T05:19:38.300155+0000","last_clean":"2024-03-26T05:19:38.300155+0000","last_became_active":"2024-03-26T05:15:22.616949+0000","last_became_peered":"2024-03-26T05:15:22.616949+0000","last_unstale":"2024-03-26T05:19:38.300155+0000","last_undegraded":"2024-03-26T05:19:38.300155+0000","last_fullsized":"2024-03-26T05:19:38.300155+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:16:46.320738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:16:46.320738+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"426'5024","reported_seq":5070,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:39.878480+0000","last_change":"2024-03-26T05:18:36.689562+0000","last_active":"2024-03-26T05:19:39.878480+0000","last_peered":"2024-03-26T05:19:39.878480+0000","last_clean":"2024-03-26T05:19:39.878480+0000","last_became_active":"2024-03-26T05:18:36.689247+0000","last_became_peered":"2024-03-26T05:18:36.689247+0000","last_unstale":"2024-03-26T05:19:39.878480+0000","last_undegraded":"2024-03-26T05:19:39.878480+0000","last_fullsized":"2024-03-26T05:19:39.878480+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":5024,"log_dups_size":0,"ondisk_log_size":5024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:42.468275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329252864,"num_objects":5024,"num_object_clones":0,"num_object_copies":15072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5024,"num_write_kb":321536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"254'25","reported_seq":632,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.308442+0000","last_change":"2024-03-26T05:15:48.646310+0000","last_active":"2024-03-26T05:19:38.308442+0000","last_peered":"2024-03-26T05:19:38.308442+0000","last_clean":"2024-03-26T05:10:30.649037+0000","last_became_active":"2024-03-26T05:15:48.646310+0000","last_became_peered":"2024-03-26T05:15:48.646310+0000","last_unstale":"2024-03-26T05:19:38.308442+0000","last_undegraded":"2024-03-26T05:15:48.632255+0000","last_fullsized":"2024-03-26T05:15:48.632148+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:13:04.195003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:04.195003+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.900245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":6}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":769,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.315932+0000","last_change":"2024-03-26T05:15:48.641615+0000","last_active":"2024-03-26T05:19:38.315932+0000","last_peered":"2024-03-26T05:19:38.315932+0000","last_clean":"2024-03-26T05:12:19.732296+0000","last_became_active":"2024-03-26T05:15:48.641615+0000","last_became_peered":"2024-03-26T05:15:48.641615+0000","last_unstale":"2024-03-26T05:19:38.315932+0000","last_undegraded":"2024-03-26T05:19:38.315932+0000","last_fullsized":"2024-03-26T05:15:48.632821+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:29.465975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.465975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:39.002433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":466,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.298606+0000","last_change":"2024-03-26T05:15:48.643225+0000","last_active":"2024-03-26T05:19:38.298606+0000","last_peered":"2024-03-26T05:19:38.298606+0000","last_clean":"2024-03-26T05:15:46.630428+0000","last_became_active":"2024-03-26T05:15:48.643225+0000","last_became_peered":"2024-03-26T05:15:48.643225+0000","last_unstale":"2024-03-26T05:19:38.298606+0000","last_undegraded":"2024-03-26T05:19:38.298606+0000","last_fullsized":"2024-03-26T05:15:48.630436+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:56.632109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:13:07.883082+0000","last_clean_scrub_stamp":"2024-03-26T05:14:56.632109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":789,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.304535+0000","last_change":"2024-03-26T05:16:41.068268+0000","last_active":"2024-03-26T05:19:38.304535+0000","last_peered":"2024-03-26T05:19:38.304535+0000","last_clean":"2024-03-26T05:19:38.304535+0000","last_became_active":"2024-03-26T05:07:59.957995+0000","last_became_peered":"2024-03-26T05:07:59.957995+0000","last_unstale":"2024-03-26T05:19:38.304535+0000","last_undegraded":"2024-03-26T05:19:38.304535+0000","last_fullsized":"2024-03-26T05:19:38.304535+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:41.068156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:16:41.068156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"426'5148","reported_seq":5194,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.097899+0000","last_change":"2024-03-26T05:18:36.697223+0000","last_active":"2024-03-26T05:19:41.097899+0000","last_peered":"2024-03-26T05:19:41.097899+0000","last_clean":"2024-03-26T05:19:41.097899+0000","last_became_active":"2024-03-26T05:18:36.696863+0000","last_became_peered":"2024-03-26T05:18:36.696863+0000","last_unstale":"2024-03-26T05:19:41.097899+0000","last_undegraded":"2024-03-26T05:19:41.097899+0000","last_fullsized":"2024-03-26T05:19:41.097899+0000","mapping_epoch":408,"log_start":"415'400","ondisk_log_start":"415'400","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4748,"log_dups_size":0,"ondisk_log_size":4748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:54.695988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337379328,"num_objects":5148,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":329472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"292'34","reported_seq":846,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.308322+0000","last_change":"2024-03-26T05:15:48.644946+0000","last_active":"2024-03-26T05:19:38.308322+0000","last_peered":"2024-03-26T05:19:38.308322+0000","last_clean":"2024-03-26T05:15:46.609501+0000","last_became_active":"2024-03-26T05:15:48.644946+0000","last_became_peered":"2024-03-26T05:15:48.644946+0000","last_unstale":"2024-03-26T05:19:38.308322+0000","last_undegraded":"2024-03-26T05:15:48.636746+0000","last_fullsized":"2024-03-26T05:15:48.636639+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'34","last_scrub_stamp":"2024-03-26T05:15:43.090549+0000","last_deep_scrub":"292'34","last_deep_scrub_stamp":"2024-03-26T05:12:23.778630+0000","last_clean_scrub_stamp":"2024-03-26T05:15:43.090549+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503594+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":790,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.306589+0000","last_change":"2024-03-26T05:17:42.933401+0000","last_active":"2024-03-26T05:19:38.306589+0000","last_peered":"2024-03-26T05:19:38.306589+0000","last_clean":"2024-03-26T05:19:38.306589+0000","last_became_active":"2024-03-26T05:15:32.626265+0000","last_became_peered":"2024-03-26T05:15:32.626265+0000","last_unstale":"2024-03-26T05:19:38.306589+0000","last_undegraded":"2024-03-26T05:19:38.306589+0000","last_fullsized":"2024-03-26T05:19:38.306589+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:42.933293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:17:42.933293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"343'11","reported_seq":654,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.298007+0000","last_change":"2024-03-26T05:15:48.648004+0000","last_active":"2024-03-26T05:19:38.298007+0000","last_peered":"2024-03-26T05:19:38.298007+0000","last_clean":"2024-03-26T05:07:58.484584+0000","last_became_active":"2024-03-26T05:15:48.648004+0000","last_became_peered":"2024-03-26T05:15:48.648004+0000","last_unstale":"2024-03-26T05:19:38.298007+0000","last_undegraded":"2024-03-26T05:15:48.629532+0000","last_fullsized":"2024-03-26T05:15:48.629408+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:14:03.822423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:03.822423+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:43.376491+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":471,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.298769+0000","last_change":"2024-03-26T05:18:49.287721+0000","last_active":"2024-03-26T05:19:38.298769+0000","last_peered":"2024-03-26T05:19:38.298769+0000","last_clean":"2024-03-26T05:19:38.298769+0000","last_became_active":"2024-03-26T05:08:00.005676+0000","last_became_peered":"2024-03-26T05:08:00.005676+0000","last_unstale":"2024-03-26T05:19:38.298769+0000","last_undegraded":"2024-03-26T05:19:38.298769+0000","last_fullsized":"2024-03-26T05:19:38.298769+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:49.287596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_clean_scrub_stamp":"2024-03-26T05:18:49.287596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:01.056828+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"88'128","reported_seq":594,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.297606+0000","last_change":"2024-03-26T05:16:48.345452+0000","last_active":"2024-03-26T05:19:38.297606+0000","last_peered":"2024-03-26T05:19:38.297606+0000","last_clean":"2024-03-26T05:19:38.297606+0000","last_became_active":"2024-03-26T05:15:22.622542+0000","last_became_peered":"2024-03-26T05:15:22.622542+0000","last_unstale":"2024-03-26T05:19:38.297606+0000","last_undegraded":"2024-03-26T05:19:38.297606+0000","last_fullsized":"2024-03-26T05:19:38.297606+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:16:48.345307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:16:48.345307+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"426'5081","reported_seq":5127,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:40.869391+0000","last_change":"2024-03-26T05:18:36.691585+0000","last_active":"2024-03-26T05:19:40.869391+0000","last_peered":"2024-03-26T05:19:40.869391+0000","last_clean":"2024-03-26T05:19:40.869391+0000","last_became_active":"2024-03-26T05:18:36.691270+0000","last_became_peered":"2024-03-26T05:18:36.691270+0000","last_unstale":"2024-03-26T05:19:40.869391+0000","last_undegraded":"2024-03-26T05:19:40.869391+0000","last_fullsized":"2024-03-26T05:19:40.869391+0000","mapping_epoch":408,"log_start":"415'200","ondisk_log_start":"415'200","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:42.272292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332988416,"num_objects":5081,"num_object_clones":0,"num_object_copies":15243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5081,"num_write_kb":325184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":754,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.309745+0000","last_change":"2024-03-26T05:15:48.640538+0000","last_active":"2024-03-26T05:19:38.309745+0000","last_peered":"2024-03-26T05:19:38.309745+0000","last_clean":"2024-03-26T05:12:19.732211+0000","last_became_active":"2024-03-26T05:15:48.640538+0000","last_became_peered":"2024-03-26T05:15:48.640538+0000","last_unstale":"2024-03-26T05:19:38.309745+0000","last_undegraded":"2024-03-26T05:19:38.309745+0000","last_fullsized":"2024-03-26T05:15:48.628482+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.411139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.411139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"254'52","reported_seq":879,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.306734+0000","last_change":"2024-03-26T05:17:53.968129+0000","last_active":"2024-03-26T05:19:38.306734+0000","last_peered":"2024-03-26T05:19:38.306734+0000","last_clean":"2024-03-26T05:19:38.306734+0000","last_became_active":"2024-03-26T05:15:32.636466+0000","last_became_peered":"2024-03-26T05:15:32.636466+0000","last_unstale":"2024-03-26T05:19:38.306734+0000","last_undegraded":"2024-03-26T05:19:38.306734+0000","last_fullsized":"2024-03-26T05:19:38.306734+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'52","last_scrub_stamp":"2024-03-26T05:17:53.967981+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:17:53.967981+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":790,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.313671+0000","last_change":"2024-03-26T05:18:36.798166+0000","last_active":"2024-03-26T05:19:38.313671+0000","last_peered":"2024-03-26T05:19:38.313671+0000","last_clean":"2024-03-26T05:19:38.313671+0000","last_became_active":"2024-03-26T05:15:32.636983+0000","last_became_peered":"2024-03-26T05:15:32.636983+0000","last_unstale":"2024-03-26T05:19:38.313671+0000","last_undegraded":"2024-03-26T05:19:38.313671+0000","last_fullsized":"2024-03-26T05:19:38.313671+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.798083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.798083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:53.110193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":177,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.309769+0000","last_change":"2024-03-26T05:15:48.635004+0000","last_active":"2024-03-26T05:19:38.309769+0000","last_peered":"2024-03-26T05:19:38.309769+0000","last_clean":"2024-03-26T05:00:27.943471+0000","last_became_active":"2024-03-26T05:15:48.635004+0000","last_became_peered":"2024-03-26T05:15:48.635004+0000","last_unstale":"2024-03-26T05:19:38.309769+0000","last_undegraded":"2024-03-26T05:19:38.309769+0000","last_fullsized":"2024-03-26T05:15:48.630635+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:54.923744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:13:54.923744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"88'128","reported_seq":556,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.316901+0000","last_change":"2024-03-26T05:15:48.644026+0000","last_active":"2024-03-26T05:19:38.316901+0000","last_peered":"2024-03-26T05:19:38.316901+0000","last_clean":"2024-03-26T05:01:05.801809+0000","last_became_active":"2024-03-26T05:15:48.644026+0000","last_became_peered":"2024-03-26T05:15:48.644026+0000","last_unstale":"2024-03-26T05:19:38.316901+0000","last_undegraded":"2024-03-26T05:19:38.316901+0000","last_fullsized":"2024-03-26T05:15:48.633510+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"426'5035","reported_seq":5078,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:41.121882+0000","last_change":"2024-03-26T05:18:51.975262+0000","last_active":"2024-03-26T05:19:41.121882+0000","last_peered":"2024-03-26T05:19:41.121882+0000","last_clean":"2024-03-26T05:18:36.613885+0000","last_became_active":"2024-03-26T05:18:36.688394+0000","last_became_peered":"2024-03-26T05:18:36.688394+0000","last_unstale":"2024-03-26T05:19:41.121882+0000","last_undegraded":"2024-03-26T05:18:51.974639+0000","last_fullsized":"2024-03-26T05:18:36.685186+0000","mapping_epoch":408,"log_start":"415'300","ondisk_log_start":"415'300","created":408,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:01.955780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329973760,"num_objects":5035,"num_object_clones":0,"num_object_copies":15105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5035,"num_write_kb":322240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":5035}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"343'9","reported_seq":1010,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.306939+0000","last_change":"2024-03-26T05:15:48.648617+0000","last_active":"2024-03-26T05:19:38.306939+0000","last_peered":"2024-03-26T05:19:38.306939+0000","last_clean":"2024-03-26T05:15:46.625052+0000","last_became_active":"2024-03-26T05:15:48.648617+0000","last_became_peered":"2024-03-26T05:15:48.648617+0000","last_unstale":"2024-03-26T05:19:38.306939+0000","last_undegraded":"2024-03-26T05:15:48.628872+0000","last_fullsized":"2024-03-26T05:15:48.628757+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:13:56.721250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:13:56.721250+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"254'34","reported_seq":691,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.297333+0000","last_change":"2024-03-26T05:15:48.656188+0000","last_active":"2024-03-26T05:19:38.297333+0000","last_peered":"2024-03-26T05:19:38.297333+0000","last_clean":"2024-03-26T05:10:30.652936+0000","last_became_active":"2024-03-26T05:15:48.656188+0000","last_became_peered":"2024-03-26T05:15:48.656188+0000","last_unstale":"2024-03-26T05:19:38.297333+0000","last_undegraded":"2024-03-26T05:15:48.628982+0000","last_fullsized":"2024-03-26T05:15:48.628847+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'34","last_scrub_stamp":"2024-03-26T05:14:59.971454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:59.971454+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":804,"reported_epoch":427,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:41.338627+0000","last_change":"2024-03-26T05:15:48.646902+0000","last_active":"2024-03-26T05:19:41.338627+0000","last_peered":"2024-03-26T05:19:41.338627+0000","last_clean":"2024-03-26T05:15:46.622472+0000","last_became_active":"2024-03-26T05:15:48.646902+0000","last_became_peered":"2024-03-26T05:15:48.646902+0000","last_unstale":"2024-03-26T05:19:41.338627+0000","last_undegraded":"2024-03-26T05:15:48.640683+0000","last_fullsized":"2024-03-26T05:15:48.640556+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:12.113993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:12.113993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":697,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.303984+0000","last_change":"2024-03-26T05:18:01.721458+0000","last_active":"2024-03-26T05:19:38.303984+0000","last_peered":"2024-03-26T05:19:38.303984+0000","last_clean":"2024-03-26T05:19:38.303984+0000","last_became_active":"2024-03-26T05:15:32.630135+0000","last_became_peered":"2024-03-26T05:15:32.630135+0000","last_unstale":"2024-03-26T05:19:38.303984+0000","last_undegraded":"2024-03-26T05:19:38.303984+0000","last_fullsized":"2024-03-26T05:19:38.303984+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:01.721323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:01.721323+0000","last_clean_scrub_stamp":"2024-03-26T05:18:01.721323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:43.313887+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"396'129","reported_seq":609,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.304421+0000","last_change":"2024-03-26T05:17:59.038747+0000","last_active":"2024-03-26T05:19:38.304421+0000","last_peered":"2024-03-26T05:19:38.304421+0000","last_clean":"2024-03-26T05:19:38.304421+0000","last_became_active":"2024-03-26T05:15:33.617734+0000","last_became_peered":"2024-03-26T05:15:33.617734+0000","last_unstale":"2024-03-26T05:19:38.304421+0000","last_undegraded":"2024-03-26T05:19:38.304421+0000","last_fullsized":"2024-03-26T05:19:38.304421+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"396'129","last_scrub_stamp":"2024-03-26T05:17:59.038611+0000","last_deep_scrub":"396'129","last_deep_scrub_stamp":"2024-03-26T05:17:59.038611+0000","last_clean_scrub_stamp":"2024-03-26T05:17:59.038611+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"426'4981","reported_seq":5024,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:40.924247+0000","last_change":"2024-03-26T05:18:51.991654+0000","last_active":"2024-03-26T05:19:40.924247+0000","last_peered":"2024-03-26T05:19:40.924247+0000","last_clean":"2024-03-26T05:18:36.613885+0000","last_became_active":"2024-03-26T05:18:36.687244+0000","last_became_peered":"2024-03-26T05:18:36.687244+0000","last_unstale":"2024-03-26T05:19:40.924247+0000","last_undegraded":"2024-03-26T05:18:51.985208+0000","last_fullsized":"2024-03-26T05:18:36.684553+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:51.837284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326434816,"num_objects":4981,"num_object_clones":0,"num_object_copies":14943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4981,"num_write_kb":318784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":4981}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"216'11","reported_seq":1063,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.343782+0000","last_change":"2024-03-26T05:18:36.306700+0000","last_active":"2024-03-26T05:19:41.343782+0000","last_peered":"2024-03-26T05:19:41.343782+0000","last_clean":"2024-03-26T05:19:41.343782+0000","last_became_active":"2024-03-26T05:07:59.994682+0000","last_became_peered":"2024-03-26T05:07:59.994682+0000","last_unstale":"2024-03-26T05:19:41.343782+0000","last_undegraded":"2024-03-26T05:19:41.343782+0000","last_fullsized":"2024-03-26T05:19:41.343782+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:18:36.306612+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.306612+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:56.086275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"254'26","reported_seq":842,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.317484+0000","last_change":"2024-03-26T05:15:48.638710+0000","last_active":"2024-03-26T05:19:38.317484+0000","last_peered":"2024-03-26T05:19:38.317484+0000","last_clean":"2024-03-26T05:15:46.626893+0000","last_became_active":"2024-03-26T05:15:48.638710+0000","last_became_peered":"2024-03-26T05:15:48.638710+0000","last_unstale":"2024-03-26T05:19:38.317484+0000","last_undegraded":"2024-03-26T05:15:48.631213+0000","last_fullsized":"2024-03-26T05:15:48.631104+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:14:05.219767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:05.219767+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:39.002463+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":471,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.298943+0000","last_change":"2024-03-26T05:18:35.144324+0000","last_active":"2024-03-26T05:19:38.298943+0000","last_peered":"2024-03-26T05:19:38.298943+0000","last_clean":"2024-03-26T05:19:38.298943+0000","last_became_active":"2024-03-26T05:07:59.960768+0000","last_became_peered":"2024-03-26T05:07:59.960768+0000","last_unstale":"2024-03-26T05:19:38.298943+0000","last_undegraded":"2024-03-26T05:19:38.298943+0000","last_fullsized":"2024-03-26T05:19:38.298943+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:35.144207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:35.144207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:50.487657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":789,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.336460+0000","last_change":"2024-03-26T05:18:02.541892+0000","last_active":"2024-03-26T05:19:41.336460+0000","last_peered":"2024-03-26T05:19:41.336460+0000","last_clean":"2024-03-26T05:19:41.336460+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:19:41.336460+0000","last_undegraded":"2024-03-26T05:19:41.336460+0000","last_fullsized":"2024-03-26T05:19:41.336460+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:02.541760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:18:02.541760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"88'128","reported_seq":540,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.308333+0000","last_change":"2024-03-26T05:17:40.189131+0000","last_active":"2024-03-26T05:19:38.308333+0000","last_peered":"2024-03-26T05:19:38.308333+0000","last_clean":"2024-03-26T05:19:38.308333+0000","last_became_active":"2024-03-26T05:15:33.617538+0000","last_became_peered":"2024-03-26T05:15:33.617538+0000","last_unstale":"2024-03-26T05:19:38.308333+0000","last_undegraded":"2024-03-26T05:19:38.308333+0000","last_fullsized":"2024-03-26T05:19:38.308333+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:17:40.188989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:17:40.188989+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.900381+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"426'5167","reported_seq":5209,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:41.086805+0000","last_change":"2024-03-26T05:18:51.976678+0000","last_active":"2024-03-26T05:19:41.086805+0000","last_peered":"2024-03-26T05:19:41.086805+0000","last_clean":"2024-03-26T05:18:36.613885+0000","last_became_active":"2024-03-26T05:18:36.691138+0000","last_became_peered":"2024-03-26T05:18:36.691138+0000","last_unstale":"2024-03-26T05:19:41.086805+0000","last_undegraded":"2024-03-26T05:18:51.976157+0000","last_fullsized":"2024-03-26T05:18:36.684718+0000","mapping_epoch":408,"log_start":"415'400","ondisk_log_start":"415'400","created":408,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4767,"log_dups_size":0,"ondisk_log_size":4767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:49.554406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338624512,"num_objects":5167,"num_object_clones":0,"num_object_copies":15501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5167,"num_write_kb":330688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5167}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":825,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.307822+0000","last_change":"2024-03-26T05:18:02.129424+0000","last_active":"2024-03-26T05:19:38.307822+0000","last_peered":"2024-03-26T05:19:38.307822+0000","last_clean":"2024-03-26T05:19:38.307822+0000","last_became_active":"2024-03-26T05:07:59.991484+0000","last_became_peered":"2024-03-26T05:07:59.991484+0000","last_unstale":"2024-03-26T05:19:38.307822+0000","last_undegraded":"2024-03-26T05:19:38.307822+0000","last_fullsized":"2024-03-26T05:19:38.307822+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:02.129294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:18:02.129294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.900474+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"254'44","reported_seq":930,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.338067+0000","last_change":"2024-03-26T05:18:43.255390+0000","last_active":"2024-03-26T05:19:41.338067+0000","last_peered":"2024-03-26T05:19:41.338067+0000","last_clean":"2024-03-26T05:19:41.338067+0000","last_became_active":"2024-03-26T05:15:32.634000+0000","last_became_peered":"2024-03-26T05:15:32.634000+0000","last_unstale":"2024-03-26T05:19:41.338067+0000","last_undegraded":"2024-03-26T05:19:41.338067+0000","last_fullsized":"2024-03-26T05:19:41.338067+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'44","last_scrub_stamp":"2024-03-26T05:18:43.255284+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:18:43.255284+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:02.962391+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":818,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.309479+0000","last_change":"2024-03-26T05:17:44.674568+0000","last_active":"2024-03-26T05:19:38.309479+0000","last_peered":"2024-03-26T05:19:38.309479+0000","last_clean":"2024-03-26T05:19:38.309479+0000","last_became_active":"2024-03-26T05:08:00.003099+0000","last_became_peered":"2024-03-26T05:08:00.003099+0000","last_unstale":"2024-03-26T05:19:38.309479+0000","last_undegraded":"2024-03-26T05:19:38.309479+0000","last_fullsized":"2024-03-26T05:19:38.309479+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:44.674133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:11:29.514915+0000","last_clean_scrub_stamp":"2024-03-26T05:17:44.674133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503685+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":786,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.310384+0000","last_change":"2024-03-26T05:17:43.809862+0000","last_active":"2024-03-26T05:19:38.310384+0000","last_peered":"2024-03-26T05:19:38.310384+0000","last_clean":"2024-03-26T05:19:38.310384+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:19:38.310384+0000","last_undegraded":"2024-03-26T05:19:38.310384+0000","last_fullsized":"2024-03-26T05:19:38.310384+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:43.809723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:17:43.809723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503636+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"88'128","reported_seq":542,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.308389+0000","last_change":"2024-03-26T05:18:40.004354+0000","last_active":"2024-03-26T05:19:38.308389+0000","last_peered":"2024-03-26T05:19:38.308389+0000","last_clean":"2024-03-26T05:19:38.308389+0000","last_became_active":"2024-03-26T05:15:22.624234+0000","last_became_peered":"2024-03-26T05:15:22.624234+0000","last_unstale":"2024-03-26T05:19:38.308389+0000","last_undegraded":"2024-03-26T05:19:38.308389+0000","last_fullsized":"2024-03-26T05:19:38.308389+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:18:40.004244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:18:40.004244+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:55.713409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"426'4873","reported_seq":4919,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:39.992720+0000","last_change":"2024-03-26T05:18:36.692740+0000","last_active":"2024-03-26T05:19:39.992720+0000","last_peered":"2024-03-26T05:19:39.992720+0000","last_clean":"2024-03-26T05:19:39.992720+0000","last_became_active":"2024-03-26T05:18:36.689513+0000","last_became_peered":"2024-03-26T05:18:36.689513+0000","last_unstale":"2024-03-26T05:19:39.992720+0000","last_undegraded":"2024-03-26T05:19:39.992720+0000","last_fullsized":"2024-03-26T05:19:39.992720+0000","mapping_epoch":408,"log_start":"415'400","ondisk_log_start":"415'400","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4473,"log_dups_size":0,"ondisk_log_size":4473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:53.396154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319356928,"num_objects":4873,"num_object_clones":0,"num_object_copies":14619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4873,"num_write_kb":311872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":471,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.298760+0000","last_change":"2024-03-26T05:18:37.186066+0000","last_active":"2024-03-26T05:19:38.298760+0000","last_peered":"2024-03-26T05:19:38.298760+0000","last_clean":"2024-03-26T05:19:38.298760+0000","last_became_active":"2024-03-26T05:07:59.999990+0000","last_became_peered":"2024-03-26T05:07:59.999990+0000","last_unstale":"2024-03-26T05:19:38.298760+0000","last_undegraded":"2024-03-26T05:19:38.298760+0000","last_fullsized":"2024-03-26T05:19:38.298760+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:37.185975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:18:37.185975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"254'30","reported_seq":878,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.338525+0000","last_change":"2024-03-26T05:18:00.501152+0000","last_active":"2024-03-26T05:19:41.338525+0000","last_peered":"2024-03-26T05:19:41.338525+0000","last_clean":"2024-03-26T05:19:41.338525+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:19:41.338525+0000","last_undegraded":"2024-03-26T05:19:41.338525+0000","last_fullsized":"2024-03-26T05:19:41.338525+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'30","last_scrub_stamp":"2024-03-26T05:18:00.501027+0000","last_deep_scrub":"187'18","last_deep_scrub_stamp":"2024-03-26T05:06:49.027756+0000","last_clean_scrub_stamp":"2024-03-26T05:18:00.501027+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":1656,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:39.193337+0000","last_change":"2024-03-26T05:17:46.664147+0000","last_active":"2024-03-26T05:19:39.193337+0000","last_peered":"2024-03-26T05:19:39.193337+0000","last_clean":"2024-03-26T05:19:39.193337+0000","last_became_active":"2024-03-26T05:15:32.633061+0000","last_became_peered":"2024-03-26T05:15:32.633061+0000","last_unstale":"2024-03-26T05:19:39.193337+0000","last_undegraded":"2024-03-26T05:19:39.193337+0000","last_fullsized":"2024-03-26T05:19:39.193337+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:17:46.664002+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:17:46.664002+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:43.313856+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":313,"num_read_kb":308,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":510,"reported_epoch":427,"state":"active+undersized","last_fresh":"2024-03-26T05:19:41.337690+0000","last_change":"2024-03-26T05:15:48.659891+0000","last_active":"2024-03-26T05:19:41.337690+0000","last_peered":"2024-03-26T05:19:41.337690+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:15:48.659891+0000","last_became_peered":"2024-03-26T05:15:48.659891+0000","last_unstale":"2024-03-26T05:19:41.337690+0000","last_undegraded":"2024-03-26T05:19:41.337690+0000","last_fullsized":"2024-03-26T05:15:48.638273+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:16.863389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.863389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":806,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.307988+0000","last_change":"2024-03-26T05:18:02.774271+0000","last_active":"2024-03-26T05:19:38.307988+0000","last_peered":"2024-03-26T05:19:38.307988+0000","last_clean":"2024-03-26T05:19:38.307988+0000","last_became_active":"2024-03-26T05:08:00.001657+0000","last_became_peered":"2024-03-26T05:08:00.001657+0000","last_unstale":"2024-03-26T05:19:38.307988+0000","last_undegraded":"2024-03-26T05:19:38.307988+0000","last_fullsized":"2024-03-26T05:19:38.307988+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:02.773898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:18:02.773898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:41.564070+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"88'128","reported_seq":598,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.303388+0000","last_change":"2024-03-26T05:15:48.647616+0000","last_active":"2024-03-26T05:19:38.303388+0000","last_peered":"2024-03-26T05:19:38.303388+0000","last_clean":"2024-03-26T05:15:46.610234+0000","last_became_active":"2024-03-26T05:15:48.647616+0000","last_became_peered":"2024-03-26T05:15:48.647616+0000","last_unstale":"2024-03-26T05:19:38.303388+0000","last_undegraded":"2024-03-26T05:19:38.303388+0000","last_fullsized":"2024-03-26T05:15:48.634886+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:15:44.054471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:15:44.054471+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503607+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"426'5018","reported_seq":5064,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:39.995749+0000","last_change":"2024-03-26T05:18:36.695086+0000","last_active":"2024-03-26T05:19:39.995749+0000","last_peered":"2024-03-26T05:19:39.995749+0000","last_clean":"2024-03-26T05:19:39.995749+0000","last_became_active":"2024-03-26T05:18:36.693460+0000","last_became_peered":"2024-03-26T05:18:36.693460+0000","last_unstale":"2024-03-26T05:19:39.995749+0000","last_undegraded":"2024-03-26T05:19:39.995749+0000","last_fullsized":"2024-03-26T05:19:39.995749+0000","mapping_epoch":408,"log_start":"415'600","ondisk_log_start":"415'600","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4418,"log_dups_size":0,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:04.340777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328859648,"num_objects":5018,"num_object_clones":0,"num_object_copies":15054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5018,"num_write_kb":321152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":718,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.304303+0000","last_change":"2024-03-26T05:17:41.527426+0000","last_active":"2024-03-26T05:19:38.304303+0000","last_peered":"2024-03-26T05:19:38.304303+0000","last_clean":"2024-03-26T05:19:38.304303+0000","last_became_active":"2024-03-26T05:15:32.637320+0000","last_became_peered":"2024-03-26T05:15:32.637320+0000","last_unstale":"2024-03-26T05:19:38.304303+0000","last_undegraded":"2024-03-26T05:19:38.304303+0000","last_fullsized":"2024-03-26T05:19:38.304303+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:41.527348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:14.948209+0000","last_clean_scrub_stamp":"2024-03-26T05:17:41.527348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:43.313838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"343'39","reported_seq":1088,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.308339+0000","last_change":"2024-03-26T05:15:48.640970+0000","last_active":"2024-03-26T05:19:38.308339+0000","last_peered":"2024-03-26T05:19:38.308339+0000","last_clean":"2024-03-26T05:15:46.625517+0000","last_became_active":"2024-03-26T05:15:48.640970+0000","last_became_peered":"2024-03-26T05:15:48.640970+0000","last_unstale":"2024-03-26T05:19:38.308339+0000","last_undegraded":"2024-03-26T05:15:48.633766+0000","last_fullsized":"2024-03-26T05:15:48.633639+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'38","last_scrub_stamp":"2024-03-26T05:15:46.434771+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:15:46.434771+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.900343+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":810,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.304404+0000","last_change":"2024-03-26T05:18:46.748037+0000","last_active":"2024-03-26T05:19:38.304404+0000","last_peered":"2024-03-26T05:19:38.304404+0000","last_clean":"2024-03-26T05:19:38.304404+0000","last_became_active":"2024-03-26T05:07:59.960061+0000","last_became_peered":"2024-03-26T05:07:59.960061+0000","last_unstale":"2024-03-26T05:19:38.304404+0000","last_undegraded":"2024-03-26T05:19:38.304404+0000","last_fullsized":"2024-03-26T05:19:38.304404+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:46.747932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:46.747932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:57.090439+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":973,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.303583+0000","last_change":"2024-03-26T05:18:58.577783+0000","last_active":"2024-03-26T05:19:38.303583+0000","last_peered":"2024-03-26T05:19:38.303583+0000","last_clean":"2024-03-26T05:19:38.303583+0000","last_became_active":"2024-03-26T05:15:32.632929+0000","last_became_peered":"2024-03-26T05:15:32.632929+0000","last_unstale":"2024-03-26T05:19:38.303583+0000","last_undegraded":"2024-03-26T05:19:38.303583+0000","last_fullsized":"2024-03-26T05:19:38.303583+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:18:58.577676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:18:58.577676+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:18.067570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":474,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.298837+0000","last_change":"2024-03-26T05:15:48.643476+0000","last_active":"2024-03-26T05:19:38.298837+0000","last_peered":"2024-03-26T05:19:38.298837+0000","last_clean":"2024-03-26T05:15:46.630583+0000","last_became_active":"2024-03-26T05:15:48.643476+0000","last_became_peered":"2024-03-26T05:15:48.643476+0000","last_unstale":"2024-03-26T05:19:38.298837+0000","last_undegraded":"2024-03-26T05:19:38.298837+0000","last_fullsized":"2024-03-26T05:15:48.632538+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:14.528992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:14.528992+0000","last_clean_scrub_stamp":"2024-03-26T05:15:14.528992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"426'5023","reported_seq":5069,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:39.968582+0000","last_change":"2024-03-26T05:18:36.696055+0000","last_active":"2024-03-26T05:19:39.968582+0000","last_peered":"2024-03-26T05:19:39.968582+0000","last_clean":"2024-03-26T05:19:39.968582+0000","last_became_active":"2024-03-26T05:18:36.695726+0000","last_became_peered":"2024-03-26T05:18:36.695726+0000","last_unstale":"2024-03-26T05:19:39.968582+0000","last_undegraded":"2024-03-26T05:19:39.968582+0000","last_fullsized":"2024-03-26T05:19:39.968582+0000","mapping_epoch":408,"log_start":"415'600","ondisk_log_start":"415'600","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4423,"log_dups_size":0,"ondisk_log_size":4423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:53.895271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329187328,"num_objects":5023,"num_object_clones":0,"num_object_copies":15069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5023,"num_write_kb":321472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":810,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.316904+0000","last_change":"2024-03-26T05:17:51.080728+0000","last_active":"2024-03-26T05:19:38.316904+0000","last_peered":"2024-03-26T05:19:38.316904+0000","last_clean":"2024-03-26T05:19:38.316904+0000","last_became_active":"2024-03-26T05:15:32.625188+0000","last_became_peered":"2024-03-26T05:15:32.625188+0000","last_unstale":"2024-03-26T05:19:38.316904+0000","last_undegraded":"2024-03-26T05:19:38.316904+0000","last_fullsized":"2024-03-26T05:19:38.316904+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:51.080386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:17:51.080386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:39.002594+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"254'23","reported_seq":612,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.318671+0000","last_change":"2024-03-26T05:18:01.180911+0000","last_active":"2024-03-26T05:19:38.318671+0000","last_peered":"2024-03-26T05:19:38.318671+0000","last_clean":"2024-03-26T05:19:38.318671+0000","last_became_active":"2024-03-26T05:08:00.001398+0000","last_became_peered":"2024-03-26T05:08:00.001398+0000","last_unstale":"2024-03-26T05:19:38.318671+0000","last_undegraded":"2024-03-26T05:19:38.318671+0000","last_fullsized":"2024-03-26T05:19:38.318671+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'23","last_scrub_stamp":"2024-03-26T05:18:01.180771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:18:01.180771+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:39.002570+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":815,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.316215+0000","last_change":"2024-03-26T05:18:03.193191+0000","last_active":"2024-03-26T05:19:38.316215+0000","last_peered":"2024-03-26T05:19:38.316215+0000","last_clean":"2024-03-26T05:19:38.316215+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:19:38.316215+0000","last_undegraded":"2024-03-26T05:19:38.316215+0000","last_fullsized":"2024-03-26T05:19:38.316215+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:03.193105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:03.193105+0000","last_clean_scrub_stamp":"2024-03-26T05:18:03.193105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:39.002541+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":798,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.307391+0000","last_change":"2024-03-26T05:17:37.515235+0000","last_active":"2024-03-26T05:19:38.307391+0000","last_peered":"2024-03-26T05:19:38.307391+0000","last_clean":"2024-03-26T05:19:38.307391+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:19:38.307391+0000","last_undegraded":"2024-03-26T05:19:38.307391+0000","last_fullsized":"2024-03-26T05:19:38.307391+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:37.515108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:17:37.515108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503661+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":666,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.304359+0000","last_change":"2024-03-26T05:18:41.624630+0000","last_active":"2024-03-26T05:19:38.304359+0000","last_peered":"2024-03-26T05:19:38.304359+0000","last_clean":"2024-03-26T05:19:38.304359+0000","last_became_active":"2024-03-26T05:15:32.633633+0000","last_became_peered":"2024-03-26T05:15:32.633633+0000","last_unstale":"2024-03-26T05:19:38.304359+0000","last_undegraded":"2024-03-26T05:19:38.304359+0000","last_fullsized":"2024-03-26T05:19:38.304359+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:41.624502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:18:41.624502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:04.113557+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"88'128","reported_seq":957,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.303898+0000","last_change":"2024-03-26T05:15:48.652567+0000","last_active":"2024-03-26T05:19:38.303898+0000","last_peered":"2024-03-26T05:19:38.303898+0000","last_clean":"2024-03-26T05:15:46.623182+0000","last_became_active":"2024-03-26T05:15:48.652567+0000","last_became_peered":"2024-03-26T05:15:48.652567+0000","last_unstale":"2024-03-26T05:19:38.303898+0000","last_undegraded":"2024-03-26T05:15:48.634567+0000","last_fullsized":"2024-03-26T05:15:48.634420+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.293436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"426'4939","reported_seq":4985,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:40.007224+0000","last_change":"2024-03-26T05:18:36.696360+0000","last_active":"2024-03-26T05:19:40.007224+0000","last_peered":"2024-03-26T05:19:40.007224+0000","last_clean":"2024-03-26T05:19:40.007224+0000","last_became_active":"2024-03-26T05:18:36.695687+0000","last_became_peered":"2024-03-26T05:18:36.695687+0000","last_unstale":"2024-03-26T05:19:40.007224+0000","last_undegraded":"2024-03-26T05:19:40.007224+0000","last_fullsized":"2024-03-26T05:19:40.007224+0000","mapping_epoch":408,"log_start":"415'500","ondisk_log_start":"415'500","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:44.555288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323682304,"num_objects":4939,"num_object_clones":0,"num_object_copies":14817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4939,"num_write_kb":316096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":788,"reported_epoch":427,"state":"active+undersized","last_fresh":"2024-03-26T05:19:41.338703+0000","last_change":"2024-03-26T05:15:48.645632+0000","last_active":"2024-03-26T05:19:41.338703+0000","last_peered":"2024-03-26T05:19:41.338703+0000","last_clean":"2024-03-26T05:15:46.622114+0000","last_became_active":"2024-03-26T05:15:48.645632+0000","last_became_peered":"2024-03-26T05:15:48.645632+0000","last_unstale":"2024-03-26T05:19:41.338703+0000","last_undegraded":"2024-03-26T05:19:41.338703+0000","last_fullsized":"2024-03-26T05:15:48.638911+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:14.131427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:14.131427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"254'46","reported_seq":1115,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.309616+0000","last_change":"2024-03-26T05:18:43.564529+0000","last_active":"2024-03-26T05:19:38.309616+0000","last_peered":"2024-03-26T05:19:38.309616+0000","last_clean":"2024-03-26T05:19:38.309616+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:19:38.309616+0000","last_undegraded":"2024-03-26T05:19:38.309616+0000","last_fullsized":"2024-03-26T05:19:38.309616+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'46","last_scrub_stamp":"2024-03-26T05:18:43.564424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:18:43.564424+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:48.845780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":177,"reported_epoch":427,"state":"active+undersized","last_fresh":"2024-03-26T05:19:41.338063+0000","last_change":"2024-03-26T05:15:48.646887+0000","last_active":"2024-03-26T05:19:41.338063+0000","last_peered":"2024-03-26T05:19:41.338063+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:15:48.646887+0000","last_became_peered":"2024-03-26T05:15:48.646887+0000","last_unstale":"2024-03-26T05:19:41.338063+0000","last_undegraded":"2024-03-26T05:19:41.338063+0000","last_fullsized":"2024-03-26T05:15:48.641087+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:02.881800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:02.881800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":475,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.298403+0000","last_change":"2024-03-26T05:18:41.097551+0000","last_active":"2024-03-26T05:19:38.298403+0000","last_peered":"2024-03-26T05:19:38.298403+0000","last_clean":"2024-03-26T05:19:38.298403+0000","last_became_active":"2024-03-26T05:07:59.998511+0000","last_became_peered":"2024-03-26T05:07:59.998511+0000","last_unstale":"2024-03-26T05:19:38.298403+0000","last_undegraded":"2024-03-26T05:19:38.298403+0000","last_fullsized":"2024-03-26T05:19:38.298403+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:18:41.097424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:18:41.097424+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:53.839561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":765,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.305791+0000","last_change":"2024-03-26T05:15:48.637195+0000","last_active":"2024-03-26T05:19:38.305791+0000","last_peered":"2024-03-26T05:19:38.305791+0000","last_clean":"2024-03-26T05:15:46.628089+0000","last_became_active":"2024-03-26T05:15:48.637195+0000","last_became_peered":"2024-03-26T05:15:48.637195+0000","last_unstale":"2024-03-26T05:19:38.305791+0000","last_undegraded":"2024-03-26T05:19:38.305791+0000","last_fullsized":"2024-03-26T05:15:48.630113+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:31.518660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:31.518660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"88'128","reported_seq":594,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.297214+0000","last_change":"2024-03-26T05:16:47.319821+0000","last_active":"2024-03-26T05:19:38.297214+0000","last_peered":"2024-03-26T05:19:38.297214+0000","last_clean":"2024-03-26T05:19:38.297214+0000","last_became_active":"2024-03-26T05:15:22.621431+0000","last_became_peered":"2024-03-26T05:15:22.621431+0000","last_unstale":"2024-03-26T05:19:38.297214+0000","last_undegraded":"2024-03-26T05:19:38.297214+0000","last_fullsized":"2024-03-26T05:19:38.297214+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:16:47.319697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:16:47.319697+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"426'5051","reported_seq":5097,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:40.818393+0000","last_change":"2024-03-26T05:18:36.688177+0000","last_active":"2024-03-26T05:19:40.818393+0000","last_peered":"2024-03-26T05:19:40.818393+0000","last_clean":"2024-03-26T05:19:40.818393+0000","last_became_active":"2024-03-26T05:18:36.687838+0000","last_became_peered":"2024-03-26T05:18:36.687838+0000","last_unstale":"2024-03-26T05:19:40.818393+0000","last_undegraded":"2024-03-26T05:19:40.818393+0000","last_fullsized":"2024-03-26T05:19:40.818393+0000","mapping_epoch":408,"log_start":"415'200","ondisk_log_start":"415'200","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:58.702948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331022336,"num_objects":5051,"num_object_clones":0,"num_object_copies":15153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":323264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"254'21","reported_seq":853,"reported_epoch":427,"state":"active+clean+remapped","last_fresh":"2024-03-26T05:19:41.338695+0000","last_change":"2024-03-26T05:18:31.264945+0000","last_active":"2024-03-26T05:19:41.338695+0000","last_peered":"2024-03-26T05:19:41.338695+0000","last_clean":"2024-03-26T05:19:41.338695+0000","last_became_active":"2024-03-26T05:17:40.868563+0000","last_became_peered":"2024-03-26T05:17:40.868563+0000","last_unstale":"2024-03-26T05:19:41.338695+0000","last_undegraded":"2024-03-26T05:19:41.338695+0000","last_fullsized":"2024-03-26T05:19:41.338695+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:18:31.264666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:18:31.264666+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:55.656620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2,1],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":471,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.298200+0000","last_change":"2024-03-26T05:18:33.144296+0000","last_active":"2024-03-26T05:19:38.298200+0000","last_peered":"2024-03-26T05:19:38.298200+0000","last_clean":"2024-03-26T05:19:38.298200+0000","last_became_active":"2024-03-26T05:07:59.959978+0000","last_became_peered":"2024-03-26T05:07:59.959978+0000","last_unstale":"2024-03-26T05:19:38.298200+0000","last_undegraded":"2024-03-26T05:19:38.298200+0000","last_fullsized":"2024-03-26T05:19:38.298200+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:33.144190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:33.144190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:44.258280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":792,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.316037+0000","last_change":"2024-03-26T05:15:48.638309+0000","last_active":"2024-03-26T05:19:38.316037+0000","last_peered":"2024-03-26T05:19:38.316037+0000","last_clean":"2024-03-26T05:15:46.625678+0000","last_became_active":"2024-03-26T05:15:48.638309+0000","last_became_peered":"2024-03-26T05:15:48.638309+0000","last_unstale":"2024-03-26T05:19:38.316037+0000","last_undegraded":"2024-03-26T05:19:38.316037+0000","last_fullsized":"2024-03-26T05:15:48.631169+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:52.371799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:52.371799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:39.002487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":781,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.338178+0000","last_change":"2024-03-26T05:17:44.467349+0000","last_active":"2024-03-26T05:19:41.338178+0000","last_peered":"2024-03-26T05:19:41.338178+0000","last_clean":"2024-03-26T05:19:41.338178+0000","last_became_active":"2024-03-26T05:15:32.639943+0000","last_became_peered":"2024-03-26T05:15:32.639943+0000","last_unstale":"2024-03-26T05:19:41.338178+0000","last_undegraded":"2024-03-26T05:19:41.338178+0000","last_fullsized":"2024-03-26T05:19:41.338178+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:44.467219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:17:44.467219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"426'4982","reported_seq":5028,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:40.861739+0000","last_change":"2024-03-26T05:18:36.684064+0000","last_active":"2024-03-26T05:19:40.861739+0000","last_peered":"2024-03-26T05:19:40.861739+0000","last_clean":"2024-03-26T05:19:40.861739+0000","last_became_active":"2024-03-26T05:18:36.683707+0000","last_became_peered":"2024-03-26T05:18:36.683707+0000","last_unstale":"2024-03-26T05:19:40.861739+0000","last_undegraded":"2024-03-26T05:19:40.861739+0000","last_fullsized":"2024-03-26T05:19:40.861739+0000","mapping_epoch":408,"log_start":"415'100","ondisk_log_start":"415'100","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326500352,"num_objects":4982,"num_object_clones":0,"num_object_copies":14946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4982,"num_write_kb":318848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"343'51","reported_seq":886,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.307750+0000","last_change":"2024-03-26T05:18:50.837517+0000","last_active":"2024-03-26T05:19:38.307750+0000","last_peered":"2024-03-26T05:19:38.307750+0000","last_clean":"2024-03-26T05:19:38.307750+0000","last_became_active":"2024-03-26T05:15:32.626640+0000","last_became_peered":"2024-03-26T05:15:32.626640+0000","last_unstale":"2024-03-26T05:19:38.307750+0000","last_undegraded":"2024-03-26T05:19:38.307750+0000","last_fullsized":"2024-03-26T05:19:38.307750+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'51","last_scrub_stamp":"2024-03-26T05:18:50.837412+0000","last_deep_scrub":"343'51","last_deep_scrub_stamp":"2024-03-26T05:17:45.197613+0000","last_clean_scrub_stamp":"2024-03-26T05:18:50.837412+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:16.369953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":686,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.304718+0000","last_change":"2024-03-26T05:18:46.539574+0000","last_active":"2024-03-26T05:19:38.304718+0000","last_peered":"2024-03-26T05:19:38.304718+0000","last_clean":"2024-03-26T05:19:38.304718+0000","last_became_active":"2024-03-26T05:15:32.640861+0000","last_became_peered":"2024-03-26T05:15:32.640861+0000","last_unstale":"2024-03-26T05:19:38.304718+0000","last_undegraded":"2024-03-26T05:19:38.304718+0000","last_fullsized":"2024-03-26T05:19:38.304718+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:46.539478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:46.539478+0000","last_clean_scrub_stamp":"2024-03-26T05:18:46.539478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:12.213897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":809,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.316961+0000","last_change":"2024-03-26T05:17:53.117292+0000","last_active":"2024-03-26T05:19:38.316961+0000","last_peered":"2024-03-26T05:19:38.316961+0000","last_clean":"2024-03-26T05:19:38.316961+0000","last_became_active":"2024-03-26T05:08:00.000009+0000","last_became_peered":"2024-03-26T05:08:00.000009+0000","last_unstale":"2024-03-26T05:19:38.316961+0000","last_undegraded":"2024-03-26T05:19:38.316961+0000","last_fullsized":"2024-03-26T05:19:38.316961+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:53.117165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:17:53.117165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:39.002619+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":689,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.303190+0000","last_change":"2024-03-26T05:17:54.623803+0000","last_active":"2024-03-26T05:19:38.303190+0000","last_peered":"2024-03-26T05:19:38.303190+0000","last_clean":"2024-03-26T05:19:38.303190+0000","last_became_active":"2024-03-26T05:15:32.633481+0000","last_became_peered":"2024-03-26T05:15:32.633481+0000","last_unstale":"2024-03-26T05:19:38.303190+0000","last_undegraded":"2024-03-26T05:19:38.303190+0000","last_fullsized":"2024-03-26T05:19:38.303190+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:54.623723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_clean_scrub_stamp":"2024-03-26T05:17:54.623723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"427'5127","reported_seq":5175,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.722089+0000","last_change":"2024-03-26T05:18:36.695182+0000","last_active":"2024-03-26T05:19:41.722089+0000","last_peered":"2024-03-26T05:19:41.722089+0000","last_clean":"2024-03-26T05:19:41.722089+0000","last_became_active":"2024-03-26T05:18:36.693789+0000","last_became_peered":"2024-03-26T05:18:36.693789+0000","last_unstale":"2024-03-26T05:19:41.722089+0000","last_undegraded":"2024-03-26T05:19:41.722089+0000","last_fullsized":"2024-03-26T05:19:41.722089+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":5127,"log_dups_size":0,"ondisk_log_size":5127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:56.057517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336003072,"num_objects":5127,"num_object_clones":0,"num_object_copies":15381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5127,"num_write_kb":328128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":797,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.315111+0000","last_change":"2024-03-26T05:18:52.271198+0000","last_active":"2024-03-26T05:19:38.315111+0000","last_peered":"2024-03-26T05:19:38.315111+0000","last_clean":"2024-03-26T05:19:38.315111+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:19:38.315111+0000","last_undegraded":"2024-03-26T05:19:38.315111+0000","last_fullsized":"2024-03-26T05:19:38.315111+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:52.271117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:52.271117+0000","last_clean_scrub_stamp":"2024-03-26T05:18:52.271117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:12.705032+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"254'37","reported_seq":652,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.310796+0000","last_change":"2024-03-26T05:15:48.645453+0000","last_active":"2024-03-26T05:19:38.310796+0000","last_peered":"2024-03-26T05:19:38.310796+0000","last_clean":"2024-03-26T05:10:30.473392+0000","last_became_active":"2024-03-26T05:15:48.645453+0000","last_became_peered":"2024-03-26T05:15:48.645453+0000","last_unstale":"2024-03-26T05:19:38.310796+0000","last_undegraded":"2024-03-26T05:15:48.634195+0000","last_fullsized":"2024-03-26T05:15:48.634068+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-26T05:14:55.963773+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_clean_scrub_stamp":"2024-03-26T05:14:55.963773+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503529+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":703,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.304517+0000","last_change":"2024-03-26T05:15:48.650267+0000","last_active":"2024-03-26T05:19:38.304517+0000","last_peered":"2024-03-26T05:19:38.304517+0000","last_clean":"2024-03-26T05:15:46.624664+0000","last_became_active":"2024-03-26T05:15:48.650267+0000","last_became_peered":"2024-03-26T05:15:48.650267+0000","last_unstale":"2024-03-26T05:19:38.304517+0000","last_undegraded":"2024-03-26T05:19:38.304517+0000","last_fullsized":"2024-03-26T05:15:48.633889+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:20.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:20.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":778,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.316343+0000","last_change":"2024-03-26T05:15:48.646428+0000","last_active":"2024-03-26T05:19:38.316343+0000","last_peered":"2024-03-26T05:19:38.316343+0000","last_clean":"2024-03-26T05:15:46.626191+0000","last_became_active":"2024-03-26T05:15:48.646428+0000","last_became_peered":"2024-03-26T05:15:48.646428+0000","last_unstale":"2024-03-26T05:19:38.316343+0000","last_undegraded":"2024-03-26T05:19:38.316343+0000","last_fullsized":"2024-03-26T05:15:48.631162+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:51.030982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:51.030982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"426'4944","reported_seq":4987,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:40.407768+0000","last_change":"2024-03-26T05:18:51.979584+0000","last_active":"2024-03-26T05:19:40.407768+0000","last_peered":"2024-03-26T05:19:40.407768+0000","last_clean":"2024-03-26T05:18:36.613885+0000","last_became_active":"2024-03-26T05:18:36.691882+0000","last_became_peered":"2024-03-26T05:18:36.691882+0000","last_unstale":"2024-03-26T05:19:40.407768+0000","last_undegraded":"2024-03-26T05:18:51.973872+0000","last_fullsized":"2024-03-26T05:18:36.679509+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4944,"log_dups_size":0,"ondisk_log_size":4944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:50.183741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324009984,"num_objects":4944,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4944,"num_write_kb":316416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":4944}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"343'11","reported_seq":888,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.303494+0000","last_change":"2024-03-26T05:15:48.645351+0000","last_active":"2024-03-26T05:19:38.303494+0000","last_peered":"2024-03-26T05:19:38.303494+0000","last_clean":"2024-03-26T05:15:46.624511+0000","last_became_active":"2024-03-26T05:15:48.645351+0000","last_became_peered":"2024-03-26T05:15:48.645351+0000","last_unstale":"2024-03-26T05:19:38.303494+0000","last_undegraded":"2024-03-26T05:15:48.635536+0000","last_fullsized":"2024-03-26T05:15:48.635413+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:12:03.961391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:03.961391+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:43.313796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"254'48","reported_seq":519,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.299863+0000","last_change":"2024-03-26T05:17:53.297423+0000","last_active":"2024-03-26T05:19:38.299863+0000","last_peered":"2024-03-26T05:19:38.299863+0000","last_clean":"2024-03-26T05:19:38.299863+0000","last_became_active":"2024-03-26T05:15:32.620512+0000","last_became_peered":"2024-03-26T05:15:32.620512+0000","last_unstale":"2024-03-26T05:19:38.299863+0000","last_undegraded":"2024-03-26T05:19:38.299863+0000","last_fullsized":"2024-03-26T05:19:38.299863+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'48","last_scrub_stamp":"2024-03-26T05:17:53.297292+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:17:53.297292+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":692,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.304615+0000","last_change":"2024-03-26T05:15:48.651187+0000","last_active":"2024-03-26T05:19:38.304615+0000","last_peered":"2024-03-26T05:19:38.304615+0000","last_clean":"2024-03-26T05:15:46.625050+0000","last_became_active":"2024-03-26T05:15:48.651187+0000","last_became_peered":"2024-03-26T05:15:48.651187+0000","last_unstale":"2024-03-26T05:19:38.304615+0000","last_undegraded":"2024-03-26T05:15:48.633222+0000","last_fullsized":"2024-03-26T05:15:48.633108+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.959120+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":475,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.297978+0000","last_change":"2024-03-26T05:18:44.137144+0000","last_active":"2024-03-26T05:19:38.297978+0000","last_peered":"2024-03-26T05:19:38.297978+0000","last_clean":"2024-03-26T05:19:38.297978+0000","last_became_active":"2024-03-26T05:08:00.000166+0000","last_became_peered":"2024-03-26T05:08:00.000166+0000","last_unstale":"2024-03-26T05:19:38.297978+0000","last_undegraded":"2024-03-26T05:19:38.297978+0000","last_fullsized":"2024-03-26T05:19:38.297978+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:18:44.137037+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:18:44.137037+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:48.571387+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"254'18","reported_seq":815,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.314024+0000","last_change":"2024-03-26T05:18:04.990205+0000","last_active":"2024-03-26T05:19:38.314024+0000","last_peered":"2024-03-26T05:19:38.314024+0000","last_clean":"2024-03-26T05:19:38.314024+0000","last_became_active":"2024-03-26T05:15:32.640871+0000","last_became_peered":"2024-03-26T05:15:32.640871+0000","last_unstale":"2024-03-26T05:19:38.314024+0000","last_undegraded":"2024-03-26T05:19:38.314024+0000","last_fullsized":"2024-03-26T05:19:38.314024+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'18","last_scrub_stamp":"2024-03-26T05:18:04.990075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:18:04.990075+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":839,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.307475+0000","last_change":"2024-03-26T05:17:50.148050+0000","last_active":"2024-03-26T05:19:38.307475+0000","last_peered":"2024-03-26T05:19:38.307475+0000","last_clean":"2024-03-26T05:19:38.307475+0000","last_became_active":"2024-03-26T05:07:59.992217+0000","last_became_peered":"2024-03-26T05:07:59.992217+0000","last_unstale":"2024-03-26T05:19:38.307475+0000","last_undegraded":"2024-03-26T05:19:38.307475+0000","last_fullsized":"2024-03-26T05:19:38.307475+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:17:50.147911+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:17:50.147911+0000","last_clean_scrub_stamp":"2024-03-26T05:17:50.147911+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.900428+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":470,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.298198+0000","last_change":"2024-03-26T05:18:36.146429+0000","last_active":"2024-03-26T05:19:38.298198+0000","last_peered":"2024-03-26T05:19:38.298198+0000","last_clean":"2024-03-26T05:19:38.298198+0000","last_became_active":"2024-03-26T05:07:59.997434+0000","last_became_peered":"2024-03-26T05:07:59.997434+0000","last_unstale":"2024-03-26T05:19:38.298198+0000","last_undegraded":"2024-03-26T05:19:38.298198+0000","last_fullsized":"2024-03-26T05:19:38.298198+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.146286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.146286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:53.473390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":685,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.304408+0000","last_change":"2024-03-26T05:15:48.643985+0000","last_active":"2024-03-26T05:19:38.304408+0000","last_peered":"2024-03-26T05:19:38.304408+0000","last_clean":"2024-03-26T05:15:46.624057+0000","last_became_active":"2024-03-26T05:15:48.643985+0000","last_became_peered":"2024-03-26T05:15:48.643985+0000","last_unstale":"2024-03-26T05:19:38.304408+0000","last_undegraded":"2024-03-26T05:19:38.304408+0000","last_fullsized":"2024-03-26T05:15:48.635169+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:07.006139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.006139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"254'27","reported_seq":846,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.341307+0000","last_change":"2024-03-26T05:17:53.504402+0000","last_active":"2024-03-26T05:19:41.341307+0000","last_peered":"2024-03-26T05:19:41.341307+0000","last_clean":"2024-03-26T05:19:41.341307+0000","last_became_active":"2024-03-26T05:15:32.634434+0000","last_became_peered":"2024-03-26T05:15:32.634434+0000","last_unstale":"2024-03-26T05:19:41.341307+0000","last_undegraded":"2024-03-26T05:19:41.341307+0000","last_fullsized":"2024-03-26T05:19:41.341307+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:17:53.504269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:17:53.504269+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":659,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.303499+0000","last_change":"2024-03-26T05:15:48.646040+0000","last_active":"2024-03-26T05:19:38.303499+0000","last_peered":"2024-03-26T05:19:38.303499+0000","last_clean":"2024-03-26T05:15:46.624763+0000","last_became_active":"2024-03-26T05:15:48.646040+0000","last_became_peered":"2024-03-26T05:15:48.646040+0000","last_unstale":"2024-03-26T05:19:38.303499+0000","last_undegraded":"2024-03-26T05:19:38.303499+0000","last_fullsized":"2024-03-26T05:15:48.635818+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.939576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.939576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":781,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.313870+0000","last_change":"2024-03-26T05:18:03.064748+0000","last_active":"2024-03-26T05:19:38.313870+0000","last_peered":"2024-03-26T05:19:38.313870+0000","last_clean":"2024-03-26T05:19:38.313870+0000","last_became_active":"2024-03-26T05:15:32.641031+0000","last_became_peered":"2024-03-26T05:15:32.641031+0000","last_unstale":"2024-03-26T05:19:38.313870+0000","last_undegraded":"2024-03-26T05:19:38.313870+0000","last_fullsized":"2024-03-26T05:19:38.313870+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:03.064638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:18:03.064638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":789,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.317643+0000","last_change":"2024-03-26T05:15:48.639802+0000","last_active":"2024-03-26T05:19:38.317643+0000","last_peered":"2024-03-26T05:19:38.317643+0000","last_clean":"2024-03-26T05:15:46.626518+0000","last_became_active":"2024-03-26T05:15:48.639802+0000","last_became_peered":"2024-03-26T05:15:48.639802+0000","last_unstale":"2024-03-26T05:19:38.317643+0000","last_undegraded":"2024-03-26T05:19:38.317643+0000","last_fullsized":"2024-03-26T05:15:48.633216+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:13.142714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:13.142714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:39.002449+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"254'26","reported_seq":492,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.298850+0000","last_change":"2024-03-26T05:16:51.291839+0000","last_active":"2024-03-26T05:19:38.298850+0000","last_peered":"2024-03-26T05:19:38.298850+0000","last_clean":"2024-03-26T05:19:38.298850+0000","last_became_active":"2024-03-26T05:15:32.621049+0000","last_became_peered":"2024-03-26T05:15:32.621049+0000","last_unstale":"2024-03-26T05:19:38.298850+0000","last_undegraded":"2024-03-26T05:19:38.298850+0000","last_fullsized":"2024-03-26T05:19:38.298850+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:16:51.291743+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:16:51.291743+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":809,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.337537+0000","last_change":"2024-03-26T05:18:51.276677+0000","last_active":"2024-03-26T05:19:41.337537+0000","last_peered":"2024-03-26T05:19:41.337537+0000","last_clean":"2024-03-26T05:19:41.337537+0000","last_became_active":"2024-03-26T05:15:32.632387+0000","last_became_peered":"2024-03-26T05:15:32.632387+0000","last_unstale":"2024-03-26T05:19:41.337537+0000","last_undegraded":"2024-03-26T05:19:41.337537+0000","last_fullsized":"2024-03-26T05:19:41.337537+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:51.276565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:45.352678+0000","last_clean_scrub_stamp":"2024-03-26T05:18:51.276565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:18.013893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"336'11","reported_seq":1028,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.316684+0000","last_change":"2024-03-26T05:17:37.976715+0000","last_active":"2024-03-26T05:19:38.316684+0000","last_peered":"2024-03-26T05:19:38.316684+0000","last_clean":"2024-03-26T05:19:38.316684+0000","last_became_active":"2024-03-26T05:15:32.636392+0000","last_became_peered":"2024-03-26T05:15:32.636392+0000","last_unstale":"2024-03-26T05:19:38.316684+0000","last_undegraded":"2024-03-26T05:19:38.316684+0000","last_fullsized":"2024-03-26T05:19:38.316684+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-03-26T05:17:37.976316+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:17:37.976316+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":497,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.318468+0000","last_change":"2024-03-26T05:15:48.641096+0000","last_active":"2024-03-26T05:19:38.318468+0000","last_peered":"2024-03-26T05:19:38.318468+0000","last_clean":"2024-03-26T05:07:55.690747+0000","last_became_active":"2024-03-26T05:15:48.641096+0000","last_became_peered":"2024-03-26T05:15:48.641096+0000","last_unstale":"2024-03-26T05:19:38.318468+0000","last_undegraded":"2024-03-26T05:19:38.318468+0000","last_fullsized":"2024-03-26T05:15:48.634411+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:06.096138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:06.096138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:39.002512+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"345'39","reported_seq":855,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.304617+0000","last_change":"2024-03-26T05:17:41.001273+0000","last_active":"2024-03-26T05:19:38.304617+0000","last_peered":"2024-03-26T05:19:38.304617+0000","last_clean":"2024-03-26T05:19:38.304617+0000","last_became_active":"2024-03-26T05:15:32.626448+0000","last_became_peered":"2024-03-26T05:15:32.626448+0000","last_unstale":"2024-03-26T05:19:38.304617+0000","last_undegraded":"2024-03-26T05:19:38.304617+0000","last_fullsized":"2024-03-26T05:19:38.304617+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'39","last_scrub_stamp":"2024-03-26T05:17:41.001143+0000","last_deep_scrub":"345'39","last_deep_scrub_stamp":"2024-03-26T05:15:56.674423+0000","last_clean_scrub_stamp":"2024-03-26T05:17:41.001143+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":693,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.304043+0000","last_change":"2024-03-26T05:15:48.641873+0000","last_active":"2024-03-26T05:19:38.304043+0000","last_peered":"2024-03-26T05:19:38.304043+0000","last_clean":"2024-03-26T05:15:46.623430+0000","last_became_active":"2024-03-26T05:15:48.641873+0000","last_became_peered":"2024-03-26T05:15:48.641873+0000","last_unstale":"2024-03-26T05:19:38.304043+0000","last_undegraded":"2024-03-26T05:19:38.304043+0000","last_fullsized":"2024-03-26T05:15:48.632198+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:01.962048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:01.962048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"216'10","reported_seq":614,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.299058+0000","last_change":"2024-03-26T05:18:40.082860+0000","last_active":"2024-03-26T05:19:38.299058+0000","last_peered":"2024-03-26T05:19:38.299058+0000","last_clean":"2024-03-26T05:19:38.299058+0000","last_became_active":"2024-03-26T05:08:00.005648+0000","last_became_peered":"2024-03-26T05:08:00.005648+0000","last_unstale":"2024-03-26T05:19:38.299058+0000","last_undegraded":"2024-03-26T05:19:38.299058+0000","last_fullsized":"2024-03-26T05:19:38.299058+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:18:40.082750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:18:40.082750+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:57.009265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":788,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.338431+0000","last_change":"2024-03-26T05:17:59.508871+0000","last_active":"2024-03-26T05:19:41.338431+0000","last_peered":"2024-03-26T05:19:41.338431+0000","last_clean":"2024-03-26T05:19:41.338431+0000","last_became_active":"2024-03-26T05:15:32.631444+0000","last_became_peered":"2024-03-26T05:15:32.631444+0000","last_unstale":"2024-03-26T05:19:41.338431+0000","last_undegraded":"2024-03-26T05:19:41.338431+0000","last_fullsized":"2024-03-26T05:19:41.338431+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:59.508745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:59.508745+0000","last_clean_scrub_stamp":"2024-03-26T05:17:59.508745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"254'24","reported_seq":625,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.307844+0000","last_change":"2024-03-26T05:15:48.645289+0000","last_active":"2024-03-26T05:19:38.307844+0000","last_peered":"2024-03-26T05:19:38.307844+0000","last_clean":"2024-03-26T05:10:30.628152+0000","last_became_active":"2024-03-26T05:15:48.645289+0000","last_became_peered":"2024-03-26T05:15:48.645289+0000","last_unstale":"2024-03-26T05:19:38.307844+0000","last_undegraded":"2024-03-26T05:15:48.632524+0000","last_fullsized":"2024-03-26T05:15:48.632421+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'24","last_scrub_stamp":"2024-03-26T05:13:56.875610+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:13:56.875610+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.900259+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":761,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.304346+0000","last_change":"2024-03-26T05:15:48.642442+0000","last_active":"2024-03-26T05:19:38.304346+0000","last_peered":"2024-03-26T05:19:38.304346+0000","last_clean":"2024-03-26T05:12:19.733579+0000","last_became_active":"2024-03-26T05:15:48.642442+0000","last_became_peered":"2024-03-26T05:15:48.642442+0000","last_unstale":"2024-03-26T05:19:38.304346+0000","last_undegraded":"2024-03-26T05:19:38.304346+0000","last_fullsized":"2024-03-26T05:15:48.631873+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:21.515294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:21.515294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":817,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.303061+0000","last_change":"2024-03-26T05:18:55.496931+0000","last_active":"2024-03-26T05:19:38.303061+0000","last_peered":"2024-03-26T05:19:38.303061+0000","last_clean":"2024-03-26T05:19:38.303061+0000","last_became_active":"2024-03-26T05:08:00.001601+0000","last_became_peered":"2024-03-26T05:08:00.001601+0000","last_unstale":"2024-03-26T05:19:38.303061+0000","last_undegraded":"2024-03-26T05:19:38.303061+0000","last_fullsized":"2024-03-26T05:19:38.303061+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:55.496800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:55.496800+0000","last_clean_scrub_stamp":"2024-03-26T05:18:55.496800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:03.107317+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":789,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.309520+0000","last_change":"2024-03-26T05:18:38.817607+0000","last_active":"2024-03-26T05:19:38.309520+0000","last_peered":"2024-03-26T05:19:38.309520+0000","last_clean":"2024-03-26T05:19:38.309520+0000","last_became_active":"2024-03-26T05:07:59.999703+0000","last_became_peered":"2024-03-26T05:07:59.999703+0000","last_unstale":"2024-03-26T05:19:38.309520+0000","last_undegraded":"2024-03-26T05:19:38.309520+0000","last_fullsized":"2024-03-26T05:19:38.309520+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:38.817507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:18:38.817507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:04.867025+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"254'15","reported_seq":826,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.303148+0000","last_change":"2024-03-26T05:18:46.589571+0000","last_active":"2024-03-26T05:19:38.303148+0000","last_peered":"2024-03-26T05:19:38.303148+0000","last_clean":"2024-03-26T05:19:38.303148+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:19:38.303148+0000","last_undegraded":"2024-03-26T05:19:38.303148+0000","last_fullsized":"2024-03-26T05:19:38.303148+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'15","last_scrub_stamp":"2024-03-26T05:18:46.589484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:18:46.589484+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:54.542824+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":665,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.304413+0000","last_change":"2024-03-26T05:15:48.639023+0000","last_active":"2024-03-26T05:19:38.304413+0000","last_peered":"2024-03-26T05:19:38.304413+0000","last_clean":"2024-03-26T05:15:46.624142+0000","last_became_active":"2024-03-26T05:15:48.639023+0000","last_became_peered":"2024-03-26T05:15:48.639023+0000","last_unstale":"2024-03-26T05:19:38.304413+0000","last_undegraded":"2024-03-26T05:19:38.304413+0000","last_fullsized":"2024-03-26T05:15:48.631660+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:33.016072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:11:33.016072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":781,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.316163+0000","last_change":"2024-03-26T05:17:31.960294+0000","last_active":"2024-03-26T05:19:38.316163+0000","last_peered":"2024-03-26T05:19:38.316163+0000","last_clean":"2024-03-26T05:19:38.316163+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:19:38.316163+0000","last_undegraded":"2024-03-26T05:19:38.316163+0000","last_fullsized":"2024-03-26T05:19:38.316163+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:31.959647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:17:31.959647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":808,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.336321+0000","last_change":"2024-03-26T05:18:38.324660+0000","last_active":"2024-03-26T05:19:41.336321+0000","last_peered":"2024-03-26T05:19:41.336321+0000","last_clean":"2024-03-26T05:19:41.336321+0000","last_became_active":"2024-03-26T05:07:59.994210+0000","last_became_peered":"2024-03-26T05:07:59.994210+0000","last_unstale":"2024-03-26T05:19:41.336321+0000","last_undegraded":"2024-03-26T05:19:41.336321+0000","last_fullsized":"2024-03-26T05:19:41.336321+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:38.324541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:18:38.324541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:03.252964+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"254'25","reported_seq":829,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.309938+0000","last_change":"2024-03-26T05:17:38.978325+0000","last_active":"2024-03-26T05:19:38.309938+0000","last_peered":"2024-03-26T05:19:38.309938+0000","last_clean":"2024-03-26T05:19:38.309938+0000","last_became_active":"2024-03-26T05:15:32.636359+0000","last_became_peered":"2024-03-26T05:15:32.636359+0000","last_unstale":"2024-03-26T05:19:38.309938+0000","last_undegraded":"2024-03-26T05:19:38.309938+0000","last_fullsized":"2024-03-26T05:19:38.309938+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:17:38.978187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:17:38.978187+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":471,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.298276+0000","last_change":"2024-03-26T05:18:57.311724+0000","last_active":"2024-03-26T05:19:38.298276+0000","last_peered":"2024-03-26T05:19:38.298276+0000","last_clean":"2024-03-26T05:19:38.298276+0000","last_became_active":"2024-03-26T05:15:32.620559+0000","last_became_peered":"2024-03-26T05:15:32.620559+0000","last_unstale":"2024-03-26T05:19:38.298276+0000","last_undegraded":"2024-03-26T05:19:38.298276+0000","last_fullsized":"2024-03-26T05:19:38.298276+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:57.311648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:18:57.311648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:20.115823+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":702,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.303560+0000","last_change":"2024-03-26T05:15:48.640423+0000","last_active":"2024-03-26T05:19:38.303560+0000","last_peered":"2024-03-26T05:19:38.303560+0000","last_clean":"2024-03-26T05:15:46.625977+0000","last_became_active":"2024-03-26T05:15:48.640423+0000","last_became_peered":"2024-03-26T05:15:48.640423+0000","last_unstale":"2024-03-26T05:19:38.303560+0000","last_undegraded":"2024-03-26T05:19:38.303560+0000","last_fullsized":"2024-03-26T05:15:48.633132+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:23.957542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:11:23.957542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":785,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.316523+0000","last_change":"2024-03-26T05:18:53.017319+0000","last_active":"2024-03-26T05:19:38.316523+0000","last_peered":"2024-03-26T05:19:38.316523+0000","last_clean":"2024-03-26T05:19:38.316523+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:19:38.316523+0000","last_undegraded":"2024-03-26T05:19:38.316523+0000","last_fullsized":"2024-03-26T05:19:38.316523+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:53.017168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:53.017168+0000","last_clean_scrub_stamp":"2024-03-26T05:18:53.017168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:22.402074+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"254'10","reported_seq":807,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.306502+0000","last_change":"2024-03-26T05:18:56.877975+0000","last_active":"2024-03-26T05:19:38.306502+0000","last_peered":"2024-03-26T05:19:38.306502+0000","last_clean":"2024-03-26T05:19:38.306502+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:19:38.306502+0000","last_undegraded":"2024-03-26T05:19:38.306502+0000","last_fullsized":"2024-03-26T05:19:38.306502+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'10","last_scrub_stamp":"2024-03-26T05:18:56.877852+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:18:56.877852+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:10.911919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":819,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.337942+0000","last_change":"2024-03-26T05:18:40.334085+0000","last_active":"2024-03-26T05:19:41.337942+0000","last_peered":"2024-03-26T05:19:41.337942+0000","last_clean":"2024-03-26T05:19:41.337942+0000","last_became_active":"2024-03-26T05:07:59.993996+0000","last_became_peered":"2024-03-26T05:07:59.993996+0000","last_unstale":"2024-03-26T05:19:41.337942+0000","last_undegraded":"2024-03-26T05:19:41.337942+0000","last_fullsized":"2024-03-26T05:19:41.337942+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:40.333981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:11.070341+0000","last_clean_scrub_stamp":"2024-03-26T05:18:40.333981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:08.880203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":784,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.307778+0000","last_change":"2024-03-26T05:15:48.643994+0000","last_active":"2024-03-26T05:19:38.307778+0000","last_peered":"2024-03-26T05:19:38.307778+0000","last_clean":"2024-03-26T05:15:46.616068+0000","last_became_active":"2024-03-26T05:15:48.643994+0000","last_became_peered":"2024-03-26T05:15:48.643994+0000","last_unstale":"2024-03-26T05:19:38.307778+0000","last_undegraded":"2024-03-26T05:19:38.307778+0000","last_fullsized":"2024-03-26T05:15:48.631480+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.693592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.693592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.900226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":686,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.303698+0000","last_change":"2024-03-26T05:15:48.641318+0000","last_active":"2024-03-26T05:19:38.303698+0000","last_peered":"2024-03-26T05:19:38.303698+0000","last_clean":"2024-03-26T05:15:46.623140+0000","last_became_active":"2024-03-26T05:15:48.641318+0000","last_became_peered":"2024-03-26T05:15:48.641318+0000","last_unstale":"2024-03-26T05:19:38.303698+0000","last_undegraded":"2024-03-26T05:19:38.303698+0000","last_fullsized":"2024-03-26T05:15:48.634268+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:10.020572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:10.020572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"413'31","reported_seq":926,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.308384+0000","last_change":"2024-03-26T05:18:46.881168+0000","last_active":"2024-03-26T05:19:38.308384+0000","last_peered":"2024-03-26T05:19:38.308384+0000","last_clean":"2024-03-26T05:19:38.308384+0000","last_became_active":"2024-03-26T05:07:59.962007+0000","last_became_peered":"2024-03-26T05:07:59.962007+0000","last_unstale":"2024-03-26T05:19:38.308384+0000","last_undegraded":"2024-03-26T05:19:38.308384+0000","last_fullsized":"2024-03-26T05:19:38.308384+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'31","last_scrub_stamp":"2024-03-26T05:18:46.881054+0000","last_deep_scrub":"413'31","last_deep_scrub_stamp":"2024-03-26T05:18:46.881054+0000","last_clean_scrub_stamp":"2024-03-26T05:18:46.881054+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:56.894344+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":796,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.306405+0000","last_change":"2024-03-26T05:18:47.776529+0000","last_active":"2024-03-26T05:19:38.306405+0000","last_peered":"2024-03-26T05:19:38.306405+0000","last_clean":"2024-03-26T05:19:38.306405+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:19:38.306405+0000","last_undegraded":"2024-03-26T05:19:38.306405+0000","last_fullsized":"2024-03-26T05:19:38.306405+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:47.776444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:47.776444+0000","last_clean_scrub_stamp":"2024-03-26T05:18:47.776444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:04.967599+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":786,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.342291+0000","last_change":"2024-03-26T05:17:41.380529+0000","last_active":"2024-03-26T05:19:41.342291+0000","last_peered":"2024-03-26T05:19:41.342291+0000","last_clean":"2024-03-26T05:19:41.342291+0000","last_became_active":"2024-03-26T05:15:32.634716+0000","last_became_peered":"2024-03-26T05:15:32.634716+0000","last_unstale":"2024-03-26T05:19:41.342291+0000","last_undegraded":"2024-03-26T05:19:41.342291+0000","last_fullsized":"2024-03-26T05:19:41.342291+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:41.380448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:17:41.380448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":791,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.303443+0000","last_change":"2024-03-26T05:15:48.644977+0000","last_active":"2024-03-26T05:19:38.303443+0000","last_peered":"2024-03-26T05:19:38.303443+0000","last_clean":"2024-03-26T05:15:46.610540+0000","last_became_active":"2024-03-26T05:15:48.644977+0000","last_became_peered":"2024-03-26T05:15:48.644977+0000","last_unstale":"2024-03-26T05:19:38.303443+0000","last_undegraded":"2024-03-26T05:19:38.303443+0000","last_fullsized":"2024-03-26T05:15:48.636267+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:00.151355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.151355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503555+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'32","reported_seq":854,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.342224+0000","last_change":"2024-03-26T05:18:57.256490+0000","last_active":"2024-03-26T05:19:41.342224+0000","last_peered":"2024-03-26T05:19:41.342224+0000","last_clean":"2024-03-26T05:19:41.342224+0000","last_became_active":"2024-03-26T05:15:32.634216+0000","last_became_peered":"2024-03-26T05:15:32.634216+0000","last_unstale":"2024-03-26T05:19:41.342224+0000","last_undegraded":"2024-03-26T05:19:41.342224+0000","last_fullsized":"2024-03-26T05:19:41.342224+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'32","last_scrub_stamp":"2024-03-26T05:18:57.255727+0000","last_deep_scrub":"290'32","last_deep_scrub_stamp":"2024-03-26T05:12:15.970313+0000","last_clean_scrub_stamp":"2024-03-26T05:18:57.255727+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:12.795177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":792,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.310590+0000","last_change":"2024-03-26T05:15:48.647499+0000","last_active":"2024-03-26T05:19:38.310590+0000","last_peered":"2024-03-26T05:19:38.310590+0000","last_clean":"2024-03-26T05:15:46.609101+0000","last_became_active":"2024-03-26T05:15:48.647499+0000","last_became_peered":"2024-03-26T05:15:48.647499+0000","last_unstale":"2024-03-26T05:19:38.310590+0000","last_undegraded":"2024-03-26T05:15:48.634583+0000","last_fullsized":"2024-03-26T05:15:48.634474+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:49.948818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:49.948818+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"343'11","reported_seq":1047,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.308453+0000","last_change":"2024-03-26T05:15:48.644163+0000","last_active":"2024-03-26T05:19:38.308453+0000","last_peered":"2024-03-26T05:19:38.308453+0000","last_clean":"2024-03-26T05:15:46.619140+0000","last_became_active":"2024-03-26T05:15:48.644163+0000","last_became_peered":"2024-03-26T05:15:48.644163+0000","last_unstale":"2024-03-26T05:19:38.308453+0000","last_undegraded":"2024-03-26T05:15:48.636521+0000","last_fullsized":"2024-03-26T05:15:48.636379+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'10","last_scrub_stamp":"2024-03-26T05:15:38.516660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:15:38.516660+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.900323+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":713,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.304060+0000","last_change":"2024-03-26T05:17:50.645743+0000","last_active":"2024-03-26T05:19:38.304060+0000","last_peered":"2024-03-26T05:19:38.304060+0000","last_clean":"2024-03-26T05:19:38.304060+0000","last_became_active":"2024-03-26T05:15:32.636624+0000","last_became_peered":"2024-03-26T05:15:32.636624+0000","last_unstale":"2024-03-26T05:19:38.304060+0000","last_undegraded":"2024-03-26T05:19:38.304060+0000","last_fullsized":"2024-03-26T05:19:38.304060+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-26T05:17:50.645496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:17:50.645496+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"322'32","reported_seq":831,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.304803+0000","last_change":"2024-03-26T05:15:48.643576+0000","last_active":"2024-03-26T05:19:38.304803+0000","last_peered":"2024-03-26T05:19:38.304803+0000","last_clean":"2024-03-26T05:15:46.628715+0000","last_became_active":"2024-03-26T05:15:48.643576+0000","last_became_peered":"2024-03-26T05:15:48.643576+0000","last_unstale":"2024-03-26T05:19:38.304803+0000","last_undegraded":"2024-03-26T05:15:48.629218+0000","last_fullsized":"2024-03-26T05:15:48.629123+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'32","last_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_deep_scrub":"322'32","last_deep_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_clean_scrub_stamp":"2024-03-26T05:15:10.715786+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":176,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.302799+0000","last_change":"2024-03-26T05:15:48.650757+0000","last_active":"2024-03-26T05:19:38.302799+0000","last_peered":"2024-03-26T05:19:38.302799+0000","last_clean":"2024-03-26T05:00:21.748689+0000","last_became_active":"2024-03-26T05:15:48.650757+0000","last_became_peered":"2024-03-26T05:15:48.650757+0000","last_unstale":"2024-03-26T05:19:38.302799+0000","last_undegraded":"2024-03-26T05:19:38.302799+0000","last_fullsized":"2024-03-26T05:15:48.636772+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:03.047441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:15:03.047441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":176,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.304713+0000","last_change":"2024-03-26T05:15:48.638667+0000","last_active":"2024-03-26T05:19:38.304713+0000","last_peered":"2024-03-26T05:19:38.304713+0000","last_clean":"2024-03-26T05:00:25.933152+0000","last_became_active":"2024-03-26T05:15:48.638667+0000","last_became_peered":"2024-03-26T05:15:48.638667+0000","last_unstale":"2024-03-26T05:19:38.304713+0000","last_undegraded":"2024-03-26T05:19:38.304713+0000","last_fullsized":"2024-03-26T05:15:48.630074+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:04.830989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:04.830989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":798,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.313946+0000","last_change":"2024-03-26T05:17:57.254262+0000","last_active":"2024-03-26T05:19:38.313946+0000","last_peered":"2024-03-26T05:19:38.313946+0000","last_clean":"2024-03-26T05:19:38.313946+0000","last_became_active":"2024-03-26T05:07:59.959705+0000","last_became_peered":"2024-03-26T05:07:59.959705+0000","last_unstale":"2024-03-26T05:19:38.313946+0000","last_undegraded":"2024-03-26T05:19:38.313946+0000","last_fullsized":"2024-03-26T05:19:38.313946+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:57.254006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:57.254006+0000","last_clean_scrub_stamp":"2024-03-26T05:17:57.254006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"254'27","reported_seq":843,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.317390+0000","last_change":"2024-03-26T05:17:54.067855+0000","last_active":"2024-03-26T05:19:38.317390+0000","last_peered":"2024-03-26T05:19:38.317390+0000","last_clean":"2024-03-26T05:19:38.317390+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:19:38.317390+0000","last_undegraded":"2024-03-26T05:19:38.317390+0000","last_fullsized":"2024-03-26T05:19:38.317390+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:17:54.067701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:17:54.067701+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:39.002647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":795,"reported_epoch":427,"state":"active+undersized","last_fresh":"2024-03-26T05:19:41.338369+0000","last_change":"2024-03-26T05:15:48.652795+0000","last_active":"2024-03-26T05:19:41.338369+0000","last_peered":"2024-03-26T05:19:41.338369+0000","last_clean":"2024-03-26T05:15:46.621710+0000","last_became_active":"2024-03-26T05:15:48.652795+0000","last_became_peered":"2024-03-26T05:15:48.652795+0000","last_unstale":"2024-03-26T05:19:41.338369+0000","last_undegraded":"2024-03-26T05:19:41.338369+0000","last_fullsized":"2024-03-26T05:15:48.642501+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:04.980730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:12:04.980730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":783,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.303235+0000","last_change":"2024-03-26T05:15:48.648471+0000","last_active":"2024-03-26T05:19:38.303235+0000","last_peered":"2024-03-26T05:19:38.303235+0000","last_clean":"2024-03-26T05:15:46.610002+0000","last_became_active":"2024-03-26T05:15:48.648471+0000","last_became_peered":"2024-03-26T05:15:48.648471+0000","last_unstale":"2024-03-26T05:19:38.303235+0000","last_undegraded":"2024-03-26T05:19:38.303235+0000","last_fullsized":"2024-03-26T05:15:48.637209+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:11.912203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:15:11.912203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503580+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":505,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.308355+0000","last_change":"2024-03-26T05:15:48.646231+0000","last_active":"2024-03-26T05:19:38.308355+0000","last_peered":"2024-03-26T05:19:38.308355+0000","last_clean":"2024-03-26T05:07:55.689953+0000","last_became_active":"2024-03-26T05:15:48.646231+0000","last_became_peered":"2024-03-26T05:15:48.646231+0000","last_unstale":"2024-03-26T05:19:38.308355+0000","last_undegraded":"2024-03-26T05:19:38.308355+0000","last_fullsized":"2024-03-26T05:15:48.631854+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:05.787712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:14:05.787712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.900272+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"321'16","reported_seq":762,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.316014+0000","last_change":"2024-03-26T05:15:48.646822+0000","last_active":"2024-03-26T05:19:38.316014+0000","last_peered":"2024-03-26T05:19:38.316014+0000","last_clean":"2024-03-26T05:15:00.994414+0000","last_became_active":"2024-03-26T05:15:48.646822+0000","last_became_peered":"2024-03-26T05:15:48.646822+0000","last_unstale":"2024-03-26T05:19:38.316014+0000","last_undegraded":"2024-03-26T05:15:48.630770+0000","last_fullsized":"2024-03-26T05:15:48.630637+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'16","last_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_deep_scrub":"321'16","last_deep_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.995735+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":470,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.298297+0000","last_change":"2024-03-26T05:18:50.218189+0000","last_active":"2024-03-26T05:19:38.298297+0000","last_peered":"2024-03-26T05:19:38.298297+0000","last_clean":"2024-03-26T05:19:38.298297+0000","last_became_active":"2024-03-26T05:08:00.011262+0000","last_became_peered":"2024-03-26T05:08:00.011262+0000","last_unstale":"2024-03-26T05:19:38.298297+0000","last_undegraded":"2024-03-26T05:19:38.298297+0000","last_fullsized":"2024-03-26T05:19:38.298297+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:50.218114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:00.635308+0000","last_clean_scrub_stamp":"2024-03-26T05:18:50.218114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:11.874689+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":701,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.303991+0000","last_change":"2024-03-26T05:15:48.652929+0000","last_active":"2024-03-26T05:19:38.303991+0000","last_peered":"2024-03-26T05:19:38.303991+0000","last_clean":"2024-03-26T05:15:46.623242+0000","last_became_active":"2024-03-26T05:15:48.652929+0000","last_became_peered":"2024-03-26T05:15:48.652929+0000","last_unstale":"2024-03-26T05:19:38.303991+0000","last_undegraded":"2024-03-26T05:19:38.303991+0000","last_fullsized":"2024-03-26T05:15:48.634670+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_clean_scrub_stamp":"2024-03-26T05:12:18.959687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":470,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.297358+0000","last_change":"2024-03-26T05:18:38.242285+0000","last_active":"2024-03-26T05:19:38.297358+0000","last_peered":"2024-03-26T05:19:38.297358+0000","last_clean":"2024-03-26T05:19:38.297358+0000","last_became_active":"2024-03-26T05:07:59.998777+0000","last_became_peered":"2024-03-26T05:07:59.998777+0000","last_unstale":"2024-03-26T05:19:38.297358+0000","last_undegraded":"2024-03-26T05:19:38.297358+0000","last_fullsized":"2024-03-26T05:19:38.297358+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:38.242171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:18:38.242171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:49.190395+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"254'27","reported_seq":638,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.309459+0000","last_change":"2024-03-26T05:15:48.647529+0000","last_active":"2024-03-26T05:19:38.309459+0000","last_peered":"2024-03-26T05:19:38.309459+0000","last_clean":"2024-03-26T05:10:30.566889+0000","last_became_active":"2024-03-26T05:15:48.647529+0000","last_became_peered":"2024-03-26T05:15:48.647529+0000","last_unstale":"2024-03-26T05:19:38.309459+0000","last_undegraded":"2024-03-26T05:15:48.629088+0000","last_fullsized":"2024-03-26T05:15:48.628959+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:13:22.026354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:22.026354+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":465,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.299208+0000","last_change":"2024-03-26T05:15:48.636186+0000","last_active":"2024-03-26T05:19:38.299208+0000","last_peered":"2024-03-26T05:19:38.299208+0000","last_clean":"2024-03-26T05:15:46.631730+0000","last_became_active":"2024-03-26T05:15:48.636186+0000","last_became_peered":"2024-03-26T05:15:48.636186+0000","last_unstale":"2024-03-26T05:19:38.299208+0000","last_undegraded":"2024-03-26T05:19:38.299208+0000","last_fullsized":"2024-03-26T05:15:48.629342+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:17.792745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:17.792745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":471,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.297716+0000","last_change":"2024-03-26T05:18:05.269739+0000","last_active":"2024-03-26T05:19:38.297716+0000","last_peered":"2024-03-26T05:19:38.297716+0000","last_clean":"2024-03-26T05:19:38.297716+0000","last_became_active":"2024-03-26T05:15:32.622003+0000","last_became_peered":"2024-03-26T05:15:32.622003+0000","last_unstale":"2024-03-26T05:19:38.297716+0000","last_undegraded":"2024-03-26T05:19:38.297716+0000","last_fullsized":"2024-03-26T05:19:38.297716+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:05.269620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:18:05.269620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":788,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.336824+0000","last_change":"2024-03-26T05:17:52.510073+0000","last_active":"2024-03-26T05:19:41.336824+0000","last_peered":"2024-03-26T05:19:41.336824+0000","last_clean":"2024-03-26T05:19:41.336824+0000","last_became_active":"2024-03-26T05:15:32.640052+0000","last_became_peered":"2024-03-26T05:15:32.640052+0000","last_unstale":"2024-03-26T05:19:41.336824+0000","last_undegraded":"2024-03-26T05:19:41.336824+0000","last_fullsized":"2024-03-26T05:19:41.336824+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:52.509951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:17:52.509951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"254'51","reported_seq":875,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.310036+0000","last_change":"2024-03-26T05:17:29.903595+0000","last_active":"2024-03-26T05:19:38.310036+0000","last_peered":"2024-03-26T05:19:38.310036+0000","last_clean":"2024-03-26T05:19:38.310036+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:19:38.310036+0000","last_undegraded":"2024-03-26T05:19:38.310036+0000","last_fullsized":"2024-03-26T05:19:38.310036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'51","last_scrub_stamp":"2024-03-26T05:17:29.903436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:17:29.903436+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":790,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.305526+0000","last_change":"2024-03-26T05:16:30.881766+0000","last_active":"2024-03-26T05:19:38.305526+0000","last_peered":"2024-03-26T05:19:38.305526+0000","last_clean":"2024-03-26T05:19:38.305526+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:19:38.305526+0000","last_undegraded":"2024-03-26T05:19:38.305526+0000","last_fullsized":"2024-03-26T05:19:38.305526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:16:30.881617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:16:30.881617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":176,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.307536+0000","last_change":"2024-03-26T05:15:48.639904+0000","last_active":"2024-03-26T05:19:38.307536+0000","last_peered":"2024-03-26T05:19:38.307536+0000","last_clean":"2024-03-26T05:00:25.933152+0000","last_became_active":"2024-03-26T05:15:48.639904+0000","last_became_peered":"2024-03-26T05:15:48.639904+0000","last_unstale":"2024-03-26T05:19:38.307536+0000","last_undegraded":"2024-03-26T05:19:38.307536+0000","last_fullsized":"2024-03-26T05:15:48.630870+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:59.872240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:13:59.872240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.900286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":769,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.316253+0000","last_change":"2024-03-26T05:17:36.113579+0000","last_active":"2024-03-26T05:19:38.316253+0000","last_peered":"2024-03-26T05:19:38.316253+0000","last_clean":"2024-03-26T05:19:38.316253+0000","last_became_active":"2024-03-26T05:15:32.637717+0000","last_became_peered":"2024-03-26T05:15:32.637717+0000","last_unstale":"2024-03-26T05:19:38.316253+0000","last_undegraded":"2024-03-26T05:19:38.316253+0000","last_fullsized":"2024-03-26T05:19:38.316253+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:36.113309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:17:36.113309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"254'21","reported_seq":626,"reported_epoch":426,"state":"active+undersized+degraded","last_fresh":"2024-03-26T05:19:38.307949+0000","last_change":"2024-03-26T05:15:48.639602+0000","last_active":"2024-03-26T05:19:38.307949+0000","last_peered":"2024-03-26T05:19:38.307949+0000","last_clean":"2024-03-26T05:10:30.455391+0000","last_became_active":"2024-03-26T05:15:48.639602+0000","last_became_peered":"2024-03-26T05:15:48.639602+0000","last_unstale":"2024-03-26T05:19:38.307949+0000","last_undegraded":"2024-03-26T05:15:48.632373+0000","last_fullsized":"2024-03-26T05:15:48.632263+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:15:04.002977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:15:04.002977+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:40.900307+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":813,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-26T05:19:41.337712+0000","last_change":"2024-03-26T05:18:34.300102+0000","last_active":"2024-03-26T05:19:41.337712+0000","last_peered":"2024-03-26T05:19:41.337712+0000","last_clean":"2024-03-26T05:19:41.337712+0000","last_became_active":"2024-03-26T05:07:59.994431+0000","last_became_peered":"2024-03-26T05:07:59.994431+0000","last_unstale":"2024-03-26T05:19:41.337712+0000","last_undegraded":"2024-03-26T05:19:41.337712+0000","last_fullsized":"2024-03-26T05:19:41.337712+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:34.300011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:34.300011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:53.486867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":801,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-26T05:19:38.310490+0000","last_change":"2024-03-26T05:17:36.539795+0000","last_active":"2024-03-26T05:19:38.310490+0000","last_peered":"2024-03-26T05:19:38.310490+0000","last_clean":"2024-03-26T05:19:38.310490+0000","last_became_active":"2024-03-26T05:15:32.629027+0000","last_became_peered":"2024-03-26T05:15:32.629027+0000","last_unstale":"2024-03-26T05:19:38.310490+0000","last_undegraded":"2024-03-26T05:19:38.310490+0000","last_fullsized":"2024-03-26T05:19:38.310490+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:17:36.539278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:17:36.539278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:19:38.503710+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":497,"reported_epoch":426,"state":"active+undersized","last_fresh":"2024-03-26T05:19:38.297725+0000","last_change":"2024-03-26T05:15:48.649557+0000","last_active":"2024-03-26T05:19:38.297725+0000","last_peered":"2024-03-26T05:19:38.297725+0000","last_clean":"2024-03-26T05:07:55.689482+0000","last_became_active":"2024-03-26T05:15:48.649557+0000","last_became_peered":"2024-03-26T05:15:48.649557+0000","last_unstale":"2024-03-26T05:19:38.297725+0000","last_undegraded":"2024-03-26T05:19:38.297725+0000","last_fullsized":"2024-03-26T05:15:48.628605+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:53.055153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:53.055153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5258084352,"num_objects":80232,"num_object_clones":0,"num_object_copies":240696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80232,"num_write_kb":5134848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14471987200,"data_stored":14471987200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76532,"ondisk_log_size":76532,"up":44,"acting":44,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":313,"num_read_kb":308,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697708,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":54,"num_write":33,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":322020,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":30,"acting":30,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":82,"acting":82,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":80,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1405,"num_read_kb":1130,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":982,"ondisk_log_size":982,"up":82,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":81,"acting":81,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":83,"acting":83,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":335,"seq":1438814044211,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2196568,"kb_used_data":1922572,"kb_used_omap":4,"kb_used_meta":273979,"kb_avail":91548584,"statfs":{"total":95995035648,"available":93745750016,"internally_reserved":0,"allocated":1968713728,"data_stored":1967816304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":280555176},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332337,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3121156,"kb_used_data":2534652,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":90623996,"statfs":{"total":95995035648,"available":92798971904,"internally_reserved":0,"allocated":2595483648,"data_stored":2594601092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":600570310},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575115,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3292204,"kb_used_data":2575204,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":90452948,"statfs":{"total":95995035648,"available":92623818752,"internally_reserved":0,"allocated":2637008896,"data_stored":2636086314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738640,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2353776,"kb_used_data":1614564,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":91391376,"statfs":{"total":95995035648,"available":93584769024,"internally_reserved":0,"allocated":1653313536,"data_stored":1652400014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411590,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2526740,"kb_used_data":1913864,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":91218412,"statfs":{"total":95995035648,"available":93407653888,"internally_reserved":0,"allocated":1959796736,"data_stored":1958902690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117248,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2975912,"kb_used_data":2208924,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":90769240,"statfs":{"total":95995035648,"available":92947701760,"internally_reserved":0,"allocated":2261938176,"data_stored":2261023876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":785381831},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920973,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3139256,"kb_used_data":2751280,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":90605896,"statfs":{"total":95995035648,"available":92780437504,"internally_reserved":0,"allocated":2817310720,"data_stored":2816894270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659382,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2123856,"kb_used_data":1280516,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":91621296,"statfs":{"total":95995035648,"available":93820207104,"internally_reserved":0,"allocated":1311248384,"data_stored":1310358506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1658454016,"data_stored":1658454016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2657943552,"data_stored":2657943552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1983774720,"data_stored":1983774720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2282029056,"data_stored":2282029056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1309081600,"data_stored":1309081600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2597978112,"data_stored":2597978112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1982726144,"data_stored":1982726144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:19:45.177 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-03-26T05:19:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:45.200 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:45.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:45.298 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:19:45.298 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:19:45.298 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:19:45.298 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:19:45.298 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.298 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109994 9.c 9:34e37923:::benchmark_data_smithi080_136801_object109993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-26T05:19:45.298 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.792060+0000", 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 439.27249443199997, 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0406930759999999, 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109994 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792060+0000", 2024-03-26T05:19:45.299 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.300 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.300 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.300 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.300 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792060+0000", 2024-03-26T05:19:45.300 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.300 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.300 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.300 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.300 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792062+0000", 2024-03-26T05:19:45.300 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7889999999999999e-06 2024-03-26T05:19:45.300 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.301 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.301 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.301 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792085+0000", 2024-03-26T05:19:45.301 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3445999999999999e-05 2024-03-26T05:19:45.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.301 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.301 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.301 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792099+0000", 2024-03-26T05:19:45.301 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4372000000000001e-05 2024-03-26T05:19:45.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.301 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.301 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792110+0000", 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0739e-05 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792816+0000", 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00070612300000000002 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.792992+0000", 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017620600000000001 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.302 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.303 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,7", 2024-03-26T05:19:45.303 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.793149+0000", 2024-03-26T05:19:45.303 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015638199999999999 2024-03-26T05:19:45.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.303 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.303 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.303 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.809680+0000", 2024-03-26T05:19:45.303 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016531033000000001 2024-03-26T05:19:45.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.303 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.809693+0000", 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2781e-05 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.821562+0000", 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.01186924 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for degraded object", 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.822506+0000", 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00094436699999999999 2024-03-26T05:19:45.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.832661+0000", 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.01015492 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.832753+0000", 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1677999999999999e-05 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.305 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109996 9.c 9:318dcee9:::benchmark_data_smithi080_136801_object109995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.817916+0000", 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 439.246638029, 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.016490052, 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109996 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.306 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817916+0000", 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817916+0000", 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817917+0000", 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1600000000000001e-07 2024-03-26T05:19:45.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817942+0000", 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.5422000000000002e-05 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817950+0000", 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.977e-06 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.817956+0000", 2024-03-26T05:19:45.308 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.263e-06 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822354+0000", 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0043976070000000004 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822365+0000", 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.098e-05 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.309 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.822966+0000", 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0006012339999999 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.823106+0000", 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013994699999999999 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4", 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.823382+0000", 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00027621999999999998 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.310 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.832803+0000", 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0094210860000000004 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.834257+0000", 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014535419999999999 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.834309+0000", 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2330999999999998e-05 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.311 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.834406+0000", 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6927000000000002e-05 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109989 9.3 9:c1ef3764:::benchmark_data_smithi080_136801_object109988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.822224+0000", 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 439.24233004799999, 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.002023533, 2024-03-26T05:19:45.312 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109989 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822224+0000", 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.313 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822224+0000", 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822227+0000", 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.8040000000000002e-06 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822261+0000", 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3615000000000003e-05 2024-03-26T05:19:45.314 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822275+0000", 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4759e-05 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822287+0000", 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1419e-05 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822922+0000", 2024-03-26T05:19:45.315 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00063540799999999996 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822932+0000", 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5510000000000005e-06 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.822466+0000", 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99953455999999996 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.316 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.824248+0000", 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001781417 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109993 9.3 9:c281200c:::benchmark_data_smithi080_136801_object109992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.822314+0000", 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 439.24224058499999, 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0022707790000001, 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:19:45.317 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109993 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822314+0000", 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822314+0000", 2024-03-26T05:19:45.318 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822314+0000", 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6000000000000003e-07 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822356+0000", 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1310999999999998e-05 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.319 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822363+0000", 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9940000000000003e-06 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822368+0000", 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7200000000000003e-06 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822959+0000", 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00059071500000000001 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.320 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822965+0000", 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0800000000000002e-06 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.824517+0000", 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0015518800000001 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.824584+0000", 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7418999999999999e-05 2024-03-26T05:19:45.321 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.322 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.322 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.322 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.322 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.322 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109998 9.3 9:cd2a8d52:::benchmark_data_smithi080_136801_object109997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:19:45.322 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.822378+0000", 2024-03-26T05:19:45.322 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 439.24217619199999, 2024-03-26T05:19:45.322 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0183832850000001, 2024-03-26T05:19:45.322 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.322 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.322 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.322 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109998 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822378+0000", 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822378+0000", 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822378+0000", 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6699999999999999e-07 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822401+0000", 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2480000000000002e-05 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822407+0000", 2024-03-26T05:19:45.324 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5139999999999998e-06 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822413+0000", 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9549999999999997e-06 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822988+0000", 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00057432800000000003 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.325 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.326 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.822994+0000", 2024-03-26T05:19:45.326 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9800000000000003e-06 2024-03-26T05:19:45.326 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.326 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.326 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.326 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.824610+0000", 2024-03-26T05:19:45.326 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.001616391 2024-03-26T05:19:45.326 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.326 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.326 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.326 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.826015+0000", 2024-03-26T05:19:45.326 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0014048019999999999 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1", 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.826423+0000", 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00040815099999999999 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840598+0000", 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014175045000000001 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.327 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840625+0000", 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6840000000000001e-05 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840672+0000", 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6598999999999999e-05 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840761+0000", 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.9733000000000003e-05 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.328 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:109999 9.3 9:c8cd8b0a:::benchmark_data_smithi080_136801_object109998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.829248+0000", 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 439.23530597299998, 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0177361709999999, 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 109999 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.329 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.330 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.330 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.330 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829248+0000", 2024-03-26T05:19:45.330 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.330 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.330 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.330 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.330 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829248+0000", 2024-03-26T05:19:45.330 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.330 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.330 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.330 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829251+0000", 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.706e-06 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829301+0000", 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0448000000000002e-05 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829327+0000", 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6041999999999999e-05 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.331 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829340+0000", 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2177999999999999e-05 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829560+0000", 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00022066499999999999 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.829571+0000", 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1049999999999999e-05 2024-03-26T05:19:45.332 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.830957+0000", 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.00138595 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.831076+0000", 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118507 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1", 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.831217+0000", 2024-03-26T05:19:45.333 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014082499999999999 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840769+0000", 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0095528499999999999 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.846840+0000", 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0060710260000000002 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.334 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.846896+0000", 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5721999999999999e-05 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.846984+0000", 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8201999999999993e-05 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:110007 9.1 9:894e6b0c:::benchmark_data_smithi080_136801_object110006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:19:45.335 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.844428+0000", 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 439.22012621900001, 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97646292999999995, 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 110007 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844428+0000", 2024-03-26T05:19:45.336 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844428+0000", 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844430+0000", 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.68e-06 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.337 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844464+0000", 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4585000000000002e-05 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844488+0000", 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3312e-05 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844497+0000", 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4709999999999996e-06 2024-03-26T05:19:45.338 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844569+0000", 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1805999999999997e-05 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.844578+0000", 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5799999999999998e-06 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.811307+0000", 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96672887299999999 2024-03-26T05:19:45.339 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.811467+0000", 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016000899999999999 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 6", 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.813451+0000", 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0019838109999999998 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.340 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817336+0000", 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0038850600000000001 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820757+0000", 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0034208289999999998 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820805+0000", 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7953e-05 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.341 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.342 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820891+0000", 2024-03-26T05:19:45.342 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5960999999999999e-05 2024-03-26T05:19:45.342 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.342 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.342 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.342 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.342 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.342 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:110010 9.3 9:cd2ac3f8:::benchmark_data_smithi080_136801_object110009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:19:45.342 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.848965+0000", 2024-03-26T05:19:45.342 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 439.21558916800001, 2024-03-26T05:19:45.342 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99831434500000005, 2024-03-26T05:19:45.342 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 110010 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848965+0000", 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.343 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848965+0000", 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848966+0000", 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.381e-06 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848987+0000", 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0312000000000001e-05 2024-03-26T05:19:45.344 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.848999+0000", 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2170999999999999e-05 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.849009+0000", 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6700000000000006e-06 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.849085+0000", 2024-03-26T05:19:45.345 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6884000000000005e-05 2024-03-26T05:19:45.346 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.346 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.346 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.346 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.849096+0000", 2024-03-26T05:19:45.346 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0179e-05 2024-03-26T05:19:45.346 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.346 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.346 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.346 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.833721+0000", 2024-03-26T05:19:45.346 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98462529899999995 2024-03-26T05:19:45.346 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.346 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.833851+0000", 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012981200000000001 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1", 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.834023+0000", 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017234300000000001 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.840776+0000", 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0067526269999999998 2024-03-26T05:19:45.347 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.847155+0000", 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0063788459999999996 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.847201+0000", 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6702000000000001e-05 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.847279+0000", 2024-03-26T05:19:45.348 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8119000000000002e-05 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.25268.0:110013 9.1 9:8edffc06:::benchmark_data_smithi080_136801_object110012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:12:25.857433+0000", 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 439.20712134899998, 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96361645399999996, 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.25268", 2024-03-26T05:19:45.349 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 110013 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857433+0000", 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857433+0000", 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857434+0000", 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.178e-06 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857452+0000", 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7561e-05 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857463+0000", 2024-03-26T05:19:45.351 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1099e-05 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857470+0000", 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1999999999999997e-06 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857537+0000", 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6946999999999994e-05 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.352 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:25.857547+0000", 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0669000000000001e-05 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817405+0000", 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.95985719199999997 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817539+0000", 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013459600000000001 2024-03-26T05:19:45.353 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 6", 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.817693+0000", 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015364900000000001 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820443+0000", 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002750438 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.354 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820935+0000", 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00049171399999999997 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.820979+0000", 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3495000000000003e-05 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:12:26.821049+0000", 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0716000000000005e-05 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.355 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:205421 10.15 10:ac3cc547:::benchmark_data_smithi080_161990_object93698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:53.712849+0000", 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 111.351704756, 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99141917899999998, 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 205421 2024-03-26T05:19:45.356 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712849+0000", 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712849+0000", 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.357 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712851+0000", 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9410000000000001e-06 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712856+0000", 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5310000000000003e-06 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712868+0000", 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1763000000000001e-05 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.358 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712875+0000", 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6839999999999996e-06 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712938+0000", 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3108999999999995e-05 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712955+0000", 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6254000000000001e-05 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.359 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.699335+0000", 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98638049900000002 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.699435+0000", 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9513999999999996e-05 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.699750+0000", 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00031550600000000001 2024-03-26T05:19:45.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704128+0000", 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0043781779999999999 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704137+0000", 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8300000000000002e-06 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704171+0000", 2024-03-26T05:19:45.361 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3618000000000002e-05 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704236+0000", 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5011999999999994e-05 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704269+0000", 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2740000000000002e-05 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.362 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:205443 10.15 10:a8127e64:::benchmark_data_smithi080_161990_object93720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:53.730200+0000", 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 111.33435410200001, 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97445138600000003, 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 205443 2024-03-26T05:19:45.363 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730200+0000", 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730200+0000", 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.364 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730202+0000", 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6840000000000001e-06 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730208+0000", 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2029999999999997e-06 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730224+0000", 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6398000000000001e-05 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.365 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730233+0000", 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5290000000000008e-06 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730292+0000", 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9163000000000003e-05 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730301+0000", 2024-03-26T05:19:45.366 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.149e-06 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.701474+0000", 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97117258299999998 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.701589+0000", 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115262 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:19:45.367 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.701769+0000", 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000180373 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704432+0000", 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002662897 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704480+0000", 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7416999999999997e-05 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.368 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704558+0000", 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7880999999999995e-05 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704608+0000", 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0430000000000003e-05 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704651+0000", 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3417000000000001e-05 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.369 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:205446 10.15 10:aad44559:::benchmark_data_smithi080_161990_object93723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:53.730986+0000", 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 111.333568301, 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97380755900000004, 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 205446 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.370 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730986+0000", 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730986+0000", 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.371 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730987+0000", 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.71e-07 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730990+0000", 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3179999999999998e-06 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.731001+0000", 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1192999999999999e-05 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.372 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.731009+0000", 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7000000000000008e-06 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.731197+0000", 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018755200000000001 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.731206+0000", 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0799999999999995e-06 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.373 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.702711+0000", 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97150552199999995 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.702790+0000", 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9139000000000002e-05 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.703033+0000", 2024-03-26T05:19:45.374 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00024264999999999999 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704542+0000", 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001509355 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704661+0000", 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118347 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.375 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704718+0000", 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7617000000000001e-05 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704771+0000", 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2320000000000001e-05 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704793+0000", 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2895e-05 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.376 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:205463 10.15 10:ab1253e4:::benchmark_data_smithi080_161990_object93740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:53.740050+0000", 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 111.32450463799999, 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.974296261, 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 205463 2024-03-26T05:19:45.377 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740050+0000", 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740050+0000", 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.378 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740051+0000", 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.482e-06 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740056+0000", 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2429999999999996e-06 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740070+0000", 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3883000000000001e-05 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.379 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740079+0000", 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8899999999999996e-06 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740141+0000", 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1835999999999995e-05 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740151+0000", 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9629999999999993e-06 2024-03-26T05:19:45.380 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704843+0000", 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96469201199999999 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704956+0000", 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011328 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:19:45.381 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.705176+0000", 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021946499999999999 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.708658+0000", 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0034822059999999998 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.712329+0000", 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036716689999999998 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.382 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.714249+0000", 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0019193299999999999 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.714300+0000", 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1372999999999998e-05 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.714346+0000", 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5629e-05 2024-03-26T05:19:45.383 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:31399 11.8 11:19cc2409:::benchmark_data_smithi080_189724_object31398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:05.721634+0000", 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 39.342920653, 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.199969274, 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:19:45.384 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31399 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721634+0000", 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721634+0000", 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.385 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721634+0000", 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0800000000000004e-07 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721661+0000", 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.7093999999999998e-05 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721669+0000", 2024-03-26T05:19:45.386 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9480000000000008e-06 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721675+0000", 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2929999999999997e-06 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725183+0000", 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0035076420000000001 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.387 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725315+0000", 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013223999999999999 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725476+0000", 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016097999999999999 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.728367+0000", 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0028904690000000001 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.388 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.921455+0000", 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.193088282 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.921492+0000", 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7187999999999997e-05 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.921553+0000", 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0257000000000003e-05 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.389 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.921603+0000", 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0272999999999998e-05 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:31400 11.a 11:5879cd22:::benchmark_data_smithi080_189724_object31399:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:05.721685+0000", 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 39.342869241999999, 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.201225987, 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.390 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31400 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721685+0000", 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721685+0000", 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.391 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721685+0000", 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4799999999999999e-07 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721710+0000", 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4139e-05 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721716+0000", 2024-03-26T05:19:45.392 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9820000000000002e-06 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721722+0000", 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5929999999999996e-06 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.723746+0000", 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0020242960000000001 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.393 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.723880+0000", 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013347700000000001 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.724061+0000", 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018120300000000001 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.728551+0000", 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0044897000000000001 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.728715+0000", 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016376699999999999 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.922804+0000", 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.194089664 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.922867+0000", 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2892999999999996e-05 2024-03-26T05:19:45.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.922911+0000", 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3825000000000001e-05 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:31403 11.a 11:5744ed5a:::benchmark_data_smithi080_189724_object31402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:05.721786+0000", 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 39.342767993000002, 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.20170707399999999, 2024-03-26T05:19:45.396 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31403 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721786+0000", 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.397 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721786+0000", 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721787+0000", 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.46e-07 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721810+0000", 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3743e-05 2024-03-26T05:19:45.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721817+0000", 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7410000000000003e-06 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721823+0000", 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5879999999999997e-06 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725820+0000", 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0039973659999999996 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.726021+0000", 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000200721 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.726194+0000", 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000172895 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.922973+0000", 2024-03-26T05:19:45.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.196779763 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923041+0000", 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.7848000000000005e-05 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923053+0000", 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2128e-05 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923196+0000", 2024-03-26T05:19:45.401 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000142788 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923493+0000", 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00029704699999999999 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:31412 11.a 11:524e5acf:::benchmark_data_smithi080_189724_object31411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:05.721830+0000", 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 39.342724463000003, 2024-03-26T05:19:45.402 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.20363530299999999, 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31412 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721830+0000", 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.403 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721830+0000", 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721830+0000", 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6699999999999999e-07 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.404 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721865+0000", 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4601999999999998e-05 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721872+0000", 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5340000000000002e-06 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721878+0000", 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7219999999999996e-06 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.728760+0000", 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0068817519999999997 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.729030+0000", 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00027042499999999997 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.921481+0000", 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.19245103599999999 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.406 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923500+0000", 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0020189380000000001 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.925083+0000", 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0015828629999999999 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.925267+0000", 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000183766 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.925415+0000", 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014801899999999999 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.925465+0000", 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0179000000000002e-05 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.408 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:31423 11.a 11:5a11d067:::benchmark_data_smithi080_189724_object31422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:05.725877+0000", 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 39.338677097000001, 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.20839273799999999, 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31423 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.409 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725877+0000", 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725877+0000", 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725878+0000", 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1689999999999999e-06 2024-03-26T05:19:45.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725898+0000", 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9647e-05 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725916+0000", 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8454e-05 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725924+0000", 2024-03-26T05:19:45.411 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8760000000000005e-06 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923529+0000", 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.197604479 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923660+0000", 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013092900000000001 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:19:45.412 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923799+0000", 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000139159 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.925515+0000", 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0017163219999999999 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.928641+0000", 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0031254880000000001 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.413 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.934157+0000", 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055162539999999999 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.934221+0000", 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4067999999999997e-05 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.934270+0000", 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8893000000000001e-05 2024-03-26T05:19:45.414 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:31428 11.8 11:1ec91d57:::benchmark_data_smithi080_189724_object31427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:05.728981+0000", 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 39.335572792000001, 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.198745634, 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31428 2024-03-26T05:19:45.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.728981+0000", 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.728981+0000", 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.728985+0000", 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9299999999999996e-06 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.729005+0000", 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9922e-05 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.729015+0000", 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6639999999999993e-06 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.417 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.729021+0000", 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4370000000000004e-06 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923897+0000", 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.194875674 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.924043+0000", 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000145833 2024-03-26T05:19:45.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.924184+0000", 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014133799999999999 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.926803+0000", 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0026189569999999999 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.927346+0000", 2024-03-26T05:19:45.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00054241599999999999 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.927633+0000", 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00028726400000000002 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.927683+0000", 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0504999999999998e-05 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.420 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.927727+0000", 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3693999999999999e-05 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:31434 11.a 11:55f34270:::benchmark_data_smithi080_189724_object31433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:05.731458+0000", 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 39.333095950000001, 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.20302163500000001, 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:19:45.421 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31434 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.731458+0000", 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:19:45.422 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.731458+0000", 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.731460+0000", 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5600000000000001e-06 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.731483+0000", 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3099e-05 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.423 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:19:45.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.921450+0000", 2024-03-26T05:19:45.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.189967 2024-03-26T05:19:45.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:19:45.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.921463+0000", 2024-03-26T05:19:45.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3086e-05 2024-03-26T05:19:45.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:19:45.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.925546+0000", 2024-03-26T05:19:45.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0040828139999999997 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.925703+0000", 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000157069 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.927317+0000", 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0016144919999999999 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.425 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.928686+0000", 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0013688369999999999 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.930267+0000", 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001580922 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.934397+0000", 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0041298230000000003 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.426 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.934443+0000", 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6399000000000001e-05 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.934480+0000", 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6534e-05 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:19:45.427 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:19:45.428 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:45.454 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:45.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:45.556 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:45.598 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:45.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:45.668 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:19:45.700 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:45.754 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:45.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:45.856 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:45.920 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:45.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:45.971 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 54 16 85939 85923 99.4258 64.875 0.01704 0.0100532 2024-03-26T05:19:46.021 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:46.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:45 smithi132 ceph-mon[33500]: pgmap v902: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 685 B/s rd, 52 MiB/s wr, 837 op/s; 20217/241389 objects degraded (8.375%); 5/241389 objects misplaced (0.002%) 2024-03-26T05:19:46.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:45 smithi132 ceph-mon[33500]: 6.1 scrub starts 2024-03-26T05:19:46.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:45 smithi132 ceph-mon[33500]: 6.1 scrub ok 2024-03-26T05:19:46.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:19:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:19:46.050 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:46.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:46.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:46.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:45 smithi080 ceph-mon[29691]: pgmap v902: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 685 B/s rd, 52 MiB/s wr, 837 op/s; 20217/241389 objects degraded (8.375%); 5/241389 objects misplaced (0.002%) 2024-03-26T05:19:46.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:45 smithi080 ceph-mon[29691]: 6.1 scrub starts 2024-03-26T05:19:46.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:45 smithi080 ceph-mon[29691]: 6.1 scrub ok 2024-03-26T05:19:46.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:45 smithi080 ceph-mon[40404]: pgmap v902: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 685 B/s rd, 52 MiB/s wr, 837 op/s; 20217/241389 objects degraded (8.375%); 5/241389 objects misplaced (0.002%) 2024-03-26T05:19:46.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:45 smithi080 ceph-mon[40404]: 6.1 scrub starts 2024-03-26T05:19:46.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:45 smithi080 ceph-mon[40404]: 6.1 scrub ok 2024-03-26T05:19:46.152 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:46.179 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:46.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:46.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:46.281 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:46.344 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:46.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:46.445 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:46.486 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:46.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:46.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:46.617 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:46.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:46.695 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T05:19:46.719 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:46.780 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:46.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:46.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:46.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[29691]: 1.1 scrub starts 2024-03-26T05:19:46.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[29691]: 1.1 scrub ok 2024-03-26T05:19:46.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[29691]: 3.10 scrub starts 2024-03-26T05:19:46.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[29691]: 3.10 scrub ok 2024-03-26T05:19:46.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:46.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[29691]: pgmap v903: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 5.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s; 21535/256299 objects degraded (8.402%); 5/256299 objects misplaced (0.002%) 2024-03-26T05:19:46.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[29691]: 6.c scrub starts 2024-03-26T05:19:46.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[29691]: 6.c scrub ok 2024-03-26T05:19:46.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 20217/241389 objects degraded (8.375%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:46.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[40404]: 1.1 scrub starts 2024-03-26T05:19:46.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[40404]: 1.1 scrub ok 2024-03-26T05:19:46.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[40404]: 3.10 scrub starts 2024-03-26T05:19:46.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[40404]: 3.10 scrub ok 2024-03-26T05:19:46.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:46.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[40404]: pgmap v903: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 5.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s; 21535/256299 objects degraded (8.402%); 5/256299 objects misplaced (0.002%) 2024-03-26T05:19:46.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[40404]: 6.c scrub starts 2024-03-26T05:19:46.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[40404]: 6.c scrub ok 2024-03-26T05:19:46.817 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:46 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 20217/241389 objects degraded (8.375%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:46.883 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:46.918 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:46.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:46.971 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 55 15 87093 87078 98.9305 72.1875 0.00861378 0.0101029 2024-03-26T05:19:47.019 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:46 smithi132 ceph-mon[33500]: 1.1 scrub starts 2024-03-26T05:19:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:46 smithi132 ceph-mon[33500]: 1.1 scrub ok 2024-03-26T05:19:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:46 smithi132 ceph-mon[33500]: 3.10 scrub starts 2024-03-26T05:19:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:46 smithi132 ceph-mon[33500]: 3.10 scrub ok 2024-03-26T05:19:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:46 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:46 smithi132 ceph-mon[33500]: pgmap v903: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 5.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s; 21535/256299 objects degraded (8.402%); 5/256299 objects misplaced (0.002%) 2024-03-26T05:19:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:46 smithi132 ceph-mon[33500]: 6.c scrub starts 2024-03-26T05:19:47.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:46 smithi132 ceph-mon[33500]: 6.c scrub ok 2024-03-26T05:19:47.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:46 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 20217/241389 objects degraded (8.375%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:47.050 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:47.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:47.152 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:47.183 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:47.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:47.285 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:47.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:47.334 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:47.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:47.435 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:47.483 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:47.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:47.584 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:47.625 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:47.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:47.729 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:47.765 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:47.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:47.867 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:47.903 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:47.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:47.971 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 56 16 88217 88201 98.4172 70.1875 0.013284 0.0101559 2024-03-26T05:19:48.005 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:47 smithi132 ceph-mon[33500]: 1.a scrub starts 2024-03-26T05:19:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:47 smithi132 ceph-mon[33500]: 1.a scrub ok 2024-03-26T05:19:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:47 smithi132 ceph-mon[33500]: 3.0 scrub starts 2024-03-26T05:19:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:47 smithi132 ceph-mon[33500]: 3.0 scrub ok 2024-03-26T05:19:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:47 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:19:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:47 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:19:48.044 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:48.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:48.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:48.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:47 smithi080 ceph-mon[29691]: 1.a scrub starts 2024-03-26T05:19:48.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:47 smithi080 ceph-mon[29691]: 1.a scrub ok 2024-03-26T05:19:48.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:47 smithi080 ceph-mon[29691]: 3.0 scrub starts 2024-03-26T05:19:48.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:47 smithi080 ceph-mon[29691]: 3.0 scrub ok 2024-03-26T05:19:48.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:47 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:19:48.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:47 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:19:48.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:47 smithi080 ceph-mon[40404]: 1.a scrub starts 2024-03-26T05:19:48.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:47 smithi080 ceph-mon[40404]: 1.a scrub ok 2024-03-26T05:19:48.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:47 smithi080 ceph-mon[40404]: 3.0 scrub starts 2024-03-26T05:19:48.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:47 smithi080 ceph-mon[40404]: 3.0 scrub ok 2024-03-26T05:19:48.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:47 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:19:48.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:47 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:19:48.146 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:48.183 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:48.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:48.285 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:48.310 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:48.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:48.411 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:48.449 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:48.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:48.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:48.551 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:48.589 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:48.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:48.691 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:48.727 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:48.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:48.828 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:48.870 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:48.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:48.972 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 57 16 89338 89322 97.9197 70.0625 0.0233204 0.0102078 2024-03-26T05:19:48.972 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:49.011 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:19:49.020 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:49.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:49.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:49.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:48 smithi132 ceph-mon[33500]: 2.0 scrub starts 2024-03-26T05:19:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:48 smithi132 ceph-mon[33500]: 2.0 scrub ok 2024-03-26T05:19:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:48 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:19:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:48 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:19:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:48 smithi132 ceph-mon[33500]: pgmap v904: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 5.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.13k op/s; 21535/257280 objects degraded (8.370%); 5/257280 objects misplaced (0.002%) 2024-03-26T05:19:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:48 smithi132 ceph-mon[33500]: 4.19 scrub starts 2024-03-26T05:19:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:48 smithi132 ceph-mon[33500]: 4.19 scrub ok 2024-03-26T05:19:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:48 smithi132 ceph-mon[33500]: 5.1e scrub starts 2024-03-26T05:19:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:48 smithi132 ceph-mon[33500]: 5.1e scrub ok 2024-03-26T05:19:49.050 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[29691]: 2.0 scrub starts 2024-03-26T05:19:49.051 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[29691]: 2.0 scrub ok 2024-03-26T05:19:49.051 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:19:49.051 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:19:49.051 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[29691]: pgmap v904: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 5.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.13k op/s; 21535/257280 objects degraded (8.370%); 5/257280 objects misplaced (0.002%) 2024-03-26T05:19:49.051 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[29691]: 4.19 scrub starts 2024-03-26T05:19:49.051 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[29691]: 4.19 scrub ok 2024-03-26T05:19:49.051 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[29691]: 5.1e scrub starts 2024-03-26T05:19:49.051 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[29691]: 5.1e scrub ok 2024-03-26T05:19:49.051 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[40404]: 2.0 scrub starts 2024-03-26T05:19:49.051 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[40404]: 2.0 scrub ok 2024-03-26T05:19:49.051 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:19:49.052 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:19:49.052 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[40404]: pgmap v904: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 5.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.13k op/s; 21535/257280 objects degraded (8.370%); 5/257280 objects misplaced (0.002%) 2024-03-26T05:19:49.052 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[40404]: 4.19 scrub starts 2024-03-26T05:19:49.052 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[40404]: 4.19 scrub ok 2024-03-26T05:19:49.052 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[40404]: 5.1e scrub starts 2024-03-26T05:19:49.052 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:48 smithi080 ceph-mon[40404]: 5.1e scrub ok 2024-03-26T05:19:49.121 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:49.156 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:49.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:49.170 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:49.258 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:49.274 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:19:49.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:49.295 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:49.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:49.313 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:19:49.357 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:19:49.396 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:49.397 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:19:49.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:49.441 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:49.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:49.445 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:19:49.482 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:19:49.534 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:19:49.534 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:19:49.534 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:19:49.534 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:19:49.534 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:19:49.534 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:19:49.534 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:19:49.534 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:19:49.534 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:19:49.542 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:49.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:49.582 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:49.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:49.683 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:49.714 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:49.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:49.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:49.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:49.859 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:49.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:49.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:49 smithi080 ceph-mon[40404]: 6.8 scrub starts 2024-03-26T05:19:49.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:49 smithi080 ceph-mon[40404]: 6.8 scrub ok 2024-03-26T05:19:49.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:49 smithi080 ceph-mon[40404]: 3.4 scrub starts 2024-03-26T05:19:49.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:49 smithi080 ceph-mon[40404]: 3.4 scrub ok 2024-03-26T05:19:49.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:49 smithi080 ceph-mon[40404]: 11.1 scrub starts 2024-03-26T05:19:49.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:49 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3034218903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:19:49.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:49 smithi080 ceph-mon[29691]: 6.8 scrub starts 2024-03-26T05:19:49.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:49 smithi080 ceph-mon[29691]: 6.8 scrub ok 2024-03-26T05:19:49.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:49 smithi080 ceph-mon[29691]: 3.4 scrub starts 2024-03-26T05:19:49.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:49 smithi080 ceph-mon[29691]: 3.4 scrub ok 2024-03-26T05:19:49.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:49 smithi080 ceph-mon[29691]: 11.1 scrub starts 2024-03-26T05:19:49.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:49 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3034218903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:19:49.948 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:49 smithi132 ceph-mon[33500]: 6.8 scrub starts 2024-03-26T05:19:49.948 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:49 smithi132 ceph-mon[33500]: 6.8 scrub ok 2024-03-26T05:19:49.948 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:49 smithi132 ceph-mon[33500]: 3.4 scrub starts 2024-03-26T05:19:49.948 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:49 smithi132 ceph-mon[33500]: 3.4 scrub ok 2024-03-26T05:19:49.948 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:49 smithi132 ceph-mon[33500]: 11.1 scrub starts 2024-03-26T05:19:49.948 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:49 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3034218903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:19:49.961 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:49.971 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 58 16 90399 90383 97.3747 66.3125 0.00886194 0.0102643 2024-03-26T05:19:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:50.004 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:50.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:50.105 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:50.132 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:50.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:50.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:50.234 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:50.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:50.272 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:50.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:50.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:50.401 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:50.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:50.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:50.530 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:50.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:50.632 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:50.668 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:50.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:50.770 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:50.808 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:50.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:50.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:50.824 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:19:50.910 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:50.946 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:50.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:50.972 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 59 16 91460 91444 96.8482 66.3125 0.00723314 0.0103202 2024-03-26T05:19:51.048 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:51.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[29691]: 6.3 scrub starts 2024-03-26T05:19:51.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:19:51.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:19:51.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[29691]: 6.3 scrub ok 2024-03-26T05:19:51.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[29691]: osdmap e428: 8 total, 7 up, 8 in 2024-03-26T05:19:51.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[29691]: pgmap v906: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 577 B/s rd, 54 MiB/s wr, 872 op/s; 21535/263463 objects degraded (8.174%); 5/263463 objects misplaced (0.002%) 2024-03-26T05:19:51.066 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[29691]: 6.9 scrub starts 2024-03-26T05:19:51.066 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[29691]: 6.9 scrub ok 2024-03-26T05:19:51.066 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/255030352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:19:51.066 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:19:51.066 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[40404]: 6.3 scrub starts 2024-03-26T05:19:51.066 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:19:51.066 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:19:51.066 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[40404]: 6.3 scrub ok 2024-03-26T05:19:51.066 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[40404]: osdmap e428: 8 total, 7 up, 8 in 2024-03-26T05:19:51.066 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[40404]: pgmap v906: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 577 B/s rd, 54 MiB/s wr, 872 op/s; 21535/263463 objects degraded (8.174%); 5/263463 objects misplaced (0.002%) 2024-03-26T05:19:51.067 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[40404]: 6.9 scrub starts 2024-03-26T05:19:51.067 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[40404]: 6.9 scrub ok 2024-03-26T05:19:51.067 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/255030352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:19:51.067 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:50 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:19:51.079 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:51.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:51.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:51.084 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:50 smithi132 ceph-mon[33500]: 6.3 scrub starts 2024-03-26T05:19:51.084 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:50 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:19:51.084 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:50 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:19:51.084 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:50 smithi132 ceph-mon[33500]: 6.3 scrub ok 2024-03-26T05:19:51.084 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:50 smithi132 ceph-mon[33500]: osdmap e428: 8 total, 7 up, 8 in 2024-03-26T05:19:51.084 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:50 smithi132 ceph-mon[33500]: pgmap v906: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 577 B/s rd, 54 MiB/s wr, 872 op/s; 21535/263463 objects degraded (8.174%); 5/263463 objects misplaced (0.002%) 2024-03-26T05:19:51.084 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:50 smithi132 ceph-mon[33500]: 6.9 scrub starts 2024-03-26T05:19:51.084 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:50 smithi132 ceph-mon[33500]: 6.9 scrub ok 2024-03-26T05:19:51.084 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:50 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/255030352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:19:51.084 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:50 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:19:51.117 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:19:51.117 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:19:51.181 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:51.212 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:51.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:51.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:51.314 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:51.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:51.349 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:51.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:51.450 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:51.476 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:19:51.477 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:19:51.519 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:51.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:51.621 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:51.657 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:51.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:51.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:51.759 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:51.791 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:51.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:51.893 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:51.928 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:51.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:51.972 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:19:51.983316+0000 min lat: 0.0012195 max lat: 0.2094 avg lat: 0.0103747 2024-03-26T05:19:51.972 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:19:51.972 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 60 16 92527 92511 96.3455 66.6875 0.0157237 0.0103747 2024-03-26T05:19:52.030 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:52.065 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:52.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:52.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[29691]: 11.1 scrub ok 2024-03-26T05:19:52.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:19:52.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/255030352' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:19:52.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[29691]: osdmap e429: 8 total, 7 up, 8 in 2024-03-26T05:19:52.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[29691]: 2.1 scrub starts 2024-03-26T05:19:52.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[29691]: 2.1 scrub ok 2024-03-26T05:19:52.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:52.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 21535/263463 objects degraded (8.174%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:52.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[40404]: 11.1 scrub ok 2024-03-26T05:19:52.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:19:52.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/255030352' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:19:52.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[40404]: osdmap e429: 8 total, 7 up, 8 in 2024-03-26T05:19:52.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[40404]: 2.1 scrub starts 2024-03-26T05:19:52.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[40404]: 2.1 scrub ok 2024-03-26T05:19:52.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:52.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:51 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 21535/263463 objects degraded (8.174%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:52.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:52.254 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:52.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:52.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:51 smithi132 ceph-mon[33500]: 11.1 scrub ok 2024-03-26T05:19:52.282 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:51 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:19:52.282 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:51 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/255030352' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:19:52.282 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:51 smithi132 ceph-mon[33500]: osdmap e429: 8 total, 7 up, 8 in 2024-03-26T05:19:52.282 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:51 smithi132 ceph-mon[33500]: 2.1 scrub starts 2024-03-26T05:19:52.282 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:51 smithi132 ceph-mon[33500]: 2.1 scrub ok 2024-03-26T05:19:52.282 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:52.282 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:51 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 21535/263463 objects degraded (8.174%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:52.322 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T05:19:52.355 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:52.408 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:52.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:52.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:52.510 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:52.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:52.544 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:52.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:52.645 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:52.691 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:52.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:52.692 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:19:52.792 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:52.820 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:52.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:52.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:52.922 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:52.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:52.972 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:52.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:52.972 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 61 16 93662 93646 95.9289 70.9375 0.00359805 0.0104183 2024-03-26T05:19:53.028 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[29691]: 1.8 scrub starts 2024-03-26T05:19:53.028 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[29691]: 1.8 scrub ok 2024-03-26T05:19:53.028 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[29691]: 5.19 scrub starts 2024-03-26T05:19:53.028 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[29691]: 5.19 scrub ok 2024-03-26T05:19:53.028 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[29691]: pgmap v908: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s; 22934/273876 objects degraded (8.374%); 5/273876 objects misplaced (0.002%) 2024-03-26T05:19:53.028 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[29691]: 3.b scrub starts 2024-03-26T05:19:53.029 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[29691]: 3.b scrub ok 2024-03-26T05:19:53.029 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:53.029 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:53.029 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:19:53.029 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[40404]: 1.8 scrub starts 2024-03-26T05:19:53.029 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[40404]: 1.8 scrub ok 2024-03-26T05:19:53.029 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[40404]: 5.19 scrub starts 2024-03-26T05:19:53.029 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[40404]: 5.19 scrub ok 2024-03-26T05:19:53.029 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[40404]: pgmap v908: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s; 22934/273876 objects degraded (8.374%); 5/273876 objects misplaced (0.002%) 2024-03-26T05:19:53.029 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[40404]: 3.b scrub starts 2024-03-26T05:19:53.029 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[40404]: 3.b scrub ok 2024-03-26T05:19:53.029 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:53.030 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:53.030 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:52 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:19:53.073 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:53.102 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:53.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:53.119 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:19:53.204 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:53.232 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:53.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:53.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:53.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:52 smithi132 ceph-mon[33500]: 1.8 scrub starts 2024-03-26T05:19:53.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:52 smithi132 ceph-mon[33500]: 1.8 scrub ok 2024-03-26T05:19:53.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:52 smithi132 ceph-mon[33500]: 5.19 scrub starts 2024-03-26T05:19:53.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:52 smithi132 ceph-mon[33500]: 5.19 scrub ok 2024-03-26T05:19:53.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:52 smithi132 ceph-mon[33500]: pgmap v908: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s; 22934/273876 objects degraded (8.374%); 5/273876 objects misplaced (0.002%) 2024-03-26T05:19:53.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:52 smithi132 ceph-mon[33500]: 3.b scrub starts 2024-03-26T05:19:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:52 smithi132 ceph-mon[33500]: 3.b scrub ok 2024-03-26T05:19:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:52 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:52 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:52 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:19:53.334 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:53.416 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:53.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:53.519 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:53.566 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:53.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:53.598 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:53.668 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:53.699 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:53.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:53.801 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:53.834 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:53.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:53.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:53.960 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:53.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:53.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:53.972 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 62 16 94585 94569 95.3121 57.6875 0.00357117 0.0104847 2024-03-26T05:19:54.023 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:53 smithi080 ceph-mon[29691]: 6.4 scrub starts 2024-03-26T05:19:54.023 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:53 smithi080 ceph-mon[29691]: 6.4 scrub ok 2024-03-26T05:19:54.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:53 smithi080 ceph-mon[29691]: 6.1b scrub starts 2024-03-26T05:19:54.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:53 smithi080 ceph-mon[29691]: 6.16 scrub starts 2024-03-26T05:19:54.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:53 smithi080 ceph-mon[29691]: 6.16 scrub ok 2024-03-26T05:19:54.024 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:53 smithi080 ceph-mon[29691]: 6.1b scrub ok 2024-03-26T05:19:54.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:53 smithi080 ceph-mon[40404]: 6.4 scrub starts 2024-03-26T05:19:54.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:53 smithi080 ceph-mon[40404]: 6.4 scrub ok 2024-03-26T05:19:54.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:53 smithi080 ceph-mon[40404]: 6.1b scrub starts 2024-03-26T05:19:54.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:53 smithi080 ceph-mon[40404]: 6.16 scrub starts 2024-03-26T05:19:54.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:53 smithi080 ceph-mon[40404]: 6.16 scrub ok 2024-03-26T05:19:54.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:53 smithi080 ceph-mon[40404]: 6.1b scrub ok 2024-03-26T05:19:54.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:53 smithi132 ceph-mon[33500]: 6.4 scrub starts 2024-03-26T05:19:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:53 smithi132 ceph-mon[33500]: 6.4 scrub ok 2024-03-26T05:19:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:53 smithi132 ceph-mon[33500]: 6.1b scrub starts 2024-03-26T05:19:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:53 smithi132 ceph-mon[33500]: 6.16 scrub starts 2024-03-26T05:19:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:53 smithi132 ceph-mon[33500]: 6.16 scrub ok 2024-03-26T05:19:54.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:53 smithi132 ceph-mon[33500]: 6.1b scrub ok 2024-03-26T05:19:54.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:54.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:54.091 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:54.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:54.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:54.232 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:19:54.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:54.234 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:54.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:54.313 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:19:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:19:54.336 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:54.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:54.379 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:54.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:54.480 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:54.505 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:54.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:54.606 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:54.663 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:54.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:54.764 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:54.832 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:54.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:54.862 INFO:teuthology.orchestra.run.smithi080.stderr:set pool 11 pgp_num to 16 2024-03-26T05:19:54.934 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:54.974 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 63 16 95478 95462 94.6818 55.8125 0.0341634 0.0105558 2024-03-26T05:19:54.979 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:54.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:55.083 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:55.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[40404]: 4.15 scrub starts 2024-03-26T05:19:55.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[40404]: 4.15 scrub ok 2024-03-26T05:19:55.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2335135161' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T05:19:55.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T05:19:55.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[40404]: 4.7 scrub starts 2024-03-26T05:19:55.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[40404]: 4.7 scrub ok 2024-03-26T05:19:55.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[40404]: pgmap v909: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 732 op/s; 22934/273876 objects degraded (8.374%); 5/273876 objects misplaced (0.002%) 2024-03-26T05:19:55.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[40404]: 4.16 deep-scrub starts 2024-03-26T05:19:55.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[40404]: 3.8 scrub starts 2024-03-26T05:19:55.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[40404]: 4.16 deep-scrub ok 2024-03-26T05:19:55.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[40404]: 3.8 scrub ok 2024-03-26T05:19:55.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[29691]: 4.15 scrub starts 2024-03-26T05:19:55.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[29691]: 4.15 scrub ok 2024-03-26T05:19:55.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2335135161' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T05:19:55.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T05:19:55.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[29691]: 4.7 scrub starts 2024-03-26T05:19:55.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[29691]: 4.7 scrub ok 2024-03-26T05:19:55.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[29691]: pgmap v909: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 732 op/s; 22934/273876 objects degraded (8.374%); 5/273876 objects misplaced (0.002%) 2024-03-26T05:19:55.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[29691]: 4.16 deep-scrub starts 2024-03-26T05:19:55.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[29691]: 3.8 scrub starts 2024-03-26T05:19:55.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[29691]: 4.16 deep-scrub ok 2024-03-26T05:19:55.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:54 smithi080 ceph-mon[29691]: 3.8 scrub ok 2024-03-26T05:19:55.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:55.124 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:55.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:55.225 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:55.268 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:55.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:55.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:54 smithi132 ceph-mon[33500]: 4.15 scrub starts 2024-03-26T05:19:55.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:54 smithi132 ceph-mon[33500]: 4.15 scrub ok 2024-03-26T05:19:55.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:54 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2335135161' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T05:19:55.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:54 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T05:19:55.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:54 smithi132 ceph-mon[33500]: 4.7 scrub starts 2024-03-26T05:19:55.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:54 smithi132 ceph-mon[33500]: 4.7 scrub ok 2024-03-26T05:19:55.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:54 smithi132 ceph-mon[33500]: pgmap v909: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 732 op/s; 22934/273876 objects degraded (8.374%); 5/273876 objects misplaced (0.002%) 2024-03-26T05:19:55.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:54 smithi132 ceph-mon[33500]: 4.16 deep-scrub starts 2024-03-26T05:19:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:54 smithi132 ceph-mon[33500]: 3.8 scrub starts 2024-03-26T05:19:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:54 smithi132 ceph-mon[33500]: 4.16 deep-scrub ok 2024-03-26T05:19:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:54 smithi132 ceph-mon[33500]: 3.8 scrub ok 2024-03-26T05:19:55.369 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:55.419 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:55.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:55.520 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:55.575 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:55.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:55.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:55.677 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:55.716 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:55.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:55.817 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:55.869 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:55.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:55.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:55.971 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:55.974 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 64 16 96421 96405 94.1233 58.9375 0.00456928 0.0106176 2024-03-26T05:19:56.002 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:56.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:56.008 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[40404]: 4.8 scrub starts 2024-03-26T05:19:56.008 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[40404]: 4.8 scrub ok 2024-03-26T05:19:56.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[40404]: 6.1c deep-scrub starts 2024-03-26T05:19:56.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[40404]: 6.1c deep-scrub ok 2024-03-26T05:19:56.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T05:19:56.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[40404]: osdmap e430: 8 total, 7 up, 8 in 2024-03-26T05:19:56.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[40404]: 5.6 scrub starts 2024-03-26T05:19:56.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[40404]: 5.6 scrub ok 2024-03-26T05:19:56.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[40404]: 5.13 scrub starts 2024-03-26T05:19:56.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[40404]: 5.13 scrub ok 2024-03-26T05:19:56.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[29691]: 4.8 scrub starts 2024-03-26T05:19:56.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[29691]: 4.8 scrub ok 2024-03-26T05:19:56.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[29691]: 6.1c deep-scrub starts 2024-03-26T05:19:56.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[29691]: 6.1c deep-scrub ok 2024-03-26T05:19:56.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T05:19:56.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[29691]: osdmap e430: 8 total, 7 up, 8 in 2024-03-26T05:19:56.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[29691]: 5.6 scrub starts 2024-03-26T05:19:56.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[29691]: 5.6 scrub ok 2024-03-26T05:19:56.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[29691]: 5.13 scrub starts 2024-03-26T05:19:56.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:55 smithi080 ceph-mon[29691]: 5.13 scrub ok 2024-03-26T05:19:56.105 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:56.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:56.154 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:56.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:56.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:56.282 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:56.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:56.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:19:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:19:56.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:55 smithi132 ceph-mon[33500]: 4.8 scrub starts 2024-03-26T05:19:56.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:55 smithi132 ceph-mon[33500]: 4.8 scrub ok 2024-03-26T05:19:56.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:55 smithi132 ceph-mon[33500]: 6.1c deep-scrub starts 2024-03-26T05:19:56.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:55 smithi132 ceph-mon[33500]: 6.1c deep-scrub ok 2024-03-26T05:19:56.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:55 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T05:19:56.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:55 smithi132 ceph-mon[33500]: osdmap e430: 8 total, 7 up, 8 in 2024-03-26T05:19:56.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:55 smithi132 ceph-mon[33500]: 5.6 scrub starts 2024-03-26T05:19:56.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:55 smithi132 ceph-mon[33500]: 5.6 scrub ok 2024-03-26T05:19:56.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:55 smithi132 ceph-mon[33500]: 5.13 scrub starts 2024-03-26T05:19:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:55 smithi132 ceph-mon[33500]: 5.13 scrub ok 2024-03-26T05:19:56.382 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:56.408 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:56.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:56.510 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:56.540 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:56.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:56.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:56.642 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:56.681 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:56.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:56.782 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:56.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:56.824 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:56.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:56.926 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:56.963 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:56.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:56.974 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 65 16 97404 97388 93.6205 61.4375 0.00612655 0.0106741 2024-03-26T05:19:57.065 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:57.104 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:57.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:57.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[40404]: 11.f scrub starts 2024-03-26T05:19:57.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:57.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[40404]: pgmap v911: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.6 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 24146/288126 objects degraded (8.380%); 5/288126 objects misplaced (0.002%) 2024-03-26T05:19:57.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[40404]: 5.14 scrub starts 2024-03-26T05:19:57.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[40404]: 5.14 scrub ok 2024-03-26T05:19:57.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 22934/273876 objects degraded (8.374%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:57.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[40404]: osdmap e431: 8 total, 7 up, 8 in 2024-03-26T05:19:57.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[29691]: 11.f scrub starts 2024-03-26T05:19:57.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:57.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[29691]: pgmap v911: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.6 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 24146/288126 objects degraded (8.380%); 5/288126 objects misplaced (0.002%) 2024-03-26T05:19:57.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[29691]: 5.14 scrub starts 2024-03-26T05:19:57.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[29691]: 5.14 scrub ok 2024-03-26T05:19:57.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 22934/273876 objects degraded (8.374%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:57.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:56 smithi080 ceph-mon[29691]: osdmap e431: 8 total, 7 up, 8 in 2024-03-26T05:19:57.207 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:57.244 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:57.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:56 smithi132 ceph-mon[33500]: 11.f scrub starts 2024-03-26T05:19:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:56 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:19:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:56 smithi132 ceph-mon[33500]: pgmap v911: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.6 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 24146/288126 objects degraded (8.380%); 5/288126 objects misplaced (0.002%) 2024-03-26T05:19:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:56 smithi132 ceph-mon[33500]: 5.14 scrub starts 2024-03-26T05:19:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:56 smithi132 ceph-mon[33500]: 5.14 scrub ok 2024-03-26T05:19:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:56 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 22934/273876 objects degraded (8.374%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:19:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:56 smithi132 ceph-mon[33500]: osdmap e431: 8 total, 7 up, 8 in 2024-03-26T05:19:57.346 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:57.387 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:57.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:57.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:57.522 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:57.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:57.624 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:57.663 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:57.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:57.765 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:57.803 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:57.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:57.905 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:57.914 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:57 smithi132 ceph-mon[33500]: 5.1b scrub starts 2024-03-26T05:19:57.915 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:57 smithi132 ceph-mon[33500]: 5.1b scrub ok 2024-03-26T05:19:57.915 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:57 smithi132 ceph-mon[33500]: 1.6 scrub starts 2024-03-26T05:19:57.915 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:57 smithi132 ceph-mon[33500]: 1.6 scrub ok 2024-03-26T05:19:57.915 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:57 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1115309780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:19:57.915 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:57 smithi132 ceph-mon[33500]: 4.11 scrub starts 2024-03-26T05:19:57.915 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:57 smithi132 ceph-mon[33500]: 4.11 scrub ok 2024-03-26T05:19:57.942 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:57.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:57.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:57.961 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:19:57.961 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:19:57.961 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:19:57.961 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:19:57.961 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:19:57.961 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:19:57.961 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:19:57.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:19:57.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:19:57.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:19:57.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:19:57.962 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:19:57.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:19:57.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:19:57.962 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:19:57.974 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 66 16 98495 98479 93.2351 68.1875 0.0213687 0.0107205 2024-03-26T05:19:58.044 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:19:58.070 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:58.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:19:58.073 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:19:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:19:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2024-03-26T05:19:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2024-03-26T05:19:58.073 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:19:58.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[40404]: 5.1b scrub starts 2024-03-26T05:19:58.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[40404]: 5.1b scrub ok 2024-03-26T05:19:58.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[40404]: 1.6 scrub starts 2024-03-26T05:19:58.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[40404]: 1.6 scrub ok 2024-03-26T05:19:58.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1115309780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:19:58.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[40404]: 4.11 scrub starts 2024-03-26T05:19:58.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[40404]: 4.11 scrub ok 2024-03-26T05:19:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[29691]: 5.1b scrub starts 2024-03-26T05:19:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[29691]: 5.1b scrub ok 2024-03-26T05:19:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[29691]: 1.6 scrub starts 2024-03-26T05:19:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[29691]: 1.6 scrub ok 2024-03-26T05:19:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1115309780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:19:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[29691]: 4.11 scrub starts 2024-03-26T05:19:58.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:57 smithi080 ceph-mon[29691]: 4.11 scrub ok 2024-03-26T05:19:58.172 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:19:58.207 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:58.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:19:58.312 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:58.339 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:58.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:58.440 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:58.476 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:58.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:58.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:58.578 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:58.619 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:58.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:58.720 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:58.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:58.749 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:58.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:58.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T05:19:58.849 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:19:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:58.887 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:58.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:19:58.975 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 67 16 99535 99519 92.8136 65 0.0202719 0.0107697 2024-03-26T05:19:58.988 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:19:59.026 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:59.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:59.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:19:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:58 smithi132 ceph-mon[33500]: 11.f scrub ok 2024-03-26T05:19:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:58 smithi132 ceph-mon[33500]: 5.7 scrub starts 2024-03-26T05:19:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:58 smithi132 ceph-mon[33500]: 5.7 scrub ok 2024-03-26T05:19:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:58 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:19:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:58 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:19:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:58 smithi132 ceph-mon[33500]: 5.b scrub starts 2024-03-26T05:19:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:58 smithi132 ceph-mon[33500]: 5.b scrub ok 2024-03-26T05:19:59.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:58 smithi132 ceph-mon[33500]: pgmap v913: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 24146/289005 objects degraded (8.355%); 5/289005 objects misplaced (0.002%) 2024-03-26T05:19:59.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:58 smithi132 ceph-mon[33500]: 4.14 scrub starts 2024-03-26T05:19:59.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:58 smithi132 ceph-mon[33500]: 4.14 scrub ok 2024-03-26T05:19:59.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[40404]: 11.f scrub ok 2024-03-26T05:19:59.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[40404]: 5.7 scrub starts 2024-03-26T05:19:59.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[40404]: 5.7 scrub ok 2024-03-26T05:19:59.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:19:59.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:19:59.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[40404]: 5.b scrub starts 2024-03-26T05:19:59.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[40404]: 5.b scrub ok 2024-03-26T05:19:59.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[40404]: pgmap v913: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 24146/289005 objects degraded (8.355%); 5/289005 objects misplaced (0.002%) 2024-03-26T05:19:59.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[40404]: 4.14 scrub starts 2024-03-26T05:19:59.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[40404]: 4.14 scrub ok 2024-03-26T05:19:59.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[29691]: 11.f scrub ok 2024-03-26T05:19:59.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[29691]: 5.7 scrub starts 2024-03-26T05:19:59.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[29691]: 5.7 scrub ok 2024-03-26T05:19:59.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:19:59.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:19:59.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[29691]: 5.b scrub starts 2024-03-26T05:19:59.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[29691]: 5.b scrub ok 2024-03-26T05:19:59.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[29691]: pgmap v913: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 5.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 24146/289005 objects degraded (8.355%); 5/289005 objects misplaced (0.002%) 2024-03-26T05:19:59.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[29691]: 4.14 scrub starts 2024-03-26T05:19:59.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:58 smithi080 ceph-mon[29691]: 4.14 scrub ok 2024-03-26T05:19:59.129 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:59.129 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:19:59.169 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:59.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:59.271 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:59.299 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:59.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:59.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:59.401 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:19:59.444 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:19:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:59.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:19:59.546 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:59.575 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:59.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:59.677 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:19:59.704 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:59.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:19:59.806 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:19:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:19:59.925 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:19:59.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:19:59.975 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 68 16 100381 100365 92.2263 52.875 0.00370864 0.0108253 2024-03-26T05:20:00.025 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:00.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:00.066 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:00.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:00.167 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:00.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:00.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:59 smithi132 ceph-mon[33500]: 3.c scrub starts 2024-03-26T05:20:00.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:59 smithi132 ceph-mon[33500]: 3.c scrub ok 2024-03-26T05:20:00.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:19:59 smithi132 ceph-mon[33500]: 11.0 scrub starts 2024-03-26T05:20:00.300 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:00.339 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:00.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:00.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:59 smithi080 ceph-mon[29691]: 3.c scrub starts 2024-03-26T05:20:00.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:59 smithi080 ceph-mon[29691]: 3.c scrub ok 2024-03-26T05:20:00.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:19:59 smithi080 ceph-mon[29691]: 11.0 scrub starts 2024-03-26T05:20:00.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:59 smithi080 ceph-mon[40404]: 3.c scrub starts 2024-03-26T05:20:00.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:59 smithi080 ceph-mon[40404]: 3.c scrub ok 2024-03-26T05:20:00.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:19:59 smithi080 ceph-mon[40404]: 11.0 scrub starts 2024-03-26T05:20:00.441 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:00.485 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:00.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:00.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:00.624 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:00.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:00.726 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:00.757 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:00.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:00.859 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:00.884 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:00.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:00.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:00.944 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:20:00.945 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:20:00.945 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:20:00.945 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:20:00.945 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.945 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.25268.0:109969 9.a 9:57b77978:::benchmark_data_smithi080_136801_object109968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-26T05:20:00.945 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:12:25.740475+0000", 2024-03-26T05:20:00.945 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 455.21082138399998, 2024-03-26T05:20:00.945 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.080896952, 2024-03-26T05:20:00.945 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:00.945 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:20:00.945 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.25268", 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 109969 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.740475+0000", 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:00.946 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.740475+0000", 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.740476+0000", 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2160000000000001e-06 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.740539+0000", 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3102999999999995e-05 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:00.947 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.740550+0000", 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1123e-05 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.740556+0000", 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9730000000000002e-06 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.756730+0000", 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016173646999999999 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.948 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.756843+0000", 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000113331 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,7", 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.757009+0000", 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016554699999999999 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.760750+0000", 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0037407439999999998 2024-03-26T05:20:00.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.760858+0000", 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001086 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.817544+0000", 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056685724 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:00.950 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.817553+0000", 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2429999999999994e-06 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.821285+0000", 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0037314399999999 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.821372+0000", 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7261000000000004e-05 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:00.951 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.952 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.952 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.952 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.25268.0:110001 9.7 9:e773ec00:::benchmark_data_smithi080_136801_object110000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:20:00.952 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:12:25.831731+0000", 2024-03-26T05:20:00.952 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 455.11956528600001, 2024-03-26T05:20:00.952 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0580222290000001, 2024-03-26T05:20:00.952 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:00.952 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:00.952 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:00.952 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.25268", 2024-03-26T05:20:00.952 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:20:00.952 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 110001 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.831731+0000", 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.831731+0000", 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.831732+0000", 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5000000000000001e-07 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.831799+0000", 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7101999999999996e-05 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.831807+0000", 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8590000000000005e-06 2024-03-26T05:20:00.954 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.831812+0000", 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.95e-06 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.831841+0000", 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.8705e-05 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.831846+0000", 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0520000000000001e-06 2024-03-26T05:20:00.955 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.956 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.956 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.956 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.837500+0000", 2024-03-26T05:20:00.956 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.005654397 2024-03-26T05:20:00.956 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.956 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.956 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:00.956 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.837584+0000", 2024-03-26T05:20:00.956 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3719000000000003e-05 2024-03-26T05:20:00.956 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.956 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3", 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.837687+0000", 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000102693 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.848122+0000", 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010435012 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.889627+0000", 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041505351000000003 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.957 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.889680+0000", 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2966000000000001e-05 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.889753+0000", 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3473000000000006e-05 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.958 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.959 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: 1.9 scrub starts 2024-03-26T05:20:00.959 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: 1.9 scrub ok 2024-03-26T05:20:00.959 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: 4.1c scrub starts 2024-03-26T05:20:00.959 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: 4.1c scrub ok 2024-03-26T05:20:00.959 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 24146/289005 objects degraded (8.355%), 25 pgs degraded, 62 pgs undersized 2024-03-26T05:20:00.959 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: [WRN] OSD_DOWN: 1 osds down 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: osd.4 (root=default,host=smithi132) is down 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: [WRN] PG_DEGRADED: Degraded data redundancy: 24146/289005 objects degraded (8.355%), 25 pgs degraded, 62 pgs undersized 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 1.2 is stuck undersized for 4m, current state active+undersized+degraded, last acting [7,0] 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 1.3 is stuck undersized for 4m, current state active+undersized, last acting [1,6] 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 1.5 is stuck undersized for 4m, current state active+undersized, last acting [2,6] 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 3.1 is stuck undersized for 4m, current state active+undersized, last acting [0,5] 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 3.3 is stuck undersized for 4m, current state active+undersized, last acting [0,6] 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 3.6 is stuck undersized for 4m, current state active+undersized+degraded, last acting [0,1] 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 3.9 is stuck undersized for 4m, current state active+undersized, last acting [2,7] 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 3.e is stuck undersized for 4m, current state active+undersized, last acting [7,1] 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 3.f is stuck undersized for 4m, current state active+undersized+degraded, last acting [7,0] 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 3.11 is stuck undersized for 4m, current state active+undersized, last acting [7,6] 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 3.13 is stuck undersized for 4m, current state active+undersized, last acting [7,2] 2024-03-26T05:20:00.960 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 3.14 is stuck undersized for 4m, current state active+undersized, last acting [7,6] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 3.15 is stuck undersized for 4m, current state active+undersized, last acting [7,3] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 3.19 is stuck undersized for 4m, current state active+undersized+degraded, last acting [1,3] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 3.1a is stuck undersized for 4m, current state active+undersized, last acting [1,2] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 4.1 is stuck undersized for 4m, current state active+undersized+degraded, last acting [5,6] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 4.2 is stuck undersized for 4m, current state active+undersized+degraded, last acting [2,5] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 4.5 is stuck undersized for 4m, current state active+undersized+degraded, last acting [6,7] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 4.9 is stuck undersized for 4m, current state active+undersized+degraded, last acting [1,3] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 4.c is stuck undersized for 4m, current state active+undersized+degraded, last acting [3,6] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 4.d is stuck undersized for 4m, current state active+undersized+degraded, last acting [2,1] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 4.e is stuck undersized for 4m, current state active+undersized+degraded, last acting [6,1] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 4.f is stuck undersized for 4m, current state active+undersized+degraded, last acting [1,3] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 4.13 is stuck undersized for 4m, current state active+undersized+degraded, last acting [6,1] 2024-03-26T05:20:00.961 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 4.1d is stuck undersized for 4m, current state active+undersized+degraded, last acting [3,1] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 5.0 is stuck undersized for 4m, current state active+undersized+degraded, last acting [3,1] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 5.1 is stuck undersized for 4m, current state active+undersized, last acting [3,7] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 5.5 is stuck undersized for 4m, current state active+undersized, last acting [0,1] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 5.9 is stuck undersized for 4m, current state active+undersized+degraded, last acting [7,6] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 5.a is stuck undersized for 4m, current state active+undersized, last acting [2,3] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 5.c is stuck undersized for 4m, current state active+undersized, last acting [1,0] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 5.e is stuck undersized for 4m, current state active+undersized+degraded, last acting [5,0] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 5.f is stuck undersized for 4m, current state active+undersized, last acting [5,6] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 5.10 is stuck undersized for 4m, current state active+undersized, last acting [7,3] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 5.11 is stuck undersized for 4m, current state active+undersized, last acting [6,7] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 5.1f is stuck undersized for 4m, current state active+undersized+degraded, last acting [6,7] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 6.2 is stuck undersized for 4m, current state active+undersized, last acting [3,2] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 6.6 is stuck undersized for 4m, current state active+undersized, last acting [3,7] 2024-03-26T05:20:00.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 6.7 is stuck undersized for 4m, current state active+undersized, last acting [5,3] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 6.b is stuck undersized for 4m, current state active+undersized, last acting [3,1] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 6.e is stuck undersized for 4m, current state active+undersized, last acting [1,2] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 6.f is stuck undersized for 4m, current state active+undersized, last acting [2,3] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 6.12 is stuck undersized for 4m, current state active+undersized, last acting [7,2] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 6.14 is stuck undersized for 4m, current state active+undersized, last acting [2,7] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 6.15 is stuck undersized for 4m, current state active+undersized, last acting [7,6] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 6.17 is stuck undersized for 4m, current state active+undersized, last acting [2,5] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 6.1d is stuck undersized for 4m, current state active+undersized, last acting [1,5] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 11.7 is stuck undersized for 81s, current state active+undersized+degraded, last acting [5,1] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 11.c is stuck undersized for 81s, current state active+undersized+degraded, last acting [1,2] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 11.d is stuck undersized for 81s, current state active+undersized+degraded, last acting [2,6] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pg 11.e is stuck undersized for 81s, current state active+undersized+degraded, last acting [1,7] 2024-03-26T05:20:00.963 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:00 smithi132 ceph-mon[33500]: pgmap v914: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 859 op/s; 24146/294507 objects degraded (8.199%); 5/294507 objects misplaced (0.002%) 2024-03-26T05:20:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.25268.0:110003 9.7 9:e1ebcc93:::benchmark_data_smithi080_136801_object110002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:20:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:12:25.832500+0000", 2024-03-26T05:20:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 455.11879627000002, 2024-03-26T05:20:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.057727952, 2024-03-26T05:20:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.25268", 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 110003 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.832500+0000", 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.832500+0000", 2024-03-26T05:20:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.832501+0000", 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.39e-07 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.832558+0000", 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7590000000000003e-05 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.832565+0000", 2024-03-26T05:20:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0589999999999997e-06 2024-03-26T05:20:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.832570+0000", 2024-03-26T05:20:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6959999999999996e-06 2024-03-26T05:20:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.832602+0000", 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2113000000000002e-05 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.832609+0000", 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6150000000000002e-06 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.839787+0000", 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0071782709999999 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.839874+0000", 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6786000000000006e-05 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3", 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.839980+0000", 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010626800000000001 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.848125+0000", 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081453580000000001 2024-03-26T05:20:00.969 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.890140+0000", 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.042015018000000001 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.890176+0000", 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5397e-05 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:00.970 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.890228+0000", 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2342000000000001e-05 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.25268.0:110008 9.7 9:e5350627:::benchmark_data_smithi080_136801_object110007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:12:25.844422+0000", 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 455.10687422900003, 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.045942277, 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.25268", 2024-03-26T05:20:00.971 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 110008 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.844422+0000", 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.844422+0000", 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.844424+0000", 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.2259999999999999e-06 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.844471+0000", 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7129000000000002e-05 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.844482+0000", 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0481e-05 2024-03-26T05:20:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.844505+0000", 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.2795000000000001e-05 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.844542+0000", 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7190000000000001e-05 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.844547+0000", 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4620000000000004e-06 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.841812+0000", 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.997265026 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.841887+0000", 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4833999999999995e-05 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3", 2024-03-26T05:20:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.841990+0000", 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000102562 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.848130+0000", 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006140164 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.890266+0000", 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.042135888000000003 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.890301+0000", 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5206000000000001e-05 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.890364+0000", 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3313999999999997e-05 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.25268.0:110012 9.a 9:5e46a4c8:::benchmark_data_smithi080_136801_object110011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:12:25.854397+0000", 2024-03-26T05:20:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 455.09689904700002, 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.97536776000000003, 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.25268", 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1073783467", 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 110012 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.854397+0000", 2024-03-26T05:20:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.854397+0000", 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.854398+0000", 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1650000000000001e-06 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.854432+0000", 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.3049000000000002e-05 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.854444+0000", 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1997000000000001e-05 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.854450+0000", 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.122e-06 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.860730+0000", 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0062805320000000001 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:25.860739+0000", 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0329999999999997e-06 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.821566+0000", 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.960827188 2024-03-26T05:20:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.821693+0000", 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012700799999999999 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.821851+0000", 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000157646 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.827528+0000", 2024-03-26T05:20:00.982 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0056765720000000004 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.829613+0000", 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020854630000000001 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.829671+0000", 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7995000000000003e-05 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:12:26.829765+0000", 2024-03-26T05:20:00.983 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.399e-05 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141729 10.9 10:90ac0485:::benchmark_data_smithi080_161990_object30006:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.720442+0000", 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 190.230854491, 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.012933866, 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:00.984 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141729 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720442+0000", 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720442+0000", 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.985 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720444+0000", 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.526e-06 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720455+0000", 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0433e-05 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720479+0000", 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.4596999999999998e-05 2024-03-26T05:20:00.986 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720494+0000", 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4133e-05 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.809189+0000", 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.088695464000000002 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.809199+0000", 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5419999999999994e-06 2024-03-26T05:20:00.987 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.733307+0000", 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92410795400000001 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.733376+0000", 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9216999999999998e-05 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141730 10.9 10:92d423bb:::benchmark_data_smithi080_161990_object30007:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.720524+0000", 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 190.23077234900001, 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.013281876, 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141730 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720524+0000", 2024-03-26T05:20:00.989 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.990 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:00.990 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720524+0000", 2024-03-26T05:20:00.990 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.990 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:00.990 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720524+0000", 2024-03-26T05:20:00.990 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5699999999999998e-07 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720527+0000", 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.3939999999999999e-06 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720536+0000", 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4809999999999995e-06 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720542+0000", 2024-03-26T05:20:00.991 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1649999999999997e-06 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.809225+0000", 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.088683159999999997 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.809232+0000", 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.503e-06 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.992 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.733739+0000", 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92450684699999996 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.733806+0000", 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6869e-05 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141748 10.16 10:6ac9489a:::benchmark_data_smithi080_161990_object30025:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.720548+0000", 2024-03-26T05:20:00.993 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 190.23074853899999, 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.018819301, 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141748 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720548+0000", 2024-03-26T05:20:00.994 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720548+0000", 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720548+0000", 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6899999999999998e-07 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720551+0000", 2024-03-26T05:20:00.995 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.5399999999999998e-06 2024-03-26T05:20:00.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.996 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:00.996 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720560+0000", 2024-03-26T05:20:00.996 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2979999999999997e-06 2024-03-26T05:20:00.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.996 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:00.996 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720565+0000", 2024-03-26T05:20:00.996 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7489999999999998e-06 2024-03-26T05:20:00.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.809269+0000", 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.088704029000000004 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.809276+0000", 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5989999999999996e-06 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.730276+0000", 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92099931300000004 2024-03-26T05:20:00.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.730401+0000", 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000125217 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.730700+0000", 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00029868899999999999 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:00.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.732755+0000", 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002055781 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739245+0000", 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00648987 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739287+0000", 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1913999999999999e-05 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:00.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739367+0000", 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9932999999999995e-05 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158392 10.11 10:8a795ca1:::benchmark_data_smithi080_161990_object46669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.596536+0000", 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 171.354760672, 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.094042502, 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:01.000 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158392 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596536+0000", 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596536+0000", 2024-03-26T05:20:01.001 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596538+0000", 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.885e-06 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596542+0000", 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7700000000000001e-06 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596555+0000", 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.292e-05 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596563+0000", 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9459999999999998e-06 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596625+0000", 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1779999999999995e-05 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596639+0000", 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3797e-05 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.687392+0000", 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0907529970000001 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.687505+0000", 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000113009 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:20:01.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.687732+0000", 2024-03-26T05:20:01.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00022680399999999999 2024-03-26T05:20:01.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690443+0000", 2024-03-26T05:20:01.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002711364 2024-03-26T05:20:01.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:01.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690462+0000", 2024-03-26T05:20:01.005 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:01.007 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 69 16 101307 101291 91.7285 57.875 0.00943633 0.0108969 2024-03-26T05:20:01.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.878e-05 2024-03-26T05:20:01.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690490+0000", 2024-03-26T05:20:01.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.7838e-05 2024-03-26T05:20:01.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:01.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690536+0000", 2024-03-26T05:20:01.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6758000000000002e-05 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690578+0000", 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1854000000000003e-05 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158416 10.11 10:8ed20479:::benchmark_data_smithi080_161990_object46693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.609295+0000", 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 171.34200137100001, 2024-03-26T05:20:01.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.083529985, 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158416 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609295+0000", 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:01.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609295+0000", 2024-03-26T05:20:01.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:01.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609296+0000", 2024-03-26T05:20:01.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.395e-06 2024-03-26T05:20:01.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609301+0000", 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1089999999999997e-06 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609315+0000", 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3568e-05 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609323+0000", 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9330000000000001e-06 2024-03-26T05:20:01.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609969+0000", 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00064633900000000001 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609977+0000", 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7929999999999998e-06 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.689332+0000", 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0793544639999999 2024-03-26T05:20:01.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.689422+0000", 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0303999999999995e-05 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.689556+0000", 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000133931 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.691785+0000", 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0022293230000000001 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692668+0000", 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088328400000000002 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692722+0000", 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3233999999999999e-05 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692781+0000", 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8940000000000002e-05 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692825+0000", 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4367999999999999e-05 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158447 10.11 10:8efb7744:::benchmark_data_smithi080_161990_object46724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.633507+0000", 2024-03-26T05:20:01.016 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 171.317789839, 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0649222920000001, 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158447 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633507+0000", 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633507+0000", 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633507+0000", 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.23e-07 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633511+0000", 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.754e-06 2024-03-26T05:20:01.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633521+0000", 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0077e-05 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633527+0000", 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633578+0000", 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1239999999999997e-05 2024-03-26T05:20:01.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633588+0000", 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6649999999999998e-06 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690639+0000", 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0570507950000001 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690756+0000", 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000117679 2024-03-26T05:20:01.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690962+0000", 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00020548699999999999 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692888+0000", 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0019258369999999999 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.694025+0000", 2024-03-26T05:20:01.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011376870000000001 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698342+0000", 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043164609999999997 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698390+0000", 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7627999999999999e-05 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698429+0000", 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.926e-05 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158477 10.11 10:8cf6ea54:::benchmark_data_smithi080_161990_object46754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.657292+0000", 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 171.29400479, 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0412966560000001, 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:01.023 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158477 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657292+0000", 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657292+0000", 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657293+0000", 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3599999999999999e-06 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657298+0000", 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657312+0000", 2024-03-26T05:20:01.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3521e-05 2024-03-26T05:20:01.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:01.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657321+0000", 2024-03-26T05:20:01.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7779999999999996e-06 2024-03-26T05:20:01.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657384+0000", 2024-03-26T05:20:01.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3275999999999995e-05 2024-03-26T05:20:01.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657396+0000", 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1783999999999999e-05 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.691824+0000", 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.034427921 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.691892+0000", 2024-03-26T05:20:01.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8178999999999999e-05 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692026+0000", 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013440300000000001 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.694032+0000", 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020056230000000002 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.694070+0000", 2024-03-26T05:20:01.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8408999999999998e-05 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698509+0000", 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.004438727 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698560+0000", 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0816000000000003e-05 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698588+0000", 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.8438000000000001e-05 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158504 10.11 10:89d6d76b:::benchmark_data_smithi080_161990_object46781:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.677034+0000", 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 171.274262083, 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.021703617, 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:20:01.030 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158504 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677034+0000", 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677034+0000", 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677036+0000", 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4610000000000001e-06 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677041+0000", 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6219999999999997e-06 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677056+0000", 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4630999999999999e-05 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677064+0000", 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8560000000000007e-06 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677122+0000", 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7734e-05 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677131+0000", 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0080000000000008e-06 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692954+0000", 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.015823565 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.693026+0000", 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1594999999999995e-05 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.693185+0000", 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015974699999999999 2024-03-26T05:20:01.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.695963+0000", 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0027773009999999998 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.695973+0000", 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8139999999999995e-06 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698666+0000", 2024-03-26T05:20:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0026932979999999998 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698703+0000", 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6817999999999999e-05 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698738+0000", 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5166999999999997e-05 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158528 10.11 10:8c53284c:::benchmark_data_smithi080_161990_object46805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.696752+0000", 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 171.25454410699999, 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.002081698, 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158528 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696752+0000", 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696752+0000", 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696753+0000", 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.159e-06 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696758+0000", 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0189999999999997e-06 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696776+0000", 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.7456999999999999e-05 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696784+0000", 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3739999999999998e-06 2024-03-26T05:20:01.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696840+0000", 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6116000000000003e-05 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696923+0000", 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2599e-05 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.694099+0000", 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.99717579599999995 2024-03-26T05:20:01.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.694175+0000", 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6039999999999997e-05 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.694339+0000", 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000164637 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.695999+0000", 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001659607 2024-03-26T05:20:01.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.697054+0000", 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00105466 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698783+0000", 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001729405 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698811+0000", 2024-03-26T05:20:01.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.7582000000000001e-05 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698834+0000", 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.3247e-05 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158578 10.11 10:89bd8808:::benchmark_data_smithi080_161990_object46855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.740795+0000", 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 171.21050109800001, 2024-03-26T05:20:01.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.96429717400000003, 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158578 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740795+0000", 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740795+0000", 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740797+0000", 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3030000000000001e-06 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740802+0000", 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1270000000000002e-06 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:01.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740816+0000", 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4089e-05 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740823+0000", 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5190000000000003e-06 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740884+0000", 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0424999999999998e-05 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740896+0000", 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2548000000000001e-05 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.695076+0000", 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.95417941799999995 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.695139+0000", 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3326999999999999e-05 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.695273+0000", 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000133516 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698468+0000", 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.003195163 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698747+0000", 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00027941899999999999 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.704990+0000", 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0062431179999999998 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.705049+0000", 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8473000000000001e-05 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.705092+0000", 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3729000000000001e-05 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.049 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.050 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.050 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:98521 11.7 11:e44025d9:::benchmark_data_smithi080_189724_object98520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-26T05:20:01.050 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:19:58.004976+0000", 2024-03-26T05:20:01.050 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 2.9463202530000001, 2024-03-26T05:20:01.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.094111021000000003, 2024-03-26T05:20:01.050 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:01.050 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:01.050 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:01.050 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:20:01.050 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 98521 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.004976+0000", 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.004976+0000", 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.004978+0000", 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.531e-06 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.005025+0000", 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7240999999999997e-05 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.005040+0000", 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5536000000000002e-05 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.005048+0000", 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8879999999999997e-06 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.005190+0000", 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000142053 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.005326+0000", 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013528300000000001 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.005476+0000", 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015056999999999999 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.010583+0000", 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0051068650000000004 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.098900+0000", 2024-03-26T05:20:01.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.088316749999999999 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.098973+0000", 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2737000000000005e-05 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:58.099087+0000", 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000114567 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:99585 11.7 11:e043a4f7:::benchmark_data_smithi080_189724_object99584:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:19:59.037083+0000", 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 1.914213245, 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10897438, 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 99585 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:01.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.037083+0000", 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.037083+0000", 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.037084+0000", 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3e-06 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:01.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.037125+0000", 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1004000000000002e-05 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.037141+0000", 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5387e-05 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.037148+0000", 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.148e-06 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.037200+0000", 2024-03-26T05:20:01.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2330000000000002e-05 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.037353+0000", 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015281100000000001 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.037506+0000", 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015328100000000001 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.040480+0000", 2024-03-26T05:20:01.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002973542 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.145895+0000", 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10541539699999999 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.145971+0000", 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5703000000000003e-05 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.146057+0000", 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6477000000000006e-05 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:99596 11.7 11:eb1bdbcd:::benchmark_data_smithi080_189724_object99595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:19:59.047871+0000", 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 1.9034253539999999, 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.102689927, 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:01.064 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 99596 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.047871+0000", 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.047871+0000", 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:01.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.047873+0000", 2024-03-26T05:20:01.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.0150000000000002e-06 2024-03-26T05:20:01.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:01.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.047921+0000", 2024-03-26T05:20:01.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8131999999999999e-05 2024-03-26T05:20:01.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.047936+0000", 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5139e-05 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.047946+0000", 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5610000000000004e-06 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.048011+0000", 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5097000000000004e-05 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.048173+0000", 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000162106 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.048348+0000", 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017476300000000001 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.054100+0000", 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.005751796 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.150380+0000", 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.096280629000000006 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.150459+0000", 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9029000000000005e-05 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.150561+0000", 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010166 2024-03-26T05:20:01.069 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:99962 11.7 11:e29bb502:::benchmark_data_smithi080_189724_object99961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:19:59.470814+0000", 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 1.480482418, 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.091205210999999994, 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:20:01.070 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 99962 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.470814+0000", 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.470814+0000", 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:01.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.470815+0000", 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.057e-06 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.470865+0000", 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9659e-05 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.470879+0000", 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4209000000000001e-05 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.470887+0000", 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5539999999999998e-06 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.470954+0000", 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6577999999999999e-05 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.471088+0000", 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013369799999999999 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.471234+0000", 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014587700000000001 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.473814+0000", 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0025803250000000001 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.561847+0000", 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.088033001 2024-03-26T05:20:01.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.561908+0000", 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0933000000000001e-05 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.562019+0000", 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011132 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.075 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:99971 11.7 11:ee55b956:::benchmark_data_smithi080_189724_object99970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:19:59.480318+0000", 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 1.4709781610000001, 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.081913076000000001, 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 99971 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:01.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.480318+0000", 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.480318+0000", 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.480319+0000", 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.28e-06 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:01.077 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.480370+0000", 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0216999999999997e-05 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.480382+0000", 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2343e-05 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.480390+0000", 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7870000000000002e-06 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.480443+0000", 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2864000000000001e-05 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.480595+0000", 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015254900000000001 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.480749+0000", 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015383500000000001 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.483358+0000", 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00260926 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.562144+0000", 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.078785506000000005 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.562181+0000", 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6922000000000003e-05 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:19:59.562231+0000", 2024-03-26T05:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0513000000000002e-05 2024-03-26T05:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:20:01.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:01.083 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:01.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:01.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: 1.9 scrub starts 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: 1.9 scrub ok 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: 4.1c scrub starts 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: 4.1c scrub ok 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 24146/289005 objects degraded (8.355%), 25 pgs degraded, 62 pgs undersized 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: [WRN] OSD_DOWN: 1 osds down 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: osd.4 (root=default,host=smithi132) is down 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: [WRN] PG_DEGRADED: Degraded data redundancy: 24146/289005 objects degraded (8.355%), 25 pgs degraded, 62 pgs undersized 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 1.2 is stuck undersized for 4m, current state active+undersized+degraded, last acting [7,0] 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 1.3 is stuck undersized for 4m, current state active+undersized, last acting [1,6] 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 1.5 is stuck undersized for 4m, current state active+undersized, last acting [2,6] 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 3.1 is stuck undersized for 4m, current state active+undersized, last acting [0,5] 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 3.3 is stuck undersized for 4m, current state active+undersized, last acting [0,6] 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 3.6 is stuck undersized for 4m, current state active+undersized+degraded, last acting [0,1] 2024-03-26T05:20:01.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 3.9 is stuck undersized for 4m, current state active+undersized, last acting [2,7] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 3.e is stuck undersized for 4m, current state active+undersized, last acting [7,1] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 3.f is stuck undersized for 4m, current state active+undersized+degraded, last acting [7,0] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 3.11 is stuck undersized for 4m, current state active+undersized, last acting [7,6] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 3.13 is stuck undersized for 4m, current state active+undersized, last acting [7,2] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 3.14 is stuck undersized for 4m, current state active+undersized, last acting [7,6] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 3.15 is stuck undersized for 4m, current state active+undersized, last acting [7,3] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 3.19 is stuck undersized for 4m, current state active+undersized+degraded, last acting [1,3] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 3.1a is stuck undersized for 4m, current state active+undersized, last acting [1,2] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 4.1 is stuck undersized for 4m, current state active+undersized+degraded, last acting [5,6] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 4.2 is stuck undersized for 4m, current state active+undersized+degraded, last acting [2,5] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 4.5 is stuck undersized for 4m, current state active+undersized+degraded, last acting [6,7] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 4.9 is stuck undersized for 4m, current state active+undersized+degraded, last acting [1,3] 2024-03-26T05:20:01.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 4.c is stuck undersized for 4m, current state active+undersized+degraded, last acting [3,6] 2024-03-26T05:20:01.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 4.d is stuck undersized for 4m, current state active+undersized+degraded, last acting [2,1] 2024-03-26T05:20:01.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 4.e is stuck undersized for 4m, current state active+undersized+degraded, last acting [6,1] 2024-03-26T05:20:01.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 4.f is stuck undersized for 4m, current state active+undersized+degraded, last acting [1,3] 2024-03-26T05:20:01.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 4.13 is stuck undersized for 4m, current state active+undersized+degraded, last acting [6,1] 2024-03-26T05:20:01.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: 1.9 scrub starts 2024-03-26T05:20:01.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: 1.9 scrub ok 2024-03-26T05:20:01.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: 4.1c scrub starts 2024-03-26T05:20:01.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: 4.1c scrub ok 2024-03-26T05:20:01.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 24146/289005 objects degraded (8.355%), 25 pgs degraded, 62 pgs undersized 2024-03-26T05:20:01.119 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: [WRN] OSD_DOWN: 1 osds down 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: osd.4 (root=default,host=smithi132) is down 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: [WRN] PG_DEGRADED: Degraded data redundancy: 24146/289005 objects degraded (8.355%), 25 pgs degraded, 62 pgs undersized 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 1.2 is stuck undersized for 4m, current state active+undersized+degraded, last acting [7,0] 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 1.3 is stuck undersized for 4m, current state active+undersized, last acting [1,6] 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 1.5 is stuck undersized for 4m, current state active+undersized, last acting [2,6] 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 3.1 is stuck undersized for 4m, current state active+undersized, last acting [0,5] 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 3.3 is stuck undersized for 4m, current state active+undersized, last acting [0,6] 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 3.6 is stuck undersized for 4m, current state active+undersized+degraded, last acting [0,1] 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 3.9 is stuck undersized for 4m, current state active+undersized, last acting [2,7] 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 3.e is stuck undersized for 4m, current state active+undersized, last acting [7,1] 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 3.f is stuck undersized for 4m, current state active+undersized+degraded, last acting [7,0] 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 3.11 is stuck undersized for 4m, current state active+undersized, last acting [7,6] 2024-03-26T05:20:01.120 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 3.13 is stuck undersized for 4m, current state active+undersized, last acting [7,2] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 3.14 is stuck undersized for 4m, current state active+undersized, last acting [7,6] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 3.15 is stuck undersized for 4m, current state active+undersized, last acting [7,3] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 3.19 is stuck undersized for 4m, current state active+undersized+degraded, last acting [1,3] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 3.1a is stuck undersized for 4m, current state active+undersized, last acting [1,2] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 4.1 is stuck undersized for 4m, current state active+undersized+degraded, last acting [5,6] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 4.2 is stuck undersized for 4m, current state active+undersized+degraded, last acting [2,5] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 4.5 is stuck undersized for 4m, current state active+undersized+degraded, last acting [6,7] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 4.9 is stuck undersized for 4m, current state active+undersized+degraded, last acting [1,3] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 4.c is stuck undersized for 4m, current state active+undersized+degraded, last acting [3,6] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 4.d is stuck undersized for 4m, current state active+undersized+degraded, last acting [2,1] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 4.e is stuck undersized for 4m, current state active+undersized+degraded, last acting [6,1] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 4.f is stuck undersized for 4m, current state active+undersized+degraded, last acting [1,3] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 4.13 is stuck undersized for 4m, current state active+undersized+degraded, last acting [6,1] 2024-03-26T05:20:01.121 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 4.1d is stuck undersized for 4m, current state active+undersized+degraded, last acting [3,1] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 5.0 is stuck undersized for 4m, current state active+undersized+degraded, last acting [3,1] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 5.1 is stuck undersized for 4m, current state active+undersized, last acting [3,7] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 5.5 is stuck undersized for 4m, current state active+undersized, last acting [0,1] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 5.9 is stuck undersized for 4m, current state active+undersized+degraded, last acting [7,6] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 5.a is stuck undersized for 4m, current state active+undersized, last acting [2,3] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 5.c is stuck undersized for 4m, current state active+undersized, last acting [1,0] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 5.e is stuck undersized for 4m, current state active+undersized+degraded, last acting [5,0] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 5.f is stuck undersized for 4m, current state active+undersized, last acting [5,6] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 5.10 is stuck undersized for 4m, current state active+undersized, last acting [7,3] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 5.11 is stuck undersized for 4m, current state active+undersized, last acting [6,7] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 5.1f is stuck undersized for 4m, current state active+undersized+degraded, last acting [6,7] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 6.2 is stuck undersized for 4m, current state active+undersized, last acting [3,2] 2024-03-26T05:20:01.122 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 6.6 is stuck undersized for 4m, current state active+undersized, last acting [3,7] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 6.7 is stuck undersized for 4m, current state active+undersized, last acting [5,3] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 6.b is stuck undersized for 4m, current state active+undersized, last acting [3,1] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 6.e is stuck undersized for 4m, current state active+undersized, last acting [1,2] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 6.f is stuck undersized for 4m, current state active+undersized, last acting [2,3] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 6.12 is stuck undersized for 4m, current state active+undersized, last acting [7,2] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 6.14 is stuck undersized for 4m, current state active+undersized, last acting [2,7] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 6.15 is stuck undersized for 4m, current state active+undersized, last acting [7,6] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 6.17 is stuck undersized for 4m, current state active+undersized, last acting [2,5] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 6.1d is stuck undersized for 4m, current state active+undersized, last acting [1,5] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 11.7 is stuck undersized for 81s, current state active+undersized+degraded, last acting [5,1] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 11.c is stuck undersized for 81s, current state active+undersized+degraded, last acting [1,2] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 11.d is stuck undersized for 81s, current state active+undersized+degraded, last acting [2,6] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pg 11.e is stuck undersized for 81s, current state active+undersized+degraded, last acting [1,7] 2024-03-26T05:20:01.123 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[40404]: pgmap v914: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 859 op/s; 24146/294507 objects degraded (8.199%); 5/294507 objects misplaced (0.002%) 2024-03-26T05:20:01.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 4.1d is stuck undersized for 4m, current state active+undersized+degraded, last acting [3,1] 2024-03-26T05:20:01.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 5.0 is stuck undersized for 4m, current state active+undersized+degraded, last acting [3,1] 2024-03-26T05:20:01.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 5.1 is stuck undersized for 4m, current state active+undersized, last acting [3,7] 2024-03-26T05:20:01.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 5.5 is stuck undersized for 4m, current state active+undersized, last acting [0,1] 2024-03-26T05:20:01.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 5.9 is stuck undersized for 4m, current state active+undersized+degraded, last acting [7,6] 2024-03-26T05:20:01.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 5.a is stuck undersized for 4m, current state active+undersized, last acting [2,3] 2024-03-26T05:20:01.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 5.c is stuck undersized for 4m, current state active+undersized, last acting [1,0] 2024-03-26T05:20:01.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 5.e is stuck undersized for 4m, current state active+undersized+degraded, last acting [5,0] 2024-03-26T05:20:01.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 5.f is stuck undersized for 4m, current state active+undersized, last acting [5,6] 2024-03-26T05:20:01.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 5.10 is stuck undersized for 4m, current state active+undersized, last acting [7,3] 2024-03-26T05:20:01.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 5.11 is stuck undersized for 4m, current state active+undersized, last acting [6,7] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 5.1f is stuck undersized for 4m, current state active+undersized+degraded, last acting [6,7] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 6.2 is stuck undersized for 4m, current state active+undersized, last acting [3,2] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 6.6 is stuck undersized for 4m, current state active+undersized, last acting [3,7] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 6.7 is stuck undersized for 4m, current state active+undersized, last acting [5,3] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 6.b is stuck undersized for 4m, current state active+undersized, last acting [3,1] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 6.e is stuck undersized for 4m, current state active+undersized, last acting [1,2] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 6.f is stuck undersized for 4m, current state active+undersized, last acting [2,3] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 6.12 is stuck undersized for 4m, current state active+undersized, last acting [7,2] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 6.14 is stuck undersized for 4m, current state active+undersized, last acting [2,7] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 6.15 is stuck undersized for 4m, current state active+undersized, last acting [7,6] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 6.17 is stuck undersized for 4m, current state active+undersized, last acting [2,5] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 6.1d is stuck undersized for 4m, current state active+undersized, last acting [1,5] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 11.7 is stuck undersized for 81s, current state active+undersized+degraded, last acting [5,1] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 11.c is stuck undersized for 81s, current state active+undersized+degraded, last acting [1,2] 2024-03-26T05:20:01.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 11.d is stuck undersized for 81s, current state active+undersized+degraded, last acting [2,6] 2024-03-26T05:20:01.126 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pg 11.e is stuck undersized for 81s, current state active+undersized+degraded, last acting [1,7] 2024-03-26T05:20:01.126 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:00 smithi080 ceph-mon[29691]: pgmap v914: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 859 op/s; 24146/294507 objects degraded (8.199%); 5/294507 objects misplaced (0.002%) 2024-03-26T05:20:01.184 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:01.220 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:01.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:01.322 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:01.351 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:01.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:01.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:01.453 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:01.484 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:01.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:01.586 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:01.618 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:01.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:01.720 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:01.744 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:01.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:01.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:01.856 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T05:20:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:01.875 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:01.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:01.975 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 70 16 102310 102294 91.3136 62.6875 0.00749559 0.0109466 2024-03-26T05:20:01.976 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:02.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:02.010 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:02.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:02.111 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:02.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:01 smithi080 ceph-mon[40404]: 4.4 scrub starts 2024-03-26T05:20:02.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:01 smithi080 ceph-mon[40404]: 4.4 scrub ok 2024-03-26T05:20:02.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:01 smithi080 ceph-mon[40404]: 11.0 scrub ok 2024-03-26T05:20:02.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:01 smithi080 ceph-mon[40404]: 6.1f deep-scrub starts 2024-03-26T05:20:02.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:01 smithi080 ceph-mon[40404]: 6.1f deep-scrub ok 2024-03-26T05:20:02.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:01 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 24146/294507 objects degraded (8.199%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:20:02.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:01 smithi080 ceph-mon[29691]: 4.4 scrub starts 2024-03-26T05:20:02.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:01 smithi080 ceph-mon[29691]: 4.4 scrub ok 2024-03-26T05:20:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:01 smithi080 ceph-mon[29691]: 11.0 scrub ok 2024-03-26T05:20:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:01 smithi080 ceph-mon[29691]: 6.1f deep-scrub starts 2024-03-26T05:20:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:01 smithi080 ceph-mon[29691]: 6.1f deep-scrub ok 2024-03-26T05:20:02.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:01 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 24146/294507 objects degraded (8.199%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:20:02.149 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:02.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:02.238 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:20:02.254 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:02.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:01 smithi132 ceph-mon[33500]: 4.4 scrub starts 2024-03-26T05:20:02.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:01 smithi132 ceph-mon[33500]: 4.4 scrub ok 2024-03-26T05:20:02.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:01 smithi132 ceph-mon[33500]: 11.0 scrub ok 2024-03-26T05:20:02.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:01 smithi132 ceph-mon[33500]: 6.1f deep-scrub starts 2024-03-26T05:20:02.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:01 smithi132 ceph-mon[33500]: 6.1f deep-scrub ok 2024-03-26T05:20:02.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:01 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 24146/294507 objects degraded (8.199%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:20:02.299 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:02.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:02.387 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:02.401 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:02.457 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:02.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:02.558 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:02.602 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:02.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:02.703 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:02.743 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:02.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:02.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:02.815 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:20:02.816 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:20:02.845 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:02.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:02.884 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:02.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:02.975 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 71 16 103494 103478 91.0697 74 0.00703045 0.0109763 2024-03-26T05:20:02.985 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:03.038 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:03.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:03.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:02 smithi080 ceph-mon[29691]: 4.12 scrub starts 2024-03-26T05:20:03.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:02 smithi080 ceph-mon[29691]: 4.12 scrub ok 2024-03-26T05:20:03.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:02 smithi080 ceph-mon[29691]: pgmap v915: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s; 25376/303666 objects degraded (8.357%); 5/303666 objects misplaced (0.002%) 2024-03-26T05:20:03.113 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:02 smithi080 ceph-mon[40404]: 4.12 scrub starts 2024-03-26T05:20:03.113 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:02 smithi080 ceph-mon[40404]: 4.12 scrub ok 2024-03-26T05:20:03.113 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:02 smithi080 ceph-mon[40404]: pgmap v915: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s; 25376/303666 objects degraded (8.357%); 5/303666 objects misplaced (0.002%) 2024-03-26T05:20:03.140 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:03.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:03.206 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:03.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:03.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:02 smithi132 ceph-mon[33500]: 4.12 scrub starts 2024-03-26T05:20:03.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:02 smithi132 ceph-mon[33500]: 4.12 scrub ok 2024-03-26T05:20:03.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:02 smithi132 ceph-mon[33500]: pgmap v915: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s; 25376/303666 objects degraded (8.357%); 5/303666 objects misplaced (0.002%) 2024-03-26T05:20:03.307 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:03.346 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:03.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:03.448 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:03.481 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:03.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:03.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:03.587 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:03.630 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:03.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:03.732 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:03.770 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:03.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:03.871 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:03.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:03.912 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:03.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:03.975 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 72 16 104712 104696 90.8621 76.125 0.00945885 0.0110014 2024-03-26T05:20:04.013 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:04.054 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:04.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:04.155 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:04.191 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:04.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:04.220 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:20:04.220 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:20:04.220 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.220 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:104974 11.3 11:ca922857:::benchmark_data_smithi080_189724_object104973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-26T05:20:04.220 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:04.221518+0000", 2024-03-26T05:20:04.220 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.0064955880000000001, 2024-03-26T05:20:04.220 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0065301730000000002, 2024-03-26T05:20:04.220 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:04.220 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:20:04.220 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 104974 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.221518+0000", 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.221518+0000", 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.222 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:04.222 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.221519+0000", 2024-03-26T05:20:04.222 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.798e-06 2024-03-26T05:20:04.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.222 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:04.222 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.221559+0000", 2024-03-26T05:20:04.222 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0061999999999997e-05 2024-03-26T05:20:04.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.222 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.221571+0000", 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1605e-05 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.221579+0000", 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8529999999999992e-06 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.221643+0000", 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4463999999999998e-05 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.223 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.224 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:04.224 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.221791+0000", 2024-03-26T05:20:04.224 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000148121 2024-03-26T05:20:04.224 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.224 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.224 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,7", 2024-03-26T05:20:04.224 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.222532+0000", 2024-03-26T05:20:04.224 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00074069499999999998 2024-03-26T05:20:04.224 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.224 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.224 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:20:04.224 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.226178+0000", 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0036458939999999998 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227767+0000", 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0015889350000000001 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228036+0000", 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00026932600000000002 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.225 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.15930.0:104973 11.8 e431/408 11:1f81fb0f:::benchmark_data_smithi080_189724_object104972:head v 431'6616, mlcod=431'6614)", 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:04.224452+0000", 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.003560821, 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.003712632, 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.224452+0000", 2024-03-26T05:20:04.226 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.224452+0000", 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.224453+0000", 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6300000000000004e-07 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.227 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.224488+0000", 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.4826999999999997e-05 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.224503+0000", 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5367999999999999e-05 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.224511+0000", 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7349999999999996e-06 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.228 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.224571+0000", 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0078999999999999e-05 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.224598+0000", 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.6769000000000001e-05 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.229 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.230 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.15930.0:104977 11.8 e431/408 11:1028b80f:::benchmark_data_smithi080_189724_object104976:head v 431'6617, mlcod=431'6614)", 2024-03-26T05:20:04.230 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:04.226893+0000", 2024-03-26T05:20:04.230 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.001119955, 2024-03-26T05:20:04.230 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0013347190000000001, 2024-03-26T05:20:04.230 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:04.230 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:20:04.230 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:04.230 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.230 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:04.230 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.226893+0000", 2024-03-26T05:20:04.230 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.230 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.226893+0000", 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.226894+0000", 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5199999999999996e-07 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:04.231 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.226931+0000", 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7197000000000003e-05 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.226943+0000", 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1856e-05 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.226949+0000", 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3659999999999997e-06 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.232 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227426+0000", 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00047714300000000002 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227446+0000", 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.9882e-05 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:104980 11.1 11:82595694:::benchmark_data_smithi080_189724_object104979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:04.227140+0000", 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.00087279299999999996, 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011575240000000001, 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 104980 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:04.234 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.235 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:04.235 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227140+0000", 2024-03-26T05:20:04.235 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.235 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.235 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.235 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:04.235 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227140+0000", 2024-03-26T05:20:04.235 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.235 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.235 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.235 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:04.235 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227142+0000", 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.432e-06 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227172+0000", 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0091e-05 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227181+0000", 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3230000000000004e-06 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.236 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227188+0000", 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6359999999999997e-06 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227314+0000", 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012623199999999999 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227462+0000", 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014761200000000001 2024-03-26T05:20:04.237 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,3", 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227852+0000", 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000390627 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:104981 11.1 11.9b112da1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:04.227200+0000", 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.00081340099999999995, 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011743109999999999, 2024-03-26T05:20:04.238 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "queued for pg", 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 104981 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227200+0000", 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.239 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227200+0000", 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227200+0000", 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.68e-07 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227227+0000", 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.6486e-05 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.240 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227234+0000", 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9659999999999999e-06 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227240+0000", 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8200000000000002e-06 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.15930.0:104966 11.4 e431/408 11:23176ada:::benchmark_data_smithi080_189724_object104965:head v 431'6630, mlcod=431'6628)", 2024-03-26T05:20:04.241 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:04.227638+0000", 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.00037532100000000002, 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079252200000000004, 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227638+0000", 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:04.242 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227638+0000", 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227638+0000", 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2900000000000004e-07 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227660+0000", 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.2146000000000001e-05 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.243 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227672+0000", 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1127999999999999e-05 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227679+0000", 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1629999999999999e-06 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227725+0000", 2024-03-26T05:20:04.244 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6026000000000003e-05 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.227768+0000", 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3142000000000001e-05 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "replica_scrub(pg: 11.4,from:0'0,to:431'6611,epoch:431/408,start:11:21729a3a:::benchmark_data_smithi080_189724_object36197:0,end:11:21ab60e6:::benchmark_data_smithi080_189724_object66300:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:04.228374+0000", 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 4294967295.9996395, 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000129908, 2024-03-26T05:20:04.245 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "queued for pg", 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228374+0000", 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228374+0000", 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.247 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:04.247 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228374+0000", 2024-03-26T05:20:04.247 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6000000000000004e-07 2024-03-26T05:20:04.247 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.247 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.247 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:04.247 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228379+0000", 2024-03-26T05:20:04.247 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0520000000000001e-06 2024-03-26T05:20:04.247 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.247 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.247 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:04.247 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228391+0000", 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1681e-05 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228400+0000", 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5550000000000008e-06 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.15930.0:104979 11.8 e431/408)", 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:04.228470+0000", 2024-03-26T05:20:04.248 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 4294967295.9995427, 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9114000000000002e-05, 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "queued for pg", 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228470+0000", 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228470+0000", 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:20:04.249 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228471+0000", 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4700000000000002e-07 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228499+0000", 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.8345999999999999e-05 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228511+0000", 2024-03-26T05:20:04.250 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2125e-05 2024-03-26T05:20:04.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:20:04.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:20:04.251 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:20:04.251 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:04.228517+0000", 2024-03-26T05:20:04.251 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9880000000000001e-06 2024-03-26T05:20:04.251 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.251 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:20:04.251 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.251 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:20:04.251 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-03-26T05:20:04.251 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 8 2024-03-26T05:20:04.251 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:20:04.281 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:04 smithi080 ceph-mon[29691]: 11.4 scrub starts 2024-03-26T05:20:04.281 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:20:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:20:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:20:04.282 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:04 smithi080 ceph-mon[40404]: 11.4 scrub starts 2024-03-26T05:20:04.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:04 smithi132 ceph-mon[33500]: 11.4 scrub starts 2024-03-26T05:20:04.293 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:04.335 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:04.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:04.437 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:04.482 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:04.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:04.584 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:04.619 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:04.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:04.720 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:04.757 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:04.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:04.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:04.817 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:20:04.859 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:04.894 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:04.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:04.975 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 73 16 105766 105750 90.5197 65.875 0.0132204 0.011042 2024-03-26T05:20:04.995 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:05.030 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:05.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:05.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:05.112 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T05:20:05.132 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:05.194 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:05.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:05.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:05.296 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:05.326 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:05.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:05.402 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:05.428 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:05.446 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:20:05.458 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:05.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:05.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:05.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:05 smithi132 ceph-mon[33500]: pgmap v916: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 25376/303666 objects degraded (8.357%); 5/303666 objects misplaced (0.002%) 2024-03-26T05:20:05.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:05 smithi132 ceph-mon[33500]: 6.a scrub starts 2024-03-26T05:20:05.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:05 smithi132 ceph-mon[33500]: 6.a scrub ok 2024-03-26T05:20:05.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:05 smithi132 ceph-mon[33500]: 4.17 scrub starts 2024-03-26T05:20:05.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:05 smithi132 ceph-mon[33500]: 4.17 scrub ok 2024-03-26T05:20:05.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:05 smithi132 ceph-mon[33500]: 6.0 deep-scrub starts 2024-03-26T05:20:05.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:05 smithi132 ceph-mon[33500]: 6.0 deep-scrub ok 2024-03-26T05:20:05.546 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[29691]: pgmap v916: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 25376/303666 objects degraded (8.357%); 5/303666 objects misplaced (0.002%) 2024-03-26T05:20:05.547 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[29691]: 6.a scrub starts 2024-03-26T05:20:05.547 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[29691]: 6.a scrub ok 2024-03-26T05:20:05.547 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[29691]: 4.17 scrub starts 2024-03-26T05:20:05.547 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[29691]: 4.17 scrub ok 2024-03-26T05:20:05.547 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[29691]: 6.0 deep-scrub starts 2024-03-26T05:20:05.547 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[29691]: 6.0 deep-scrub ok 2024-03-26T05:20:05.547 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[40404]: pgmap v916: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 25376/303666 objects degraded (8.357%); 5/303666 objects misplaced (0.002%) 2024-03-26T05:20:05.547 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[40404]: 6.a scrub starts 2024-03-26T05:20:05.547 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[40404]: 6.a scrub ok 2024-03-26T05:20:05.547 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[40404]: 4.17 scrub starts 2024-03-26T05:20:05.547 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[40404]: 4.17 scrub ok 2024-03-26T05:20:05.547 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[40404]: 6.0 deep-scrub starts 2024-03-26T05:20:05.548 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:05 smithi080 ceph-mon[40404]: 6.0 deep-scrub ok 2024-03-26T05:20:05.560 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:05.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:05.614 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:05.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:05.715 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:05.755 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:05.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:05.857 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:05.885 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:05.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:05.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:05.975 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 74 16 106669 106653 90.0591 56.4375 0.0204121 0.0110994 2024-03-26T05:20:05.987 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:06.017 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:06.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:06.118 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:06.150 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:06.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:06.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:20:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:20:06.256 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:06.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:06.311 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:06.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:06.412 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:06.442 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:06.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:06.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:06 smithi132 ceph-mon[33500]: 11.2 scrub starts 2024-03-26T05:20:06.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:06 smithi132 ceph-mon[33500]: 4.0 scrub starts 2024-03-26T05:20:06.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:06 smithi132 ceph-mon[33500]: 4.0 scrub ok 2024-03-26T05:20:06.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:06 smithi132 ceph-mon[33500]: 4.6 deep-scrub starts 2024-03-26T05:20:06.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:06 smithi132 ceph-mon[33500]: 4.6 deep-scrub ok 2024-03-26T05:20:06.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:06 smithi132 ceph-mon[33500]: 5.12 scrub starts 2024-03-26T05:20:06.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:06 smithi132 ceph-mon[33500]: 5.12 scrub ok 2024-03-26T05:20:06.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:06 smithi132 ceph-mon[33500]: 11.4 scrub ok 2024-03-26T05:20:06.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:06.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:20:06.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:06 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2705657245' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:20:06.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:06 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:20:06.544 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:06.575 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:06.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:06.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[29691]: 11.2 scrub starts 2024-03-26T05:20:06.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[29691]: 4.0 scrub starts 2024-03-26T05:20:06.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[29691]: 4.0 scrub ok 2024-03-26T05:20:06.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[29691]: 4.6 deep-scrub starts 2024-03-26T05:20:06.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[29691]: 4.6 deep-scrub ok 2024-03-26T05:20:06.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[29691]: 5.12 scrub starts 2024-03-26T05:20:06.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[29691]: 5.12 scrub ok 2024-03-26T05:20:06.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[29691]: 11.4 scrub ok 2024-03-26T05:20:06.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:06.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:20:06.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2705657245' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:20:06.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:20:06.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[40404]: 11.2 scrub starts 2024-03-26T05:20:06.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[40404]: 4.0 scrub starts 2024-03-26T05:20:06.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[40404]: 4.0 scrub ok 2024-03-26T05:20:06.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[40404]: 4.6 deep-scrub starts 2024-03-26T05:20:06.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[40404]: 4.6 deep-scrub ok 2024-03-26T05:20:06.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[40404]: 5.12 scrub starts 2024-03-26T05:20:06.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[40404]: 5.12 scrub ok 2024-03-26T05:20:06.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[40404]: 11.4 scrub ok 2024-03-26T05:20:06.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:06.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:20:06.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2705657245' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:20:06.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:06 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:20:06.676 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:06.716 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:06.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:06.774 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:20:06.818 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:06.866 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:06.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:06.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:06.978 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 75 16 107700 107684 89.7175 64.4375 0.00723798 0.0111412 2024-03-26T05:20:06.996 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:06.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:07.098 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:07.130 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:07.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:07.232 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:07.262 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:07.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:07.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:07.364 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:07.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:07.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[29691]: pgmap v917: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s; 26746/318798 objects degraded (8.390%); 5/318798 objects misplaced (0.002%) 2024-03-26T05:20:07.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[29691]: 6.19 scrub starts 2024-03-26T05:20:07.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[29691]: 6.19 scrub ok 2024-03-26T05:20:07.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 25376/303666 objects degraded (8.357%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:20:07.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:20:07.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:20:07.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[29691]: osdmap e432: 8 total, 7 up, 8 in 2024-03-26T05:20:07.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:07.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[40404]: pgmap v917: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s; 26746/318798 objects degraded (8.390%); 5/318798 objects misplaced (0.002%) 2024-03-26T05:20:07.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[40404]: 6.19 scrub starts 2024-03-26T05:20:07.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[40404]: 6.19 scrub ok 2024-03-26T05:20:07.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 25376/303666 objects degraded (8.357%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:20:07.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:20:07.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:20:07.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:07 smithi080 ceph-mon[40404]: osdmap e432: 8 total, 7 up, 8 in 2024-03-26T05:20:07.386 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T05:20:07.387 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:20:07.387 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T05:20:07.387 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd out 4 2024-03-26T05:20:07.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:07.398 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:07.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:07.468 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:07 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:07.468 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:07 smithi132 ceph-mon[33500]: pgmap v917: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s; 26746/318798 objects degraded (8.390%); 5/318798 objects misplaced (0.002%) 2024-03-26T05:20:07.469 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:07 smithi132 ceph-mon[33500]: 6.19 scrub starts 2024-03-26T05:20:07.469 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:07 smithi132 ceph-mon[33500]: 6.19 scrub ok 2024-03-26T05:20:07.469 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:07 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 25376/303666 objects degraded (8.357%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:20:07.469 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:07 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:20:07.469 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:07 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:20:07.469 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:07 smithi132 ceph-mon[33500]: osdmap e432: 8 total, 7 up, 8 in 2024-03-26T05:20:07.499 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:07.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:07.549 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:07.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:07.637 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:20:07.637 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:20:07.637 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2024-03-26T05:20:07.637 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2024-03-26T05:20:07.637 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:20:07.649 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:07.684 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:07.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:07.785 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:07.820 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:07.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:07.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:07.921 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:07.952 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:07.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:07.975 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 76 16 108818 108802 89.4564 69.875 0.010356 0.0111743 2024-03-26T05:20:08.053 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:08.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:08.077 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:08.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:08.177 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:08.206 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:08.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:08.307 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:08.342 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:08.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:08.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:08.406 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:08 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:20:08.407 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:08 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:20:08.444 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:08.446 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T05:20:08.488 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:08.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:08.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:08.506 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:08 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:20:08.506 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:08 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:20:08.507 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:08 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:20:08.507 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:08 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:20:08.578 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:08.590 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:08.622 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:08.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:08.647 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:20:08.691 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:20:08.724 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:08.736 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:20:08.763 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:08.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:08.774 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:20:08.786 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:20:08.815 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:20:08.855 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:20:08.865 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:08.895 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:20:08.895 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:20:08.895 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:20:08.895 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:20:08.895 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:20:08.896 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:20:08.896 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:20:08.896 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:20:08.896 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:20:08.898 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:08.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:08.976 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 77 16 109750 109734 89.0511 58.25 0.00718255 0.0112252 2024-03-26T05:20:09.000 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:09.040 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:09.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:09.142 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:09.171 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:09.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:09.273 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:09.318 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:09.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:09.418 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:09 smithi080 ceph-mon[40404]: pgmap v919: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s; 26746/319752 objects degraded (8.365%); 5/319752 objects misplaced (0.002%) 2024-03-26T05:20:09.418 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:09 smithi080 ceph-mon[40404]: osdmap e433: 8 total, 7 up, 8 in 2024-03-26T05:20:09.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:09 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/858374140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:20:09.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:09 smithi080 ceph-mon[29691]: pgmap v919: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s; 26746/319752 objects degraded (8.365%); 5/319752 objects misplaced (0.002%) 2024-03-26T05:20:09.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:09 smithi080 ceph-mon[29691]: osdmap e433: 8 total, 7 up, 8 in 2024-03-26T05:20:09.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:09 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/858374140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:20:09.419 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:09.466 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:09.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:09.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:09 smithi132 ceph-mon[33500]: pgmap v919: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s; 26746/319752 objects degraded (8.365%); 5/319752 objects misplaced (0.002%) 2024-03-26T05:20:09.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:09 smithi132 ceph-mon[33500]: osdmap e433: 8 total, 7 up, 8 in 2024-03-26T05:20:09.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:09 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/858374140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:20:09.567 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:09.601 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:09.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:09.703 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:09.728 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:09.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:09.830 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:09.857 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:09.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:09.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:09.959 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:09.975 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 78 16 110954 110938 88.8741 75.25 0.0156228 0.0112467 2024-03-26T05:20:09.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:09.989 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:09.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:10.091 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:10.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:10.135 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:10.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:20:10.156 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:20:10.236 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:10.271 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:10.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:10.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:10.373 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:10.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:10.413 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:10.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:10.515 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:10.554 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:10.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:10.656 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:10.689 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:10.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:10.717 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:20:10.718 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:20:10.790 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:10.829 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:10.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:10.931 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:10.975 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:10.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:10.976 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 79 16 112169 112153 88.7103 75.9375 0.0136361 0.0112682 2024-03-26T05:20:11.077 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:11.116 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:11.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:11.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:11.193 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:20:11.218 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:11.246 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:11.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:11.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:11.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:11.393 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:11.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:11.445 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T05:20:11.495 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:11.535 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:11 smithi080 ceph-mon[29691]: pgmap v921: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 6.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 915 op/s; 26746/325623 objects degraded (8.214%); 5/325623 objects misplaced (0.002%) 2024-03-26T05:20:11.535 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:11 smithi080 ceph-mon[40404]: pgmap v921: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 6.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 915 op/s; 26746/325623 objects degraded (8.214%); 5/325623 objects misplaced (0.002%) 2024-03-26T05:20:11.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:11.541 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:11.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:11.642 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:11.675 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:11.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:11.776 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:11.784 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:20:11.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:11 smithi132 ceph-mon[33500]: pgmap v921: 158 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 37 active+undersized, 94 active+clean, 25 active+undersized+degraded; 6.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 915 op/s; 26746/325623 objects degraded (8.214%); 5/325623 objects misplaced (0.002%) 2024-03-26T05:20:11.814 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:11.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:11.915 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:11.945 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:11.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:11.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:11.976 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:2024-03-26T05:20:11.987197+0000 min lat: 0.0012195 max lat: 0.2094 avg lat: 0.011294 2024-03-26T05:20:11.976 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T05:20:11.976 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 80 16 113330 113314 88.5084 72.5625 0.020707 0.011294 2024-03-26T05:20:12.047 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:12.085 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:12.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:12.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:12.158 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:20:12.188 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:12.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:12.224 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:12.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:12.325 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:12.352 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:12.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:12.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:12.454 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:12.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:12.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:12.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:12.508 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:12 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 26746/325623 objects degraded (8.214%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:20:12.508 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:12 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 26746/325623 objects degraded (8.214%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:20:12.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:12 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 26746/325623 objects degraded (8.214%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T05:20:12.600 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:12.627 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:12.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:12.712 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:12.729 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:12.775 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:12.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:12.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:12.877 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:12.925 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:12.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:12.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:12.976 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 81 16 114334 114318 88.1902 62.75 0.0100449 0.0113335 2024-03-26T05:20:13.027 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:13.055 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:13.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:13.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:13.158 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:13.188 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:13.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:13.290 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:13.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:13.338 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:13.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:13.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:13 smithi080 ceph-mon[40404]: pgmap v922: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 28098/336048 objects degraded (8.361%); 5/336048 objects misplaced (0.001%) 2024-03-26T05:20:13.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:13 smithi080 ceph-mon[29691]: pgmap v922: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 28098/336048 objects degraded (8.361%); 5/336048 objects misplaced (0.001%) 2024-03-26T05:20:13.439 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:13.486 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:13 smithi132 ceph-mon[33500]: pgmap v922: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 28098/336048 objects degraded (8.361%); 5/336048 objects misplaced (0.001%) 2024-03-26T05:20:13.486 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:13 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:20:13.486 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:13 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:20:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:13.488 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:13.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:13.589 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:13.626 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:13.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:13.728 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:13.770 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:13.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:13.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:13 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:20:13.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:13 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:20:13.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:13 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:20:13.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:13 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:20:13.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:13 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a[29687]: 2024-03-26T05:20:13.421+0000 7f70d2504700 -1 mon.a@0(leader).osd e434 definitely_dead 0 2024-03-26T05:20:13.872 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:13.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:13.896 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:13.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:13.976 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 82 16 115237 115221 87.8029 56.4375 0.00352791 0.0113637 2024-03-26T05:20:13.997 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:14.026 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:14.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:14.114 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:20:14.114 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:20:14.127 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:14.167 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:14.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:14.268 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:14.309 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:14.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:14.337 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:20:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:20:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:20:14.367 INFO:teuthology.orchestra.run.smithi080.stderr:marked out osd.4. 2024-03-26T05:20:14.411 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:14.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:14.444 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:14.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:14.502 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:14 smithi132 ceph-mon[33500]: osdmap e434: 8 total, 7 up, 8 in 2024-03-26T05:20:14.502 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:14 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1135306632' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-26T05:20:14.502 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:14 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-26T05:20:14.502 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:14 smithi132 ceph-mon[33500]: Client client.admin marked osd.4 out, after it was down for 265 seconds 2024-03-26T05:20:14.545 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:14.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:14.595 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:14 smithi080 ceph-mon[40404]: osdmap e434: 8 total, 7 up, 8 in 2024-03-26T05:20:14.595 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:14 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1135306632' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-26T05:20:14.595 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:14 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-26T05:20:14.595 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:14 smithi080 ceph-mon[40404]: Client client.admin marked osd.4 out, after it was down for 265 seconds 2024-03-26T05:20:14.595 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:14 smithi080 ceph-mon[29691]: osdmap e434: 8 total, 7 up, 8 in 2024-03-26T05:20:14.596 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:14 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1135306632' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-26T05:20:14.596 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:14 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-26T05:20:14.596 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:14 smithi080 ceph-mon[29691]: Client client.admin marked osd.4 out, after it was down for 265 seconds 2024-03-26T05:20:14.672 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:14.697 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:14.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:14.798 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:14.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:14.823 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:14.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:14.855 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T05:20:14.924 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:14.947 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:14.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:14.976 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 83 16 115710 115694 87.1012 29.5625 0.00282897 0.0114011 2024-03-26T05:20:15.050 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:15.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:15.074 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:15.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:15.100 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:20:15.175 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:15.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:15.200 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:15.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:15.301 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:15.325 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:15.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:15.426 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:15.455 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:15.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:15.531 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:15 smithi132 ceph-mon[33500]: pgmap v924: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 688 B/s rd, 46 MiB/s wr, 731 op/s; 28098/336048 objects degraded (8.361%); 5/336048 objects misplaced (0.001%) 2024-03-26T05:20:15.531 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:15 smithi132 ceph-mon[33500]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T05:20:15.531 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:15 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-26T05:20:15.531 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:15 smithi132 ceph-mon[33500]: osdmap e435: 8 total, 7 up, 7 in 2024-03-26T05:20:15.558 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:15.582 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:15.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:15.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:15 smithi080 ceph-mon[29691]: pgmap v924: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 688 B/s rd, 46 MiB/s wr, 731 op/s; 28098/336048 objects degraded (8.361%); 5/336048 objects misplaced (0.001%) 2024-03-26T05:20:15.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:15 smithi080 ceph-mon[29691]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T05:20:15.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:15 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-26T05:20:15.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:15 smithi080 ceph-mon[29691]: osdmap e435: 8 total, 7 up, 7 in 2024-03-26T05:20:15.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:15 smithi080 ceph-mon[40404]: pgmap v924: 158 pgs: 1 active+clean+remapped, 37 active+undersized, 95 active+clean, 25 active+undersized+degraded; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 688 B/s rd, 46 MiB/s wr, 731 op/s; 28098/336048 objects degraded (8.361%); 5/336048 objects misplaced (0.001%) 2024-03-26T05:20:15.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:15 smithi080 ceph-mon[40404]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T05:20:15.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:15 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-26T05:20:15.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:15 smithi080 ceph-mon[40404]: osdmap e435: 8 total, 7 up, 7 in 2024-03-26T05:20:15.683 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:15.708 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:15.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:15.809 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:15.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:15.836 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:15.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:15.937 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:15.961 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:15.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:15.976 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 84 16 115710 115694 86.0644 0 - 0.0114011 2024-03-26T05:20:16.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:16.087 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:16.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:16.189 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:16.214 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:16.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:16.290 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:20:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:20:16.315 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:16.339 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:16.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:16.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:16.441 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:16.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:16.470 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:16.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:16.531 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:16 smithi080 ceph-mon[40404]: osdmap e436: 8 total, 7 up, 7 in 2024-03-26T05:20:16.531 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:16 smithi080 ceph-mon[40404]: osdmap e437: 8 total, 7 up, 7 in 2024-03-26T05:20:16.531 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:16 smithi080 ceph-mon[29691]: osdmap e436: 8 total, 7 up, 7 in 2024-03-26T05:20:16.531 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:16 smithi080 ceph-mon[29691]: osdmap e437: 8 total, 7 up, 7 in 2024-03-26T05:20:16.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:16 smithi132 ceph-mon[33500]: osdmap e436: 8 total, 7 up, 7 in 2024-03-26T05:20:16.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:16 smithi132 ceph-mon[33500]: osdmap e437: 8 total, 7 up, 7 in 2024-03-26T05:20:16.571 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:16.614 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:16.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:16.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:16.716 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:16.757 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:16.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:16.859 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:16.893 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:16.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:16.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:16.976 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 85 16 116155 116139 85.3791 13.9062 0.0100604 0.0117073 2024-03-26T05:20:16.995 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:17.028 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:17.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:17.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:17.130 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:17.142 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:17.157 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:17.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:17.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:17.259 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:17.282 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:20:17.282 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:20:17.310 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:17.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:17.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:17.411 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:17.443 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:17.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:17.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:17.449 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:20:17.449 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:20:17.449 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:17.449 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[40404]: pgmap v927: 158 pgs: 1 active+recovery_wait+degraded, 7 remapped+peering, 19 peering, 1 active+clean+remapped, 4 active+undersized, 125 active+clean, 1 active+undersized+degraded; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.7 KiB/s rd, 75 MiB/s wr, 1.21k op/s; 2/347310 objects degraded (0.001%); 5/347310 objects misplaced (0.001%); 242 B/s, 6 objects/s recovering 2024-03-26T05:20:17.450 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:20:17.450 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:20:17.450 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[40404]: 11.7 starting backfill to osd.0 from (0'0,0'0] MAX to 434'7036 2024-03-26T05:20:17.450 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[40404]: 11.c starting backfill to osd.7 from (0'0,0'0] MAX to 434'7371 2024-03-26T05:20:17.450 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[40404]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 434'7245 2024-03-26T05:20:17.450 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[40404]: 11.d starting backfill to osd.1 from (0'0,0'0] MAX to 434'7234 2024-03-26T05:20:17.450 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 2/347310 objects degraded (0.001%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T05:20:17.451 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[40404]: osdmap e438: 8 total, 7 up, 7 in 2024-03-26T05:20:17.451 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:20:17.451 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:20:17.451 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:17.452 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[29691]: pgmap v927: 158 pgs: 1 active+recovery_wait+degraded, 7 remapped+peering, 19 peering, 1 active+clean+remapped, 4 active+undersized, 125 active+clean, 1 active+undersized+degraded; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.7 KiB/s rd, 75 MiB/s wr, 1.21k op/s; 2/347310 objects degraded (0.001%); 5/347310 objects misplaced (0.001%); 242 B/s, 6 objects/s recovering 2024-03-26T05:20:17.452 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:20:17.452 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:20:17.452 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[29691]: 11.7 starting backfill to osd.0 from (0'0,0'0] MAX to 434'7036 2024-03-26T05:20:17.452 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[29691]: 11.c starting backfill to osd.7 from (0'0,0'0] MAX to 434'7371 2024-03-26T05:20:17.452 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[29691]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 434'7245 2024-03-26T05:20:17.452 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[29691]: 11.d starting backfill to osd.1 from (0'0,0'0] MAX to 434'7234 2024-03-26T05:20:17.453 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 2/347310 objects degraded (0.001%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T05:20:17.453 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:17 smithi080 ceph-mon[29691]: osdmap e438: 8 total, 7 up, 7 in 2024-03-26T05:20:17.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:17 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:20:17.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:17 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:20:17.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:17 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:17.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:17 smithi132 ceph-mon[33500]: pgmap v927: 158 pgs: 1 active+recovery_wait+degraded, 7 remapped+peering, 19 peering, 1 active+clean+remapped, 4 active+undersized, 125 active+clean, 1 active+undersized+degraded; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.7 KiB/s rd, 75 MiB/s wr, 1.21k op/s; 2/347310 objects degraded (0.001%); 5/347310 objects misplaced (0.001%); 242 B/s, 6 objects/s recovering 2024-03-26T05:20:17.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:17 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:20:17.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:17 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:20:17.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:17 smithi132 ceph-mon[33500]: 11.7 starting backfill to osd.0 from (0'0,0'0] MAX to 434'7036 2024-03-26T05:20:17.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:17 smithi132 ceph-mon[33500]: 11.c starting backfill to osd.7 from (0'0,0'0] MAX to 434'7371 2024-03-26T05:20:17.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:17 smithi132 ceph-mon[33500]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 434'7245 2024-03-26T05:20:17.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:17 smithi132 ceph-mon[33500]: 11.d starting backfill to osd.1 from (0'0,0'0] MAX to 434'7234 2024-03-26T05:20:17.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:17 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 2/347310 objects degraded (0.001%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T05:20:17.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:17 smithi132 ceph-mon[33500]: osdmap e438: 8 total, 7 up, 7 in 2024-03-26T05:20:17.546 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:17.590 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:17.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:17.692 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:17.725 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:20:17.733 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:17.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:17.835 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:17.866 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:17.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:17.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:17.968 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:17.977 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 86 16 117130 117114 85.0946 60.9375 0.00806967 0.0117465 2024-03-26T05:20:18.010 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:18.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:18.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:18.112 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:18.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:18.151 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:18.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:18.252 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:18.275 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T05:20:18.294 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:18.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:18.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:18.395 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:18.437 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:18.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:18.539 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:18.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:18 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4098893255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:20:18.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:18 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:20:18.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:18 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4098893255' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:20:18.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:18 smithi132 ceph-mon[33500]: osdmap e439: 8 total, 7 up, 7 in 2024-03-26T05:20:18.574 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:18.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:18.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:18.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:18 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4098893255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:20:18.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:18 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:20:18.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:18 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4098893255' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:20:18.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:18 smithi080 ceph-mon[40404]: osdmap e439: 8 total, 7 up, 7 in 2024-03-26T05:20:18.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:18 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4098893255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:20:18.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:18 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:20:18.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:18 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4098893255' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:20:18.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:18 smithi080 ceph-mon[29691]: osdmap e439: 8 total, 7 up, 7 in 2024-03-26T05:20:18.618 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:20:18.676 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:18.725 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:18.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:18.827 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:18.867 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:18.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:18.968 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:18.977 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 87 16 118185 118169 84.8744 65.9375 0.011383 0.0117773 2024-03-26T05:20:19.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:19.012 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:19.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:19.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:19.164 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:19.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:19.266 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:19.307 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:19.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:19.408 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:19.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:19.457 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:19.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:19.559 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:19.588 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:19.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:19.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:19.690 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:19.726 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:19.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:19.827 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:19.856 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:19.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:19.958 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:19.977 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 88 16 119184 119168 84.6193 62.4375 0.0218002 0.0118126 2024-03-26T05:20:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:19.994 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:19.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:20.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:19 smithi132 ceph-mon[33500]: pgmap v931: 158 pgs: 1 active+recovery_wait+degraded, 7 remapped+peering, 19 peering, 131 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.3 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 1/347817 objects degraded (0.000%); 417 B/s, 12 objects/s recovering 2024-03-26T05:20:20.097 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:20.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:19 smithi080 ceph-mon[29691]: pgmap v931: 158 pgs: 1 active+recovery_wait+degraded, 7 remapped+peering, 19 peering, 131 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.3 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 1/347817 objects degraded (0.000%); 417 B/s, 12 objects/s recovering 2024-03-26T05:20:20.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:19 smithi080 ceph-mon[40404]: pgmap v931: 158 pgs: 1 active+recovery_wait+degraded, 7 remapped+peering, 19 peering, 131 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.3 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 1/347817 objects degraded (0.000%); 417 B/s, 12 objects/s recovering 2024-03-26T05:20:20.132 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:20.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:20.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:20.234 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:20.274 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:20.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:20.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:20.376 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:20.406 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:20.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:20.507 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:20.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:20.548 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:20.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:20.562 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2024-03-26T05:20:20.562 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:20:20.649 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:20.675 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:20.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:20.755 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:20:20.776 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:20.805 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:20.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:20.906 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:20.971 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:20.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:20.977 INFO:tasks.radosbench.radosbench.0.smithi080.stdout: 89 16 120220 120204 84.3958 64.75 0.00892605 0.0118442 2024-03-26T05:20:21.073 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:21.101 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:20:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:20:20.808+0000 7f2fcc605700 -1 mgr get_metadata_python Requested missing service osd.4 2024-03-26T05:20:21.102 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:20:21.125 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:21.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:21.147 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:20:21.191 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:20:21.228 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:21.236 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:20:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:21.275 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:21.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:21.276 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:20:21.316 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:20:21.356 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:20:21.356 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:20:21.356 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:20:21.357 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:20:21.357 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:20:21.357 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:20:21.357 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:20:21.357 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:20:21.357 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:20:21.376 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:21.389 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T05:20:21.426 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:21.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:21.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:21.529 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:21.575 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7, 5] out_osds: [4] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T05:20:21.575 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:20:21.576 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 6, 7, 5] 2024-03-26T05:20:21.576 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd out 5 2024-03-26T05:20:21.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:21.577 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:21.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:21.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:21.776 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:21.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:21.877 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:21.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:21.918 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:21.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:21.952 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:21 smithi080 ceph-mon[29691]: pgmap v932: 158 pgs: 1 active+recovery_wait+degraded, 7 remapped+peering, 150 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 724 op/s; 1/351885 objects degraded (0.000%); 520 B/s, 19 objects/s recovering 2024-03-26T05:20:21.952 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:21.952 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:20:21.952 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:21 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1352988729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:20:21.952 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:21 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 1/351885 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-26T05:20:21.953 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:21 smithi080 ceph-mon[40404]: pgmap v932: 158 pgs: 1 active+recovery_wait+degraded, 7 remapped+peering, 150 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 724 op/s; 1/351885 objects degraded (0.000%); 520 B/s, 19 objects/s recovering 2024-03-26T05:20:21.953 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:21.953 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:20:21.953 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:21 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1352988729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:20:21.953 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:21 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 1/351885 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-26T05:20:21.978 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Total time run: 90.0047 2024-03-26T05:20:21.978 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Total writes made: 121107 2024-03-26T05:20:21.978 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Write size: 65536 2024-03-26T05:20:21.978 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Object size: 65536 2024-03-26T05:20:21.978 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Bandwidth (MB/sec): 84.0977 2024-03-26T05:20:21.978 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev Bandwidth: 33.6761 2024-03-26T05:20:21.978 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max bandwidth (MB/sec): 217.25 2024-03-26T05:20:21.978 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min bandwidth (MB/sec): 0 2024-03-26T05:20:21.978 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Average IOPS: 1345 2024-03-26T05:20:21.978 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev IOPS: 538.829 2024-03-26T05:20:21.978 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max IOPS: 3476 2024-03-26T05:20:21.979 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min IOPS: 0 2024-03-26T05:20:21.979 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Average Latency(s): 0.0118871 2024-03-26T05:20:21.979 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Stddev Latency(s): 0.0254258 2024-03-26T05:20:21.979 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Max latency(s): 2.12503 2024-03-26T05:20:21.979 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Min latency(s): 0.0012195 2024-03-26T05:20:21.983 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T05:20:22.019 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:22.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:21 smithi132 ceph-mon[33500]: pgmap v932: 158 pgs: 1 active+recovery_wait+degraded, 7 remapped+peering, 150 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 724 op/s; 1/351885 objects degraded (0.000%); 520 B/s, 19 objects/s recovering 2024-03-26T05:20:22.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:22.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:20:22.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:21 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1352988729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:20:22.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:21 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 1/351885 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-26T05:20:22.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:22.065 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:22.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:22.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:22.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:22.203 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:22.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:22.304 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:22.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:22.333 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:22.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:20:22.410 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:20:22.411 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:22.433 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:22.485 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:22.506 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:22.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:22.607 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:22.650 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:22.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:22.753 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:22.786 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:22.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:22.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:22.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:22 smithi080 ceph-mon[29691]: pgmap v933: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 851 B/s rd, 45 MiB/s wr, 725 op/s; 29057/360393 objects degraded (8.063%); 9.8 MiB/s, 164 objects/s recovering 2024-03-26T05:20:22.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:22 smithi080 ceph-mon[29691]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:20:22.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:22 smithi080 ceph-mon[40404]: pgmap v933: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 851 B/s rd, 45 MiB/s wr, 725 op/s; 29057/360393 objects degraded (8.063%); 9.8 MiB/s, 164 objects/s recovering 2024-03-26T05:20:22.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:22 smithi080 ceph-mon[40404]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:20:22.888 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:22.914 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:22.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:23.015 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:23.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:23.054 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:23.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:23.155 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:23.198 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:23.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:23.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:23.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:22 smithi132 ceph-mon[33500]: pgmap v933: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 851 B/s rd, 45 MiB/s wr, 725 op/s; 29057/360393 objects degraded (8.063%); 9.8 MiB/s, 164 objects/s recovering 2024-03-26T05:20:23.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:22 smithi132 ceph-mon[33500]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:20:23.300 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:23.345 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:23.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:23.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:23.447 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:23.484 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:23.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:23.585 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:23.615 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:23.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:23.717 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:23.748 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:23.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:23.850 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:23.889 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:23.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:23.935 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:23.992 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:24.040 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:24.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:24.142 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:24.170 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:24.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:24.255 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:20:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:20:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:20:24.272 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:24.313 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:24.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:24.412 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:20:24.415 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:24.500 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:24.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:24.601 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:24.671 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:24.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:24.772 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:24.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:24.805 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:24.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:24.906 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:24.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:25.036 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:25.087 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:25.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:25.189 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:25.216 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:25.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:25.318 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:25.346 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:25.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:25.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:25.448 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:25.475 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:25.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:25.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:25.496 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:25 smithi080 ceph-mon[29691]: pgmap v934: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 650 B/s rd, 35 MiB/s wr, 554 op/s; 29057/360393 objects degraded (8.063%); 7.5 MiB/s, 126 objects/s recovering 2024-03-26T05:20:25.496 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:25 smithi080 ceph-mon[40404]: pgmap v934: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 650 B/s rd, 35 MiB/s wr, 554 op/s; 29057/360393 objects degraded (8.063%); 7.5 MiB/s, 126 objects/s recovering 2024-03-26T05:20:25.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:25 smithi132 ceph-mon[33500]: pgmap v934: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 650 B/s rd, 35 MiB/s wr, 554 op/s; 29057/360393 objects degraded (8.063%); 7.5 MiB/s, 126 objects/s recovering 2024-03-26T05:20:25.577 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:25.605 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:25.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:25.707 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:25.733 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:25.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:25.834 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:25.877 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:25.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:25.979 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:26.024 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:26.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:26.115 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:26.126 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:26.181 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:26.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:26.264 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:20:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:20:26.283 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:26.293 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:20:26.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:26.325 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:26.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:26.426 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:26.455 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:26.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:26.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:26.535 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:26 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1370163670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:20:26.535 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:26 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:20:26.535 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:26 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1370163670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:20:26.536 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:26 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:20:26.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:26 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1370163670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:20:26.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:26 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:20:26.557 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:26.609 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:26.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:26.710 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:26.739 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:26.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:26.841 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:26.883 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:26.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:26.985 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:27.034 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:27.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:27.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:27.148 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:27.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:27.191 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:27.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:27.292 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:27.323 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:27.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:27.425 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:27.452 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:27.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:27.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:27 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:27.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:27 smithi132 ceph-mon[33500]: pgmap v935: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 880 op/s; 28722/354798 objects degraded (8.095%); 6.2 MiB/s, 104 objects/s recovering 2024-03-26T05:20:27.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:27 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:20:27.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:27 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:20:27.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:27 smithi132 ceph-mon[33500]: osdmap e440: 8 total, 7 up, 7 in 2024-03-26T05:20:27.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:27 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 28722/354798 objects degraded (8.095%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:27.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:27 smithi132 ceph-mon[33500]: osdmap e441: 8 total, 7 up, 7 in 2024-03-26T05:20:27.554 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:27.586 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:27.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:27.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:27.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:27.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[29691]: pgmap v935: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 880 op/s; 28722/354798 objects degraded (8.095%); 6.2 MiB/s, 104 objects/s recovering 2024-03-26T05:20:27.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:20:27.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:20:27.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[29691]: osdmap e440: 8 total, 7 up, 7 in 2024-03-26T05:20:27.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 28722/354798 objects degraded (8.095%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:27.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[29691]: osdmap e441: 8 total, 7 up, 7 in 2024-03-26T05:20:27.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:27.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[40404]: pgmap v935: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 880 op/s; 28722/354798 objects degraded (8.095%); 6.2 MiB/s, 104 objects/s recovering 2024-03-26T05:20:27.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:20:27.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:20:27.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[40404]: osdmap e440: 8 total, 7 up, 7 in 2024-03-26T05:20:27.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 28722/354798 objects degraded (8.095%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:27.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:27 smithi080 ceph-mon[40404]: osdmap e441: 8 total, 7 up, 7 in 2024-03-26T05:20:27.688 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:27.728 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:27.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:27.829 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:27.873 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:27.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:27.975 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:28.017 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:28.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:28.119 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:28.164 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:28.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:28.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:28.266 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:28.293 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:28.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:28.396 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:28.423 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:28.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:28.434 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:20:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:20:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.434 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15930.0:128373 11.e e441/436 11:7960194a:::benchmark_data_smithi080_189724_object7263:head v 441'8051, mlcod=441'8048)", 2024-03-26T05:20:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:28.440229+0000", 2024-03-26T05:20:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0034743230000000001, 2024-03-26T05:20:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0035048319999999998, 2024-03-26T05:20:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:20:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:20:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:20:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:20:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440229+0000", 2024-03-26T05:20:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:20:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440229+0000", 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440229+0000", 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.75e-07 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440234+0000", 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1680000000000003e-06 2024-03-26T05:20:28.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:20:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440246+0000", 2024-03-26T05:20:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1970999999999999e-05 2024-03-26T05:20:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:20:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440253+0000", 2024-03-26T05:20:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4030000000000004e-06 2024-03-26T05:20:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.437 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:20:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440354+0000", 2024-03-26T05:20:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010166 2024-03-26T05:20:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:20:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440374+0000", 2024-03-26T05:20:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9862999999999999e-05 2024-03-26T05:20:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:20:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:20:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:20:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "MOSDPGPush(11.e 441/436 [PushOp(11:71f9eef5:::benchmark_data_smithi080_189724_object26584:head, version: 417'1682, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:71f9eef5:::benchmark_data_smithi080_189724_object26584:head@417'1682, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T05:20:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:28.440713+0000", 2024-03-26T05:20:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.0029898099999999999, 2024-03-26T05:20:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0031088690000000002, 2024-03-26T05:20:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:20:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:20:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:20:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:20:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440713+0000", 2024-03-26T05:20:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:20:28.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440713+0000", 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440714+0000", 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6400000000000002e-07 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440749+0000", 2024-03-26T05:20:28.440 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5672e-05 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440768+0000", 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8406999999999998e-05 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.440784+0000", 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6748e-05 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:20:28.441 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443736+0000", 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0029515079999999998 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443772+0000", 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6350999999999998e-05 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15930.0:128386 11.9 e441/408 11:9b54f935:::benchmark_data_smithi080_189724_object7276:head v 441'7921, mlcod=441'7919)", 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:28.443121+0000", 2024-03-26T05:20:28.442 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.00058197399999999997, 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00076950300000000005, 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443121+0000", 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443121+0000", 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:20:28.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443122+0000", 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5499999999999997e-07 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443129+0000", 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3549999999999999e-06 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:20:28.444 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443140+0000", 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0540000000000001e-05 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443147+0000", 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2969999999999998e-06 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443366+0000", 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000219075 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:20:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443386+0000", 2024-03-26T05:20:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0094000000000001e-05 2024-03-26T05:20:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:20:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:20:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:20:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_repop(client.15930.0:128383 11.b e441/408 11:da5318ee:::benchmark_data_smithi080_189724_object7273:head v 441'7944, mlcod=441'7939)", 2024-03-26T05:20:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:28.443170+0000", 2024-03-26T05:20:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.00053235800000000003, 2024-03-26T05:20:28.446 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00077921399999999997, 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "started", 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443170+0000", 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443170+0000", 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.447 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443171+0000", 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3200000000000001e-07 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443174+0000", 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5860000000000001e-06 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443181+0000", 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9999999999999999e-06 2024-03-26T05:20:28.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443186+0000", 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6789999999999996e-06 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443253+0000", 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6511000000000001e-05 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:20:28.449 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.443272+0000", 2024-03-26T05:20:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9434000000000001e-05 2024-03-26T05:20:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:20:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:20:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:20:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:128364 11.6 11:6e31b1a0:::benchmark_data_smithi080_189724_object7254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-03-26T05:20:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:28.414862+0000", 2024-03-26T05:20:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 0.028840679000000001, 2024-03-26T05:20:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.029151644000000001, 2024-03-26T05:20:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:20:28.450 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "waiting for sub ops", 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 128364 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.414862+0000", 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:20:28.451 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.414862+0000", 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.414864+0000", 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6440000000000001e-06 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.414869+0000", 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5860000000000004e-06 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.414885+0000", 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5341000000000001e-05 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.414893+0000", 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5569999999999996e-06 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.415396+0000", 2024-03-26T05:20:28.453 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00050314800000000003 2024-03-26T05:20:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:20:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.415706+0000", 2024-03-26T05:20:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00030933799999999998 2024-03-26T05:20:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,2", 2024-03-26T05:20:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.416067+0000", 2024-03-26T05:20:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00036125600000000001 2024-03-26T05:20:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.454 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.430420+0000", 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014353024000000001 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:28.433993+0000", 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0035734030000000002 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:20:28.455 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 5 2024-03-26T05:20:28.456 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:20:28.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:28 smithi080 ceph-mon[29691]: 3.16 scrub starts 2024-03-26T05:20:28.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:28 smithi080 ceph-mon[29691]: 3.16 scrub ok 2024-03-26T05:20:28.472 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:28 smithi080 ceph-mon[40404]: 3.16 scrub starts 2024-03-26T05:20:28.472 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:28 smithi080 ceph-mon[40404]: 3.16 scrub ok 2024-03-26T05:20:28.524 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:28.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:28 smithi132 ceph-mon[33500]: 3.16 scrub starts 2024-03-26T05:20:28.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:28 smithi132 ceph-mon[33500]: 3.16 scrub ok 2024-03-26T05:20:28.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:28.567 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:28.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:28.668 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:28.710 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:28.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:28.811 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:28.849 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:28.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:28.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:28.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:28 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a[29687]: 2024-03-26T05:20:28.579+0000 7f70d2504700 -1 mon.a@0(leader).osd e441 definitely_dead 0 2024-03-26T05:20:28.952 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:28.985 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:28.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:29.086 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:29.125 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:29.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:29.226 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:29.257 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:29.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:29.301 INFO:teuthology.orchestra.run.smithi080.stderr:marked out osd.5. 2024-03-26T05:20:29.357 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:29.398 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:29.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:29.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:29.406 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T05:20:29.501 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:29.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:29.530 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:29.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:29.566 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[29691]: 1.5 scrub starts 2024-03-26T05:20:29.566 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[29691]: 1.5 scrub ok 2024-03-26T05:20:29.566 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[29691]: pgmap v938: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.4 KiB/s rd, 32 MiB/s wr, 1.03k op/s; 27147/351435 objects degraded (7.725%); 16 MiB/s, 255 objects/s recovering 2024-03-26T05:20:29.566 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[29691]: 6.13 scrub starts 2024-03-26T05:20:29.567 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4187146463' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T05:20:29.567 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T05:20:29.567 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[29691]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-26T05:20:29.567 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[29691]: 6.13 scrub ok 2024-03-26T05:20:29.567 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[40404]: 1.5 scrub starts 2024-03-26T05:20:29.567 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[40404]: 1.5 scrub ok 2024-03-26T05:20:29.567 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[40404]: pgmap v938: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.4 KiB/s rd, 32 MiB/s wr, 1.03k op/s; 27147/351435 objects degraded (7.725%); 16 MiB/s, 255 objects/s recovering 2024-03-26T05:20:29.567 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[40404]: 6.13 scrub starts 2024-03-26T05:20:29.567 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4187146463' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T05:20:29.567 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T05:20:29.568 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[40404]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-26T05:20:29.568 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:29 smithi080 ceph-mon[40404]: 6.13 scrub ok 2024-03-26T05:20:29.631 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:29.655 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:29.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:29.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:29.756 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:29.786 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:29.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:29.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:29.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:29 smithi132 ceph-mon[33500]: 1.5 scrub starts 2024-03-26T05:20:29.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:29 smithi132 ceph-mon[33500]: 1.5 scrub ok 2024-03-26T05:20:29.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:29 smithi132 ceph-mon[33500]: pgmap v938: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.4 KiB/s rd, 32 MiB/s wr, 1.03k op/s; 27147/351435 objects degraded (7.725%); 16 MiB/s, 255 objects/s recovering 2024-03-26T05:20:29.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:29 smithi132 ceph-mon[33500]: 6.13 scrub starts 2024-03-26T05:20:29.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:29 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4187146463' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T05:20:29.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:29 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T05:20:29.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:29 smithi132 ceph-mon[33500]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-26T05:20:29.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:29 smithi132 ceph-mon[33500]: 6.13 scrub ok 2024-03-26T05:20:29.882 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:20:29.888 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:29.912 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:29.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:30.013 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:30.037 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:30.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:30.139 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:30.162 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:30.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:30.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:30.265 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:30.295 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:30.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:30.396 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:30.421 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:30.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:30.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:30.523 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:30.590 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:30 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-26T05:20:30.590 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:30 smithi080 ceph-mon[40404]: osdmap e442: 8 total, 7 up, 6 in 2024-03-26T05:20:30.591 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:30 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-26T05:20:30.591 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:30 smithi080 ceph-mon[29691]: osdmap e442: 8 total, 7 up, 6 in 2024-03-26T05:20:30.617 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:30.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:30.719 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:30.745 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:30.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:30.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:30 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-26T05:20:30.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:30 smithi132 ceph-mon[33500]: osdmap e442: 8 total, 7 up, 6 in 2024-03-26T05:20:30.847 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:30.880 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:30.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:30.982 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:31.009 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:31.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:31.112 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:31.140 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:31.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:31.241 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:31.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:31.266 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:31.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:31.367 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:31.397 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:31.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:31.477 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:31 smithi080 ceph-mon[40404]: 3.9 scrub starts 2024-03-26T05:20:31.477 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:31 smithi080 ceph-mon[40404]: 3.9 scrub ok 2024-03-26T05:20:31.478 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:31 smithi080 ceph-mon[40404]: pgmap v940: 158 pgs: 20 peering, 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 134 active+clean; 7.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 13 MiB/s wr, 1.20k op/s; 27147/346050 objects degraded (7.845%); 12 MiB/s, 183 objects/s recovering 2024-03-26T05:20:31.478 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:31 smithi080 ceph-mon[40404]: osdmap e443: 8 total, 7 up, 6 in 2024-03-26T05:20:31.478 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:31 smithi080 ceph-mon[40404]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-26T05:20:31.478 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:31 smithi080 ceph-mon[40404]: osdmap e444: 8 total, 7 up, 6 in 2024-03-26T05:20:31.478 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:31 smithi080 ceph-mon[29691]: 3.9 scrub starts 2024-03-26T05:20:31.478 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:31 smithi080 ceph-mon[29691]: 3.9 scrub ok 2024-03-26T05:20:31.478 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:31 smithi080 ceph-mon[29691]: pgmap v940: 158 pgs: 20 peering, 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 134 active+clean; 7.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 13 MiB/s wr, 1.20k op/s; 27147/346050 objects degraded (7.845%); 12 MiB/s, 183 objects/s recovering 2024-03-26T05:20:31.478 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:31 smithi080 ceph-mon[29691]: osdmap e443: 8 total, 7 up, 6 in 2024-03-26T05:20:31.478 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:31 smithi080 ceph-mon[29691]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-26T05:20:31.478 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:31 smithi080 ceph-mon[29691]: osdmap e444: 8 total, 7 up, 6 in 2024-03-26T05:20:31.498 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:31.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:31.524 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:31.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:31.586 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:31.625 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:31.637 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:31.662 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:31.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:31.676 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:20:31.718 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:20:31.750 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:20:31.764 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:31.786 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:20:31.789 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:31.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:31.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:31 smithi132 ceph-mon[33500]: 3.9 scrub starts 2024-03-26T05:20:31.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:31 smithi132 ceph-mon[33500]: 3.9 scrub ok 2024-03-26T05:20:31.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:31 smithi132 ceph-mon[33500]: pgmap v940: 158 pgs: 20 peering, 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 134 active+clean; 7.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 13 MiB/s wr, 1.20k op/s; 27147/346050 objects degraded (7.845%); 12 MiB/s, 183 objects/s recovering 2024-03-26T05:20:31.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:31 smithi132 ceph-mon[33500]: osdmap e443: 8 total, 7 up, 6 in 2024-03-26T05:20:31.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:31 smithi132 ceph-mon[33500]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-26T05:20:31.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:31 smithi132 ceph-mon[33500]: osdmap e444: 8 total, 7 up, 6 in 2024-03-26T05:20:31.824 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:20:31.864 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:20:31.891 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:31.902 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:20:31.902 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:20:31.902 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:20:31.902 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:20:31.902 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:20:31.902 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:20:31.902 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:20:31.902 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:20:31.902 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:20:31.916 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:31.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:32.018 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:32.047 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:32.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:32.148 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:32.191 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:32.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:32.292 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:32.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:32.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:32.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[29691]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 441'8029 2024-03-26T05:20:32.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[29691]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 441'8123 2024-03-26T05:20:32.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[29691]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 441'8162 2024-03-26T05:20:32.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[29691]: 11.7 continuing backfill to osd.0 from (420'2998,441'7902] MIN to 441'7902 2024-03-26T05:20:32.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[29691]: 11.7 starting backfill to osd.7 from (0'0,0'0] MAX to 441'7902 2024-03-26T05:20:32.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3193361609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:20:32.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 27147/346050 objects degraded (7.845%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:32.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[29691]: osdmap e445: 8 total, 7 up, 6 in 2024-03-26T05:20:32.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[40404]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 441'8029 2024-03-26T05:20:32.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[40404]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 441'8123 2024-03-26T05:20:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[40404]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 441'8162 2024-03-26T05:20:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[40404]: 11.7 continuing backfill to osd.0 from (420'2998,441'7902] MIN to 441'7902 2024-03-26T05:20:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[40404]: 11.7 starting backfill to osd.7 from (0'0,0'0] MAX to 441'7902 2024-03-26T05:20:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3193361609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:20:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 27147/346050 objects degraded (7.845%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:32.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:32 smithi080 ceph-mon[40404]: osdmap e445: 8 total, 7 up, 6 in 2024-03-26T05:20:32.426 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:32.464 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:32.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:32.566 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:32.598 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:32.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:32.699 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:32.741 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:32.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:32.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:32 smithi132 ceph-mon[33500]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 441'8029 2024-03-26T05:20:32.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:32 smithi132 ceph-mon[33500]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 441'8123 2024-03-26T05:20:32.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:32 smithi132 ceph-mon[33500]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 441'8162 2024-03-26T05:20:32.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:32 smithi132 ceph-mon[33500]: 11.7 continuing backfill to osd.0 from (420'2998,441'7902] MIN to 441'7902 2024-03-26T05:20:32.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:32 smithi132 ceph-mon[33500]: 11.7 starting backfill to osd.7 from (0'0,0'0] MAX to 441'7902 2024-03-26T05:20:32.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:32 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3193361609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:20:32.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:32 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 27147/346050 objects degraded (7.845%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:32.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:32 smithi132 ceph-mon[33500]: osdmap e445: 8 total, 7 up, 6 in 2024-03-26T05:20:32.843 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:32.878 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:32.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:32.980 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:33.004 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:33.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:33.105 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:33.133 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:33.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:33.235 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:33.262 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:33.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:33.364 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:33.392 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:33.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:33.494 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:33.522 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:33.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:33.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:33 smithi080 ceph-mon[29691]: 6.14 scrub starts 2024-03-26T05:20:33.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:33 smithi080 ceph-mon[29691]: 6.14 scrub ok 2024-03-26T05:20:33.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:33 smithi080 ceph-mon[29691]: pgmap v944: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 6 remapped+peering, 21 peering, 2 active+undersized+degraded+remapped+backfilling, 128 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 64 KiB/s rd, 109 KiB/s wr, 1.13k op/s; 16759/339054 objects degraded (4.943%); 41 MiB/s, 665 objects/s recovering 2024-03-26T05:20:33.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:33 smithi080 ceph-mon[29691]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:20:33.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:33 smithi080 ceph-mon[29691]: osdmap e446: 8 total, 7 up, 6 in 2024-03-26T05:20:33.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:33 smithi080 ceph-mon[40404]: 6.14 scrub starts 2024-03-26T05:20:33.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:33 smithi080 ceph-mon[40404]: 6.14 scrub ok 2024-03-26T05:20:33.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:33 smithi080 ceph-mon[40404]: pgmap v944: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 6 remapped+peering, 21 peering, 2 active+undersized+degraded+remapped+backfilling, 128 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 64 KiB/s rd, 109 KiB/s wr, 1.13k op/s; 16759/339054 objects degraded (4.943%); 41 MiB/s, 665 objects/s recovering 2024-03-26T05:20:33.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:33 smithi080 ceph-mon[40404]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:20:33.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:33 smithi080 ceph-mon[40404]: osdmap e446: 8 total, 7 up, 6 in 2024-03-26T05:20:33.623 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:33.652 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:33.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:33.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:33.754 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:33.783 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:33.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:33.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:33 smithi132 ceph-mon[33500]: 6.14 scrub starts 2024-03-26T05:20:33.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:33 smithi132 ceph-mon[33500]: 6.14 scrub ok 2024-03-26T05:20:33.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:33 smithi132 ceph-mon[33500]: pgmap v944: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 6 remapped+peering, 21 peering, 2 active+undersized+degraded+remapped+backfilling, 128 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 64 KiB/s rd, 109 KiB/s wr, 1.13k op/s; 16759/339054 objects degraded (4.943%); 41 MiB/s, 665 objects/s recovering 2024-03-26T05:20:33.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:33 smithi132 ceph-mon[33500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:20:33.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:33 smithi132 ceph-mon[33500]: osdmap e446: 8 total, 7 up, 6 in 2024-03-26T05:20:33.885 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:33.910 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:33.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:34.012 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:34.060 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:34.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:34.161 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:34.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:34.203 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:34.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:34.304 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:34.334 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:20:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:20:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:20:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:34.346 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:34.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:34.447 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:34.476 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:34.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:34.577 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:34.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:34 smithi080 ceph-mon[29691]: 1.a scrub starts 2024-03-26T05:20:34.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:34 smithi080 ceph-mon[29691]: 1.a scrub ok 2024-03-26T05:20:34.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:34 smithi080 ceph-mon[40404]: 1.a scrub starts 2024-03-26T05:20:34.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:34 smithi080 ceph-mon[40404]: 1.a scrub ok 2024-03-26T05:20:34.618 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:34.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:34.720 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:34.746 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:34.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:34.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:34 smithi132 ceph-mon[33500]: 1.a scrub starts 2024-03-26T05:20:34.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:34 smithi132 ceph-mon[33500]: 1.a scrub ok 2024-03-26T05:20:34.848 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:34.878 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:34.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:34.980 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:35.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:35.020 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:35.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:35.121 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:35.152 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:35.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:35.253 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:35.296 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:35.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:35.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:35.398 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:35.438 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:35.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:35.540 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:35.577 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:35.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:35.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:35 smithi080 ceph-mon[29691]: 1.8 scrub starts 2024-03-26T05:20:35.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:35 smithi080 ceph-mon[29691]: 1.8 scrub ok 2024-03-26T05:20:35.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:35 smithi080 ceph-mon[29691]: pgmap v946: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 6 remapped+peering, 21 peering, 2 active+undersized+degraded+remapped+backfilling, 128 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 64 KiB/s rd, 109 KiB/s wr, 684 op/s; 16759/339054 objects degraded (4.943%); 41 MiB/s, 667 objects/s recovering 2024-03-26T05:20:35.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:35 smithi080 ceph-mon[40404]: 1.8 scrub starts 2024-03-26T05:20:35.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:35 smithi080 ceph-mon[40404]: 1.8 scrub ok 2024-03-26T05:20:35.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:35 smithi080 ceph-mon[40404]: pgmap v946: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 6 remapped+peering, 21 peering, 2 active+undersized+degraded+remapped+backfilling, 128 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 64 KiB/s rd, 109 KiB/s wr, 684 op/s; 16759/339054 objects degraded (4.943%); 41 MiB/s, 667 objects/s recovering 2024-03-26T05:20:35.679 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:35.691 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:35 smithi132 ceph-mon[33500]: 1.8 scrub starts 2024-03-26T05:20:35.692 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:35 smithi132 ceph-mon[33500]: 1.8 scrub ok 2024-03-26T05:20:35.692 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:35 smithi132 ceph-mon[33500]: pgmap v946: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 6 remapped+peering, 21 peering, 2 active+undersized+degraded+remapped+backfilling, 128 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 64 KiB/s rd, 109 KiB/s wr, 684 op/s; 16759/339054 objects degraded (4.943%); 41 MiB/s, 667 objects/s recovering 2024-03-26T05:20:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:20:35.707 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:20:35.707 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:35.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:35.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:35.809 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:35.849 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:35.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:35.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:35.878 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7] out_osds: [4, 5] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T05:20:35.878 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:20:35.878 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-03-26T05:20:35.879 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:20:35.939 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:20:35 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:20:35.833+0000 7f2fcc605700 -1 mgr get_metadata_python Requested missing service osd.4 2024-03-26T05:20:35.951 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:36.015 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:36.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:36.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:20:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:20:36.116 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:36.141 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:36.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:36.243 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:36.272 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:36.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:36.374 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:36.403 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:36.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:36.505 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:36.543 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:36.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:36.645 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:36.684 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:36.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:36.786 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:36.826 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:36.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:36.927 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:36.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:36.966 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:36.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:37.067 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:37.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:36 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:37.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:36 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:20:37.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:37.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:36 smithi132 ceph-mon[33500]: pgmap v947: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 remapped+peering, 1 peering, 2 active+undersized+degraded+remapped+backfilling, 149 active+clean; 6.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 74 KiB/s rd, 0 B/s wr, 947 op/s; 16499/331416 objects degraded (4.978%); 20055/331416 objects misplaced (6.051%); 33 MiB/s, 540 objects/s recovering 2024-03-26T05:20:37.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:36 smithi132 ceph-mon[33500]: Health check update: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-26T05:20:37.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:36 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 16759/339054 objects degraded (4.943%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:37.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:36 smithi132 ceph-mon[33500]: osdmap e447: 8 total, 7 up, 6 in 2024-03-26T05:20:37.108 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:37.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:37.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:20:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[29691]: pgmap v947: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 remapped+peering, 1 peering, 2 active+undersized+degraded+remapped+backfilling, 149 active+clean; 6.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 74 KiB/s rd, 0 B/s wr, 947 op/s; 16499/331416 objects degraded (4.978%); 20055/331416 objects misplaced (6.051%); 33 MiB/s, 540 objects/s recovering 2024-03-26T05:20:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[29691]: Health check update: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-26T05:20:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 16759/339054 objects degraded (4.943%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[29691]: osdmap e447: 8 total, 7 up, 6 in 2024-03-26T05:20:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:20:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[40404]: pgmap v947: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 remapped+peering, 1 peering, 2 active+undersized+degraded+remapped+backfilling, 149 active+clean; 6.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 74 KiB/s rd, 0 B/s wr, 947 op/s; 16499/331416 objects degraded (4.978%); 20055/331416 objects misplaced (6.051%); 33 MiB/s, 540 objects/s recovering 2024-03-26T05:20:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[40404]: Health check update: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-26T05:20:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 16759/339054 objects degraded (4.943%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:36 smithi080 ceph-mon[40404]: osdmap e447: 8 total, 7 up, 6 in 2024-03-26T05:20:37.210 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:37.246 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:37.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:37.348 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:37.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:37.378 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:37.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:37.479 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:37.523 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:37.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:37.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:37.625 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:37.677 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:37.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:37.708 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:20:37.778 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:37.807 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:37.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:37.909 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:37.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:37.939 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:37.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:38.039 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:38.068 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:38.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:38.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:37 smithi080 ceph-mon[29691]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 7 pgs peering) 2024-03-26T05:20:38.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:37 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:20:38.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:37 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:20:38.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:37 smithi080 ceph-mon[40404]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 7 pgs peering) 2024-03-26T05:20:38.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:37 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:20:38.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:37 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:20:38.139 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:37 smithi132 ceph-mon[33500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 7 pgs peering) 2024-03-26T05:20:38.139 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:37 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:20:38.139 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:37 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:20:38.171 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:38.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:38.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:38.337 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:38.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:38.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:38.439 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:38.485 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:38.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:38.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:38.587 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:38.615 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:38.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:38.717 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:38.782 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:38.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:38.883 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:38.914 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:38.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:39.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:39.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:39.055 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:39.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:39.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:38 smithi080 ceph-mon[29691]: 5.11 scrub starts 2024-03-26T05:20:39.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:38 smithi080 ceph-mon[29691]: 5.11 scrub ok 2024-03-26T05:20:39.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:38 smithi080 ceph-mon[29691]: osdmap e448: 8 total, 7 up, 6 in 2024-03-26T05:20:39.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:38 smithi080 ceph-mon[29691]: 6.f scrub starts 2024-03-26T05:20:39.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:38 smithi080 ceph-mon[29691]: pgmap v950: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 724 op/s; 20735/326967 objects degraded (6.342%); 26126/326967 objects misplaced (7.990%); 26 MiB/s, 423 objects/s recovering 2024-03-26T05:20:39.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:38 smithi080 ceph-mon[29691]: 6.f scrub ok 2024-03-26T05:20:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:38 smithi080 ceph-mon[40404]: 5.11 scrub starts 2024-03-26T05:20:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:38 smithi080 ceph-mon[40404]: 5.11 scrub ok 2024-03-26T05:20:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:38 smithi080 ceph-mon[40404]: osdmap e448: 8 total, 7 up, 6 in 2024-03-26T05:20:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:38 smithi080 ceph-mon[40404]: 6.f scrub starts 2024-03-26T05:20:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:38 smithi080 ceph-mon[40404]: pgmap v950: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 724 op/s; 20735/326967 objects degraded (6.342%); 26126/326967 objects misplaced (7.990%); 26 MiB/s, 423 objects/s recovering 2024-03-26T05:20:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:38 smithi080 ceph-mon[40404]: 6.f scrub ok 2024-03-26T05:20:39.157 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:39.196 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:39.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:38 smithi132 ceph-mon[33500]: 5.11 scrub starts 2024-03-26T05:20:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:38 smithi132 ceph-mon[33500]: 5.11 scrub ok 2024-03-26T05:20:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:38 smithi132 ceph-mon[33500]: osdmap e448: 8 total, 7 up, 6 in 2024-03-26T05:20:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:38 smithi132 ceph-mon[33500]: 6.f scrub starts 2024-03-26T05:20:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:38 smithi132 ceph-mon[33500]: pgmap v950: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 724 op/s; 20735/326967 objects degraded (6.342%); 26126/326967 objects misplaced (7.990%); 26 MiB/s, 423 objects/s recovering 2024-03-26T05:20:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:38 smithi132 ceph-mon[33500]: 6.f scrub ok 2024-03-26T05:20:39.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:39.341 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:39.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:39.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:39.443 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:39.470 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:39.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:39.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:39.571 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:39.614 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:39.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:39.715 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:39.745 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:39.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:39.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:39.847 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:39.878 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:39.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:39.980 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:40.021 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:40.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:40.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:39 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:20:40.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:39 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:20:40.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:39 smithi080 ceph-mon[29691]: 4.d scrub starts 2024-03-26T05:20:40.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:39 smithi080 ceph-mon[29691]: 4.d scrub ok 2024-03-26T05:20:40.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:39 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:20:40.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:39 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:20:40.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:39 smithi080 ceph-mon[40404]: 4.d scrub starts 2024-03-26T05:20:40.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:39 smithi080 ceph-mon[40404]: 4.d scrub ok 2024-03-26T05:20:40.122 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:40.151 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:40.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:40.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:40.282 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:40.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:40.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:39 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:20:40.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:39 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:20:40.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:39 smithi132 ceph-mon[33500]: 4.d scrub starts 2024-03-26T05:20:40.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:39 smithi132 ceph-mon[33500]: 4.d scrub ok 2024-03-26T05:20:40.384 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:40.420 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:40.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:40.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:40.522 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:40.548 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:40.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:40.649 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:40.688 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:40.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:40.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:40.769 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:40.790 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:40.831 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:40.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:40.933 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:40.977 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:40.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:41.078 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:41.106 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:41.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:41.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:41.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:40 smithi080 ceph-mon[40404]: 4.e scrub starts 2024-03-26T05:20:41.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:40 smithi080 ceph-mon[40404]: 4.e scrub ok 2024-03-26T05:20:41.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:40 smithi080 ceph-mon[40404]: osdmap e449: 8 total, 7 up, 6 in 2024-03-26T05:20:41.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:40 smithi080 ceph-mon[40404]: pgmap v952: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.5 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 1.04k op/s; 20735/321207 objects degraded (6.455%); 25110/321207 objects misplaced (7.817%); 31 MiB/s, 495 objects/s recovering 2024-03-26T05:20:41.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:40 smithi080 ceph-mon[29691]: 4.e scrub starts 2024-03-26T05:20:41.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:40 smithi080 ceph-mon[29691]: 4.e scrub ok 2024-03-26T05:20:41.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:40 smithi080 ceph-mon[29691]: osdmap e449: 8 total, 7 up, 6 in 2024-03-26T05:20:41.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:40 smithi080 ceph-mon[29691]: pgmap v952: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.5 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 1.04k op/s; 20735/321207 objects degraded (6.455%); 25110/321207 objects misplaced (7.817%); 31 MiB/s, 495 objects/s recovering 2024-03-26T05:20:41.207 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:41.243 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:41.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:41.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:41.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:40 smithi132 ceph-mon[33500]: 4.e scrub starts 2024-03-26T05:20:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:40 smithi132 ceph-mon[33500]: 4.e scrub ok 2024-03-26T05:20:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:40 smithi132 ceph-mon[33500]: osdmap e449: 8 total, 7 up, 6 in 2024-03-26T05:20:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:40 smithi132 ceph-mon[33500]: pgmap v952: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.5 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 1.04k op/s; 20735/321207 objects degraded (6.455%); 25110/321207 objects misplaced (7.817%); 31 MiB/s, 495 objects/s recovering 2024-03-26T05:20:41.344 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:41.378 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:41.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:41.479 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:41.517 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:41.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:41.619 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:41.647 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:41.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:41.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:41.749 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:41.778 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:41.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:41.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:41.880 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:41.917 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:41.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:42.018 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:42.048 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:42.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:42.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:42.150 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:42.179 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:42.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:42.281 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:42.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:41 smithi132 ceph-mon[33500]: 4.13 scrub starts 2024-03-26T05:20:42.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:41 smithi132 ceph-mon[33500]: 4.13 scrub ok 2024-03-26T05:20:42.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:41 smithi132 ceph-mon[33500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:20:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:41 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 20735/321207 objects degraded (6.455%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:42.306 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:42.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:42.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:41 smithi080 ceph-mon[29691]: 4.13 scrub starts 2024-03-26T05:20:42.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:41 smithi080 ceph-mon[29691]: 4.13 scrub ok 2024-03-26T05:20:42.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:41 smithi080 ceph-mon[29691]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:20:42.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:41 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 20735/321207 objects degraded (6.455%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:42.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:41 smithi080 ceph-mon[40404]: 4.13 scrub starts 2024-03-26T05:20:42.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:41 smithi080 ceph-mon[40404]: 4.13 scrub ok 2024-03-26T05:20:42.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:41 smithi080 ceph-mon[40404]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:20:42.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:41 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 20735/321207 objects degraded (6.455%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:42.408 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:42.446 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:42.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:42.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:42.453 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:20:42.453 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:20:42.548 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:42.572 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:42.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:42.674 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:42.719 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:42.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:42.821 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:42.853 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:42.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:42.954 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:42.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:42.980 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:42.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:43.082 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:43.107 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:43.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:43.208 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:43.245 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:43.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:43.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:42 smithi132 ceph-mon[33500]: pgmap v953: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 18523/311883 objects degraded (5.939%); 24174/311883 objects misplaced (7.751%); 39 MiB/s, 620 objects/s recovering 2024-03-26T05:20:43.347 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:43.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:42 smithi080 ceph-mon[40404]: pgmap v953: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 18523/311883 objects degraded (5.939%); 24174/311883 objects misplaced (7.751%); 39 MiB/s, 620 objects/s recovering 2024-03-26T05:20:43.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:42 smithi080 ceph-mon[29691]: pgmap v953: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 18523/311883 objects degraded (5.939%); 24174/311883 objects misplaced (7.751%); 39 MiB/s, 620 objects/s recovering 2024-03-26T05:20:43.374 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:43.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:43.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:43.475 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:43.514 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:43.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:43.617 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:43.658 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:43.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:43.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:43.760 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:43.798 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:43.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:43.900 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:43.928 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:43.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:44.030 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:44.059 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:44.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:44.161 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:44.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:44.188 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:44.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:44.289 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:44.317 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:44.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:44.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:20:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:20:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:20:44.418 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:44.459 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:44.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:44.561 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:44.592 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:44.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:44.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:44.694 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:44.718 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:44.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:44.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:44.823 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:44.852 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:44.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:44.953 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:44.997 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:44.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:45.099 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:45.140 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:45.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:45.242 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:45.283 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:45.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:45.385 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:45.431 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:45.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:45.533 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:45.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:45 smithi132 ceph-mon[33500]: pgmap v954: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 682 B/s rd, 0 B/s wr, 714 op/s; 18523/311883 objects degraded (5.939%); 24174/311883 objects misplaced (7.751%); 14 MiB/s, 228 objects/s recovering 2024-03-26T05:20:45.561 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:45.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:45.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:45 smithi080 ceph-mon[40404]: pgmap v954: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 682 B/s rd, 0 B/s wr, 714 op/s; 18523/311883 objects degraded (5.939%); 24174/311883 objects misplaced (7.751%); 14 MiB/s, 228 objects/s recovering 2024-03-26T05:20:45.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:45 smithi080 ceph-mon[29691]: pgmap v954: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 682 B/s rd, 0 B/s wr, 714 op/s; 18523/311883 objects degraded (5.939%); 24174/311883 objects misplaced (7.751%); 14 MiB/s, 228 objects/s recovering 2024-03-26T05:20:45.663 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:45.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:45.705 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:45.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:45.806 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:45.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:45.836 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:45.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:45.937 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:45.978 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:45.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:46.080 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:46.111 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:46.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:46.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:46.213 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:46.242 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:46.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:46.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:20:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:20:46.344 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:46.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:46.373 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:46.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:46.474 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:46.502 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:46.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:46.604 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:46.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:46.634 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:46.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:46.736 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:46.782 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:46.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:46.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:46.884 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:46.912 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:46.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:47.015 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:47.056 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:47.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:47.157 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:47.185 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:47.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:47.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:47.287 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:47.316 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:47.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:47.418 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:47.456 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:47.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:47.518 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:47 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:47.518 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:47 smithi080 ceph-mon[40404]: pgmap v955: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 986 op/s; 18216/302181 objects degraded (6.028%); 22942/302181 objects misplaced (7.592%); 15 MiB/s, 246 objects/s recovering 2024-03-26T05:20:47.518 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:47 smithi080 ceph-mon[40404]: 5.18 scrub starts 2024-03-26T05:20:47.518 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:47 smithi080 ceph-mon[40404]: 5.18 scrub ok 2024-03-26T05:20:47.518 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:47 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 18216/302181 objects degraded (6.028%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:47.518 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:47 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:47.519 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:47 smithi080 ceph-mon[29691]: pgmap v955: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 986 op/s; 18216/302181 objects degraded (6.028%); 22942/302181 objects misplaced (7.592%); 15 MiB/s, 246 objects/s recovering 2024-03-26T05:20:47.519 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:47 smithi080 ceph-mon[29691]: 5.18 scrub starts 2024-03-26T05:20:47.519 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:47 smithi080 ceph-mon[29691]: 5.18 scrub ok 2024-03-26T05:20:47.519 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:47 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 18216/302181 objects degraded (6.028%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:47.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:47 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:47.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:47 smithi132 ceph-mon[33500]: pgmap v955: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 986 op/s; 18216/302181 objects degraded (6.028%); 22942/302181 objects misplaced (7.592%); 15 MiB/s, 246 objects/s recovering 2024-03-26T05:20:47.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:47 smithi132 ceph-mon[33500]: 5.18 scrub starts 2024-03-26T05:20:47.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:47 smithi132 ceph-mon[33500]: 5.18 scrub ok 2024-03-26T05:20:47.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:47 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 18216/302181 objects degraded (6.028%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:47.557 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:47.604 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:47.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:47.706 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:47.732 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:47.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:47.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:47.834 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:47.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:47.875 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:47.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:47.976 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:48.005 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:48.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:48.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:48.107 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:48.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:48.140 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:48.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:48.241 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:48.270 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:48.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:48.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:48 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1781923211' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:20:48.296 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:48 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:20:48.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:48.415 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:48.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:48.516 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:48.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:48 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1781923211' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:20:48.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:48 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:20:48.547 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:48.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:48.583 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:48 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1781923211' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:20:48.584 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:48 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:20:48.630 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T05:20:48.649 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:48.711 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:48.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:48.802 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:20:48.812 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:48.840 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:48.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:48.942 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:48.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:48.974 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:48.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:49.075 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:49.118 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:49.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:49.221 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:49.263 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:49.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:49.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:49.393 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:49.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:49.491 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:49 smithi080 ceph-mon[29691]: pgmap v956: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s; 16442/297834 objects degraded (5.521%); 22371/297834 objects misplaced (7.511%); 21 MiB/s, 331 objects/s recovering 2024-03-26T05:20:49.491 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:49 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:20:49.491 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:49 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:20:49.491 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:49 smithi080 ceph-mon[29691]: osdmap e450: 8 total, 7 up, 6 in 2024-03-26T05:20:49.492 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:49 smithi080 ceph-mon[40404]: pgmap v956: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s; 16442/297834 objects degraded (5.521%); 22371/297834 objects misplaced (7.511%); 21 MiB/s, 331 objects/s recovering 2024-03-26T05:20:49.492 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:49 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:20:49.492 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:49 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:20:49.492 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:49 smithi080 ceph-mon[40404]: osdmap e450: 8 total, 7 up, 6 in 2024-03-26T05:20:49.495 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:49.503 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:49.521 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:49.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:49.623 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:49.676 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:49.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:49.778 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:49.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:49 smithi132 ceph-mon[33500]: pgmap v956: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s; 16442/297834 objects degraded (5.521%); 22371/297834 objects misplaced (7.511%); 21 MiB/s, 331 objects/s recovering 2024-03-26T05:20:49.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:49 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:20:49.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:49 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:20:49.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:49 smithi132 ceph-mon[33500]: osdmap e450: 8 total, 7 up, 6 in 2024-03-26T05:20:49.801 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:49.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:49.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:49.902 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:49.931 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:49.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:50.033 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:50.076 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:50.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:50.178 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:50.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:50.205 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:50.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:50.307 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:50.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:50.335 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:50.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:50.436 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:50.463 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:50.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:50.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:50.565 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:50.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:50.606 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:50.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:50.707 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:50.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:50.754 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:50.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:50.856 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:50.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:50.883 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:50.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:50.984 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:51.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:51.043 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:51.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:51.115 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:20:50 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:20:50.865+0000 7f2fcc605700 -1 mgr get_metadata_python Requested missing service osd.4 2024-03-26T05:20:51.144 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:51.201 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:51.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:51.302 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:51.342 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:51.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:51.443 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:51.483 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:51.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:51.516 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:51 smithi080 ceph-mon[29691]: pgmap v958: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 968 op/s; 16442/292173 objects degraded (5.627%); 21331/292173 objects misplaced (7.301%); 21 MiB/s, 333 objects/s recovering 2024-03-26T05:20:51.516 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:51 smithi080 ceph-mon[29691]: 6.1e scrub starts 2024-03-26T05:20:51.516 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:51 smithi080 ceph-mon[29691]: 6.1e scrub ok 2024-03-26T05:20:51.516 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:51.516 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:20:51.517 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:51 smithi080 ceph-mon[40404]: pgmap v958: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 968 op/s; 16442/292173 objects degraded (5.627%); 21331/292173 objects misplaced (7.301%); 21 MiB/s, 333 objects/s recovering 2024-03-26T05:20:51.517 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:51 smithi080 ceph-mon[40404]: 6.1e scrub starts 2024-03-26T05:20:51.517 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:51 smithi080 ceph-mon[40404]: 6.1e scrub ok 2024-03-26T05:20:51.517 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:51.517 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:20:51.548 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:20:51.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:51.636 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:51.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:51.737 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:51.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:51.788 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:51.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:51.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:51 smithi132 ceph-mon[33500]: pgmap v958: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 968 op/s; 16442/292173 objects degraded (5.627%); 21331/292173 objects misplaced (7.301%); 21 MiB/s, 333 objects/s recovering 2024-03-26T05:20:51.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:51 smithi132 ceph-mon[33500]: 6.1e scrub starts 2024-03-26T05:20:51.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:51 smithi132 ceph-mon[33500]: 6.1e scrub ok 2024-03-26T05:20:51.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:51.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:20:51.889 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:51.950 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:51.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:51.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:52.052 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:52.100 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:52.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:52.201 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:52.229 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:52.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:52.331 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:52.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:52 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 16442/292173 objects degraded (5.627%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:52.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:52 smithi080 ceph-mon[29691]: osdmap e451: 8 total, 7 up, 6 in 2024-03-26T05:20:52.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:52 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 16442/292173 objects degraded (5.627%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:52.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:52 smithi080 ceph-mon[40404]: osdmap e451: 8 total, 7 up, 6 in 2024-03-26T05:20:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:52.376 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:52.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:52.477 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:52.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:52.514 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:52.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:52.615 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:52.643 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:52.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:52.745 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:52.789 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:52.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:52.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:52.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:52 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 16442/292173 objects degraded (5.627%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:52.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:52 smithi132 ceph-mon[33500]: osdmap e451: 8 total, 7 up, 6 in 2024-03-26T05:20:52.892 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:52.921 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:52.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:52.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:53.023 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:53.051 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:53.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:53.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:53.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:53.180 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:53.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:53.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:53.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:53.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:53.329 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:53.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:53.430 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:53.481 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:53.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:53.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:53.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:53.609 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:53.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:53.614 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:53 smithi080 ceph-mon[40404]: pgmap v960: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 13897/282588 objects degraded (4.918%); 20416/282588 objects misplaced (7.225%); 30 MiB/s, 475 objects/s recovering 2024-03-26T05:20:53.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:53 smithi080 ceph-mon[40404]: 3.1d scrub starts 2024-03-26T05:20:53.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:53 smithi080 ceph-mon[40404]: 3.1d scrub ok 2024-03-26T05:20:53.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:53 smithi080 ceph-mon[40404]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:20:53.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:53 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:53.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:53 smithi080 ceph-mon[29691]: pgmap v960: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 13897/282588 objects degraded (4.918%); 20416/282588 objects misplaced (7.225%); 30 MiB/s, 475 objects/s recovering 2024-03-26T05:20:53.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:53 smithi080 ceph-mon[29691]: 3.1d scrub starts 2024-03-26T05:20:53.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:53 smithi080 ceph-mon[29691]: 3.1d scrub ok 2024-03-26T05:20:53.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:53 smithi080 ceph-mon[29691]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:20:53.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:53 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:53.710 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:53.755 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:53.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:53.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:53 smithi132 ceph-mon[33500]: pgmap v960: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 13897/282588 objects degraded (4.918%); 20416/282588 objects misplaced (7.225%); 30 MiB/s, 475 objects/s recovering 2024-03-26T05:20:53.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:53 smithi132 ceph-mon[33500]: 3.1d scrub starts 2024-03-26T05:20:53.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:53 smithi132 ceph-mon[33500]: 3.1d scrub ok 2024-03-26T05:20:53.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:53 smithi132 ceph-mon[33500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:20:53.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:53 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:53.856 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:53.894 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:53.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:53.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:53.996 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:54.024 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:54.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:54.126 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:54.199 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:54.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:54.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:54.301 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:54.331 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:54.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:54.346 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:20:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:20:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:20:54.432 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:54.482 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:54.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:54.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:54.584 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:54.629 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:54.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:54.730 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:54.756 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:54.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:54.857 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:54.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:54.903 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:54.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:55.004 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:55.051 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:55.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:55.152 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:55.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:55.201 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:55.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:55.290 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:55.290 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:20:55.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:55.328 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:55.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:55.380 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:20:55.430 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:55.431 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:55 smithi080 ceph-mon[40404]: pgmap v961: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 816 op/s; 13897/282588 objects degraded (4.918%); 20416/282588 objects misplaced (7.225%); 27 MiB/s, 431 objects/s recovering 2024-03-26T05:20:55.431 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:55 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:55.431 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:55 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:55.431 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:55 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:55.431 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:20:55.432 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:55 smithi080 ceph-mon[29691]: pgmap v961: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 816 op/s; 13897/282588 objects degraded (4.918%); 20416/282588 objects misplaced (7.225%); 27 MiB/s, 431 objects/s recovering 2024-03-26T05:20:55.432 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:55 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:55.432 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:55 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:55.432 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:55 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:55.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:55.480 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:55.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:55.480 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:20:55.526 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:20:55.578 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:20:55.581 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:55.627 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:55.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:55.638 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:20:55.690 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:20:55.691 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:20:55.691 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:20:55.691 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:20:55.691 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:20:55.691 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:20:55.691 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:20:55.691 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:20:55.691 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:20:55.728 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:55.765 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:55.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:55.791 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:20:55.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:55 smithi132 ceph-mon[33500]: pgmap v961: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 816 op/s; 13897/282588 objects degraded (4.918%); 20416/282588 objects misplaced (7.225%); 27 MiB/s, 431 objects/s recovering 2024-03-26T05:20:55.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:55 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:55.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:55 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:55.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:55 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:20:55.808 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:20:55.867 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:55.895 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:55.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:55.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:55.997 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:56.023 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:56.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:56.125 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:56.152 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:56.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:56.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:56.280 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:56.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:56.290 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:20:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:20:56.382 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:56.393 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:56 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1967261715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:20:56.393 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:56 smithi080 ceph-mon[40404]: 4.1 scrub starts 2024-03-26T05:20:56.393 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:56 smithi080 ceph-mon[40404]: 4.1 scrub ok 2024-03-26T05:20:56.394 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:56 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1967261715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:20:56.394 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:56 smithi080 ceph-mon[29691]: 4.1 scrub starts 2024-03-26T05:20:56.394 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:56 smithi080 ceph-mon[29691]: 4.1 scrub ok 2024-03-26T05:20:56.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:56.435 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:56.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:56.536 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:56.575 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:56.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:56.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:56.677 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:56.714 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:56.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:56.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:56.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:56 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1967261715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:20:56.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:56 smithi132 ceph-mon[33500]: 4.1 scrub starts 2024-03-26T05:20:56.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:56 smithi132 ceph-mon[33500]: 4.1 scrub ok 2024-03-26T05:20:56.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:56.839 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:56.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:56.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:56.941 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:56.982 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:56.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:57.083 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:57.125 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:57.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:57.227 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:57.271 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:57.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:57.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:57.400 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:57.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:57.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:57.502 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:57.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:57.543 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:57.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:57.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:57 smithi080 ceph-mon[29691]: from='client.16056 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:20:57.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:57 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:57.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:57 smithi080 ceph-mon[29691]: pgmap v962: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 13234/273273 objects degraded (4.843%); 19258/273273 objects misplaced (7.047%); 24 MiB/s, 377 objects/s recovering 2024-03-26T05:20:57.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:57 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 13234/273273 objects degraded (4.843%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:57.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:57 smithi080 ceph-mon[40404]: from='client.16056 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:20:57.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:57 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:57.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:57 smithi080 ceph-mon[40404]: pgmap v962: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 13234/273273 objects degraded (4.843%); 19258/273273 objects misplaced (7.047%); 24 MiB/s, 377 objects/s recovering 2024-03-26T05:20:57.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:57 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 13234/273273 objects degraded (4.843%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:57.644 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:57.671 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:57.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:57.772 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:57.783 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:20:57.783 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:20:57.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:57 smithi132 ceph-mon[33500]: from='client.16056 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:20:57.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:57 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:20:57.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:57 smithi132 ceph-mon[33500]: pgmap v962: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 13234/273273 objects degraded (4.843%); 19258/273273 objects misplaced (7.047%); 24 MiB/s, 377 objects/s recovering 2024-03-26T05:20:57.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:57 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 13234/273273 objects degraded (4.843%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:20:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:57.814 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:57.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:57.915 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:57.956 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:57.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:57.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:58.058 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:20:58.087 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:58.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:20:58.189 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:58.216 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:58.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:58.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:58.318 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:58.347 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:58.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:58.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:20:58.449 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:20:58.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:58.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:20:58.503 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":961,"stamp":"2024-03-26T05:20:54.229606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6158662850,"num_objects":94196,"num_object_clones":0,"num_object_copies":282588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13897,"num_objects_misplaced":20416,"num_objects_unfound":0,"num_objects_dirty":94196,"num_whiteouts":0,"num_read":2276,"num_read_kb":2014,"num_write":149486,"num_write_kb":7751334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12993,"num_bytes_recovered":834733613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63940,"ondisk_log_size":63940,"up":474,"acting":470,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":20121816,"kb_used_data":15933660,"kb_used_omap":12,"kb_used_meta":4187955,"kb_avail":542349096,"statfs":{"total":575970213888,"available":555365474304,"internally_reserved":0,"allocated":16316067840,"data_stored":16309922536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12412,"internal_metadata":4288466820},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-428015616,"num_objects":-6531,"num_object_clones":0,"num_object_copies":-19593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4319,"num_objects_misplaced":-2526,"num_objects_unfound":0,"num_objects_dirty":-6531,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6531,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3454,"num_bytes_recovered":226361344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004127"},"pg_stats":[{"pgid":"11.3","version":"450'9205","reported_seq":9301,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:49.872438+0000","last_change":"2024-03-26T05:18:36.688678+0000","last_active":"2024-03-26T05:20:49.872438+0000","last_peered":"2024-03-26T05:20:49.872438+0000","last_clean":"2024-03-26T05:20:49.872438+0000","last_became_active":"2024-03-26T05:18:36.688371+0000","last_became_peered":"2024-03-26T05:18:36.688371+0000","last_unstale":"2024-03-26T05:20:49.872438+0000","last_undegraded":"2024-03-26T05:20:49.872438+0000","last_fullsized":"2024-03-26T05:20:49.872438+0000","mapping_epoch":408,"log_start":"427'5338","ondisk_log_start":"427'5338","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382664704,"num_objects":5839,"num_object_clones":0,"num_object_copies":17517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9205,"num_write_kb":481408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"443'30","reported_seq":701,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.313273+0000","last_change":"2024-03-26T05:20:15.461803+0000","last_active":"2024-03-26T05:20:48.313273+0000","last_peered":"2024-03-26T05:20:48.313273+0000","last_clean":"2024-03-26T05:20:48.313273+0000","last_became_active":"2024-03-26T05:20:15.369000+0000","last_became_peered":"2024-03-26T05:20:15.369000+0000","last_unstale":"2024-03-26T05:20:48.313273+0000","last_undegraded":"2024-03-26T05:20:48.313273+0000","last_fullsized":"2024-03-26T05:20:48.313273+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'20","last_scrub_stamp":"2024-03-26T05:14:56.989928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:56.989928+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":860,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311036+0000","last_change":"2024-03-26T05:19:52.350482+0000","last_active":"2024-03-26T05:20:48.311036+0000","last_peered":"2024-03-26T05:20:48.311036+0000","last_clean":"2024-03-26T05:20:48.311036+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:20:48.311036+0000","last_undegraded":"2024-03-26T05:20:48.311036+0000","last_fullsized":"2024-03-26T05:20:48.311036+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:52.350328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:19:52.350328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:08.212425+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"443'14","reported_seq":1136,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.321493+0000","last_change":"2024-03-26T05:20:30.368570+0000","last_active":"2024-03-26T05:20:48.321493+0000","last_peered":"2024-03-26T05:20:48.321493+0000","last_clean":"2024-03-26T05:20:48.321493+0000","last_became_active":"2024-03-26T05:20:30.348391+0000","last_became_peered":"2024-03-26T05:20:30.348391+0000","last_unstale":"2024-03-26T05:20:48.321493+0000","last_undegraded":"2024-03-26T05:20:48.321493+0000","last_fullsized":"2024-03-26T05:20:48.321493+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'13","last_scrub_stamp":"2024-03-26T05:18:56.306025+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:18:56.306025+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":238,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.313586+0000","last_change":"2024-03-26T05:20:30.342597+0000","last_active":"2024-03-26T05:20:48.313586+0000","last_peered":"2024-03-26T05:20:48.313586+0000","last_clean":"2024-03-26T05:20:48.313586+0000","last_became_active":"2024-03-26T05:20:30.342267+0000","last_became_peered":"2024-03-26T05:20:30.342267+0000","last_unstale":"2024-03-26T05:20:48.313586+0000","last_undegraded":"2024-03-26T05:20:48.313586+0000","last_fullsized":"2024-03-26T05:20:48.313586+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:54.067506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:54.067506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"88'128","reported_seq":1027,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311455+0000","last_change":"2024-03-26T05:19:59.233794+0000","last_active":"2024-03-26T05:20:48.311455+0000","last_peered":"2024-03-26T05:20:48.311455+0000","last_clean":"2024-03-26T05:20:48.311455+0000","last_became_active":"2024-03-26T05:15:33.616505+0000","last_became_peered":"2024-03-26T05:15:33.616505+0000","last_unstale":"2024-03-26T05:20:48.311455+0000","last_undegraded":"2024-03-26T05:20:48.311455+0000","last_fullsized":"2024-03-26T05:20:48.311455+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:19:59.233670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:19:59.233670+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:10.698394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"450'9084","reported_seq":9226,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:50.960737+0000","last_change":"2024-03-26T05:20:06.852765+0000","last_active":"2024-03-26T05:20:50.960737+0000","last_peered":"2024-03-26T05:20:50.960737+0000","last_clean":"2024-03-26T05:20:50.960737+0000","last_became_active":"2024-03-26T05:18:36.692550+0000","last_became_peered":"2024-03-26T05:18:36.692550+0000","last_unstale":"2024-03-26T05:20:50.960737+0000","last_undegraded":"2024-03-26T05:20:50.960737+0000","last_fullsized":"2024-03-26T05:20:50.960737+0000","mapping_epoch":408,"log_start":"423'4460","ondisk_log_start":"423'4460","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":4299,"log_size":4624,"log_dups_size":0,"ondisk_log_size":4624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376176640,"num_objects":5740,"num_object_clones":0,"num_object_copies":17220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9084,"num_write_kb":474368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"443'45","reported_seq":740,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.319766+0000","last_change":"2024-03-26T05:20:39.141432+0000","last_active":"2024-03-26T05:20:48.319766+0000","last_peered":"2024-03-26T05:20:48.319766+0000","last_clean":"2024-03-26T05:20:48.319766+0000","last_became_active":"2024-03-26T05:20:15.381623+0000","last_became_peered":"2024-03-26T05:20:15.381623+0000","last_unstale":"2024-03-26T05:20:48.319766+0000","last_undegraded":"2024-03-26T05:20:48.319766+0000","last_fullsized":"2024-03-26T05:20:48.319766+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'45","last_scrub_stamp":"2024-03-26T05:20:39.141285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:39.141285+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:43.921867+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":889,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314224+0000","last_change":"2024-03-26T05:20:15.361409+0000","last_active":"2024-03-26T05:20:48.314224+0000","last_peered":"2024-03-26T05:20:48.314224+0000","last_clean":"2024-03-26T05:20:48.314224+0000","last_became_active":"2024-03-26T05:20:15.361072+0000","last_became_peered":"2024-03-26T05:20:15.361072+0000","last_unstale":"2024-03-26T05:20:48.314224+0000","last_undegraded":"2024-03-26T05:20:48.314224+0000","last_fullsized":"2024-03-26T05:20:48.314224+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:47.849136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:19:47.849136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:00.365555+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":839,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315040+0000","last_change":"2024-03-26T05:20:15.373542+0000","last_active":"2024-03-26T05:20:48.315040+0000","last_peered":"2024-03-26T05:20:48.315040+0000","last_clean":"2024-03-26T05:20:48.315040+0000","last_became_active":"2024-03-26T05:20:15.373227+0000","last_became_peered":"2024-03-26T05:20:15.373227+0000","last_unstale":"2024-03-26T05:20:48.315040+0000","last_undegraded":"2024-03-26T05:20:48.315040+0000","last_fullsized":"2024-03-26T05:20:48.315040+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:21.059907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:14:21.059907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":843,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.320093+0000","last_change":"2024-03-26T05:20:38.300338+0000","last_active":"2024-03-26T05:20:48.320093+0000","last_peered":"2024-03-26T05:20:48.320093+0000","last_clean":"2024-03-26T05:20:48.320093+0000","last_became_active":"2024-03-26T05:20:15.380687+0000","last_became_peered":"2024-03-26T05:20:15.380687+0000","last_unstale":"2024-03-26T05:20:48.320093+0000","last_undegraded":"2024-03-26T05:20:48.320093+0000","last_fullsized":"2024-03-26T05:20:48.320093+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:38.300223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:20:38.300223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:51.986851+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"88'128","reported_seq":421,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.320647+0000","last_change":"2024-03-26T05:20:34.153749+0000","last_active":"2024-03-26T05:20:48.320647+0000","last_peered":"2024-03-26T05:20:48.320647+0000","last_clean":"2024-03-26T05:20:48.320647+0000","last_became_active":"2024-03-26T05:20:32.754455+0000","last_became_peered":"2024-03-26T05:20:32.754455+0000","last_unstale":"2024-03-26T05:20:48.320647+0000","last_undegraded":"2024-03-26T05:20:48.320647+0000","last_fullsized":"2024-03-26T05:20:48.320647+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:20:34.153595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:20:34.153595+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:54.072523+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"450'9328","reported_seq":9489,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:49.944362+0000","last_change":"2024-03-26T05:19:50.459174+0000","last_active":"2024-03-26T05:20:49.944362+0000","last_peered":"2024-03-26T05:20:49.944362+0000","last_clean":"2024-03-26T05:20:49.944362+0000","last_became_active":"2024-03-26T05:18:36.689247+0000","last_became_peered":"2024-03-26T05:18:36.689247+0000","last_unstale":"2024-03-26T05:20:49.944362+0000","last_undegraded":"2024-03-26T05:20:49.944362+0000","last_fullsized":"2024-03-26T05:20:49.944362+0000","mapping_epoch":408,"log_start":"427'5435","ondisk_log_start":"427'5435","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'5729","last_scrub_stamp":"2024-03-26T05:19:50.459009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:19:50.459009+0000","objects_scrubbed":5667,"log_size":3893,"log_dups_size":0,"ondisk_log_size":3893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:08.521016+0000","scrub_duration":1694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392429568,"num_objects":5988,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9328,"num_write_kb":490112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"443'33","reported_seq":721,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315239+0000","last_change":"2024-03-26T05:20:39.704389+0000","last_active":"2024-03-26T05:20:48.315239+0000","last_peered":"2024-03-26T05:20:48.315239+0000","last_clean":"2024-03-26T05:20:48.315239+0000","last_became_active":"2024-03-26T05:20:15.366493+0000","last_became_peered":"2024-03-26T05:20:15.366493+0000","last_unstale":"2024-03-26T05:20:48.315239+0000","last_undegraded":"2024-03-26T05:20:48.315239+0000","last_fullsized":"2024-03-26T05:20:48.315239+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'33","last_scrub_stamp":"2024-03-26T05:20:39.704184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:39.704184+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:41.150865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":832,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.319023+0000","last_change":"2024-03-26T05:20:30.098403+0000","last_active":"2024-03-26T05:20:48.319023+0000","last_peered":"2024-03-26T05:20:48.319023+0000","last_clean":"2024-03-26T05:20:48.319023+0000","last_became_active":"2024-03-26T05:20:15.374939+0000","last_became_peered":"2024-03-26T05:20:15.374939+0000","last_unstale":"2024-03-26T05:20:48.319023+0000","last_undegraded":"2024-03-26T05:20:48.319023+0000","last_fullsized":"2024-03-26T05:20:48.319023+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:30.098264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:20:30.098264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:45.626635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":509,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314720+0000","last_change":"2024-03-26T05:20:30.335362+0000","last_active":"2024-03-26T05:20:48.314720+0000","last_peered":"2024-03-26T05:20:48.314720+0000","last_clean":"2024-03-26T05:20:48.314720+0000","last_became_active":"2024-03-26T05:20:30.335018+0000","last_became_peered":"2024-03-26T05:20:30.335018+0000","last_unstale":"2024-03-26T05:20:48.314720+0000","last_undegraded":"2024-03-26T05:20:48.314720+0000","last_fullsized":"2024-03-26T05:20:48.314720+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:56.632109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:13:07.883082+0000","last_clean_scrub_stamp":"2024-03-26T05:14:56.632109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":852,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.309550+0000","last_change":"2024-03-26T05:20:30.327361+0000","last_active":"2024-03-26T05:20:48.309550+0000","last_peered":"2024-03-26T05:20:48.309550+0000","last_clean":"2024-03-26T05:20:48.309550+0000","last_became_active":"2024-03-26T05:20:30.327060+0000","last_became_peered":"2024-03-26T05:20:30.327060+0000","last_unstale":"2024-03-26T05:20:48.309550+0000","last_undegraded":"2024-03-26T05:20:48.309550+0000","last_fullsized":"2024-03-26T05:20:48.309550+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:46.337028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:46.337028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:08.664724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"450'9379","reported_seq":9547,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.295944+0000","last_change":"2024-03-26T05:20:00.971938+0000","last_active":"2024-03-26T05:20:51.295944+0000","last_peered":"2024-03-26T05:20:51.295944+0000","last_clean":"2024-03-26T05:20:51.295944+0000","last_became_active":"2024-03-26T05:18:36.696863+0000","last_became_peered":"2024-03-26T05:18:36.696863+0000","last_unstale":"2024-03-26T05:20:51.295944+0000","last_undegraded":"2024-03-26T05:20:51.295944+0000","last_fullsized":"2024-03-26T05:20:51.295944+0000","mapping_epoch":408,"log_start":"427'5669","ondisk_log_start":"427'5669","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'6421","last_scrub_stamp":"2024-03-26T05:20:00.971796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:20:00.971796+0000","objects_scrubbed":6351,"log_size":3710,"log_dups_size":0,"ondisk_log_size":3710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:29.459048+0000","scrub_duration":2311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384892928,"num_objects":5873,"num_object_clones":0,"num_object_copies":17619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9379,"num_write_kb":488064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"443'46","reported_seq":935,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.320070+0000","last_change":"2024-03-26T05:20:15.465505+0000","last_active":"2024-03-26T05:20:48.320070+0000","last_peered":"2024-03-26T05:20:48.320070+0000","last_clean":"2024-03-26T05:20:48.320070+0000","last_became_active":"2024-03-26T05:20:15.378070+0000","last_became_peered":"2024-03-26T05:20:15.378070+0000","last_unstale":"2024-03-26T05:20:48.320070+0000","last_undegraded":"2024-03-26T05:20:48.320070+0000","last_fullsized":"2024-03-26T05:20:48.320070+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'34","last_scrub_stamp":"2024-03-26T05:15:43.090549+0000","last_deep_scrub":"292'34","last_deep_scrub_stamp":"2024-03-26T05:12:23.778630+0000","last_clean_scrub_stamp":"2024-03-26T05:15:43.090549+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":847,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.316985+0000","last_change":"2024-03-26T05:19:54.430984+0000","last_active":"2024-03-26T05:20:48.316985+0000","last_peered":"2024-03-26T05:20:48.316985+0000","last_clean":"2024-03-26T05:20:48.316985+0000","last_became_active":"2024-03-26T05:15:32.626265+0000","last_became_peered":"2024-03-26T05:15:32.626265+0000","last_unstale":"2024-03-26T05:20:48.316985+0000","last_undegraded":"2024-03-26T05:20:48.316985+0000","last_fullsized":"2024-03-26T05:20:48.316985+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:54.430734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.430734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:54.987158+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"443'13","reported_seq":62,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.758880+0000","last_change":"2024-03-26T05:20:30.358019+0000","last_active":"2024-03-26T05:20:51.758880+0000","last_peered":"2024-03-26T05:20:51.758880+0000","last_clean":"2024-03-26T05:20:51.758880+0000","last_became_active":"2024-03-26T05:20:30.350797+0000","last_became_peered":"2024-03-26T05:20:30.350797+0000","last_unstale":"2024-03-26T05:20:51.758880+0000","last_undegraded":"2024-03-26T05:20:51.758880+0000","last_fullsized":"2024-03-26T05:20:51.758880+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:14:03.822423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:03.822423+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":148,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.313380+0000","last_change":"2024-03-26T05:20:30.342843+0000","last_active":"2024-03-26T05:20:48.313380+0000","last_peered":"2024-03-26T05:20:48.313380+0000","last_clean":"2024-03-26T05:20:48.313380+0000","last_became_active":"2024-03-26T05:20:30.342410+0000","last_became_peered":"2024-03-26T05:20:30.342410+0000","last_unstale":"2024-03-26T05:20:48.313380+0000","last_undegraded":"2024-03-26T05:20:48.313380+0000","last_fullsized":"2024-03-26T05:20:48.313380+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:49.287596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_clean_scrub_stamp":"2024-03-26T05:18:49.287596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"88'128","reported_seq":417,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.318961+0000","last_change":"2024-03-26T05:20:33.139347+0000","last_active":"2024-03-26T05:20:48.318961+0000","last_peered":"2024-03-26T05:20:48.318961+0000","last_clean":"2024-03-26T05:20:48.318961+0000","last_became_active":"2024-03-26T05:20:32.753941+0000","last_became_peered":"2024-03-26T05:20:32.753941+0000","last_unstale":"2024-03-26T05:20:48.318961+0000","last_undegraded":"2024-03-26T05:20:48.318961+0000","last_fullsized":"2024-03-26T05:20:48.318961+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:20:33.139232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:20:33.139232+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:59.863137+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"450'9328","reported_seq":9492,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:50.884623+0000","last_change":"2024-03-26T05:19:57.196304+0000","last_active":"2024-03-26T05:20:50.884623+0000","last_peered":"2024-03-26T05:20:50.884623+0000","last_clean":"2024-03-26T05:20:50.884623+0000","last_became_active":"2024-03-26T05:18:36.691270+0000","last_became_peered":"2024-03-26T05:18:36.691270+0000","last_unstale":"2024-03-26T05:20:50.884623+0000","last_undegraded":"2024-03-26T05:20:50.884623+0000","last_fullsized":"2024-03-26T05:20:50.884623+0000","mapping_epoch":408,"log_start":"427'5598","ondisk_log_start":"427'5598","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'6148","last_scrub_stamp":"2024-03-26T05:19:57.195991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.195991+0000","objects_scrubbed":6082,"log_size":3730,"log_dups_size":0,"ondisk_log_size":3730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:07.579922+0000","scrub_duration":1951,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382730240,"num_objects":5840,"num_object_clones":0,"num_object_copies":17520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9328,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":809,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.308215+0000","last_change":"2024-03-26T05:20:15.376611+0000","last_active":"2024-03-26T05:20:48.308215+0000","last_peered":"2024-03-26T05:20:48.308215+0000","last_clean":"2024-03-26T05:20:48.308215+0000","last_became_active":"2024-03-26T05:20:15.376307+0000","last_became_peered":"2024-03-26T05:20:15.376307+0000","last_unstale":"2024-03-26T05:20:48.308215+0000","last_undegraded":"2024-03-26T05:20:48.308215+0000","last_fullsized":"2024-03-26T05:20:48.308215+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.411139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.411139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"443'64","reported_seq":954,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.317112+0000","last_change":"2024-03-26T05:20:05.275701+0000","last_active":"2024-03-26T05:20:48.317112+0000","last_peered":"2024-03-26T05:20:48.317112+0000","last_clean":"2024-03-26T05:20:48.317112+0000","last_became_active":"2024-03-26T05:15:32.636466+0000","last_became_peered":"2024-03-26T05:15:32.636466+0000","last_unstale":"2024-03-26T05:20:48.317112+0000","last_undegraded":"2024-03-26T05:20:48.317112+0000","last_fullsized":"2024-03-26T05:20:48.317112+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'52","last_scrub_stamp":"2024-03-26T05:20:05.274985+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:20:05.274985+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:25.059537+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":74,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":839,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315421+0000","last_change":"2024-03-26T05:18:36.798166+0000","last_active":"2024-03-26T05:20:48.315421+0000","last_peered":"2024-03-26T05:20:48.315421+0000","last_clean":"2024-03-26T05:20:48.315421+0000","last_became_active":"2024-03-26T05:15:32.636983+0000","last_became_peered":"2024-03-26T05:15:32.636983+0000","last_unstale":"2024-03-26T05:20:48.315421+0000","last_undegraded":"2024-03-26T05:20:48.315421+0000","last_fullsized":"2024-03-26T05:20:48.315421+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.798083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.798083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":232,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.320035+0000","last_change":"2024-03-26T05:20:15.377248+0000","last_active":"2024-03-26T05:20:48.320035+0000","last_peered":"2024-03-26T05:20:48.320035+0000","last_clean":"2024-03-26T05:20:48.320035+0000","last_became_active":"2024-03-26T05:20:15.375300+0000","last_became_peered":"2024-03-26T05:20:15.375300+0000","last_unstale":"2024-03-26T05:20:48.320035+0000","last_undegraded":"2024-03-26T05:20:48.320035+0000","last_fullsized":"2024-03-26T05:20:48.320035+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:54.923744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:13:54.923744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"88'128","reported_seq":631,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.321544+0000","last_change":"2024-03-26T05:20:32.753218+0000","last_active":"2024-03-26T05:20:48.321544+0000","last_peered":"2024-03-26T05:20:48.321544+0000","last_clean":"2024-03-26T05:20:48.321544+0000","last_became_active":"2024-03-26T05:20:32.752804+0000","last_became_peered":"2024-03-26T05:20:32.752804+0000","last_unstale":"2024-03-26T05:20:48.321544+0000","last_undegraded":"2024-03-26T05:20:48.321544+0000","last_fullsized":"2024-03-26T05:20:48.321544+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:20:28.011935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:20:28.011935+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:48.159618+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"450'9351","reported_seq":16694,"reported_epoch":450,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-03-26T05:20:51.312476+0000","last_change":"2024-03-26T05:20:30.371331+0000","last_active":"2024-03-26T05:20:51.312476+0000","last_peered":"2024-03-26T05:20:51.312476+0000","last_clean":"2024-03-26T05:18:36.613885+0000","last_became_active":"2024-03-26T05:20:16.435995+0000","last_became_peered":"2024-03-26T05:20:16.435995+0000","last_unstale":"2024-03-26T05:20:51.312476+0000","last_undegraded":"2024-03-26T05:20:16.371058+0000","last_fullsized":"2024-03-26T05:20:16.351598+0000","mapping_epoch":436,"log_start":"427'5616","ondisk_log_start":"427'5616","created":408,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382664704,"num_objects":5839,"num_object_clones":0,"num_object_copies":17517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":486080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3621,"num_bytes_recovered":237305856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":5839}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"439'10","reported_seq":1083,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.317236+0000","last_change":"2024-03-26T05:20:15.470722+0000","last_active":"2024-03-26T05:20:48.317236+0000","last_peered":"2024-03-26T05:20:48.317236+0000","last_clean":"2024-03-26T05:20:48.317236+0000","last_became_active":"2024-03-26T05:20:15.376570+0000","last_became_peered":"2024-03-26T05:20:15.376570+0000","last_unstale":"2024-03-26T05:20:48.317236+0000","last_undegraded":"2024-03-26T05:20:48.317236+0000","last_fullsized":"2024-03-26T05:20:48.317236+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:13:56.721250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:13:56.721250+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"443'48","reported_seq":580,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314832+0000","last_change":"2024-03-26T05:20:30.377241+0000","last_active":"2024-03-26T05:20:48.314832+0000","last_peered":"2024-03-26T05:20:48.314832+0000","last_clean":"2024-03-26T05:20:48.314832+0000","last_became_active":"2024-03-26T05:20:30.343622+0000","last_became_peered":"2024-03-26T05:20:30.343622+0000","last_unstale":"2024-03-26T05:20:48.314832+0000","last_undegraded":"2024-03-26T05:20:48.314832+0000","last_fullsized":"2024-03-26T05:20:48.314832+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'34","last_scrub_stamp":"2024-03-26T05:14:59.971454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:59.971454+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":63,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":863,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.758565+0000","last_change":"2024-03-26T05:20:15.464557+0000","last_active":"2024-03-26T05:20:51.758565+0000","last_peered":"2024-03-26T05:20:51.758565+0000","last_clean":"2024-03-26T05:20:51.758565+0000","last_became_active":"2024-03-26T05:20:15.379302+0000","last_became_peered":"2024-03-26T05:20:15.379302+0000","last_unstale":"2024-03-26T05:20:51.758565+0000","last_undegraded":"2024-03-26T05:20:51.758565+0000","last_fullsized":"2024-03-26T05:20:51.758565+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:12.113993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:12.113993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":753,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312049+0000","last_change":"2024-03-26T05:19:49.427510+0000","last_active":"2024-03-26T05:20:48.312049+0000","last_peered":"2024-03-26T05:20:48.312049+0000","last_clean":"2024-03-26T05:20:48.312049+0000","last_became_active":"2024-03-26T05:15:32.630135+0000","last_became_peered":"2024-03-26T05:15:32.630135+0000","last_unstale":"2024-03-26T05:20:48.312049+0000","last_undegraded":"2024-03-26T05:20:48.312049+0000","last_fullsized":"2024-03-26T05:20:48.312049+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:49.427405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:01.721323+0000","last_clean_scrub_stamp":"2024-03-26T05:19:49.427405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:02.828589+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"396'129","reported_seq":670,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.309355+0000","last_change":"2024-03-26T05:20:32.747492+0000","last_active":"2024-03-26T05:20:48.309355+0000","last_peered":"2024-03-26T05:20:48.309355+0000","last_clean":"2024-03-26T05:20:48.309355+0000","last_became_active":"2024-03-26T05:20:32.747103+0000","last_became_peered":"2024-03-26T05:20:32.747103+0000","last_unstale":"2024-03-26T05:20:48.309355+0000","last_undegraded":"2024-03-26T05:20:48.309355+0000","last_fullsized":"2024-03-26T05:20:48.309355+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"396'129","last_scrub_stamp":"2024-03-26T05:17:59.038611+0000","last_deep_scrub":"396'129","last_deep_scrub_stamp":"2024-03-26T05:17:59.038611+0000","last_clean_scrub_stamp":"2024-03-26T05:17:59.038611+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"450'9267","reported_seq":10249,"reported_epoch":450,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-03-26T05:20:50.994045+0000","last_change":"2024-03-26T05:20:46.498947+0000","last_active":"2024-03-26T05:20:50.994045+0000","last_peered":"2024-03-26T05:20:50.994045+0000","last_clean":"2024-03-26T05:18:36.613885+0000","last_became_active":"2024-03-26T05:20:16.463690+0000","last_became_peered":"2024-03-26T05:20:16.463690+0000","last_unstale":"2024-03-26T05:20:50.994045+0000","last_undegraded":"2024-03-26T05:20:16.371306+0000","last_fullsized":"2024-03-26T05:20:16.356663+0000","mapping_epoch":436,"log_start":"424'4654","ondisk_log_start":"424'4654","created":408,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388497432,"num_objects":5929,"num_object_clones":0,"num_object_copies":17787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9267,"num_write_kb":486209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":440,"num_bytes_recovered":28835840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":5929}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"443'12","reported_seq":1136,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.736917+0000","last_change":"2024-03-26T05:20:30.357087+0000","last_active":"2024-03-26T05:20:51.736917+0000","last_peered":"2024-03-26T05:20:51.736917+0000","last_clean":"2024-03-26T05:20:51.736917+0000","last_became_active":"2024-03-26T05:20:30.350209+0000","last_became_peered":"2024-03-26T05:20:30.350209+0000","last_unstale":"2024-03-26T05:20:51.736917+0000","last_undegraded":"2024-03-26T05:20:51.736917+0000","last_fullsized":"2024-03-26T05:20:51.736917+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:18:36.306612+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.306612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"443'36","reported_seq":54,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.313083+0000","last_change":"2024-03-26T05:20:30.389164+0000","last_active":"2024-03-26T05:20:48.313083+0000","last_peered":"2024-03-26T05:20:48.313083+0000","last_clean":"2024-03-26T05:20:48.313083+0000","last_became_active":"2024-03-26T05:20:30.345634+0000","last_became_peered":"2024-03-26T05:20:30.345634+0000","last_unstale":"2024-03-26T05:20:48.313083+0000","last_undegraded":"2024-03-26T05:20:48.313083+0000","last_fullsized":"2024-03-26T05:20:48.313083+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:14:05.219767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:05.219767+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":141,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315086+0000","last_change":"2024-03-26T05:20:30.342087+0000","last_active":"2024-03-26T05:20:48.315086+0000","last_peered":"2024-03-26T05:20:48.315086+0000","last_clean":"2024-03-26T05:20:48.315086+0000","last_became_active":"2024-03-26T05:20:30.341758+0000","last_became_peered":"2024-03-26T05:20:30.341758+0000","last_unstale":"2024-03-26T05:20:48.315086+0000","last_undegraded":"2024-03-26T05:20:48.315086+0000","last_fullsized":"2024-03-26T05:20:48.315086+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:35.144207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:35.144207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":846,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.735778+0000","last_change":"2024-03-26T05:20:04.436906+0000","last_active":"2024-03-26T05:20:51.735778+0000","last_peered":"2024-03-26T05:20:51.735778+0000","last_clean":"2024-03-26T05:20:51.735778+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:20:51.735778+0000","last_undegraded":"2024-03-26T05:20:51.735778+0000","last_fullsized":"2024-03-26T05:20:51.735778+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:04.436614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:20:04.436614+0000","last_clean_scrub_stamp":"2024-03-26T05:20:04.436614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:29.645219+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"88'128","reported_seq":596,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314443+0000","last_change":"2024-03-26T05:19:43.824458+0000","last_active":"2024-03-26T05:20:48.314443+0000","last_peered":"2024-03-26T05:20:48.314443+0000","last_clean":"2024-03-26T05:20:48.314443+0000","last_became_active":"2024-03-26T05:15:33.617538+0000","last_became_peered":"2024-03-26T05:15:33.617538+0000","last_unstale":"2024-03-26T05:20:48.314443+0000","last_undegraded":"2024-03-26T05:20:48.314443+0000","last_fullsized":"2024-03-26T05:20:48.314443+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:19:43.824375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.824375+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:00.790925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"450'9407","reported_seq":20354,"reported_epoch":450,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-03-26T05:20:51.311867+0000","last_change":"2024-03-26T05:20:30.356481+0000","last_active":"2024-03-26T05:20:51.311867+0000","last_peered":"2024-03-26T05:20:51.311867+0000","last_clean":"2024-03-26T05:18:36.613885+0000","last_became_active":"2024-03-26T05:20:16.423747+0000","last_became_peered":"2024-03-26T05:20:16.423747+0000","last_unstale":"2024-03-26T05:20:51.311867+0000","last_undegraded":"2024-03-26T05:20:16.371565+0000","last_fullsized":"2024-03-26T05:20:16.351124+0000","mapping_epoch":436,"log_start":"427'5638","ondisk_log_start":"427'5638","created":408,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393019392,"num_objects":5997,"num_object_clones":0,"num_object_copies":17991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9407,"num_write_kb":492928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5423,"num_bytes_recovered":355401728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5997}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":887,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314519+0000","last_change":"2024-03-26T05:20:30.343778+0000","last_active":"2024-03-26T05:20:48.314519+0000","last_peered":"2024-03-26T05:20:48.314519+0000","last_clean":"2024-03-26T05:20:48.314519+0000","last_became_active":"2024-03-26T05:20:30.343345+0000","last_became_peered":"2024-03-26T05:20:30.343345+0000","last_unstale":"2024-03-26T05:20:48.314519+0000","last_undegraded":"2024-03-26T05:20:48.314519+0000","last_fullsized":"2024-03-26T05:20:48.314519+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:46.815116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:19:46.815116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:57.760096+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"443'54","reported_seq":1019,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.736560+0000","last_change":"2024-03-26T05:20:30.369614+0000","last_active":"2024-03-26T05:20:51.736560+0000","last_peered":"2024-03-26T05:20:51.736560+0000","last_clean":"2024-03-26T05:20:51.736560+0000","last_became_active":"2024-03-26T05:20:30.347061+0000","last_became_peered":"2024-03-26T05:20:30.347061+0000","last_unstale":"2024-03-26T05:20:51.736560+0000","last_undegraded":"2024-03-26T05:20:51.736560+0000","last_fullsized":"2024-03-26T05:20:51.736560+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'44","last_scrub_stamp":"2024-03-26T05:18:43.255284+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:18:43.255284+0000","objects_scrubbed":9,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":880,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314772+0000","last_change":"2024-03-26T05:20:30.342347+0000","last_active":"2024-03-26T05:20:48.314772+0000","last_peered":"2024-03-26T05:20:48.314772+0000","last_clean":"2024-03-26T05:20:48.314772+0000","last_became_active":"2024-03-26T05:20:30.341954+0000","last_became_peered":"2024-03-26T05:20:30.341954+0000","last_unstale":"2024-03-26T05:20:48.314772+0000","last_undegraded":"2024-03-26T05:20:48.314772+0000","last_fullsized":"2024-03-26T05:20:48.314772+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:48.584746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:11:29.514915+0000","last_clean_scrub_stamp":"2024-03-26T05:19:48.584746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:12.421895+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":842,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311137+0000","last_change":"2024-03-26T05:19:44.571275+0000","last_active":"2024-03-26T05:20:48.311137+0000","last_peered":"2024-03-26T05:20:48.311137+0000","last_clean":"2024-03-26T05:20:48.311137+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:20:48.311137+0000","last_undegraded":"2024-03-26T05:20:48.311137+0000","last_fullsized":"2024-03-26T05:20:48.311137+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:44.571153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:19:44.571153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:53.112338+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"88'128","reported_seq":598,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314935+0000","last_change":"2024-03-26T05:19:56.753537+0000","last_active":"2024-03-26T05:20:48.314935+0000","last_peered":"2024-03-26T05:20:48.314935+0000","last_clean":"2024-03-26T05:20:48.314935+0000","last_became_active":"2024-03-26T05:15:22.624234+0000","last_became_peered":"2024-03-26T05:15:22.624234+0000","last_unstale":"2024-03-26T05:20:48.314935+0000","last_undegraded":"2024-03-26T05:20:48.314935+0000","last_fullsized":"2024-03-26T05:20:48.314935+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:19:56.753458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:19:56.753458+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:18.489667+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"450'9052","reported_seq":9146,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:50.137438+0000","last_change":"2024-03-26T05:18:36.692740+0000","last_active":"2024-03-26T05:20:50.137438+0000","last_peered":"2024-03-26T05:20:50.137438+0000","last_clean":"2024-03-26T05:20:50.137438+0000","last_became_active":"2024-03-26T05:18:36.689513+0000","last_became_peered":"2024-03-26T05:18:36.689513+0000","last_unstale":"2024-03-26T05:20:50.137438+0000","last_undegraded":"2024-03-26T05:20:50.137438+0000","last_fullsized":"2024-03-26T05:20:50.137438+0000","mapping_epoch":408,"log_start":"427'5497","ondisk_log_start":"427'5497","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3555,"log_dups_size":0,"ondisk_log_size":3555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386793472,"num_objects":5902,"num_object_clones":0,"num_object_copies":17706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9052,"num_write_kb":478528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":148,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.320367+0000","last_change":"2024-03-26T05:20:30.339317+0000","last_active":"2024-03-26T05:20:48.320367+0000","last_peered":"2024-03-26T05:20:48.320367+0000","last_clean":"2024-03-26T05:20:48.320367+0000","last_became_active":"2024-03-26T05:20:30.339002+0000","last_became_peered":"2024-03-26T05:20:30.339002+0000","last_unstale":"2024-03-26T05:20:48.320367+0000","last_undegraded":"2024-03-26T05:20:48.320367+0000","last_fullsized":"2024-03-26T05:20:48.320367+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:57.304687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.304687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:22.654585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"443'42","reported_seq":953,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.755190+0000","last_change":"2024-03-26T05:20:05.431539+0000","last_active":"2024-03-26T05:20:51.755190+0000","last_peered":"2024-03-26T05:20:51.755190+0000","last_clean":"2024-03-26T05:20:51.755190+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:20:51.755190+0000","last_undegraded":"2024-03-26T05:20:51.755190+0000","last_fullsized":"2024-03-26T05:20:51.755190+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'30","last_scrub_stamp":"2024-03-26T05:20:05.430706+0000","last_deep_scrub":"254'30","last_deep_scrub_stamp":"2024-03-26T05:20:05.430706+0000","last_clean_scrub_stamp":"2024-03-26T05:20:05.430706+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:31.787575+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":1782,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:49.283236+0000","last_change":"2024-03-26T05:19:47.348573+0000","last_active":"2024-03-26T05:20:49.283236+0000","last_peered":"2024-03-26T05:20:49.283236+0000","last_clean":"2024-03-26T05:20:49.283236+0000","last_became_active":"2024-03-26T05:15:32.633061+0000","last_became_peered":"2024-03-26T05:15:32.633061+0000","last_unstale":"2024-03-26T05:20:49.283236+0000","last_undegraded":"2024-03-26T05:20:49.283236+0000","last_fullsized":"2024-03-26T05:20:49.283236+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:19:47.348445+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:19:47.348445+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:04.985958+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":383,"num_read_kb":378,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":571,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.737790+0000","last_change":"2024-03-26T05:20:30.349671+0000","last_active":"2024-03-26T05:20:51.737790+0000","last_peered":"2024-03-26T05:20:51.737790+0000","last_clean":"2024-03-26T05:20:51.737790+0000","last_became_active":"2024-03-26T05:20:30.349459+0000","last_became_peered":"2024-03-26T05:20:30.349459+0000","last_unstale":"2024-03-26T05:20:51.737790+0000","last_undegraded":"2024-03-26T05:20:51.737790+0000","last_fullsized":"2024-03-26T05:20:51.737790+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:16.863389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.863389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":868,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314272+0000","last_change":"2024-03-26T05:20:30.332076+0000","last_active":"2024-03-26T05:20:48.314272+0000","last_peered":"2024-03-26T05:20:48.314272+0000","last_clean":"2024-03-26T05:20:48.314272+0000","last_became_active":"2024-03-26T05:20:30.331524+0000","last_became_peered":"2024-03-26T05:20:30.331524+0000","last_unstale":"2024-03-26T05:20:48.314272+0000","last_undegraded":"2024-03-26T05:20:48.314272+0000","last_fullsized":"2024-03-26T05:20:48.314272+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:52.594486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:52.594486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:11.188826+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"88'128","reported_seq":664,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315056+0000","last_change":"2024-03-26T05:20:32.758080+0000","last_active":"2024-03-26T05:20:48.315056+0000","last_peered":"2024-03-26T05:20:48.315056+0000","last_clean":"2024-03-26T05:20:48.315056+0000","last_became_active":"2024-03-26T05:20:32.754918+0000","last_became_peered":"2024-03-26T05:20:32.754918+0000","last_unstale":"2024-03-26T05:20:48.315056+0000","last_undegraded":"2024-03-26T05:20:48.315056+0000","last_fullsized":"2024-03-26T05:20:48.315056+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:15:44.054471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:15:44.054471+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"450'9313","reported_seq":12900,"reported_epoch":450,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T05:20:50.129486+0000","last_change":"2024-03-26T05:20:31.424738+0000","last_active":"2024-03-26T05:20:50.129486+0000","last_peered":"2024-03-26T05:20:50.129486+0000","last_clean":"2024-03-26T05:20:29.304433+0000","last_became_active":"2024-03-26T05:20:31.403994+0000","last_became_peered":"2024-03-26T05:20:31.403994+0000","last_unstale":"2024-03-26T05:20:50.129486+0000","last_undegraded":"2024-03-26T05:20:50.129486+0000","last_fullsized":"2024-03-26T05:20:50.129486+0000","mapping_epoch":443,"log_start":"429'5774","ondisk_log_start":"429'5774","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3539,"log_dups_size":0,"ondisk_log_size":3539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390135808,"num_objects":5953,"num_object_clones":0,"num_object_copies":17859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4367,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9313,"num_write_kb":488512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1743,"num_bytes_recovered":114229248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":5953}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":780,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312625+0000","last_change":"2024-03-26T05:20:30.342594+0000","last_active":"2024-03-26T05:20:48.312625+0000","last_peered":"2024-03-26T05:20:48.312625+0000","last_clean":"2024-03-26T05:20:48.312625+0000","last_became_active":"2024-03-26T05:20:30.342259+0000","last_became_peered":"2024-03-26T05:20:30.342259+0000","last_unstale":"2024-03-26T05:20:48.312625+0000","last_undegraded":"2024-03-26T05:20:48.312625+0000","last_fullsized":"2024-03-26T05:20:48.312625+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:43.408113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:14.948209+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.408113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:11.432907+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"443'52","reported_seq":1192,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315695+0000","last_change":"2024-03-26T05:20:15.456992+0000","last_active":"2024-03-26T05:20:48.315695+0000","last_peered":"2024-03-26T05:20:48.315695+0000","last_clean":"2024-03-26T05:20:48.315695+0000","last_became_active":"2024-03-26T05:20:15.372700+0000","last_became_peered":"2024-03-26T05:20:15.372700+0000","last_unstale":"2024-03-26T05:20:48.315695+0000","last_undegraded":"2024-03-26T05:20:48.315695+0000","last_fullsized":"2024-03-26T05:20:48.315695+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'38","last_scrub_stamp":"2024-03-26T05:15:46.434771+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:15:46.434771+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":864,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.319715+0000","last_change":"2024-03-26T05:20:30.350444+0000","last_active":"2024-03-26T05:20:48.319715+0000","last_peered":"2024-03-26T05:20:48.319715+0000","last_clean":"2024-03-26T05:20:48.319715+0000","last_became_active":"2024-03-26T05:20:30.349656+0000","last_became_peered":"2024-03-26T05:20:30.349656+0000","last_unstale":"2024-03-26T05:20:48.319715+0000","last_undegraded":"2024-03-26T05:20:48.319715+0000","last_fullsized":"2024-03-26T05:20:48.319715+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:46.747932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:46.747932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":1021,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312576+0000","last_change":"2024-03-26T05:18:58.577783+0000","last_active":"2024-03-26T05:20:48.312576+0000","last_peered":"2024-03-26T05:20:48.312576+0000","last_clean":"2024-03-26T05:20:48.312576+0000","last_became_active":"2024-03-26T05:15:32.632929+0000","last_became_peered":"2024-03-26T05:15:32.632929+0000","last_unstale":"2024-03-26T05:20:48.312576+0000","last_undegraded":"2024-03-26T05:20:48.312576+0000","last_fullsized":"2024-03-26T05:20:48.312576+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:18:58.577676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:18:58.577676+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":517,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315793+0000","last_change":"2024-03-26T05:20:30.347796+0000","last_active":"2024-03-26T05:20:48.315793+0000","last_peered":"2024-03-26T05:20:48.315793+0000","last_clean":"2024-03-26T05:20:48.315793+0000","last_became_active":"2024-03-26T05:20:30.346881+0000","last_became_peered":"2024-03-26T05:20:30.346881+0000","last_unstale":"2024-03-26T05:20:48.315793+0000","last_undegraded":"2024-03-26T05:20:48.315793+0000","last_fullsized":"2024-03-26T05:20:48.315793+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:14.528992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:14.528992+0000","last_clean_scrub_stamp":"2024-03-26T05:15:14.528992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"450'9335","reported_seq":9438,"reported_epoch":450,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T05:20:50.125956+0000","last_change":"2024-03-26T05:20:31.419126+0000","last_active":"2024-03-26T05:20:50.125956+0000","last_peered":"2024-03-26T05:20:50.125956+0000","last_clean":"2024-03-26T05:20:29.277491+0000","last_became_active":"2024-03-26T05:20:31.394946+0000","last_became_peered":"2024-03-26T05:20:31.394946+0000","last_unstale":"2024-03-26T05:20:50.125956+0000","last_undegraded":"2024-03-26T05:20:50.125956+0000","last_fullsized":"2024-03-26T05:20:50.125956+0000","mapping_epoch":443,"log_start":"429'5835","ondisk_log_start":"429'5835","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389480448,"num_objects":5943,"num_object_clones":0,"num_object_copies":17829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5943,"num_objects_unfound":0,"num_objects_dirty":5943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9335,"num_write_kb":488896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":5943}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":873,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.321108+0000","last_change":"2024-03-26T05:20:30.336844+0000","last_active":"2024-03-26T05:20:48.321108+0000","last_peered":"2024-03-26T05:20:48.321108+0000","last_clean":"2024-03-26T05:20:48.321108+0000","last_became_active":"2024-03-26T05:20:30.335654+0000","last_became_peered":"2024-03-26T05:20:30.335654+0000","last_unstale":"2024-03-26T05:20:48.321108+0000","last_undegraded":"2024-03-26T05:20:48.321108+0000","last_fullsized":"2024-03-26T05:20:48.321108+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:54.964851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.964851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:20.185778+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"443'27","reported_seq":694,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.320363+0000","last_change":"2024-03-26T05:20:30.365782+0000","last_active":"2024-03-26T05:20:48.320363+0000","last_peered":"2024-03-26T05:20:48.320363+0000","last_clean":"2024-03-26T05:20:48.320363+0000","last_became_active":"2024-03-26T05:20:30.345752+0000","last_became_peered":"2024-03-26T05:20:30.345752+0000","last_unstale":"2024-03-26T05:20:48.320363+0000","last_undegraded":"2024-03-26T05:20:48.320363+0000","last_fullsized":"2024-03-26T05:20:48.320363+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'23","last_scrub_stamp":"2024-03-26T05:19:53.957953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:19:53.957953+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:00.451309+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":871,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.319107+0000","last_change":"2024-03-26T05:19:50.912733+0000","last_active":"2024-03-26T05:20:48.319107+0000","last_peered":"2024-03-26T05:20:48.319107+0000","last_clean":"2024-03-26T05:20:48.319107+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:20:48.319107+0000","last_undegraded":"2024-03-26T05:20:48.319107+0000","last_fullsized":"2024-03-26T05:20:48.319107+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:50.912594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:03.193105+0000","last_clean_scrub_stamp":"2024-03-26T05:19:50.912594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:57.950051+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":855,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312299+0000","last_change":"2024-03-26T05:19:46.522718+0000","last_active":"2024-03-26T05:20:48.312299+0000","last_peered":"2024-03-26T05:20:48.312299+0000","last_clean":"2024-03-26T05:20:48.312299+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:20:48.312299+0000","last_undegraded":"2024-03-26T05:20:48.312299+0000","last_fullsized":"2024-03-26T05:20:48.312299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:46.522602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:19:46.522602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:53.162409+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":714,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312457+0000","last_change":"2024-03-26T05:18:41.624630+0000","last_active":"2024-03-26T05:20:48.312457+0000","last_peered":"2024-03-26T05:20:48.312457+0000","last_clean":"2024-03-26T05:20:48.312457+0000","last_became_active":"2024-03-26T05:15:32.633633+0000","last_became_peered":"2024-03-26T05:15:32.633633+0000","last_unstale":"2024-03-26T05:20:48.312457+0000","last_undegraded":"2024-03-26T05:20:48.312457+0000","last_fullsized":"2024-03-26T05:20:48.312457+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:41.624502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:18:41.624502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"88'128","reported_seq":1015,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311620+0000","last_change":"2024-03-26T05:20:17.750124+0000","last_active":"2024-03-26T05:20:48.311620+0000","last_peered":"2024-03-26T05:20:48.311620+0000","last_clean":"2024-03-26T05:20:48.311620+0000","last_became_active":"2024-03-26T05:20:17.749765+0000","last_became_peered":"2024-03-26T05:20:17.749765+0000","last_unstale":"2024-03-26T05:20:48.311620+0000","last_undegraded":"2024-03-26T05:20:48.311620+0000","last_fullsized":"2024-03-26T05:20:48.311620+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:48.393726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"450'9111","reported_seq":9206,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:50.120235+0000","last_change":"2024-03-26T05:18:36.696360+0000","last_active":"2024-03-26T05:20:50.120235+0000","last_peered":"2024-03-26T05:20:50.120235+0000","last_clean":"2024-03-26T05:20:50.120235+0000","last_became_active":"2024-03-26T05:18:36.695687+0000","last_became_peered":"2024-03-26T05:18:36.695687+0000","last_unstale":"2024-03-26T05:20:50.120235+0000","last_undegraded":"2024-03-26T05:20:50.120235+0000","last_fullsized":"2024-03-26T05:20:50.120235+0000","mapping_epoch":408,"log_start":"427'5526","ondisk_log_start":"427'5526","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386203648,"num_objects":5893,"num_object_clones":0,"num_object_copies":17679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9111,"num_write_kb":480128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":842,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.735482+0000","last_change":"2024-03-26T05:20:15.377798+0000","last_active":"2024-03-26T05:20:51.735482+0000","last_peered":"2024-03-26T05:20:51.735482+0000","last_clean":"2024-03-26T05:20:51.735482+0000","last_became_active":"2024-03-26T05:20:15.377492+0000","last_became_peered":"2024-03-26T05:20:15.377492+0000","last_unstale":"2024-03-26T05:20:51.735482+0000","last_undegraded":"2024-03-26T05:20:51.735482+0000","last_fullsized":"2024-03-26T05:20:51.735482+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:14.131427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:14.131427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"443'58","reported_seq":1203,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314928+0000","last_change":"2024-03-26T05:20:00.666872+0000","last_active":"2024-03-26T05:20:48.314928+0000","last_peered":"2024-03-26T05:20:48.314928+0000","last_clean":"2024-03-26T05:20:48.314928+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:20:48.314928+0000","last_undegraded":"2024-03-26T05:20:48.314928+0000","last_fullsized":"2024-03-26T05:20:48.314928+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'46","last_scrub_stamp":"2024-03-26T05:20:00.666615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:00.666615+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:22.537340+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":232,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.736463+0000","last_change":"2024-03-26T05:20:15.457605+0000","last_active":"2024-03-26T05:20:51.736463+0000","last_peered":"2024-03-26T05:20:51.736463+0000","last_clean":"2024-03-26T05:20:51.736463+0000","last_became_active":"2024-03-26T05:20:15.457163+0000","last_became_peered":"2024-03-26T05:20:15.457163+0000","last_unstale":"2024-03-26T05:20:51.736463+0000","last_undegraded":"2024-03-26T05:20:51.736463+0000","last_fullsized":"2024-03-26T05:20:51.736463+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:02.881800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:02.881800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":182,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312810+0000","last_change":"2024-03-26T05:20:30.355675+0000","last_active":"2024-03-26T05:20:48.312810+0000","last_peered":"2024-03-26T05:20:48.312810+0000","last_clean":"2024-03-26T05:20:48.312810+0000","last_became_active":"2024-03-26T05:20:30.339519+0000","last_became_peered":"2024-03-26T05:20:30.339519+0000","last_unstale":"2024-03-26T05:20:48.312810+0000","last_undegraded":"2024-03-26T05:20:48.312810+0000","last_fullsized":"2024-03-26T05:20:48.312810+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:18:41.097424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:18:41.097424+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":820,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.316827+0000","last_change":"2024-03-26T05:20:15.360008+0000","last_active":"2024-03-26T05:20:48.316827+0000","last_peered":"2024-03-26T05:20:48.316827+0000","last_clean":"2024-03-26T05:20:48.316827+0000","last_became_active":"2024-03-26T05:20:15.359520+0000","last_became_peered":"2024-03-26T05:20:15.359520+0000","last_unstale":"2024-03-26T05:20:48.316827+0000","last_undegraded":"2024-03-26T05:20:48.316827+0000","last_fullsized":"2024-03-26T05:20:48.316827+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:31.518660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:31.518660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"88'128","reported_seq":417,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.737376+0000","last_change":"2024-03-26T05:20:32.759374+0000","last_active":"2024-03-26T05:20:51.737376+0000","last_peered":"2024-03-26T05:20:51.737376+0000","last_clean":"2024-03-26T05:20:51.737376+0000","last_became_active":"2024-03-26T05:20:32.758743+0000","last_became_peered":"2024-03-26T05:20:32.758743+0000","last_unstale":"2024-03-26T05:20:51.737376+0000","last_undegraded":"2024-03-26T05:20:51.737376+0000","last_fullsized":"2024-03-26T05:20:51.737376+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:19:45.389633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:19:45.389633+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:53.555716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"450'9412","reported_seq":9584,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:50.887413+0000","last_change":"2024-03-26T05:20:05.667786+0000","last_active":"2024-03-26T05:20:50.887413+0000","last_peered":"2024-03-26T05:20:50.887413+0000","last_clean":"2024-03-26T05:20:50.887413+0000","last_became_active":"2024-03-26T05:18:36.687838+0000","last_became_peered":"2024-03-26T05:18:36.687838+0000","last_unstale":"2024-03-26T05:20:50.887413+0000","last_undegraded":"2024-03-26T05:20:50.887413+0000","last_fullsized":"2024-03-26T05:20:50.887413+0000","mapping_epoch":408,"log_start":"427'5621","ondisk_log_start":"427'5621","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'6714","last_scrub_stamp":"2024-03-26T05:20:05.667571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:20:05.667571+0000","objects_scrubbed":6680,"log_size":3791,"log_dups_size":0,"ondisk_log_size":3791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:08.759966+0000","scrub_duration":1569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391774208,"num_objects":5978,"num_object_clones":0,"num_object_copies":17934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9412,"num_write_kb":492480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"443'27","reported_seq":928,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.736795+0000","last_change":"2024-03-26T05:20:16.386890+0000","last_active":"2024-03-26T05:20:51.736795+0000","last_peered":"2024-03-26T05:20:51.736795+0000","last_clean":"2024-03-26T05:20:51.736795+0000","last_became_active":"2024-03-26T05:20:16.378745+0000","last_became_peered":"2024-03-26T05:20:16.378745+0000","last_unstale":"2024-03-26T05:20:51.736795+0000","last_undegraded":"2024-03-26T05:20:51.736795+0000","last_fullsized":"2024-03-26T05:20:51.736795+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:18:31.264666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:18:31.264666+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":141,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.313093+0000","last_change":"2024-03-26T05:20:30.327239+0000","last_active":"2024-03-26T05:20:48.313093+0000","last_peered":"2024-03-26T05:20:48.313093+0000","last_clean":"2024-03-26T05:20:48.313093+0000","last_became_active":"2024-03-26T05:20:30.326917+0000","last_became_peered":"2024-03-26T05:20:30.326917+0000","last_unstale":"2024-03-26T05:20:48.313093+0000","last_undegraded":"2024-03-26T05:20:48.313093+0000","last_fullsized":"2024-03-26T05:20:48.313093+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:58.239217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:19:58.239217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:25.709717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":847,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.319123+0000","last_change":"2024-03-26T05:20:15.381990+0000","last_active":"2024-03-26T05:20:48.319123+0000","last_peered":"2024-03-26T05:20:48.319123+0000","last_clean":"2024-03-26T05:20:48.319123+0000","last_became_active":"2024-03-26T05:20:15.380969+0000","last_became_peered":"2024-03-26T05:20:15.380969+0000","last_unstale":"2024-03-26T05:20:48.319123+0000","last_undegraded":"2024-03-26T05:20:48.319123+0000","last_fullsized":"2024-03-26T05:20:48.319123+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:52.371799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:52.371799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":838,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.738145+0000","last_change":"2024-03-26T05:19:50.540208+0000","last_active":"2024-03-26T05:20:51.738145+0000","last_peered":"2024-03-26T05:20:51.738145+0000","last_clean":"2024-03-26T05:20:51.738145+0000","last_became_active":"2024-03-26T05:15:32.639943+0000","last_became_peered":"2024-03-26T05:15:32.639943+0000","last_unstale":"2024-03-26T05:20:51.738145+0000","last_undegraded":"2024-03-26T05:20:51.738145+0000","last_fullsized":"2024-03-26T05:20:51.738145+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:50.540100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:50.540100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:04.772603+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"450'9223","reported_seq":9384,"reported_epoch":450,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T05:20:50.855190+0000","last_change":"2024-03-26T05:20:31.425123+0000","last_active":"2024-03-26T05:20:50.855190+0000","last_peered":"2024-03-26T05:20:50.855190+0000","last_clean":"2024-03-26T05:20:29.303022+0000","last_became_active":"2024-03-26T05:20:31.393181+0000","last_became_peered":"2024-03-26T05:20:31.393181+0000","last_unstale":"2024-03-26T05:20:50.855190+0000","last_undegraded":"2024-03-26T05:20:50.855190+0000","last_fullsized":"2024-03-26T05:20:50.855190+0000","mapping_epoch":443,"log_start":"427'5426","ondisk_log_start":"427'5426","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"427'5138","last_scrub_stamp":"2024-03-26T05:19:43.145684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.145684+0000","objects_scrubbed":5110,"log_size":3797,"log_dups_size":0,"ondisk_log_size":3797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":796,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381353984,"num_objects":5819,"num_object_clones":0,"num_object_copies":17457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5819,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9223,"num_write_kb":481344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"5,6,7","objects":5819}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"443'67","reported_seq":959,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314405+0000","last_change":"2024-03-26T05:18:50.837517+0000","last_active":"2024-03-26T05:20:48.314405+0000","last_peered":"2024-03-26T05:20:48.314405+0000","last_clean":"2024-03-26T05:20:48.314405+0000","last_became_active":"2024-03-26T05:15:32.626640+0000","last_became_peered":"2024-03-26T05:15:32.626640+0000","last_unstale":"2024-03-26T05:20:48.314405+0000","last_undegraded":"2024-03-26T05:20:48.314405+0000","last_fullsized":"2024-03-26T05:20:48.314405+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'51","last_scrub_stamp":"2024-03-26T05:18:50.837412+0000","last_deep_scrub":"343'51","last_deep_scrub_stamp":"2024-03-26T05:17:45.197613+0000","last_clean_scrub_stamp":"2024-03-26T05:18:50.837412+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":741,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314570+0000","last_change":"2024-03-26T05:20:30.330168+0000","last_active":"2024-03-26T05:20:48.314570+0000","last_peered":"2024-03-26T05:20:48.314570+0000","last_clean":"2024-03-26T05:20:48.314570+0000","last_became_active":"2024-03-26T05:20:30.329736+0000","last_became_peered":"2024-03-26T05:20:30.329736+0000","last_unstale":"2024-03-26T05:20:48.314570+0000","last_undegraded":"2024-03-26T05:20:48.314570+0000","last_fullsized":"2024-03-26T05:20:48.314570+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:46.539478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:46.539478+0000","last_clean_scrub_stamp":"2024-03-26T05:18:46.539478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":872,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.319577+0000","last_change":"2024-03-26T05:20:30.344308+0000","last_active":"2024-03-26T05:20:48.319577+0000","last_peered":"2024-03-26T05:20:48.319577+0000","last_clean":"2024-03-26T05:20:48.319577+0000","last_became_active":"2024-03-26T05:20:30.343819+0000","last_became_peered":"2024-03-26T05:20:30.343819+0000","last_unstale":"2024-03-26T05:20:48.319577+0000","last_undegraded":"2024-03-26T05:20:48.319577+0000","last_fullsized":"2024-03-26T05:20:48.319577+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:57.997324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.997324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:25.709329+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":746,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311218+0000","last_change":"2024-03-26T05:19:48.333412+0000","last_active":"2024-03-26T05:20:48.311218+0000","last_peered":"2024-03-26T05:20:48.311218+0000","last_clean":"2024-03-26T05:20:48.311218+0000","last_became_active":"2024-03-26T05:15:32.633481+0000","last_became_peered":"2024-03-26T05:15:32.633481+0000","last_unstale":"2024-03-26T05:20:48.311218+0000","last_undegraded":"2024-03-26T05:20:48.311218+0000","last_fullsized":"2024-03-26T05:20:48.311218+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:48.333336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_clean_scrub_stamp":"2024-03-26T05:19:48.333336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:56.768510+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"451'9333","reported_seq":9430,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.754920+0000","last_change":"2024-03-26T05:18:36.695182+0000","last_active":"2024-03-26T05:20:51.754920+0000","last_peered":"2024-03-26T05:20:51.754920+0000","last_clean":"2024-03-26T05:20:51.754920+0000","last_became_active":"2024-03-26T05:18:36.693789+0000","last_became_peered":"2024-03-26T05:18:36.693789+0000","last_unstale":"2024-03-26T05:20:51.754920+0000","last_undegraded":"2024-03-26T05:20:51.754920+0000","last_fullsized":"2024-03-26T05:20:51.754920+0000","mapping_epoch":408,"log_start":"427'5627","ondisk_log_start":"427'5627","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3706,"log_dups_size":0,"ondisk_log_size":3706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379387904,"num_objects":5789,"num_object_clones":0,"num_object_copies":17367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9333,"num_write_kb":483904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":846,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.319671+0000","last_change":"2024-03-26T05:18:52.271198+0000","last_active":"2024-03-26T05:20:48.319671+0000","last_peered":"2024-03-26T05:20:48.319671+0000","last_clean":"2024-03-26T05:20:48.319671+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:20:48.319671+0000","last_undegraded":"2024-03-26T05:20:48.319671+0000","last_fullsized":"2024-03-26T05:20:48.319671+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:52.271117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:52.271117+0000","last_clean_scrub_stamp":"2024-03-26T05:18:52.271117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"443'43","reported_seq":732,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.310546+0000","last_change":"2024-03-26T05:20:15.454067+0000","last_active":"2024-03-26T05:20:48.310546+0000","last_peered":"2024-03-26T05:20:48.310546+0000","last_clean":"2024-03-26T05:20:48.310546+0000","last_became_active":"2024-03-26T05:20:15.361123+0000","last_became_peered":"2024-03-26T05:20:15.361123+0000","last_unstale":"2024-03-26T05:20:48.310546+0000","last_undegraded":"2024-03-26T05:20:48.310546+0000","last_fullsized":"2024-03-26T05:20:48.310546+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-26T05:14:55.963773+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_clean_scrub_stamp":"2024-03-26T05:14:55.963773+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":758,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312996+0000","last_change":"2024-03-26T05:20:15.374221+0000","last_active":"2024-03-26T05:20:48.312996+0000","last_peered":"2024-03-26T05:20:48.312996+0000","last_clean":"2024-03-26T05:20:48.312996+0000","last_became_active":"2024-03-26T05:20:15.373910+0000","last_became_peered":"2024-03-26T05:20:15.373910+0000","last_unstale":"2024-03-26T05:20:48.312996+0000","last_undegraded":"2024-03-26T05:20:48.312996+0000","last_fullsized":"2024-03-26T05:20:48.312996+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:20.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:20.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":833,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311861+0000","last_change":"2024-03-26T05:20:15.377202+0000","last_active":"2024-03-26T05:20:48.311861+0000","last_peered":"2024-03-26T05:20:48.311861+0000","last_clean":"2024-03-26T05:20:48.311861+0000","last_became_active":"2024-03-26T05:20:15.376729+0000","last_became_peered":"2024-03-26T05:20:15.376729+0000","last_unstale":"2024-03-26T05:20:48.311861+0000","last_undegraded":"2024-03-26T05:20:48.311861+0000","last_fullsized":"2024-03-26T05:20:48.311861+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:51.030982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:51.030982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"450'9124","reported_seq":13727,"reported_epoch":450,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-03-26T05:20:51.305963+0000","last_change":"2024-03-26T05:20:31.423903+0000","last_active":"2024-03-26T05:20:51.305963+0000","last_peered":"2024-03-26T05:20:51.305963+0000","last_clean":"2024-03-26T05:18:36.613885+0000","last_became_active":"2024-03-26T05:20:31.374964+0000","last_became_peered":"2024-03-26T05:20:31.374964+0000","last_unstale":"2024-03-26T05:20:51.305963+0000","last_undegraded":"2024-03-26T05:20:31.327999+0000","last_fullsized":"2024-03-26T05:20:31.310608+0000","mapping_epoch":443,"log_start":"427'5399","ondisk_log_start":"427'5399","created":408,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3725,"log_dups_size":0,"ondisk_log_size":3725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369754112,"num_objects":5642,"num_object_clones":0,"num_object_copies":16926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4287,"num_objects_misplaced":4287,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9124,"num_write_kb":472512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1500,"num_bytes_recovered":98304000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5642}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"439'12","reported_seq":961,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311650+0000","last_change":"2024-03-26T05:20:15.466421+0000","last_active":"2024-03-26T05:20:48.311650+0000","last_peered":"2024-03-26T05:20:48.311650+0000","last_clean":"2024-03-26T05:20:48.311650+0000","last_became_active":"2024-03-26T05:20:15.379548+0000","last_became_peered":"2024-03-26T05:20:15.379548+0000","last_unstale":"2024-03-26T05:20:48.311650+0000","last_undegraded":"2024-03-26T05:20:48.311650+0000","last_fullsized":"2024-03-26T05:20:48.311650+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:12:03.961391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:03.961391+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"443'60","reported_seq":209,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312061+0000","last_change":"2024-03-26T05:20:30.363622+0000","last_active":"2024-03-26T05:20:48.312061+0000","last_peered":"2024-03-26T05:20:48.312061+0000","last_clean":"2024-03-26T05:20:48.312061+0000","last_became_active":"2024-03-26T05:20:30.341167+0000","last_became_peered":"2024-03-26T05:20:30.341167+0000","last_unstale":"2024-03-26T05:20:48.312061+0000","last_undegraded":"2024-03-26T05:20:48.312061+0000","last_fullsized":"2024-03-26T05:20:48.312061+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'48","last_scrub_stamp":"2024-03-26T05:19:54.208469+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.208469+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:01.316991+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":755,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.313449+0000","last_change":"2024-03-26T05:20:15.470838+0000","last_active":"2024-03-26T05:20:48.313449+0000","last_peered":"2024-03-26T05:20:48.313449+0000","last_clean":"2024-03-26T05:20:48.313449+0000","last_became_active":"2024-03-26T05:20:15.453507+0000","last_became_peered":"2024-03-26T05:20:15.453507+0000","last_unstale":"2024-03-26T05:20:48.313449+0000","last_undegraded":"2024-03-26T05:20:48.313449+0000","last_fullsized":"2024-03-26T05:20:48.313449+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.959120+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":218,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314115+0000","last_change":"2024-03-26T05:20:30.368240+0000","last_active":"2024-03-26T05:20:48.314115+0000","last_peered":"2024-03-26T05:20:48.314115+0000","last_clean":"2024-03-26T05:20:48.314115+0000","last_became_active":"2024-03-26T05:20:30.343599+0000","last_became_peered":"2024-03-26T05:20:30.343599+0000","last_unstale":"2024-03-26T05:20:48.314115+0000","last_undegraded":"2024-03-26T05:20:48.314115+0000","last_fullsized":"2024-03-26T05:20:48.314115+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:20:04.267766+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:20:04.267766+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:04.502532+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"443'24","reported_seq":881,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.316076+0000","last_change":"2024-03-26T05:20:04.274955+0000","last_active":"2024-03-26T05:20:48.316076+0000","last_peered":"2024-03-26T05:20:48.316076+0000","last_clean":"2024-03-26T05:20:48.316076+0000","last_became_active":"2024-03-26T05:15:32.640871+0000","last_became_peered":"2024-03-26T05:15:32.640871+0000","last_unstale":"2024-03-26T05:20:48.316076+0000","last_undegraded":"2024-03-26T05:20:48.316076+0000","last_fullsized":"2024-03-26T05:20:48.316076+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'18","last_scrub_stamp":"2024-03-26T05:20:04.273413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:04.273413+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:29.551846+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":32,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":905,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314028+0000","last_change":"2024-03-26T05:20:30.370885+0000","last_active":"2024-03-26T05:20:48.314028+0000","last_peered":"2024-03-26T05:20:48.314028+0000","last_clean":"2024-03-26T05:20:48.314028+0000","last_became_active":"2024-03-26T05:20:30.343651+0000","last_became_peered":"2024-03-26T05:20:30.343651+0000","last_unstale":"2024-03-26T05:20:48.314028+0000","last_undegraded":"2024-03-26T05:20:48.314028+0000","last_fullsized":"2024-03-26T05:20:48.314028+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:19:45.829142+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:17:50.147911+0000","last_clean_scrub_stamp":"2024-03-26T05:19:45.829142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:02.523386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":139,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.316693+0000","last_change":"2024-03-26T05:20:30.340915+0000","last_active":"2024-03-26T05:20:48.316693+0000","last_peered":"2024-03-26T05:20:48.316693+0000","last_clean":"2024-03-26T05:20:48.316693+0000","last_became_active":"2024-03-26T05:20:30.330443+0000","last_became_peered":"2024-03-26T05:20:30.330443+0000","last_unstale":"2024-03-26T05:20:48.316693+0000","last_undegraded":"2024-03-26T05:20:48.316693+0000","last_fullsized":"2024-03-26T05:20:48.316693+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.146286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.146286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":746,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312577+0000","last_change":"2024-03-26T05:20:30.321663+0000","last_active":"2024-03-26T05:20:48.312577+0000","last_peered":"2024-03-26T05:20:48.312577+0000","last_clean":"2024-03-26T05:20:48.312577+0000","last_became_active":"2024-03-26T05:20:30.321301+0000","last_became_peered":"2024-03-26T05:20:30.321301+0000","last_unstale":"2024-03-26T05:20:48.312577+0000","last_undegraded":"2024-03-26T05:20:48.312577+0000","last_fullsized":"2024-03-26T05:20:48.312577+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:07.006139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.006139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"443'39","reported_seq":917,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.758772+0000","last_change":"2024-03-26T05:19:54.421287+0000","last_active":"2024-03-26T05:20:51.758772+0000","last_peered":"2024-03-26T05:20:51.758772+0000","last_clean":"2024-03-26T05:20:51.758772+0000","last_became_active":"2024-03-26T05:15:32.634434+0000","last_became_peered":"2024-03-26T05:15:32.634434+0000","last_unstale":"2024-03-26T05:20:51.758772+0000","last_undegraded":"2024-03-26T05:20:51.758772+0000","last_fullsized":"2024-03-26T05:20:51.758772+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'33","last_scrub_stamp":"2024-03-26T05:19:54.421104+0000","last_deep_scrub":"429'33","last_deep_scrub_stamp":"2024-03-26T05:19:54.421104+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.421104+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:57.104914+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":714,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311807+0000","last_change":"2024-03-26T05:20:15.376147+0000","last_active":"2024-03-26T05:20:48.311807+0000","last_peered":"2024-03-26T05:20:48.311807+0000","last_clean":"2024-03-26T05:20:48.311807+0000","last_became_active":"2024-03-26T05:20:15.375661+0000","last_became_peered":"2024-03-26T05:20:15.375661+0000","last_unstale":"2024-03-26T05:20:48.311807+0000","last_undegraded":"2024-03-26T05:20:48.311807+0000","last_fullsized":"2024-03-26T05:20:48.311807+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.939576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.939576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":830,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315893+0000","last_change":"2024-03-26T05:18:03.064748+0000","last_active":"2024-03-26T05:20:48.315893+0000","last_peered":"2024-03-26T05:20:48.315893+0000","last_clean":"2024-03-26T05:20:48.315893+0000","last_became_active":"2024-03-26T05:15:32.641031+0000","last_became_peered":"2024-03-26T05:15:32.641031+0000","last_unstale":"2024-03-26T05:20:48.315893+0000","last_undegraded":"2024-03-26T05:20:48.315893+0000","last_fullsized":"2024-03-26T05:20:48.315893+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:03.064638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:18:03.064638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":851,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.319492+0000","last_change":"2024-03-26T05:20:32.095405+0000","last_active":"2024-03-26T05:20:48.319492+0000","last_peered":"2024-03-26T05:20:48.319492+0000","last_clean":"2024-03-26T05:20:48.319492+0000","last_became_active":"2024-03-26T05:20:15.379306+0000","last_became_peered":"2024-03-26T05:20:15.379306+0000","last_unstale":"2024-03-26T05:20:48.319492+0000","last_undegraded":"2024-03-26T05:20:48.319492+0000","last_fullsized":"2024-03-26T05:20:48.319492+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:32.094955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:20:32.094955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:35.084139+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"443'32","reported_seq":158,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312353+0000","last_change":"2024-03-26T05:20:30.356411+0000","last_active":"2024-03-26T05:20:48.312353+0000","last_peered":"2024-03-26T05:20:48.312353+0000","last_clean":"2024-03-26T05:20:48.312353+0000","last_became_active":"2024-03-26T05:20:30.329795+0000","last_became_peered":"2024-03-26T05:20:30.329795+0000","last_unstale":"2024-03-26T05:20:48.312353+0000","last_undegraded":"2024-03-26T05:20:48.312353+0000","last_fullsized":"2024-03-26T05:20:48.312353+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:19:53.221647+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:19:53.221647+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:58.264205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":857,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.756351+0000","last_change":"2024-03-26T05:18:51.276677+0000","last_active":"2024-03-26T05:20:51.756351+0000","last_peered":"2024-03-26T05:20:51.756351+0000","last_clean":"2024-03-26T05:20:51.756351+0000","last_became_active":"2024-03-26T05:15:32.632387+0000","last_became_peered":"2024-03-26T05:15:32.632387+0000","last_unstale":"2024-03-26T05:20:51.756351+0000","last_undegraded":"2024-03-26T05:20:51.756351+0000","last_fullsized":"2024-03-26T05:20:51.756351+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:51.276565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:45.352678+0000","last_clean_scrub_stamp":"2024-03-26T05:18:51.276565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"336'11","reported_seq":1098,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314679+0000","last_change":"2024-03-26T05:19:56.382330+0000","last_active":"2024-03-26T05:20:48.314679+0000","last_peered":"2024-03-26T05:20:48.314679+0000","last_clean":"2024-03-26T05:20:48.314679+0000","last_became_active":"2024-03-26T05:15:32.636392+0000","last_became_peered":"2024-03-26T05:15:32.636392+0000","last_unstale":"2024-03-26T05:20:48.314679+0000","last_undegraded":"2024-03-26T05:20:48.314679+0000","last_fullsized":"2024-03-26T05:20:48.314679+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-03-26T05:19:56.382205+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:19:56.382205+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:14.971262+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":558,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.320264+0000","last_change":"2024-03-26T05:20:30.347802+0000","last_active":"2024-03-26T05:20:48.320264+0000","last_peered":"2024-03-26T05:20:48.320264+0000","last_clean":"2024-03-26T05:20:48.320264+0000","last_became_active":"2024-03-26T05:20:30.344267+0000","last_became_peered":"2024-03-26T05:20:30.344267+0000","last_unstale":"2024-03-26T05:20:48.320264+0000","last_undegraded":"2024-03-26T05:20:48.320264+0000","last_fullsized":"2024-03-26T05:20:48.320264+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:06.096138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:06.096138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"443'49","reported_seq":926,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312460+0000","last_change":"2024-03-26T05:19:58.389112+0000","last_active":"2024-03-26T05:20:48.312460+0000","last_peered":"2024-03-26T05:20:48.312460+0000","last_clean":"2024-03-26T05:20:48.312460+0000","last_became_active":"2024-03-26T05:15:32.626448+0000","last_became_peered":"2024-03-26T05:15:32.626448+0000","last_unstale":"2024-03-26T05:20:48.312460+0000","last_undegraded":"2024-03-26T05:20:48.312460+0000","last_fullsized":"2024-03-26T05:20:48.312460+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'39","last_scrub_stamp":"2024-03-26T05:19:58.388933+0000","last_deep_scrub":"345'39","last_deep_scrub_stamp":"2024-03-26T05:15:56.674423+0000","last_clean_scrub_stamp":"2024-03-26T05:19:58.388933+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:01.771983+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":748,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312180+0000","last_change":"2024-03-26T05:20:15.364695+0000","last_active":"2024-03-26T05:20:48.312180+0000","last_peered":"2024-03-26T05:20:48.312180+0000","last_clean":"2024-03-26T05:20:48.312180+0000","last_became_active":"2024-03-26T05:20:15.364437+0000","last_became_peered":"2024-03-26T05:20:15.364437+0000","last_unstale":"2024-03-26T05:20:48.312180+0000","last_undegraded":"2024-03-26T05:20:48.312180+0000","last_fullsized":"2024-03-26T05:20:48.312180+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:01.962048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:01.962048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"443'11","reported_seq":215,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.313573+0000","last_change":"2024-03-26T05:20:30.356968+0000","last_active":"2024-03-26T05:20:48.313573+0000","last_peered":"2024-03-26T05:20:48.313573+0000","last_clean":"2024-03-26T05:20:48.313573+0000","last_became_active":"2024-03-26T05:20:30.338470+0000","last_became_peered":"2024-03-26T05:20:30.338470+0000","last_unstale":"2024-03-26T05:20:48.313573+0000","last_undegraded":"2024-03-26T05:20:48.313573+0000","last_fullsized":"2024-03-26T05:20:48.313573+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:18:40.082750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:18:40.082750+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":845,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.755096+0000","last_change":"2024-03-26T05:19:53.535068+0000","last_active":"2024-03-26T05:20:51.755096+0000","last_peered":"2024-03-26T05:20:51.755096+0000","last_clean":"2024-03-26T05:20:51.755096+0000","last_became_active":"2024-03-26T05:15:32.631444+0000","last_became_peered":"2024-03-26T05:15:32.631444+0000","last_unstale":"2024-03-26T05:20:51.755096+0000","last_undegraded":"2024-03-26T05:20:51.755096+0000","last_fullsized":"2024-03-26T05:20:51.755096+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:53.534717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:59.508745+0000","last_clean_scrub_stamp":"2024-03-26T05:19:53.534717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:05.454369+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"443'32","reported_seq":730,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315017+0000","last_change":"2024-03-26T05:20:40.672261+0000","last_active":"2024-03-26T05:20:48.315017+0000","last_peered":"2024-03-26T05:20:48.315017+0000","last_clean":"2024-03-26T05:20:48.315017+0000","last_became_active":"2024-03-26T05:20:30.323810+0000","last_became_peered":"2024-03-26T05:20:30.323810+0000","last_unstale":"2024-03-26T05:20:48.315017+0000","last_undegraded":"2024-03-26T05:20:48.315017+0000","last_fullsized":"2024-03-26T05:20:48.315017+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'32","last_scrub_stamp":"2024-03-26T05:20:40.672162+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:20:40.672162+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:51.097496+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":816,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312699+0000","last_change":"2024-03-26T05:20:15.372145+0000","last_active":"2024-03-26T05:20:48.312699+0000","last_peered":"2024-03-26T05:20:48.312699+0000","last_clean":"2024-03-26T05:20:48.312699+0000","last_became_active":"2024-03-26T05:20:15.371796+0000","last_became_peered":"2024-03-26T05:20:15.371796+0000","last_unstale":"2024-03-26T05:20:48.312699+0000","last_undegraded":"2024-03-26T05:20:48.312699+0000","last_fullsized":"2024-03-26T05:20:48.312699+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:21.515294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:21.515294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":879,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.313184+0000","last_change":"2024-03-26T05:20:30.332976+0000","last_active":"2024-03-26T05:20:48.313184+0000","last_peered":"2024-03-26T05:20:48.313184+0000","last_clean":"2024-03-26T05:20:48.313184+0000","last_became_active":"2024-03-26T05:20:30.332245+0000","last_became_peered":"2024-03-26T05:20:30.332245+0000","last_unstale":"2024-03-26T05:20:48.313184+0000","last_undegraded":"2024-03-26T05:20:48.313184+0000","last_fullsized":"2024-03-26T05:20:48.313184+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:05.539590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:55.496800+0000","last_clean_scrub_stamp":"2024-03-26T05:20:05.539590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:27.782687+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":843,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.307715+0000","last_change":"2024-03-26T05:20:30.346728+0000","last_active":"2024-03-26T05:20:48.307715+0000","last_peered":"2024-03-26T05:20:48.307715+0000","last_clean":"2024-03-26T05:20:48.307715+0000","last_became_active":"2024-03-26T05:20:30.346313+0000","last_became_peered":"2024-03-26T05:20:30.346313+0000","last_unstale":"2024-03-26T05:20:48.307715+0000","last_undegraded":"2024-03-26T05:20:48.307715+0000","last_fullsized":"2024-03-26T05:20:48.307715+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:38.817507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:18:38.817507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"443'21","reported_seq":891,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.313276+0000","last_change":"2024-03-26T05:20:01.620894+0000","last_active":"2024-03-26T05:20:48.313276+0000","last_peered":"2024-03-26T05:20:48.313276+0000","last_clean":"2024-03-26T05:20:48.313276+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:20:48.313276+0000","last_undegraded":"2024-03-26T05:20:48.313276+0000","last_fullsized":"2024-03-26T05:20:48.313276+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'15","last_scrub_stamp":"2024-03-26T05:20:01.620742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:01.620742+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:25.726189+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":720,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312728+0000","last_change":"2024-03-26T05:20:15.375970+0000","last_active":"2024-03-26T05:20:48.312728+0000","last_peered":"2024-03-26T05:20:48.312728+0000","last_clean":"2024-03-26T05:20:48.312728+0000","last_became_active":"2024-03-26T05:20:15.375609+0000","last_became_peered":"2024-03-26T05:20:15.375609+0000","last_unstale":"2024-03-26T05:20:48.312728+0000","last_undegraded":"2024-03-26T05:20:48.312728+0000","last_fullsized":"2024-03-26T05:20:48.312728+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:33.016072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:11:33.016072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":837,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311336+0000","last_change":"2024-03-26T05:19:55.403001+0000","last_active":"2024-03-26T05:20:48.311336+0000","last_peered":"2024-03-26T05:20:48.311336+0000","last_clean":"2024-03-26T05:20:48.311336+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:20:48.311336+0000","last_undegraded":"2024-03-26T05:20:48.311336+0000","last_fullsized":"2024-03-26T05:20:48.311336+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:55.402737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:19:55.402737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:06.358471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":862,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.735751+0000","last_change":"2024-03-26T05:20:30.344287+0000","last_active":"2024-03-26T05:20:51.735751+0000","last_peered":"2024-03-26T05:20:51.735751+0000","last_clean":"2024-03-26T05:20:51.735751+0000","last_became_active":"2024-03-26T05:20:30.343972+0000","last_became_peered":"2024-03-26T05:20:30.343972+0000","last_unstale":"2024-03-26T05:20:51.735751+0000","last_undegraded":"2024-03-26T05:20:51.735751+0000","last_fullsized":"2024-03-26T05:20:51.735751+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:38.324541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:18:38.324541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"443'33","reported_seq":897,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.308299+0000","last_change":"2024-03-26T05:19:57.364173+0000","last_active":"2024-03-26T05:20:48.308299+0000","last_peered":"2024-03-26T05:20:48.308299+0000","last_clean":"2024-03-26T05:20:48.308299+0000","last_became_active":"2024-03-26T05:15:32.636359+0000","last_became_peered":"2024-03-26T05:15:32.636359+0000","last_unstale":"2024-03-26T05:20:48.308299+0000","last_undegraded":"2024-03-26T05:20:48.308299+0000","last_fullsized":"2024-03-26T05:20:48.308299+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:19:57.363571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.363571+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:12.532419+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":291,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312506+0000","last_change":"2024-03-26T05:20:30.351327+0000","last_active":"2024-03-26T05:20:48.312506+0000","last_peered":"2024-03-26T05:20:48.312506+0000","last_clean":"2024-03-26T05:20:48.312506+0000","last_became_active":"2024-03-26T05:20:30.350957+0000","last_became_peered":"2024-03-26T05:20:30.350957+0000","last_unstale":"2024-03-26T05:20:48.312506+0000","last_undegraded":"2024-03-26T05:20:48.312506+0000","last_fullsized":"2024-03-26T05:20:48.312506+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:27.232872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:20:27.232872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:51.891500+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":757,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312541+0000","last_change":"2024-03-26T05:20:15.377844+0000","last_active":"2024-03-26T05:20:48.312541+0000","last_peered":"2024-03-26T05:20:48.312541+0000","last_clean":"2024-03-26T05:20:48.312541+0000","last_became_active":"2024-03-26T05:20:15.377506+0000","last_became_peered":"2024-03-26T05:20:15.377506+0000","last_unstale":"2024-03-26T05:20:48.312541+0000","last_undegraded":"2024-03-26T05:20:48.312541+0000","last_fullsized":"2024-03-26T05:20:48.312541+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:23.957542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:11:23.957542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":841,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311950+0000","last_change":"2024-03-26T05:20:28.593855+0000","last_active":"2024-03-26T05:20:48.311950+0000","last_peered":"2024-03-26T05:20:48.311950+0000","last_clean":"2024-03-26T05:20:48.311950+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:20:48.311950+0000","last_undegraded":"2024-03-26T05:20:48.311950+0000","last_fullsized":"2024-03-26T05:20:48.311950+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:28.593685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:53.017168+0000","last_clean_scrub_stamp":"2024-03-26T05:20:28.593685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:44.643967+0000","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"443'14","reported_seq":861,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311226+0000","last_change":"2024-03-26T05:18:56.877975+0000","last_active":"2024-03-26T05:20:48.311226+0000","last_peered":"2024-03-26T05:20:48.311226+0000","last_clean":"2024-03-26T05:20:48.311226+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:20:48.311226+0000","last_undegraded":"2024-03-26T05:20:48.311226+0000","last_fullsized":"2024-03-26T05:20:48.311226+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'10","last_scrub_stamp":"2024-03-26T05:18:56.877852+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:18:56.877852+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":874,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.736621+0000","last_change":"2024-03-26T05:20:30.351206+0000","last_active":"2024-03-26T05:20:51.736621+0000","last_peered":"2024-03-26T05:20:51.736621+0000","last_clean":"2024-03-26T05:20:51.736621+0000","last_became_active":"2024-03-26T05:20:30.350521+0000","last_became_peered":"2024-03-26T05:20:30.350521+0000","last_unstale":"2024-03-26T05:20:51.736621+0000","last_undegraded":"2024-03-26T05:20:51.736621+0000","last_fullsized":"2024-03-26T05:20:51.736621+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:40.333981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:11.070341+0000","last_clean_scrub_stamp":"2024-03-26T05:18:40.333981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":846,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314899+0000","last_change":"2024-03-26T05:20:37.670090+0000","last_active":"2024-03-26T05:20:48.314899+0000","last_peered":"2024-03-26T05:20:48.314899+0000","last_clean":"2024-03-26T05:20:48.314899+0000","last_became_active":"2024-03-26T05:20:15.372788+0000","last_became_peered":"2024-03-26T05:20:15.372788+0000","last_unstale":"2024-03-26T05:20:48.314899+0000","last_undegraded":"2024-03-26T05:20:48.314899+0000","last_fullsized":"2024-03-26T05:20:48.314899+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:37.669979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:20:37.669979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:46.100433+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":741,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311532+0000","last_change":"2024-03-26T05:20:15.364726+0000","last_active":"2024-03-26T05:20:48.311532+0000","last_peered":"2024-03-26T05:20:48.311532+0000","last_clean":"2024-03-26T05:20:48.311532+0000","last_became_active":"2024-03-26T05:20:15.364379+0000","last_became_peered":"2024-03-26T05:20:15.364379+0000","last_unstale":"2024-03-26T05:20:48.311532+0000","last_undegraded":"2024-03-26T05:20:48.311532+0000","last_fullsized":"2024-03-26T05:20:48.311532+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:10.020572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:10.020572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"443'39","reported_seq":1014,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315334+0000","last_change":"2024-03-26T05:20:30.366464+0000","last_active":"2024-03-26T05:20:48.315334+0000","last_peered":"2024-03-26T05:20:48.315334+0000","last_clean":"2024-03-26T05:20:48.315334+0000","last_became_active":"2024-03-26T05:20:30.343540+0000","last_became_peered":"2024-03-26T05:20:30.343540+0000","last_unstale":"2024-03-26T05:20:48.315334+0000","last_undegraded":"2024-03-26T05:20:48.315334+0000","last_fullsized":"2024-03-26T05:20:48.315334+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'31","last_scrub_stamp":"2024-03-26T05:19:57.767398+0000","last_deep_scrub":"413'31","last_deep_scrub_stamp":"2024-03-26T05:18:46.881054+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.767398+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:06.404789+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":845,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311031+0000","last_change":"2024-03-26T05:18:47.776529+0000","last_active":"2024-03-26T05:20:48.311031+0000","last_peered":"2024-03-26T05:20:48.311031+0000","last_clean":"2024-03-26T05:20:48.311031+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:20:48.311031+0000","last_undegraded":"2024-03-26T05:20:48.311031+0000","last_fullsized":"2024-03-26T05:20:48.311031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:47.776444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:47.776444+0000","last_clean_scrub_stamp":"2024-03-26T05:18:47.776444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":843,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.758975+0000","last_change":"2024-03-26T05:19:48.378592+0000","last_active":"2024-03-26T05:20:51.758975+0000","last_peered":"2024-03-26T05:20:51.758975+0000","last_clean":"2024-03-26T05:20:51.758975+0000","last_became_active":"2024-03-26T05:15:32.634716+0000","last_became_peered":"2024-03-26T05:15:32.634716+0000","last_unstale":"2024-03-26T05:20:51.758975+0000","last_undegraded":"2024-03-26T05:20:51.758975+0000","last_fullsized":"2024-03-26T05:20:51.758975+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:48.378485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:48.378485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:03.386020+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":852,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.313799+0000","last_change":"2024-03-26T05:20:30.342850+0000","last_active":"2024-03-26T05:20:48.313799+0000","last_peered":"2024-03-26T05:20:48.313799+0000","last_clean":"2024-03-26T05:20:48.313799+0000","last_became_active":"2024-03-26T05:20:30.342550+0000","last_became_peered":"2024-03-26T05:20:30.342550+0000","last_unstale":"2024-03-26T05:20:48.313799+0000","last_undegraded":"2024-03-26T05:20:48.313799+0000","last_fullsized":"2024-03-26T05:20:48.313799+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:00.151355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.151355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"443'42","reported_seq":918,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.758678+0000","last_change":"2024-03-26T05:18:57.256490+0000","last_active":"2024-03-26T05:20:51.758678+0000","last_peered":"2024-03-26T05:20:51.758678+0000","last_clean":"2024-03-26T05:20:51.758678+0000","last_became_active":"2024-03-26T05:15:32.634216+0000","last_became_peered":"2024-03-26T05:15:32.634216+0000","last_unstale":"2024-03-26T05:20:51.758678+0000","last_undegraded":"2024-03-26T05:20:51.758678+0000","last_fullsized":"2024-03-26T05:20:51.758678+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'32","last_scrub_stamp":"2024-03-26T05:18:57.255727+0000","last_deep_scrub":"290'32","last_deep_scrub_stamp":"2024-03-26T05:12:15.970313+0000","last_clean_scrub_stamp":"2024-03-26T05:18:57.255727+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":851,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.310616+0000","last_change":"2024-03-26T05:20:15.367466+0000","last_active":"2024-03-26T05:20:48.310616+0000","last_peered":"2024-03-26T05:20:48.310616+0000","last_clean":"2024-03-26T05:20:48.310616+0000","last_became_active":"2024-03-26T05:20:15.361116+0000","last_became_peered":"2024-03-26T05:20:15.361116+0000","last_unstale":"2024-03-26T05:20:48.310616+0000","last_undegraded":"2024-03-26T05:20:48.310616+0000","last_fullsized":"2024-03-26T05:20:48.310616+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:49.948818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:49.948818+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"439'12","reported_seq":1119,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314647+0000","last_change":"2024-03-26T05:20:15.382285+0000","last_active":"2024-03-26T05:20:48.314647+0000","last_peered":"2024-03-26T05:20:48.314647+0000","last_clean":"2024-03-26T05:20:48.314647+0000","last_became_active":"2024-03-26T05:20:15.375030+0000","last_became_peered":"2024-03-26T05:20:15.375030+0000","last_unstale":"2024-03-26T05:20:48.314647+0000","last_undegraded":"2024-03-26T05:20:48.314647+0000","last_fullsized":"2024-03-26T05:20:48.314647+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'10","last_scrub_stamp":"2024-03-26T05:15:38.516660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:15:38.516660+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"429'2","reported_seq":781,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312374+0000","last_change":"2024-03-26T05:20:30.352085+0000","last_active":"2024-03-26T05:20:48.312374+0000","last_peered":"2024-03-26T05:20:48.312374+0000","last_clean":"2024-03-26T05:20:48.312374+0000","last_became_active":"2024-03-26T05:20:30.341958+0000","last_became_peered":"2024-03-26T05:20:30.341958+0000","last_unstale":"2024-03-26T05:20:48.312374+0000","last_undegraded":"2024-03-26T05:20:48.312374+0000","last_fullsized":"2024-03-26T05:20:48.312374+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'2","last_scrub_stamp":"2024-03-26T05:19:54.275264+0000","last_deep_scrub":"429'2","last_deep_scrub_stamp":"2024-03-26T05:19:54.275264+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.275264+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:03.999270+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"443'38","reported_seq":909,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315171+0000","last_change":"2024-03-26T05:20:15.462061+0000","last_active":"2024-03-26T05:20:48.315171+0000","last_peered":"2024-03-26T05:20:48.315171+0000","last_clean":"2024-03-26T05:20:48.315171+0000","last_became_active":"2024-03-26T05:20:15.362085+0000","last_became_peered":"2024-03-26T05:20:15.362085+0000","last_unstale":"2024-03-26T05:20:48.315171+0000","last_undegraded":"2024-03-26T05:20:48.315171+0000","last_fullsized":"2024-03-26T05:20:48.315171+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'32","last_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_deep_scrub":"322'32","last_deep_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_clean_scrub_stamp":"2024-03-26T05:15:10.715786+0000","objects_scrubbed":6,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":231,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312761+0000","last_change":"2024-03-26T05:20:15.379698+0000","last_active":"2024-03-26T05:20:48.312761+0000","last_peered":"2024-03-26T05:20:48.312761+0000","last_clean":"2024-03-26T05:20:48.312761+0000","last_became_active":"2024-03-26T05:20:15.379345+0000","last_became_peered":"2024-03-26T05:20:15.379345+0000","last_unstale":"2024-03-26T05:20:48.312761+0000","last_undegraded":"2024-03-26T05:20:48.312761+0000","last_fullsized":"2024-03-26T05:20:48.312761+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:03.047441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:15:03.047441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":231,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.313222+0000","last_change":"2024-03-26T05:20:15.372365+0000","last_active":"2024-03-26T05:20:48.313222+0000","last_peered":"2024-03-26T05:20:48.313222+0000","last_clean":"2024-03-26T05:20:48.313222+0000","last_became_active":"2024-03-26T05:20:15.372060+0000","last_became_peered":"2024-03-26T05:20:15.372060+0000","last_unstale":"2024-03-26T05:20:48.313222+0000","last_undegraded":"2024-03-26T05:20:48.313222+0000","last_fullsized":"2024-03-26T05:20:48.313222+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:04.830989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:04.830989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":861,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315975+0000","last_change":"2024-03-26T05:20:30.348058+0000","last_active":"2024-03-26T05:20:48.315975+0000","last_peered":"2024-03-26T05:20:48.315975+0000","last_clean":"2024-03-26T05:20:48.315975+0000","last_became_active":"2024-03-26T05:20:30.347326+0000","last_became_peered":"2024-03-26T05:20:30.347326+0000","last_unstale":"2024-03-26T05:20:48.315975+0000","last_undegraded":"2024-03-26T05:20:48.315975+0000","last_fullsized":"2024-03-26T05:20:48.315975+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:01.576615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:20:01.576615+0000","last_clean_scrub_stamp":"2024-03-26T05:20:01.576615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:21.040865+0000","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"443'39","reported_seq":918,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.319201+0000","last_change":"2024-03-26T05:19:59.984067+0000","last_active":"2024-03-26T05:20:48.319201+0000","last_peered":"2024-03-26T05:20:48.319201+0000","last_clean":"2024-03-26T05:20:48.319201+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:20:48.319201+0000","last_undegraded":"2024-03-26T05:20:48.319201+0000","last_fullsized":"2024-03-26T05:20:48.319201+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:19:59.983973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:19:59.983973+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:23.751673+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.736657+0000","last_change":"2024-03-26T05:20:15.457277+0000","last_active":"2024-03-26T05:20:51.736657+0000","last_peered":"2024-03-26T05:20:51.736657+0000","last_clean":"2024-03-26T05:20:51.736657+0000","last_became_active":"2024-03-26T05:20:15.456698+0000","last_became_peered":"2024-03-26T05:20:15.456698+0000","last_unstale":"2024-03-26T05:20:51.736657+0000","last_undegraded":"2024-03-26T05:20:51.736657+0000","last_fullsized":"2024-03-26T05:20:51.736657+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:04.980730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:12:04.980730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":844,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.313896+0000","last_change":"2024-03-26T05:20:30.339668+0000","last_active":"2024-03-26T05:20:48.313896+0000","last_peered":"2024-03-26T05:20:48.313896+0000","last_clean":"2024-03-26T05:20:48.313896+0000","last_became_active":"2024-03-26T05:20:30.338293+0000","last_became_peered":"2024-03-26T05:20:30.338293+0000","last_unstale":"2024-03-26T05:20:48.313896+0000","last_undegraded":"2024-03-26T05:20:48.313896+0000","last_fullsized":"2024-03-26T05:20:48.313896+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:11.912203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:15:11.912203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":566,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315089+0000","last_change":"2024-03-26T05:20:30.346590+0000","last_active":"2024-03-26T05:20:48.315089+0000","last_peered":"2024-03-26T05:20:48.315089+0000","last_clean":"2024-03-26T05:20:48.315089+0000","last_became_active":"2024-03-26T05:20:30.346249+0000","last_became_peered":"2024-03-26T05:20:30.346249+0000","last_unstale":"2024-03-26T05:20:48.315089+0000","last_undegraded":"2024-03-26T05:20:48.315089+0000","last_fullsized":"2024-03-26T05:20:48.315089+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:05.787712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:14:05.787712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"443'18","reported_seq":828,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312408+0000","last_change":"2024-03-26T05:20:15.460323+0000","last_active":"2024-03-26T05:20:48.312408+0000","last_peered":"2024-03-26T05:20:48.312408+0000","last_clean":"2024-03-26T05:20:48.312408+0000","last_became_active":"2024-03-26T05:20:15.374306+0000","last_became_peered":"2024-03-26T05:20:15.374306+0000","last_unstale":"2024-03-26T05:20:48.312408+0000","last_undegraded":"2024-03-26T05:20:48.312408+0000","last_fullsized":"2024-03-26T05:20:48.312408+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'16","last_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_deep_scrub":"321'16","last_deep_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.995735+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":513,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314371+0000","last_change":"2024-03-26T05:20:30.329198+0000","last_active":"2024-03-26T05:20:48.314371+0000","last_peered":"2024-03-26T05:20:48.314371+0000","last_clean":"2024-03-26T05:20:48.314371+0000","last_became_active":"2024-03-26T05:20:30.328801+0000","last_became_peered":"2024-03-26T05:20:30.328801+0000","last_unstale":"2024-03-26T05:20:48.314371+0000","last_undegraded":"2024-03-26T05:20:48.314371+0000","last_fullsized":"2024-03-26T05:20:48.314371+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:50.218114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:00.635308+0000","last_clean_scrub_stamp":"2024-03-26T05:18:50.218114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":756,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311722+0000","last_change":"2024-03-26T05:20:15.374216+0000","last_active":"2024-03-26T05:20:48.311722+0000","last_peered":"2024-03-26T05:20:48.311722+0000","last_clean":"2024-03-26T05:20:48.311722+0000","last_became_active":"2024-03-26T05:20:15.373862+0000","last_became_peered":"2024-03-26T05:20:15.373862+0000","last_unstale":"2024-03-26T05:20:48.311722+0000","last_undegraded":"2024-03-26T05:20:48.311722+0000","last_fullsized":"2024-03-26T05:20:48.311722+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_clean_scrub_stamp":"2024-03-26T05:12:18.959687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":139,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311460+0000","last_change":"2024-03-26T05:20:30.339972+0000","last_active":"2024-03-26T05:20:48.311460+0000","last_peered":"2024-03-26T05:20:48.311460+0000","last_clean":"2024-03-26T05:20:48.311460+0000","last_became_active":"2024-03-26T05:20:30.337386+0000","last_became_peered":"2024-03-26T05:20:30.337386+0000","last_unstale":"2024-03-26T05:20:48.311460+0000","last_undegraded":"2024-03-26T05:20:48.311460+0000","last_fullsized":"2024-03-26T05:20:48.311460+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:06.363747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:20:06.363747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:29.979099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"443'33","reported_seq":716,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.319933+0000","last_change":"2024-03-26T05:20:15.472237+0000","last_active":"2024-03-26T05:20:48.319933+0000","last_peered":"2024-03-26T05:20:48.319933+0000","last_clean":"2024-03-26T05:20:48.319933+0000","last_became_active":"2024-03-26T05:20:15.378661+0000","last_became_peered":"2024-03-26T05:20:15.378661+0000","last_unstale":"2024-03-26T05:20:48.319933+0000","last_undegraded":"2024-03-26T05:20:48.319933+0000","last_fullsized":"2024-03-26T05:20:48.319933+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:13:22.026354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:22.026354+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":508,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.315664+0000","last_change":"2024-03-26T05:20:30.331681+0000","last_active":"2024-03-26T05:20:48.315664+0000","last_peered":"2024-03-26T05:20:48.315664+0000","last_clean":"2024-03-26T05:20:48.315664+0000","last_became_active":"2024-03-26T05:20:30.331329+0000","last_became_peered":"2024-03-26T05:20:30.331329+0000","last_unstale":"2024-03-26T05:20:48.315664+0000","last_undegraded":"2024-03-26T05:20:48.315664+0000","last_fullsized":"2024-03-26T05:20:48.315664+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:17.792745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:17.792745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":301,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.736309+0000","last_change":"2024-03-26T05:20:30.343688+0000","last_active":"2024-03-26T05:20:51.736309+0000","last_peered":"2024-03-26T05:20:51.736309+0000","last_clean":"2024-03-26T05:20:51.736309+0000","last_became_active":"2024-03-26T05:20:30.343356+0000","last_became_peered":"2024-03-26T05:20:30.343356+0000","last_unstale":"2024-03-26T05:20:51.736309+0000","last_undegraded":"2024-03-26T05:20:51.736309+0000","last_fullsized":"2024-03-26T05:20:51.736309+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:56.259964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:56.259964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:20.971207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":845,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.735972+0000","last_change":"2024-03-26T05:19:43.520950+0000","last_active":"2024-03-26T05:20:51.735972+0000","last_peered":"2024-03-26T05:20:51.735972+0000","last_clean":"2024-03-26T05:20:51.735972+0000","last_became_active":"2024-03-26T05:15:32.640052+0000","last_became_peered":"2024-03-26T05:15:32.640052+0000","last_unstale":"2024-03-26T05:20:51.735972+0000","last_undegraded":"2024-03-26T05:20:51.735972+0000","last_fullsized":"2024-03-26T05:20:51.735972+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:43.520824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.520824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:08.218519+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"443'63","reported_seq":950,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.308392+0000","last_change":"2024-03-26T05:19:48.357831+0000","last_active":"2024-03-26T05:20:48.308392+0000","last_peered":"2024-03-26T05:20:48.308392+0000","last_clean":"2024-03-26T05:20:48.308392+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:20:48.308392+0000","last_undegraded":"2024-03-26T05:20:48.308392+0000","last_fullsized":"2024-03-26T05:20:48.308392+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'51","last_scrub_stamp":"2024-03-26T05:19:48.356743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:19:48.356743+0000","objects_scrubbed":11,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:52.671700+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":98,"num_read_kb":85,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":847,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.316557+0000","last_change":"2024-03-26T05:19:43.320734+0000","last_active":"2024-03-26T05:20:48.316557+0000","last_peered":"2024-03-26T05:20:48.316557+0000","last_clean":"2024-03-26T05:20:48.316557+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:20:48.316557+0000","last_undegraded":"2024-03-26T05:20:48.316557+0000","last_fullsized":"2024-03-26T05:20:48.316557+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:43.320603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.320603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:07.293053+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":239,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314141+0000","last_change":"2024-03-26T05:20:46.537730+0000","last_active":"2024-03-26T05:20:48.314141+0000","last_peered":"2024-03-26T05:20:48.314141+0000","last_clean":"2024-03-26T05:20:48.314141+0000","last_became_active":"2024-03-26T05:20:15.370449+0000","last_became_peered":"2024-03-26T05:20:15.370449+0000","last_unstale":"2024-03-26T05:20:48.314141+0000","last_undegraded":"2024-03-26T05:20:48.314141+0000","last_fullsized":"2024-03-26T05:20:48.314141+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:46.537613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:20:46.537613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:53.610982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":826,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.311762+0000","last_change":"2024-03-26T05:19:53.622022+0000","last_active":"2024-03-26T05:20:48.311762+0000","last_peered":"2024-03-26T05:20:48.311762+0000","last_clean":"2024-03-26T05:20:48.311762+0000","last_became_active":"2024-03-26T05:15:32.637717+0000","last_became_peered":"2024-03-26T05:15:32.637717+0000","last_unstale":"2024-03-26T05:20:48.311762+0000","last_undegraded":"2024-03-26T05:20:48.311762+0000","last_fullsized":"2024-03-26T05:20:48.311762+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:53.621892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:53.621892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:04.009123+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"443'27","reported_seq":702,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.314502+0000","last_change":"2024-03-26T05:20:15.468927+0000","last_active":"2024-03-26T05:20:48.314502+0000","last_peered":"2024-03-26T05:20:48.314502+0000","last_clean":"2024-03-26T05:20:48.314502+0000","last_became_active":"2024-03-26T05:20:15.377639+0000","last_became_peered":"2024-03-26T05:20:15.377639+0000","last_unstale":"2024-03-26T05:20:48.314502+0000","last_undegraded":"2024-03-26T05:20:48.314502+0000","last_fullsized":"2024-03-26T05:20:48.314502+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:15:04.002977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:15:04.002977+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":867,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-26T05:20:51.736372+0000","last_change":"2024-03-26T05:20:30.349760+0000","last_active":"2024-03-26T05:20:51.736372+0000","last_peered":"2024-03-26T05:20:51.736372+0000","last_clean":"2024-03-26T05:20:51.736372+0000","last_became_active":"2024-03-26T05:20:30.349429+0000","last_became_peered":"2024-03-26T05:20:30.349429+0000","last_unstale":"2024-03-26T05:20:51.736372+0000","last_undegraded":"2024-03-26T05:20:51.736372+0000","last_fullsized":"2024-03-26T05:20:51.736372+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:34.300011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:34.300011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":864,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.310723+0000","last_change":"2024-03-26T05:20:30.337232+0000","last_active":"2024-03-26T05:20:48.310723+0000","last_peered":"2024-03-26T05:20:48.310723+0000","last_clean":"2024-03-26T05:20:48.310723+0000","last_became_active":"2024-03-26T05:20:30.336902+0000","last_became_peered":"2024-03-26T05:20:30.336902+0000","last_unstale":"2024-03-26T05:20:48.310723+0000","last_undegraded":"2024-03-26T05:20:48.310723+0000","last_fullsized":"2024-03-26T05:20:48.310723+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:51.618208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:51.618208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:20:53.362834+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":540,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T05:20:48.312637+0000","last_change":"2024-03-26T05:20:30.331664+0000","last_active":"2024-03-26T05:20:48.312637+0000","last_peered":"2024-03-26T05:20:48.312637+0000","last_clean":"2024-03-26T05:20:48.312637+0000","last_became_active":"2024-03-26T05:20:30.327890+0000","last_became_peered":"2024-03-26T05:20:30.327890+0000","last_unstale":"2024-03-26T05:20:48.312637+0000","last_undegraded":"2024-03-26T05:20:48.312637+0000","last_fullsized":"2024-03-26T05:20:48.312637+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:53.055153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:53.055153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6157959192,"num_objects":93964,"num_object_clones":0,"num_object_copies":281892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13897,"num_objects_misplaced":20416,"num_objects_unfound":0,"num_objects_dirty":93964,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148252,"num_write_kb":7750849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12727,"num_bytes_recovered":834076672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17751220224,"data_stored":17751212080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61139,"ondisk_log_size":61139,"up":48,"acting":44,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":383,"num_read_kb":378,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697708,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":54,"num_write":34,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":649700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1832,"num_read_kb":1557,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":4554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":13020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1275,"ondisk_log_size":1275,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":335,"seq":1438814044225,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3054868,"kb_used_data":2640648,"kb_used_omap":4,"kb_used_meta":414203,"kb_avail":90690284,"statfs":{"total":95995035648,"available":92866850816,"internally_reserved":0,"allocated":2704023552,"data_stored":2703007055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":424144552},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332351,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3686864,"kb_used_data":3060104,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":90058288,"statfs":{"total":95995035648,"available":92219686912,"internally_reserved":0,"allocated":3133546496,"data_stored":3132529271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":641792454},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575129,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3961708,"kb_used_data":3132196,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":89783444,"statfs":{"total":95995035648,"available":91938246656,"internally_reserved":0,"allocated":3207368704,"data_stored":3206323764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":849344965},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738654,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2970900,"kb_used_data":2145612,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":90774252,"statfs":{"total":95995035648,"available":92952834048,"internally_reserved":0,"allocated":2197106688,"data_stored":2196086045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411604,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2947724,"kb_used_data":2284800,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":90797428,"statfs":{"total":95995035648,"available":92976566272,"internally_reserved":0,"allocated":2339635200,"data_stored":2338625948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117262,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3499752,"kb_used_data":2670300,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":90245400,"statfs":{"total":95995035648,"available":92411289600,"internally_reserved":0,"allocated":2734387200,"data_stored":2733350453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":849344967},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":176093659396,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2192113664,"data_stored":2192113664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3189899264,"data_stored":3189899264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2315915264,"data_stored":2315911192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2713255936,"data_stored":2713255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1529020416,"data_stored":1529020416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3117158400,"data_stored":3117154328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2693857280,"data_stored":2693857280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:20:58.548 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-26T05:20:58.549 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool set unique_pool_4 pg_num 26 2024-03-26T05:20:58.601 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:20:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:58.659 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:58.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:20:58.760 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:58.811 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:58.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:58.913 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:58.961 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:58.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:59.063 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:59.087 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:59.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:59.188 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:59.233 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:59.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:59.335 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:59.384 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:59.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:59.484 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:20:59.524 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:59.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:59.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:20:59.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:59 smithi080 ceph-mon[40404]: pgmap v963: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 969 op/s; 11735/268992 objects degraded (4.363%); 18775/268992 objects misplaced (6.980%); 26 MiB/s, 419 objects/s recovering 2024-03-26T05:20:59.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:59 smithi080 ceph-mon[40404]: 5.f scrub starts 2024-03-26T05:20:59.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:20:59 smithi080 ceph-mon[40404]: 5.f scrub ok 2024-03-26T05:20:59.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:59 smithi080 ceph-mon[29691]: pgmap v963: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 969 op/s; 11735/268992 objects degraded (4.363%); 18775/268992 objects misplaced (6.980%); 26 MiB/s, 419 objects/s recovering 2024-03-26T05:20:59.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:59 smithi080 ceph-mon[29691]: 5.f scrub starts 2024-03-26T05:20:59.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:20:59 smithi080 ceph-mon[29691]: 5.f scrub ok 2024-03-26T05:20:59.626 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:20:59.668 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:20:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:59.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:20:59.770 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:20:59.784 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:20:59.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:59 smithi132 ceph-mon[33500]: pgmap v963: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 969 op/s; 11735/268992 objects degraded (4.363%); 18775/268992 objects misplaced (6.980%); 26 MiB/s, 419 objects/s recovering 2024-03-26T05:20:59.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:59 smithi132 ceph-mon[33500]: 5.f scrub starts 2024-03-26T05:20:59.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:20:59 smithi132 ceph-mon[33500]: 5.f scrub ok 2024-03-26T05:20:59.797 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:59.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:20:59.898 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:20:59.924 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:20:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:20:59.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:00.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:00.051 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:00.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:00.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:00.178 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:00.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:00.281 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:00.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:00.325 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:00.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:00.426 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:00.455 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:00.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:00.556 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:00.586 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:00.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:00.689 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:00.718 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:00.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:00.819 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:00.865 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:00.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:00.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:00.995 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:00.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:00.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:01.096 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:01.122 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:01.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:01.224 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:01.357 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:01.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:01.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:01.459 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:01.482 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:01.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:01.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:01.584 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:01.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:01 smithi080 ceph-mon[40404]: pgmap v964: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 948 op/s; 11735/263736 objects degraded (4.450%); 17899/263736 objects misplaced (6.787%); 26 MiB/s, 412 objects/s recovering 2024-03-26T05:21:01.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:01 smithi080 ceph-mon[29691]: pgmap v964: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 948 op/s; 11735/263736 objects degraded (4.450%); 17899/263736 objects misplaced (6.787%); 26 MiB/s, 412 objects/s recovering 2024-03-26T05:21:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:01.627 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:01.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:01.728 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:01.754 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:01.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:01.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:01 smithi132 ceph-mon[33500]: pgmap v964: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 948 op/s; 11735/263736 objects degraded (4.450%); 17899/263736 objects misplaced (6.787%); 26 MiB/s, 412 objects/s recovering 2024-03-26T05:21:01.856 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:01.887 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:01.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:01.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:01.989 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:02.016 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:02.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:02.118 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:02.141 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:02.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:02.243 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:02.280 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:02.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:02.382 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:02.412 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:02.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:02.514 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:02.545 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:02.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:02.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:02.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:02 smithi080 ceph-mon[29691]: 4.18 scrub starts 2024-03-26T05:21:02.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:02 smithi080 ceph-mon[29691]: 4.18 scrub ok 2024-03-26T05:21:02.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:02 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 11735/263736 objects degraded (4.450%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:02.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:02 smithi080 ceph-mon[40404]: 4.18 scrub starts 2024-03-26T05:21:02.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:02 smithi080 ceph-mon[40404]: 4.18 scrub ok 2024-03-26T05:21:02.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:02 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 11735/263736 objects degraded (4.450%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:02.647 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:02.695 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:02.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:02.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:02 smithi132 ceph-mon[33500]: 4.18 scrub starts 2024-03-26T05:21:02.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:02 smithi132 ceph-mon[33500]: 4.18 scrub ok 2024-03-26T05:21:02.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:02 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 11735/263736 objects degraded (4.450%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:02.797 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:02.836 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:02.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:02.938 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:02.981 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:02.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:03.083 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:03.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:03.112 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:03.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:03.213 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:03.241 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:03.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:03.343 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:03.400 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:03.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:03.502 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:03.529 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:03.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:03.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:03.631 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:03.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:03.655 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:03.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:03.756 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:03.791 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:03.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:03.893 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:03.918 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:03.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:03.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:04.020 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:04.021 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:03 smithi080 ceph-mon[40404]: pgmap v965: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 974 B/s rd, 0 B/s wr, 926 op/s; 9435/254340 objects degraded (3.710%); 17061/254340 objects misplaced (6.708%); 24 MiB/s, 389 objects/s recovering 2024-03-26T05:21:04.022 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:03 smithi080 ceph-mon[40404]: 5.1f scrub starts 2024-03-26T05:21:04.022 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:03 smithi080 ceph-mon[40404]: 5.1f scrub ok 2024-03-26T05:21:04.022 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:03 smithi080 ceph-mon[40404]: osdmap e452: 8 total, 7 up, 6 in 2024-03-26T05:21:04.022 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:03 smithi080 ceph-mon[29691]: pgmap v965: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 974 B/s rd, 0 B/s wr, 926 op/s; 9435/254340 objects degraded (3.710%); 17061/254340 objects misplaced (6.708%); 24 MiB/s, 389 objects/s recovering 2024-03-26T05:21:04.022 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:03 smithi080 ceph-mon[29691]: 5.1f scrub starts 2024-03-26T05:21:04.022 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:03 smithi080 ceph-mon[29691]: 5.1f scrub ok 2024-03-26T05:21:04.022 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:03 smithi080 ceph-mon[29691]: osdmap e452: 8 total, 7 up, 6 in 2024-03-26T05:21:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:03 smithi132 ceph-mon[33500]: pgmap v965: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 974 B/s rd, 0 B/s wr, 926 op/s; 9435/254340 objects degraded (3.710%); 17061/254340 objects misplaced (6.708%); 24 MiB/s, 389 objects/s recovering 2024-03-26T05:21:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:03 smithi132 ceph-mon[33500]: 5.1f scrub starts 2024-03-26T05:21:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:03 smithi132 ceph-mon[33500]: 5.1f scrub ok 2024-03-26T05:21:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:03 smithi132 ceph-mon[33500]: osdmap e452: 8 total, 7 up, 6 in 2024-03-26T05:21:04.043 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:04.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:04.145 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:04.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:04.174 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:04.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:04.275 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:04.300 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:04.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:04.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:04.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:21:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:21:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:21:04.402 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:04.439 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:04.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:04.540 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:04.568 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:04.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:04.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:04.670 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:04.697 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:04.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:04.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:04.799 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:04.828 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:04.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:04.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:04 smithi080 ceph-mon[29691]: 4.5 scrub starts 2024-03-26T05:21:04.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:04 smithi080 ceph-mon[29691]: 4.5 scrub ok 2024-03-26T05:21:04.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:04 smithi080 ceph-mon[29691]: osdmap e453: 8 total, 7 up, 6 in 2024-03-26T05:21:04.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:04 smithi080 ceph-mon[40404]: 4.5 scrub starts 2024-03-26T05:21:04.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:04 smithi080 ceph-mon[40404]: 4.5 scrub ok 2024-03-26T05:21:04.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:04 smithi080 ceph-mon[40404]: osdmap e453: 8 total, 7 up, 6 in 2024-03-26T05:21:04.930 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:04.958 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:04.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:05.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:04 smithi132 ceph-mon[33500]: 4.5 scrub starts 2024-03-26T05:21:05.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:04 smithi132 ceph-mon[33500]: 4.5 scrub ok 2024-03-26T05:21:05.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:04 smithi132 ceph-mon[33500]: osdmap e453: 8 total, 7 up, 6 in 2024-03-26T05:21:05.060 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:05.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:05.087 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:05.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:05.188 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:05.216 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:05.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:05.318 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:05.359 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:05.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:05.461 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:05.505 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:05.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:05.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:05.607 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:05.636 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:05.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:05.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:05.738 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:05.784 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:05.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:05.885 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:05.911 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:05.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:06.012 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:06.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:05 smithi132 ceph-mon[33500]: pgmap v967: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 942 op/s; 9435/254340 objects degraded (3.710%); 17061/254340 objects misplaced (6.708%); 26 MiB/s, 408 objects/s recovering 2024-03-26T05:21:06.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:21:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:21:06.050 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:06.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:06.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:06.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:05 smithi080 ceph-mon[40404]: pgmap v967: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 942 op/s; 9435/254340 objects degraded (3.710%); 17061/254340 objects misplaced (6.708%); 26 MiB/s, 408 objects/s recovering 2024-03-26T05:21:06.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:05 smithi080 ceph-mon[29691]: pgmap v967: 158 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 942 op/s; 9435/254340 objects degraded (3.710%); 17061/254340 objects misplaced (6.708%); 26 MiB/s, 408 objects/s recovering 2024-03-26T05:21:06.115 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:21:05 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:21:05.887+0000 7f2fcc605700 -1 mgr get_metadata_python Requested missing service osd.4 2024-03-26T05:21:06.152 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:06.181 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:06.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:06.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:06.283 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:06.320 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:06.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:06.423 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:06.454 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:06.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:06.555 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:06.584 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:06.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:06.686 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:06.733 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:06.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:06.835 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:06.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:06.869 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:06.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:06.970 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:07.005 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:07.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:07.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:21:07.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:06 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 9435/254340 objects degraded (3.710%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:07.107 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:07.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:21:07.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:06 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 9435/254340 objects degraded (3.710%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:07.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:21:07.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:06 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 9435/254340 objects degraded (3.710%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:07.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:07.155 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:07.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:07.256 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:07.292 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:07.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:07.394 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:07.426 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:07.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:07.527 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:07.570 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:07.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:07.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:07.698 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:07.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:07.800 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:07.835 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:07.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:07.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:07.964 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:07.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:08.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:07 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:07 smithi132 ceph-mon[33500]: pgmap v969: 158 pgs: 3 active+remapped+backfilling, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 966 op/s; 8678/245814 objects degraded (3.530%); 14644/245814 objects misplaced (5.957%); 35 MiB/s, 563 objects/s recovering 2024-03-26T05:21:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:07 smithi132 ceph-mon[33500]: 11.3 deep-scrub starts 2024-03-26T05:21:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:07 smithi132 ceph-mon[33500]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T05:21:08.066 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:08.102 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:08.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:08.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:07 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:08.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:07 smithi080 ceph-mon[29691]: pgmap v969: 158 pgs: 3 active+remapped+backfilling, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 966 op/s; 8678/245814 objects degraded (3.530%); 14644/245814 objects misplaced (5.957%); 35 MiB/s, 563 objects/s recovering 2024-03-26T05:21:08.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:07 smithi080 ceph-mon[29691]: 11.3 deep-scrub starts 2024-03-26T05:21:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:07 smithi080 ceph-mon[29691]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T05:21:08.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:07 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:08.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:07 smithi080 ceph-mon[40404]: pgmap v969: 158 pgs: 3 active+remapped+backfilling, 4 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 966 op/s; 8678/245814 objects degraded (3.530%); 14644/245814 objects misplaced (5.957%); 35 MiB/s, 563 objects/s recovering 2024-03-26T05:21:08.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:07 smithi080 ceph-mon[40404]: 11.3 deep-scrub starts 2024-03-26T05:21:08.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:07 smithi080 ceph-mon[40404]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T05:21:08.205 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:08.235 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:08.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:08.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:08.337 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:08.384 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:08.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:08.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:08.486 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:08.513 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:08.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:08.614 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:08.653 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:08.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:08.755 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:08.779 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:08.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:08.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:08.881 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:08.911 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:08.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:09.013 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:09.040 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:09.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:08 smithi132 ceph-mon[33500]: pgmap v970: 158 pgs: 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 905 op/s; 7390/242025 objects degraded (3.053%); 14033/242025 objects misplaced (5.798%); 40 MiB/s, 644 objects/s recovering 2024-03-26T05:21:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:09.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:09.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:08 smithi080 ceph-mon[40404]: pgmap v970: 158 pgs: 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 905 op/s; 7390/242025 objects degraded (3.053%); 14033/242025 objects misplaced (5.798%); 40 MiB/s, 644 objects/s recovering 2024-03-26T05:21:09.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:08 smithi080 ceph-mon[29691]: pgmap v970: 158 pgs: 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 905 op/s; 7390/242025 objects degraded (3.053%); 14033/242025 objects misplaced (5.798%); 40 MiB/s, 644 objects/s recovering 2024-03-26T05:21:09.144 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:09.176 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:09.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:09.277 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:09.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:09.308 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:09.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:09.409 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:09.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:09.446 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:09.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:09.547 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:09.585 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:09.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:09.687 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:09.731 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:09.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:09.833 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:09.861 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:09.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:09.963 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:09.991 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:09.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:09.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:10.034 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:10.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:09 smithi132 ceph-mon[33500]: 4.a scrub starts 2024-03-26T05:21:10.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:09 smithi132 ceph-mon[33500]: 4.a scrub ok 2024-03-26T05:21:10.094 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:10.109 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:10.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:09 smithi080 ceph-mon[40404]: 4.a scrub starts 2024-03-26T05:21:10.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:09 smithi080 ceph-mon[40404]: 4.a scrub ok 2024-03-26T05:21:10.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:09 smithi080 ceph-mon[29691]: 4.a scrub starts 2024-03-26T05:21:10.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:09 smithi080 ceph-mon[29691]: 4.a scrub ok 2024-03-26T05:21:10.148 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:10.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:10.250 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:10.276 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:10.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:10.320 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:21:10.320 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:21:10.320 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:21:10.320 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:21:10.320 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.320 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:75288 10.c 10:39a3c057:::benchmark_data_smithi080_161990_object75287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e336)", 2024-03-26T05:21:10.320 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:15:32.612902+0000", 2024-03-26T05:21:10.320 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 337.71108103400002, 2024-03-26T05:21:10.320 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.105485719, 2024-03-26T05:21:10.320 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.320 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 75288 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612902+0000", 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.321 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612902+0000", 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612903+0000", 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8899999999999999e-07 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612927+0000", 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.4192000000000001e-05 2024-03-26T05:21:10.322 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612936+0000", 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0410000000000003e-06 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612943+0000", 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.674e-06 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.323 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618645+0000", 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0057027739999999999 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618649+0000", 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8029999999999998e-06 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.711588+0000", 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0929387699999999 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.325 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:21:10.325 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.711690+0000", 2024-03-26T05:21:10.325 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010237 2024-03-26T05:21:10.325 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.325 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.325 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,7", 2024-03-26T05:21:10.325 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.711799+0000", 2024-03-26T05:21:10.325 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000108787 2024-03-26T05:21:10.325 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.325 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.325 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:21:10.325 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.715595+0000", 2024-03-26T05:21:10.326 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0037958430000000001 2024-03-26T05:21:10.326 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.326 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.326 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:21:10.326 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.717997+0000", 2024-03-26T05:21:10.326 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0024021649999999999 2024-03-26T05:21:10.326 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.326 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.326 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:21:10.326 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718266+0000", 2024-03-26T05:21:10.326 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00026850799999999998 2024-03-26T05:21:10.326 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718301+0000", 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5404999999999997e-05 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718388+0000", 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6798000000000005e-05 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:75295 10.c 10:34625f2b:::benchmark_data_smithi080_161990_object75294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e336)", 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:15:32.612952+0000", 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 337.71103119399999, 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.112282223, 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 75295 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.328 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612952+0000", 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612952+0000", 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612952+0000", 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5999999999999999e-07 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.329 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612974+0000", 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1180000000000001e-05 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612981+0000", 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6260000000000003e-06 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612988+0000", 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5050000000000004e-06 2024-03-26T05:21:10.330 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618673+0000", 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0056853609999999999 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618677+0000", 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5810000000000001e-06 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.713508+0000", 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.094830787 2024-03-26T05:21:10.331 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.713636+0000", 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000128009 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,7", 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.713806+0000", 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017006899999999999 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718010+0000", 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0042043200000000001 2024-03-26T05:21:10.332 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718223+0000", 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000213496 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725036+0000", 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0068121019999999996 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:21:10.333 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725109+0000", 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3539000000000002e-05 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725234+0000", 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012518800000000001 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:75302 10.c 10:32bd27e8:::benchmark_data_smithi080_161990_object75301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e336)", 2024-03-26T05:21:10.334 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:15:32.612996+0000", 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 337.710987257, 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.112536143, 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 75302 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612996+0000", 2024-03-26T05:21:10.335 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612996+0000", 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.612996+0000", 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7800000000000002e-07 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.336 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613092+0000", 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5562000000000004e-05 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613102+0000", 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0127e-05 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613110+0000", 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.4930000000000001e-06 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.337 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618701+0000", 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055912059999999996 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618704+0000", 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5240000000000001e-06 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.715630+0000", 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0969253430000001 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.338 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.715759+0000", 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000129696 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,7", 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.715956+0000", 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019639200000000001 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718232+0000", 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022764870000000002 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.339 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718445+0000", 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021288500000000001 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725356+0000", 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0069105149999999999 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725423+0000", 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6884000000000005e-05 2024-03-26T05:21:10.340 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725532+0000", 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000109551 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:75303 10.c 10:3d000c38:::benchmark_data_smithi080_161990_object75302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e336)", 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:15:32.613121+0000", 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 337.71086269800003, 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.112658986, 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.341 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 75303 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613121+0000", 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.342 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613121+0000", 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613121+0000", 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1699999999999998e-07 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613144+0000", 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2555e-05 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.343 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613152+0000", 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8869999999999992e-06 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.613158+0000", 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2269999999999998e-06 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618720+0000", 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055622780000000004 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.344 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:32.618724+0000", 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3239999999999996e-06 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718491+0000", 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0997661860000001 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718602+0000", 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000111718 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 4,7", 2024-03-26T05:21:10.345 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.718702+0000", 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9610000000000003e-05 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.724985+0000", 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0062834459999999998 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.724996+0000", 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0403e-05 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.346 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725598+0000", 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00060190199999999997 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725664+0000", 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6790000000000003e-05 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:15:33.725780+0000", 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115043 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.347 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141751 10.18 10.aa5d358 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:50.719618+0000", 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 259.60436538499999, 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.9921031060000001, 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141751 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.348 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.719618+0000", 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.719618+0000", 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.719618+0000", 2024-03-26T05:21:10.349 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5900000000000003e-07 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.719620+0000", 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0810000000000001e-06 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.719626+0000", 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5879999999999997e-06 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.350 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.719630+0000", 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.622e-06 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.909973+0000", 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.190342961 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:50.909981+0000", 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9889999999999992e-06 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.351 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710386+0000", 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8004045870000001 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710398+0000", 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2138e-05 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.711708+0000", 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.001309923 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.711721+0000", 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2858e-05 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141765 10.18 10.fd02f5d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.726425+0000", 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.597558128, 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9870093049999999, 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.353 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141765 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.726425+0000", 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.354 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.726425+0000", 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.726426+0000", 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3969999999999999e-06 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.726431+0000", 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5050000000000001e-06 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.355 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.726441+0000", 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0202e-05 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.726455+0000", 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3762e-05 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.734757+0000", 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0083015550000000004 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.356 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.734771+0000", 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.482e-05 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710425+0000", 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97565345599999997 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710432+0000", 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2409999999999998e-06 2024-03-26T05:21:10.357 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713413+0000", 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0029810889999999 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713434+0000", 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1277999999999999e-05 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.358 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141814 10.18 10.1bf2e6f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.770677+0000", 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.55330661900001, 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9427979639999999, 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141814 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.359 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770677+0000", 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770677+0000", 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770678+0000", 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.869e-06 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.360 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770682+0000", 2024-03-26T05:21:10.362 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.703e-06 2024-03-26T05:21:10.362 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.362 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.362 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.362 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770692+0000", 2024-03-26T05:21:10.362 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.022e-05 2024-03-26T05:21:10.362 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.362 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.362 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.362 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770700+0000", 2024-03-26T05:21:10.362 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2339999999999997e-06 2024-03-26T05:21:10.362 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770888+0000", 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000188703 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.770897+0000", 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4579999999999993e-06 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710457+0000", 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93956030400000001 2024-03-26T05:21:10.363 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710464+0000", 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5899999999999996e-06 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713466+0000", 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.003002395 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713475+0000", 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4880000000000007e-06 2024-03-26T05:21:10.364 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141838 10.18 10.28ed5a58 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.789128+0000", 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.53485543199997, 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9243787189999999, 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.365 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141838 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.789128+0000", 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.789128+0000", 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.366 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.789129+0000", 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6600000000000005e-07 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.789134+0000", 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8449999999999999e-06 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.789148+0000", 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3341e-05 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.367 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.789248+0000", 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010049599999999999 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.792405+0000", 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0031570399999999998 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.792416+0000", 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0246e-05 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.368 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710487+0000", 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.918070948 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710493+0000", 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9290000000000001e-06 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713498+0000", 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030049729999999 2024-03-26T05:21:10.369 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713507+0000", 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0350000000000001e-06 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141870 10.18 10.f77d5298 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.808382+0000", 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.51560077699997, 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9051565770000001, 2024-03-26T05:21:10.370 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141870 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.808382+0000", 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.371 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.808382+0000", 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.808384+0000", 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.598e-06 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.808390+0000", 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8250000000000001e-06 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.372 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.808402+0000", 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2278e-05 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.808411+0000", 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.439e-06 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.810206+0000", 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0017952210000000001 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.373 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.810215+0000", 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8820000000000007e-06 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710516+0000", 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.90030157399999999 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710523+0000", 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2790000000000004e-06 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.374 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713531+0000", 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030082760000001 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713539+0000", 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2050000000000002e-06 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.375 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141873 10.18 10.4c990018 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.809190+0000", 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.51479293599999, 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.90438094, 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141873 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.376 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.809190+0000", 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.809190+0000", 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.809191+0000", 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3200000000000004e-07 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.377 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.809195+0000", 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9949999999999999e-06 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.809206+0000", 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0743e-05 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.809213+0000", 2024-03-26T05:21:10.378 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3810000000000001e-06 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.810236+0000", 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010223000000000001 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.810241+0000", 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5119999999999999e-06 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.379 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710544+0000", 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.90030308199999998 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710551+0000", 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1620000000000002e-06 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713564+0000", 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030123719999999 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.380 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713571+0000", 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5610000000000001e-06 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141891 10.18 10.e5b3c9d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.825300+0000", 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.49868274599999, 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8883019509999999, 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.381 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141891 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825300+0000", 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825300+0000", 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.382 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825302+0000", 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3149999999999999e-06 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825305+0000", 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2780000000000002e-06 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825314+0000", 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5180000000000004e-06 2024-03-26T05:21:10.383 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825320+0000", 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8569999999999997e-06 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.825825+0000", 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000504262 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.826868+0000", 2024-03-26T05:21:10.384 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001043247 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710579+0000", 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.88371147800000005 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710587+0000", 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7980000000000006e-06 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713594+0000", 2024-03-26T05:21:10.385 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030070179999999 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713602+0000", 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1799999999999996e-06 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:141933 10.18 10.ccf33018 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.855386+0000", 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.46859769299999, 2024-03-26T05:21:10.386 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8582478579999999, 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 141933 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855386+0000", 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855386+0000", 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855387+0000", 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5650000000000001e-06 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855392+0000", 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9609999999999996e-06 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855408+0000", 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6242000000000001e-05 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855425+0000", 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6572e-05 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855532+0000", 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000106999 2024-03-26T05:21:10.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.855543+0000", 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0845999999999999e-05 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710612+0000", 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.85506965499999998 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710620+0000", 2024-03-26T05:21:10.390 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1550000000000001e-06 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713625+0000", 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030055760000001 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713633+0000", 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2870000000000004e-06 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.391 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142054 10.18 10.8fe3ef98 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.933106+0000", 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.390876929, 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7805604609999999, 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142054 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.392 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933106+0000", 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933106+0000", 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933108+0000", 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.257e-06 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933111+0000", 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.765e-06 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933164+0000", 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2379000000000001e-05 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933278+0000", 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000114252 2024-03-26T05:21:10.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933403+0000", 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012489399999999999 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.933412+0000", 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5000000000000005e-06 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.395 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710645+0000", 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.77723299800000001 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710652+0000", 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0910000000000004e-06 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713658+0000", 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.003005916 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.396 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713667+0000", 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4090000000000003e-06 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142075 10.18 10.efe5c358 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.945089+0000", 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.37889435300002, 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.768610056, 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.397 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142075 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945089+0000", 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945089+0000", 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945090+0000", 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.04e-06 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945094+0000", 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1400000000000002e-06 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945106+0000", 2024-03-26T05:21:10.399 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2123999999999999e-05 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945114+0000", 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8860000000000004e-06 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945149+0000", 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.5429999999999998e-05 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.945155+0000", 2024-03-26T05:21:10.400 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2279999999999998e-06 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710679+0000", 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.76552449199999995 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710687+0000", 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3350000000000001e-06 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.401 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713691+0000", 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.003004556 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713699+0000", 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8250000000000005e-06 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142081 10.18 10.809f8d38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.947216+0000", 2024-03-26T05:21:10.402 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.37676752300001, 2024-03-26T05:21:10.403 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.76651465, 2024-03-26T05:21:10.403 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.403 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.403 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.403 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.403 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:10.404 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.404 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142081 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947216+0000", 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947216+0000", 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.405 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947216+0000", 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.37e-07 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947220+0000", 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8199999999999998e-06 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947230+0000", 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2969999999999992e-06 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.406 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947236+0000", 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1990000000000003e-06 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947679+0000", 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00044369400000000001 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.947688+0000", 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2179999999999999e-06 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.407 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710712+0000", 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.76302396100000003 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710719+0000", 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2989999999999999e-06 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713723+0000", 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030039209999999 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713730+0000", 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5039999999999996e-06 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142089 10.18 10.b56a2a38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.952307+0000", 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.37167641500002, 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7614548509999999, 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.409 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142089 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952307+0000", 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952307+0000", 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952308+0000", 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.035e-06 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952312+0000", 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8619999999999996e-06 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952348+0000", 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6599e-05 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952357+0000", 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4440000000000008e-06 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952508+0000", 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015090000000000001 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.952517+0000", 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2560000000000008e-06 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710745+0000", 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.75822770399999995 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710751+0000", 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6209999999999998e-06 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713754+0000", 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030027450000001 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.413 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713762+0000", 2024-03-26T05:21:10.414 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6850000000000001e-06 2024-03-26T05:21:10.414 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.414 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.414 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.414 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.414 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.414 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142118 10.18 10.c99f83d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.414 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:51.978316+0000", 2024-03-26T05:21:10.414 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.34566682000002, 2024-03-26T05:21:10.414 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.735477129, 2024-03-26T05:21:10.414 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.415 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.415 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.415 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.415 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.415 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142118 2024-03-26T05:21:10.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.415 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.415 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.415 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978316+0000", 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978316+0000", 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978318+0000", 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7060000000000001e-06 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.416 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978323+0000", 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9629999999999997e-06 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978338+0000", 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5274e-05 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978346+0000", 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6909999999999997e-06 2024-03-26T05:21:10.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978414+0000", 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8128999999999998e-05 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:51.978428+0000", 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3898e-05 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.418 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710774+0000", 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.73234595899999999 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710781+0000", 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.204e-06 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713785+0000", 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0030040730000001 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.419 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713794+0000", 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2320000000000001e-06 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142151 10.18 10.ce0d3fd8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:52.011777+0000", 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.31220572799998, 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.702047256, 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.420 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142151 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011777+0000", 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011777+0000", 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.421 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011779+0000", 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.742e-06 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011784+0000", 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.028e-06 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011796+0000", 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1688999999999999e-05 2024-03-26T05:21:10.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011803+0000", 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.861e-06 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011860+0000", 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7661000000000002e-05 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.011869+0000", 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3420000000000008e-06 2024-03-26T05:21:10.423 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710806+0000", 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.69893755000000002 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710813+0000", 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8090000000000004e-06 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713818+0000", 2024-03-26T05:21:10.424 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.003004354 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713825+0000", 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2200000000000003e-06 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142154 10.18 10.78c53af8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:52.016807+0000", 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.30717642899998, 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6970536140000001, 2024-03-26T05:21:10.425 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142154 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016807+0000", 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.426 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016807+0000", 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016808+0000", 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.297e-06 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016813+0000", 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.724e-06 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016826+0000", 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3387000000000001e-05 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016843+0000", 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6378000000000001e-05 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016885+0000", 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2179999999999999e-05 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016892+0000", 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3289999999999996e-06 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710845+0000", 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.69395327799999995 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710854+0000", 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4749999999999993e-06 2024-03-26T05:21:10.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713853+0000", 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0029995650000001 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.713860+0000", 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0010000000000004e-06 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:142155 10.18 10.ffb63458 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:16:52.016979+0000", 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 258.30700442699998, 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.697838637, 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 142155 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.431 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016979+0000", 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016979+0000", 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016979+0000", 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3099999999999997e-07 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.432 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016983+0000", 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4920000000000002e-06 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.016992+0000", 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4520000000000003e-06 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.017000+0000", 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5259999999999996e-06 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.433 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.017031+0000", 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.0621999999999999e-05 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.017034+0000", 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9519999999999996e-06 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710879+0000", 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.69384490899999995 2024-03-26T05:21:10.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:52.710887+0000", 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1359999999999999e-06 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.714807+0000", 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.003920022 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:16:53.714817+0000", 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0895e-05 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.435 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.436 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.436 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:21:10.436 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:21:10.436 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:21:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:10.437 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:10.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:10.538 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:10.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:10.569 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:10.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:10.670 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:10.698 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:10.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:10.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:10.800 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:10.812 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:21:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:10.849 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:10.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:10.951 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:10.978 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:10.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:10.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:11.080 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:11.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:10 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:21:11.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:10 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:21:11.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:10 smithi080 ceph-mon[40404]: pgmap v971: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 714 op/s; 7390/237207 objects degraded (3.115%); 12633/237207 objects misplaced (5.326%); 35 MiB/s, 561 objects/s recovering 2024-03-26T05:21:11.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:10 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1424412789' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:21:11.116 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:10 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:21:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:10 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:21:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:10 smithi080 ceph-mon[29691]: pgmap v971: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 714 op/s; 7390/237207 objects degraded (3.115%); 12633/237207 objects misplaced (5.326%); 35 MiB/s, 561 objects/s recovering 2024-03-26T05:21:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:10 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1424412789' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:21:11.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:11.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:11.218 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:11.255 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:11.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:11.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:11.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:10 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:21:11.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:10 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:21:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:10 smithi132 ceph-mon[33500]: pgmap v971: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 714 op/s; 7390/237207 objects degraded (3.115%); 12633/237207 objects misplaced (5.326%); 35 MiB/s, 561 objects/s recovering 2024-03-26T05:21:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:10 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1424412789' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:21:11.357 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:11.393 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:11.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:11.495 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:11.533 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:11.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:11.635 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:11.676 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:11.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:11.778 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:11.819 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:11.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:11.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:11.921 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:11.958 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:11.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:12.059 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:12.087 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:12.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:12.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:11 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:21:12.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:11 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1424412789' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:21:12.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:11 smithi080 ceph-mon[29691]: osdmap e454: 8 total, 7 up, 6 in 2024-03-26T05:21:12.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:11 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 7390/237207 objects degraded (3.115%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:12.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:11 smithi080 ceph-mon[29691]: osdmap e455: 8 total, 7 up, 6 in 2024-03-26T05:21:12.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:11 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:21:12.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:11 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1424412789' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:21:12.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:11 smithi080 ceph-mon[40404]: osdmap e454: 8 total, 7 up, 6 in 2024-03-26T05:21:12.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:11 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 7390/237207 objects degraded (3.115%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:12.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:11 smithi080 ceph-mon[40404]: osdmap e455: 8 total, 7 up, 6 in 2024-03-26T05:21:12.189 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:12.217 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:12.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:12.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:12.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:11 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:21:12.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:11 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1424412789' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:21:12.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:11 smithi132 ceph-mon[33500]: osdmap e454: 8 total, 7 up, 6 in 2024-03-26T05:21:12.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:11 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 7390/237207 objects degraded (3.115%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:12.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:11 smithi132 ceph-mon[33500]: osdmap e455: 8 total, 7 up, 6 in 2024-03-26T05:21:12.320 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:12.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:12.367 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:12.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:12.468 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:12.499 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:12.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:12.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:12.601 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:12.646 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:12.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:12.748 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:12.791 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:12.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:12.893 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:12.932 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:12.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:13.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:13.067 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:13.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:13.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:12 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:21:13.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:12 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:21:13.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:12 smithi080 ceph-mon[40404]: pgmap v974: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.03k op/s; 5578/229947 objects degraded (2.426%); 11019/229947 objects misplaced (4.792%); 54 MiB/s, 868 objects/s recovering 2024-03-26T05:21:13.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:12 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:21:13.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:12 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:21:13.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:12 smithi080 ceph-mon[29691]: pgmap v974: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.03k op/s; 5578/229947 objects degraded (2.426%); 11019/229947 objects misplaced (4.792%); 54 MiB/s, 868 objects/s recovering 2024-03-26T05:21:13.169 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:13.202 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:13.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:13.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:12 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:21:13.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:12 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:21:13.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:12 smithi132 ceph-mon[33500]: pgmap v974: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.03k op/s; 5578/229947 objects degraded (2.426%); 11019/229947 objects misplaced (4.792%); 54 MiB/s, 868 objects/s recovering 2024-03-26T05:21:13.304 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:13.344 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:13.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:13.446 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:13.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:13.484 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:13.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:13.585 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:13.630 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:13.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:13.732 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:13.776 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:13.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:13.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:13.877 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:13.918 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:13.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:14.020 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:14.057 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:14.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:14.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:14.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:14.198 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:14.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:14.299 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:14.341 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:14.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:14.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:21:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:21:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:21:14.442 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:14.488 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:14.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:14.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:14.561 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T05:21:14.590 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:14.617 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:14.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:14.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:14.718 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:14.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:14.792 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:14.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:14.893 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:14.941 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:14.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:15.043 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:15.084 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:15.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:15.186 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:15.245 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:15.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:15.346 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:15.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:15 smithi080 ceph-mon[40404]: pgmap v975: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 661 op/s; 5578/229947 objects degraded (2.426%); 11019/229947 objects misplaced (4.792%); 35 MiB/s, 555 objects/s recovering 2024-03-26T05:21:15.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:15 smithi080 ceph-mon[29691]: pgmap v975: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 661 op/s; 5578/229947 objects degraded (2.426%); 11019/229947 objects misplaced (4.792%); 35 MiB/s, 555 objects/s recovering 2024-03-26T05:21:15.389 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:15.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:15.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:15.491 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:15.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:15.520 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:15.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:15.540 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:15 smithi132 ceph-mon[33500]: pgmap v975: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 661 op/s; 5578/229947 objects degraded (2.426%); 11019/229947 objects misplaced (4.792%); 35 MiB/s, 555 objects/s recovering 2024-03-26T05:21:15.621 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:15.662 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:15.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:15.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:15.764 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:15.796 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:15.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:15.898 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:15.941 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:15.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:16.043 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:16.088 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:16.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:16.190 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:16.222 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:16.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:16.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:21:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:21:16.325 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:16.357 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:16.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:16.364 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:16 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:21:16.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:16 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:21:16.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:16 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:21:16.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:16 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:21:16.457 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:16.486 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:16.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:16.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:16.615 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:16.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:16.717 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:16.745 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:16.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:16.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:16.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:16 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:21:16.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:16 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:21:16.847 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:16.895 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:16.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:16.996 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:17.026 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:17.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:17.129 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:17.172 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:17.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:17.273 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:17.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:17.344 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:17.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:17.445 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:17.462 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:17.494 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:17 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:17.494 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:17 smithi080 ceph-mon[29691]: pgmap v976: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.5 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 853 op/s; 4811/221571 objects degraded (2.171%); 8723/221571 objects misplaced (3.937%); 46 MiB/s, 733 objects/s recovering 2024-03-26T05:21:17.494 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:17 smithi080 ceph-mon[29691]: osdmap e456: 8 total, 7 up, 6 in 2024-03-26T05:21:17.494 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:17 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 4811/221571 objects degraded (2.171%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:17.494 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:17 smithi080 ceph-mon[29691]: osdmap e457: 8 total, 7 up, 6 in 2024-03-26T05:21:17.495 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:17 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:17.495 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:17 smithi080 ceph-mon[40404]: pgmap v976: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.5 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 853 op/s; 4811/221571 objects degraded (2.171%); 8723/221571 objects misplaced (3.937%); 46 MiB/s, 733 objects/s recovering 2024-03-26T05:21:17.495 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:17 smithi080 ceph-mon[40404]: osdmap e456: 8 total, 7 up, 6 in 2024-03-26T05:21:17.495 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:17 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 4811/221571 objects degraded (2.171%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:17.495 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:17 smithi080 ceph-mon[40404]: osdmap e457: 8 total, 7 up, 6 in 2024-03-26T05:21:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:17.508 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:17.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:17.609 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:17.650 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:17.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:17.715 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:17 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:17.715 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:17 smithi132 ceph-mon[33500]: pgmap v976: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.5 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 853 op/s; 4811/221571 objects degraded (2.171%); 8723/221571 objects misplaced (3.937%); 46 MiB/s, 733 objects/s recovering 2024-03-26T05:21:17.715 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:17 smithi132 ceph-mon[33500]: osdmap e456: 8 total, 7 up, 6 in 2024-03-26T05:21:17.715 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:17 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 4811/221571 objects degraded (2.171%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:17.716 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:17 smithi132 ceph-mon[33500]: osdmap e457: 8 total, 7 up, 6 in 2024-03-26T05:21:17.751 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:17.779 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:17.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:17.880 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:17.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:17.923 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:17.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:18.024 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:18.030 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:21:18.048 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:18.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:18.150 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:18.178 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:18.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:18.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:18.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:18.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:18.310 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:18.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:18.411 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:18.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:18.459 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:18.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:18.560 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:18.593 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:18.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:18.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:18.694 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:18.739 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:18.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:18.840 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:18.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:18.987 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:19.032 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:19.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:19.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:19.134 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:19.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:19.163 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:19.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:19.264 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:19.309 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:19.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:19.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:19.454 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:19.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:19.556 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:19.587 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:19.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:19.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:19.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:19 smithi080 ceph-mon[29691]: pgmap v979: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 786 B/s rd, 0 B/s wr, 680 op/s; 3818/217347 objects degraded (1.757%); 8195/217347 objects misplaced (3.770%); 32 MiB/s, 509 objects/s recovering 2024-03-26T05:21:19.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:19 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4225090292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:21:19.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:19 smithi080 ceph-mon[40404]: pgmap v979: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 786 B/s rd, 0 B/s wr, 680 op/s; 3818/217347 objects degraded (1.757%); 8195/217347 objects misplaced (3.770%); 32 MiB/s, 509 objects/s recovering 2024-03-26T05:21:19.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:19 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4225090292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:21:19.689 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:19.720 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:19.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:19.763 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:21:19.763 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:21:19.763 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:21:19.763 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:21:19.763 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:21:19.763 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:21:19.763 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:21:19.764 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:21:19.764 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:21:19.764 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:21:19.764 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:21:19.764 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:21:19.764 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:21:19.765 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:21:19.765 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:21:19.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:19 smithi132 ceph-mon[33500]: pgmap v979: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 786 B/s rd, 0 B/s wr, 680 op/s; 3818/217347 objects degraded (1.757%); 8195/217347 objects misplaced (3.770%); 32 MiB/s, 509 objects/s recovering 2024-03-26T05:21:19.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:19 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4225090292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:21:19.822 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:19.850 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:19.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:19.951 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:19.980 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:19.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:20.082 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:20.124 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:20.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:20.226 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:20.269 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:20.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:20.335 INFO:teuthology.orchestra.run.smithi080.stderr:set pool 11 pg_num to 26 2024-03-26T05:21:20.370 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:20.413 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:20.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:20.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:20.515 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:20.546 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:20.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:20.614 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:20 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4059804024' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-26T05:21:20.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:20 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4059804024' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-26T05:21:20.647 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:20.686 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:20.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:20.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:20.788 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:20.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:20 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4059804024' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-26T05:21:20.830 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:20.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:20.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:20.931 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:20.960 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:20.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:21:20.982 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:21:21.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:21.088 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:21.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:21.190 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:21.219 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:21.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:21.221 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:21.322 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:21.332 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:21:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:21:20.910+0000 7f2fcc605700 -1 mgr get_metadata_python Requested missing service osd.4 2024-03-26T05:21:21.368 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:21.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:21.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:21.470 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:21.505 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:21.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:21.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:21.607 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:21.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:21 smithi080 ceph-mon[29691]: pgmap v980: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 591 op/s; 3818/215751 objects degraded (1.770%); 8195/215751 objects misplaced (3.798%); 26 MiB/s, 414 objects/s recovering 2024-03-26T05:21:21.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:21 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4059804024' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-26T05:21:21.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:21 smithi080 ceph-mon[29691]: osdmap e458: 8 total, 7 up, 6 in 2024-03-26T05:21:21.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:21:21.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:21:21.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:21 smithi080 ceph-mon[40404]: pgmap v980: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 591 op/s; 3818/215751 objects degraded (1.770%); 8195/215751 objects misplaced (3.798%); 26 MiB/s, 414 objects/s recovering 2024-03-26T05:21:21.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:21 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4059804024' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-26T05:21:21.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:21 smithi080 ceph-mon[40404]: osdmap e458: 8 total, 7 up, 6 in 2024-03-26T05:21:21.615 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:21:21.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:21:21.650 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:21.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:21.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:21.752 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:21 smithi132 ceph-mon[33500]: pgmap v980: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 591 op/s; 3818/215751 objects degraded (1.770%); 8195/215751 objects misplaced (3.798%); 26 MiB/s, 414 objects/s recovering 2024-03-26T05:21:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:21 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4059804024' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-26T05:21:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:21 smithi132 ceph-mon[33500]: osdmap e458: 8 total, 7 up, 6 in 2024-03-26T05:21:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:21:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:21:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:21.801 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:21.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:21.902 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:21.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:21.942 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:21.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:22.043 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:22.078 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:22.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:22.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:22.180 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:22.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:22.207 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:22.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:22.308 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:22.340 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:22.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:22.442 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:22.470 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:22.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:22.483 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:22 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 3818/215751 objects degraded (1.770%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:21:22.483 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:22 smithi080 ceph-mon[40404]: osdmap e459: 8 total, 7 up, 6 in 2024-03-26T05:21:22.483 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:22 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T05:21:22.483 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:22 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 3818/215751 objects degraded (1.770%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:21:22.483 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:22 smithi080 ceph-mon[29691]: osdmap e459: 8 total, 7 up, 6 in 2024-03-26T05:21:22.483 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:22 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T05:21:22.572 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:22.607 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:22.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:22.708 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:22.742 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:22.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:22.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:22 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 3818/215751 objects degraded (1.770%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:21:22.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:22 smithi132 ceph-mon[33500]: osdmap e459: 8 total, 7 up, 6 in 2024-03-26T05:21:22.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:22 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T05:21:22.844 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:22.898 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:22.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:22.984 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:21:23.000 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:23.068 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:23.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:23.170 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:23.200 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:23.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:23.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:23.460 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:23.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:23.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:23.562 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:23.588 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:23.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:23.690 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:23.716 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:23.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:23.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:23.818 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:23.848 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:23.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:23.950 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:23.979 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:23.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:24.022 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:23 smithi080 ceph-mon[29691]: pgmap v983: 158 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 864 B/s rd, 0 B/s wr, 928 op/s; 2226/205071 objects degraded (1.085%); 5431/205071 objects misplaced (2.648%); 44 MiB/s, 702 objects/s recovering 2024-03-26T05:21:24.022 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:23 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-26T05:21:24.022 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:23 smithi080 ceph-mon[29691]: osdmap e460: 8 total, 7 up, 6 in 2024-03-26T05:21:24.022 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:23 smithi080 ceph-mon[40404]: pgmap v983: 158 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 864 B/s rd, 0 B/s wr, 928 op/s; 2226/205071 objects degraded (1.085%); 5431/205071 objects misplaced (2.648%); 44 MiB/s, 702 objects/s recovering 2024-03-26T05:21:24.022 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:23 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-26T05:21:24.023 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:23 smithi080 ceph-mon[40404]: osdmap e460: 8 total, 7 up, 6 in 2024-03-26T05:21:24.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:23 smithi132 ceph-mon[33500]: pgmap v983: 158 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 864 B/s rd, 0 B/s wr, 928 op/s; 2226/205071 objects degraded (1.085%); 5431/205071 objects misplaced (2.648%); 44 MiB/s, 702 objects/s recovering 2024-03-26T05:21:24.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:23 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-26T05:21:24.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:23 smithi132 ceph-mon[33500]: osdmap e460: 8 total, 7 up, 6 in 2024-03-26T05:21:24.081 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:24.110 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:24.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:24.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:24.212 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:24.235 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:24.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:24.338 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:24.345 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:21:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:21:24.370 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:24.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:24.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:24.472 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:24.496 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:24.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:24.598 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:24.622 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:24.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:24.724 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:24.749 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:24.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:24.850 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:24.874 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:24.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:24.976 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:25.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:25.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:24 smithi132 ceph-mon[33500]: 11.8 continuing backfill to osd.0 from (439'7535,459'11035] MIN to 459'11035 2024-03-26T05:21:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:24 smithi132 ceph-mon[33500]: 11.5 continuing backfill to osd.3 from (433'7126,459'10905] MIN to 459'10905 2024-03-26T05:21:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:24 smithi132 ceph-mon[33500]: 11.7 continuing backfill to osd.0 from (434'6999,459'10713] MIN to 459'10713 2024-03-26T05:21:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:24 smithi132 ceph-mon[33500]: 11.7 continuing backfill to osd.7 from (434'6999,459'10713] MIN to 459'10713 2024-03-26T05:21:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:24 smithi132 ceph-mon[33500]: osdmap e461: 8 total, 7 up, 6 in 2024-03-26T05:21:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:24 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T05:21:25.101 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:25.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-mon[29691]: 11.8 continuing backfill to osd.0 from (439'7535,459'11035] MIN to 459'11035 2024-03-26T05:21:25.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-mon[29691]: 11.5 continuing backfill to osd.3 from (433'7126,459'10905] MIN to 459'10905 2024-03-26T05:21:25.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-mon[29691]: 11.7 continuing backfill to osd.0 from (434'6999,459'10713] MIN to 459'10713 2024-03-26T05:21:25.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-mon[29691]: 11.7 continuing backfill to osd.7 from (434'6999,459'10713] MIN to 459'10713 2024-03-26T05:21:25.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-mon[29691]: osdmap e461: 8 total, 7 up, 6 in 2024-03-26T05:21:25.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T05:21:25.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-mon[40404]: 11.8 continuing backfill to osd.0 from (439'7535,459'11035] MIN to 459'11035 2024-03-26T05:21:25.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-mon[40404]: 11.5 continuing backfill to osd.3 from (433'7126,459'10905] MIN to 459'10905 2024-03-26T05:21:25.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-mon[40404]: 11.7 continuing backfill to osd.0 from (434'6999,459'10713] MIN to 459'10713 2024-03-26T05:21:25.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-mon[40404]: 11.7 continuing backfill to osd.7 from (434'6999,459'10713] MIN to 459'10713 2024-03-26T05:21:25.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-mon[40404]: osdmap e461: 8 total, 7 up, 6 in 2024-03-26T05:21:25.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:24 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T05:21:25.125 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:25.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:25.227 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:25.251 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:25.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:25.352 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:25.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:25.379 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:25.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:25.480 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:25.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:25.505 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:25.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:25.606 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:25.631 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:25.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:25.732 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:25.759 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:25.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:25.861 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:25.898 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:25.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:25.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:26.000 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:26.028 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:26.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:26.040 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:21:25 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:21:26.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:25 smithi132 ceph-mon[33500]: pgmap v986: 168 pgs: 10 unknown, 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 890 op/s; 2226/205071 objects degraded (1.085%); 5431/205071 objects misplaced (2.648%); 52 MiB/s, 831 objects/s recovering 2024-03-26T05:21:26.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:25 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-26T05:21:26.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:25 smithi132 ceph-mon[33500]: osdmap e462: 8 total, 7 up, 6 in 2024-03-26T05:21:26.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:25 smithi080 ceph-mon[29691]: pgmap v986: 168 pgs: 10 unknown, 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 890 op/s; 2226/205071 objects degraded (1.085%); 5431/205071 objects misplaced (2.648%); 52 MiB/s, 831 objects/s recovering 2024-03-26T05:21:26.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:25 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-26T05:21:26.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:25 smithi080 ceph-mon[29691]: osdmap e462: 8 total, 7 up, 6 in 2024-03-26T05:21:26.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:25 smithi080 ceph-mon[40404]: pgmap v986: 168 pgs: 10 unknown, 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 151 active+clean; 4.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 890 op/s; 2226/205071 objects degraded (1.085%); 5431/205071 objects misplaced (2.648%); 52 MiB/s, 831 objects/s recovering 2024-03-26T05:21:26.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:25 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-26T05:21:26.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:25 smithi080 ceph-mon[40404]: osdmap e462: 8 total, 7 up, 6 in 2024-03-26T05:21:26.130 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:26.158 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:26.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:26.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:26.260 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:26.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:26.285 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:26.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:26.386 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:26.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:26.410 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:26.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:26.512 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:26.541 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:26.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:26.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:26.643 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:26.668 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:26.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:26.769 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:26.801 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:26.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:26.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:26.903 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:26.928 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7] out_osds: [4, 5] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T05:21:26.928 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:21:26.929 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-26T05:21:26.929 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-26T05:21:26.929 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:21:26.930 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:26.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:27.032 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:27.090 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:27.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:27.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:27.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[40404]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 459'10905 2024-03-26T05:21:27.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[40404]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 459'10713 2024-03-26T05:21:27.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[40404]: 11.18 continuing backfill to osd.0 from (439'7535,459'11035] MIN to 459'11035 2024-03-26T05:21:27.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[40404]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 459'11035 2024-03-26T05:21:27.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[40404]: osdmap e463: 8 total, 7 up, 6 in 2024-03-26T05:21:27.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[40404]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 459'10905 2024-03-26T05:21:27.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[40404]: 11.17 continuing backfill to osd.7 from (434'6999,459'10713] MIN to 459'10713 2024-03-26T05:21:27.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:27.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[40404]: pgmap v989: 168 pgs: 1 activating+remapped, 1 remapped+peering, 1 peering, 8 unknown, 1 active+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 155 active+clean; 3.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1050/177315 objects degraded (0.592%); 5188/177315 objects misplaced (2.926%); 46 MiB/s, 730 objects/s recovering 2024-03-26T05:21:27.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 2226/205071 objects degraded (1.085%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:21:27.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[29691]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 459'10905 2024-03-26T05:21:27.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[29691]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 459'10713 2024-03-26T05:21:27.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[29691]: 11.18 continuing backfill to osd.0 from (439'7535,459'11035] MIN to 459'11035 2024-03-26T05:21:27.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[29691]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 459'11035 2024-03-26T05:21:27.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[29691]: osdmap e463: 8 total, 7 up, 6 in 2024-03-26T05:21:27.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[29691]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 459'10905 2024-03-26T05:21:27.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[29691]: 11.17 continuing backfill to osd.7 from (434'6999,459'10713] MIN to 459'10713 2024-03-26T05:21:27.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:27.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[29691]: pgmap v989: 168 pgs: 1 activating+remapped, 1 remapped+peering, 1 peering, 8 unknown, 1 active+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 155 active+clean; 3.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1050/177315 objects degraded (0.592%); 5188/177315 objects misplaced (2.926%); 46 MiB/s, 730 objects/s recovering 2024-03-26T05:21:27.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:26 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 2226/205071 objects degraded (1.085%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:21:27.193 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:27.218 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:27.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:27.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:26 smithi132 ceph-mon[33500]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 459'10905 2024-03-26T05:21:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:26 smithi132 ceph-mon[33500]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 459'10713 2024-03-26T05:21:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:26 smithi132 ceph-mon[33500]: 11.18 continuing backfill to osd.0 from (439'7535,459'11035] MIN to 459'11035 2024-03-26T05:21:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:26 smithi132 ceph-mon[33500]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 459'11035 2024-03-26T05:21:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:26 smithi132 ceph-mon[33500]: osdmap e463: 8 total, 7 up, 6 in 2024-03-26T05:21:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:26 smithi132 ceph-mon[33500]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 459'10905 2024-03-26T05:21:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:26 smithi132 ceph-mon[33500]: 11.17 continuing backfill to osd.7 from (434'6999,459'10713] MIN to 459'10713 2024-03-26T05:21:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:26 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:26 smithi132 ceph-mon[33500]: pgmap v989: 168 pgs: 1 activating+remapped, 1 remapped+peering, 1 peering, 8 unknown, 1 active+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 155 active+clean; 3.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1050/177315 objects degraded (0.592%); 5188/177315 objects misplaced (2.926%); 46 MiB/s, 730 objects/s recovering 2024-03-26T05:21:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:26 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 2226/205071 objects degraded (1.085%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T05:21:27.321 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:27.351 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:27.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:27.452 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:27.500 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:27.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:27.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:27.511 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:21:27.511 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:21:27.602 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:27.652 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:27.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:27.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:27.753 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:27.778 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:27.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:27.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:27.880 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:27.917 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:27.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:27.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:28.019 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:28.045 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:28.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:28.147 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:28.175 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:28.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:28.277 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:28.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: osdmap e464: 8 total, 7 up, 6 in 2024-03-26T05:21:28.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 461'11066 2024-03-26T05:21:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.12 starting backfill to osd.7 from (0'0,0'0] MAX to 461'10715 2024-03-26T05:21:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.14 starting backfill to osd.1 from (0'0,0'0] MAX to 461'11045 2024-03-26T05:21:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 461'13999 2024-03-26T05:21:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 461'11046 2024-03-26T05:21:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.13 starting backfill to osd.2 from (0'0,0'0] MAX to 461'13999 2024-03-26T05:21:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 461'10966 2024-03-26T05:21:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 461'11046 2024-03-26T05:21:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 461'11066 2024-03-26T05:21:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 461'10966 2024-03-26T05:21:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.10 starting backfill to osd.6 from (0'0,0'0] MAX to 461'11046 2024-03-26T05:21:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 461'10747 2024-03-26T05:21:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:27 smithi132 ceph-mon[33500]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 461'10747 2024-03-26T05:21:28.301 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:28.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:28.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:28.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: osdmap e464: 8 total, 7 up, 6 in 2024-03-26T05:21:28.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 461'11066 2024-03-26T05:21:28.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.12 starting backfill to osd.7 from (0'0,0'0] MAX to 461'10715 2024-03-26T05:21:28.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.14 starting backfill to osd.1 from (0'0,0'0] MAX to 461'11045 2024-03-26T05:21:28.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 461'13999 2024-03-26T05:21:28.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 461'11046 2024-03-26T05:21:28.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.13 starting backfill to osd.2 from (0'0,0'0] MAX to 461'13999 2024-03-26T05:21:28.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 461'10966 2024-03-26T05:21:28.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 461'11046 2024-03-26T05:21:28.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 461'11066 2024-03-26T05:21:28.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 461'10966 2024-03-26T05:21:28.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.10 starting backfill to osd.6 from (0'0,0'0] MAX to 461'11046 2024-03-26T05:21:28.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 461'10747 2024-03-26T05:21:28.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[40404]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 461'10747 2024-03-26T05:21:28.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: osdmap e464: 8 total, 7 up, 6 in 2024-03-26T05:21:28.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 461'11066 2024-03-26T05:21:28.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.12 starting backfill to osd.7 from (0'0,0'0] MAX to 461'10715 2024-03-26T05:21:28.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.14 starting backfill to osd.1 from (0'0,0'0] MAX to 461'11045 2024-03-26T05:21:28.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 461'13999 2024-03-26T05:21:28.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 461'11046 2024-03-26T05:21:28.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.13 starting backfill to osd.2 from (0'0,0'0] MAX to 461'13999 2024-03-26T05:21:28.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 461'10966 2024-03-26T05:21:28.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 461'11046 2024-03-26T05:21:28.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 461'11066 2024-03-26T05:21:28.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 461'10966 2024-03-26T05:21:28.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.10 starting backfill to osd.6 from (0'0,0'0] MAX to 461'11046 2024-03-26T05:21:28.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 461'10747 2024-03-26T05:21:28.368 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:27 smithi080 ceph-mon[29691]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 461'10747 2024-03-26T05:21:28.402 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:28.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:28.442 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:28.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:28.543 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:28.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:28.587 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:28.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:28.688 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:28.729 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:28.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:28.831 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:28.859 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:28.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:28.960 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:29.014 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:29.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:29.115 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:29.139 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:29.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:29.240 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:29.268 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:29.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:29.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:29.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:28 smithi132 ceph-mon[33500]: pgmap v991: 168 pgs: 1 activating+remapped, 7 remapped+peering, 1 peering, 2 active+undersized+degraded+remapped+backfill_toofull, 1 unknown, 156 active+clean; 4.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 237 op/s; 742/201891 objects degraded (0.368%); 6531/201891 objects misplaced (3.235%); 45 MiB/s, 721 objects/s recovering 2024-03-26T05:21:29.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:28 smithi080 ceph-mon[40404]: pgmap v991: 168 pgs: 1 activating+remapped, 7 remapped+peering, 1 peering, 2 active+undersized+degraded+remapped+backfill_toofull, 1 unknown, 156 active+clean; 4.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 237 op/s; 742/201891 objects degraded (0.368%); 6531/201891 objects misplaced (3.235%); 45 MiB/s, 721 objects/s recovering 2024-03-26T05:21:29.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:28 smithi080 ceph-mon[29691]: pgmap v991: 168 pgs: 1 activating+remapped, 7 remapped+peering, 1 peering, 2 active+undersized+degraded+remapped+backfill_toofull, 1 unknown, 156 active+clean; 4.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 237 op/s; 742/201891 objects degraded (0.368%); 6531/201891 objects misplaced (3.235%); 45 MiB/s, 721 objects/s recovering 2024-03-26T05:21:29.369 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:29.399 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:29.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:29.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:29.501 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:29.541 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:29.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:29.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:29.644 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:29.679 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:29.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:29.780 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:29.822 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:29.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:29.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:29.924 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:29.953 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:29.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:30.055 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:30.082 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:30.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:30.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:30.184 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:30.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:30.207 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:30.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:30.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:29 smithi132 ceph-mon[33500]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:21:30.308 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:30.346 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:30.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:30.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:29 smithi080 ceph-mon[40404]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:21:30.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:29 smithi080 ceph-mon[29691]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:21:30.448 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:30.486 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:30.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:30.588 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:30.629 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:30.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:30.730 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:30.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:30.757 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:30.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:30.858 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:30.887 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:30.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:30.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:30.989 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:31.017 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:31.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:31.119 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:31.144 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:31.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:31.245 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:31.273 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:31.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:31.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:31.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:30 smithi132 ceph-mon[33500]: pgmap v992: 168 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 7 remapped+peering, 1 peering, 2 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 4.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 220 op/s; 742/201123 objects degraded (0.369%); 8612/201123 objects misplaced (4.282%); 33 MiB/s, 537 objects/s recovering 2024-03-26T05:21:31.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:30 smithi080 ceph-mon[40404]: pgmap v992: 168 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 7 remapped+peering, 1 peering, 2 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 4.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 220 op/s; 742/201123 objects degraded (0.369%); 8612/201123 objects misplaced (4.282%); 33 MiB/s, 537 objects/s recovering 2024-03-26T05:21:31.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:30 smithi080 ceph-mon[29691]: pgmap v992: 168 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 7 remapped+peering, 1 peering, 2 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 4.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 220 op/s; 742/201123 objects degraded (0.369%); 8612/201123 objects misplaced (4.282%); 33 MiB/s, 537 objects/s recovering 2024-03-26T05:21:31.376 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:31.399 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:31.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:31.501 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:31.530 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:31.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:31.632 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:31.660 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:31.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:31.762 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:31.799 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:31.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:31.901 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:31.928 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:31.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:32.030 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:32.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:32.101 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:32.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:32.202 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:32.242 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:32.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:32.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:32.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:31 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 742/201123 objects degraded (0.369%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:32.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:31 smithi132 ceph-mon[33500]: osdmap e465: 8 total, 7 up, 6 in 2024-03-26T05:21:32.344 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:32.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:31 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 742/201123 objects degraded (0.369%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:32.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:31 smithi080 ceph-mon[29691]: osdmap e465: 8 total, 7 up, 6 in 2024-03-26T05:21:32.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:31 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 742/201123 objects degraded (0.369%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:32.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:31 smithi080 ceph-mon[40404]: osdmap e465: 8 total, 7 up, 6 in 2024-03-26T05:21:32.373 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:32.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:32.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:32.475 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:32.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:32.500 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:32.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:32.601 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:32.643 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:32.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:32.743 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:32.768 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:32.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:32.869 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:32.911 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:32.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:32.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:33.013 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:33.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:33.056 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:33.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:33.158 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:33.182 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:33.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:33.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:33.285 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:33.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:32 smithi132 ceph-mon[33500]: pgmap v994: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.6 KiB/s rd, 73 MiB/s wr, 2.37k op/s; 487/190536 objects degraded (0.256%); 29920/190536 objects misplaced (15.703%); 46 MiB/s, 742 objects/s recovering 2024-03-26T05:21:33.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:33.312 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:33.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:33.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:32 smithi080 ceph-mon[29691]: pgmap v994: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.6 KiB/s rd, 73 MiB/s wr, 2.37k op/s; 487/190536 objects degraded (0.256%); 29920/190536 objects misplaced (15.703%); 46 MiB/s, 742 objects/s recovering 2024-03-26T05:21:33.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:32 smithi080 ceph-mon[40404]: pgmap v994: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.6 KiB/s rd, 73 MiB/s wr, 2.37k op/s; 487/190536 objects degraded (0.256%); 29920/190536 objects misplaced (15.703%); 46 MiB/s, 742 objects/s recovering 2024-03-26T05:21:33.414 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:33.457 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:33.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:33.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:33.559 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:33.597 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:33.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:33.698 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:33.726 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:33.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:33.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:33.828 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:33.852 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:33.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:33.954 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:33.995 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:33.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:33.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:34.096 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:34.123 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:34.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:34.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:34.198 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:33 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:21:34.198 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:33 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:21:34.198 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:33 smithi080 ceph-mon[40404]: osdmap e466: 8 total, 7 up, 6 in 2024-03-26T05:21:34.198 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:33 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:21:34.198 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:33 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:21:34.199 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:33 smithi080 ceph-mon[29691]: osdmap e466: 8 total, 7 up, 6 in 2024-03-26T05:21:34.199 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:21:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:21:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:21:34.216 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:34.225 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:34.252 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T05:21:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:34.253 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:34.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:34.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:33 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:21:34.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:33 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:21:34.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:33 smithi132 ceph-mon[33500]: osdmap e466: 8 total, 7 up, 6 in 2024-03-26T05:21:34.354 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:34.380 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:34.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:34.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:34.482 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:34.508 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:34.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:34.610 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:34.634 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:34.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:34.735 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:34.774 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:34.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:34.876 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:34.918 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:34.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:35.022 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:35.064 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:35.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:35.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:35.192 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:34 smithi132 ceph-mon[33500]: pgmap v996: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 695 B/s rd, 0 B/s wr, 514 op/s; 487/190536 objects degraded (0.256%); 29920/190536 objects misplaced (15.703%); 23 MiB/s, 365 objects/s recovering 2024-03-26T05:21:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:35.193 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:35.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:35.265 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:34 smithi080 ceph-mon[29691]: pgmap v996: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 695 B/s rd, 0 B/s wr, 514 op/s; 487/190536 objects degraded (0.256%); 29920/190536 objects misplaced (15.703%); 23 MiB/s, 365 objects/s recovering 2024-03-26T05:21:35.265 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:34 smithi080 ceph-mon[40404]: pgmap v996: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 695 B/s rd, 0 B/s wr, 514 op/s; 487/190536 objects degraded (0.256%); 29920/190536 objects misplaced (15.703%); 23 MiB/s, 365 objects/s recovering 2024-03-26T05:21:35.294 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:35.339 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:35.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:35.440 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:35.468 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:35.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:35.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:35.570 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:35.597 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:35.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:35.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:35.699 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:35.741 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:35.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:35.843 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:35.887 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:35.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:35.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:35.989 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:36.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:36.025 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:36.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:36.126 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:36.150 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:36.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:36.251 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:36.279 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:36.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:36.290 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:21:35 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:21:36.327 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:21:35 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:21:35.943+0000 7f2fcc605700 -1 mgr get_metadata_python Requested missing service osd.4 2024-03-26T05:21:36.381 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:36.423 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:36.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:36.525 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:36.554 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:36.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:36.656 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:36.708 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:36.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:36.809 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:36.839 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:36.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:36.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:36.941 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:36.981 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:36.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:37.083 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:21:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:21:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[29691]: pgmap v997: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 802 op/s; 487/182649 objects degraded (0.267%); 28019/182649 objects misplaced (15.340%); 27 MiB/s, 435 objects/s recovering 2024-03-26T05:21:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[29691]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:21:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 487/190536 objects degraded (0.256%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[29691]: osdmap e467: 8 total, 7 up, 6 in 2024-03-26T05:21:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:21:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:21:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[40404]: pgmap v997: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 802 op/s; 487/182649 objects degraded (0.267%); 28019/182649 objects misplaced (15.340%); 27 MiB/s, 435 objects/s recovering 2024-03-26T05:21:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[40404]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:21:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 487/190536 objects degraded (0.256%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:37.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:36 smithi080 ceph-mon[40404]: osdmap e467: 8 total, 7 up, 6 in 2024-03-26T05:21:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:37.134 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:37.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:37.235 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:37.277 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:37.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:37.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:36 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:21:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:36 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:21:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:36 smithi132 ceph-mon[33500]: pgmap v997: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 802 op/s; 487/182649 objects degraded (0.267%); 28019/182649 objects misplaced (15.340%); 27 MiB/s, 435 objects/s recovering 2024-03-26T05:21:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:36 smithi132 ceph-mon[33500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:21:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:36 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 487/190536 objects degraded (0.256%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:37.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:36 smithi132 ceph-mon[33500]: osdmap e467: 8 total, 7 up, 6 in 2024-03-26T05:21:37.379 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:37.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:37.409 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:37.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:37.510 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:37.539 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:37.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:37.641 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:37.669 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:37.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:37.771 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:37.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:37.807 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:37.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:37.908 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:37.936 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:37.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:38.038 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:38.066 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:38.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:38.168 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:38.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:38.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:38.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:37 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:21:38.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:37 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:21:38.297 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:38.333 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:38.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:38.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:37 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:21:38.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:37 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:21:38.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:37 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:21:38.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:37 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:21:38.434 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:38.462 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:38.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:38.564 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:38.602 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:38.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:38.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:38.704 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:38.748 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:38.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:38.849 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:38.887 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:38.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:38.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:38.989 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:39.016 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:39.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:39.118 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:39.142 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:21:39.158 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:39.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:39.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:39.260 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:39.283 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:39.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:39.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:38 smithi132 ceph-mon[33500]: pgmap v999: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 787 B/s rd, 0 B/s wr, 835 op/s; 203/176397 objects degraded (0.115%); 23967/176397 objects misplaced (13.587%); 32 MiB/s, 517 objects/s recovering 2024-03-26T05:21:39.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:38 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/712010451' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:21:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:38 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:21:39.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:38 smithi080 ceph-mon[40404]: pgmap v999: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 787 B/s rd, 0 B/s wr, 835 op/s; 203/176397 objects degraded (0.115%); 23967/176397 objects misplaced (13.587%); 32 MiB/s, 517 objects/s recovering 2024-03-26T05:21:39.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:38 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/712010451' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:21:39.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:38 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:21:39.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:38 smithi080 ceph-mon[29691]: pgmap v999: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 787 B/s rd, 0 B/s wr, 835 op/s; 203/176397 objects degraded (0.115%); 23967/176397 objects misplaced (13.587%); 32 MiB/s, 517 objects/s recovering 2024-03-26T05:21:39.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:38 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/712010451' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:21:39.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:38 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:21:39.385 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:39.419 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:39.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:39.521 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:39.550 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:39.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:39.654 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:39.684 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:39.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:39.785 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:39.816 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:39.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:39.918 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:39.955 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:39.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:40.057 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:40.097 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:40.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:40.199 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:40.227 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:40.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:39 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:21:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:39 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:21:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:39 smithi132 ceph-mon[33500]: osdmap e468: 8 total, 7 up, 6 in 2024-03-26T05:21:40.329 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:40.355 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:40.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:40.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:39 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:21:40.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:39 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:21:40.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:39 smithi080 ceph-mon[40404]: osdmap e468: 8 total, 7 up, 6 in 2024-03-26T05:21:40.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:39 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:21:40.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:39 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:21:40.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:39 smithi080 ceph-mon[29691]: osdmap e468: 8 total, 7 up, 6 in 2024-03-26T05:21:40.457 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:40.482 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:40.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:40.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:40.584 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:40.609 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:40.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:40.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:40.711 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:40.744 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:40.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:40.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:40.846 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:40.883 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:40.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:40.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:41.011 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:41.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:41.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:41.113 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:41.140 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:41.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:41.242 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:41.269 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:41.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:41.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:40 smithi132 ceph-mon[33500]: osdmap e469: 8 total, 7 up, 6 in 2024-03-26T05:21:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:40 smithi132 ceph-mon[33500]: pgmap v1002: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 868 op/s; 203/174915 objects degraded (0.116%); 23794/174915 objects misplaced (13.603%); 24 MiB/s, 382 objects/s recovering 2024-03-26T05:21:41.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:40 smithi080 ceph-mon[40404]: osdmap e469: 8 total, 7 up, 6 in 2024-03-26T05:21:41.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:40 smithi080 ceph-mon[40404]: pgmap v1002: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 868 op/s; 203/174915 objects degraded (0.116%); 23794/174915 objects misplaced (13.603%); 24 MiB/s, 382 objects/s recovering 2024-03-26T05:21:41.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:40 smithi080 ceph-mon[29691]: osdmap e469: 8 total, 7 up, 6 in 2024-03-26T05:21:41.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:40 smithi080 ceph-mon[29691]: pgmap v1002: 168 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 868 op/s; 203/174915 objects degraded (0.116%); 23794/174915 objects misplaced (13.603%); 24 MiB/s, 382 objects/s recovering 2024-03-26T05:21:41.370 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:41.407 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:41.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:41.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:41.509 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:41.541 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:41.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:41.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:41.643 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:41.679 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:41.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:41.781 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:41.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:41.818 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:41.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:41.920 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:41.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:41.947 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:41.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:42.048 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:42.089 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:42.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:42.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:41 smithi080 ceph-mon[40404]: osdmap e470: 8 total, 7 up, 6 in 2024-03-26T05:21:42.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:41 smithi080 ceph-mon[40404]: Health check update: Degraded data redundancy: 203/174915 objects degraded (0.116%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:42.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:41 smithi080 ceph-mon[40404]: osdmap e471: 8 total, 7 up, 6 in 2024-03-26T05:21:42.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:41 smithi080 ceph-mon[29691]: osdmap e470: 8 total, 7 up, 6 in 2024-03-26T05:21:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:41 smithi080 ceph-mon[29691]: Health check update: Degraded data redundancy: 203/174915 objects degraded (0.116%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:41 smithi080 ceph-mon[29691]: osdmap e471: 8 total, 7 up, 6 in 2024-03-26T05:21:42.190 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:42.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:42.214 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:42.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:42.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:41 smithi132 ceph-mon[33500]: osdmap e470: 8 total, 7 up, 6 in 2024-03-26T05:21:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:41 smithi132 ceph-mon[33500]: Health check update: Degraded data redundancy: 203/174915 objects degraded (0.116%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T05:21:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:41 smithi132 ceph-mon[33500]: osdmap e471: 8 total, 7 up, 6 in 2024-03-26T05:21:42.315 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:42.344 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:42.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:42.445 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:42.474 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:42.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:42.575 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:42.623 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:42.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:42.724 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:42.772 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:42.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:42.874 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:42.898 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:42.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:42.999 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:43.045 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:43.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:43.148 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:43.193 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:43.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:43.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:42 smithi132 ceph-mon[33500]: pgmap v1005: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 2 peering, 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 157 active+clean; 3.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 17354/162399 objects misplaced (10.686%); 43 MiB/s, 692 objects/s recovering 2024-03-26T05:21:43.294 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:43.322 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:43.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:43.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:42 smithi080 ceph-mon[29691]: pgmap v1005: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 2 peering, 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 157 active+clean; 3.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 17354/162399 objects misplaced (10.686%); 43 MiB/s, 692 objects/s recovering 2024-03-26T05:21:43.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:42 smithi080 ceph-mon[40404]: pgmap v1005: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 2 peering, 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 157 active+clean; 3.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 17354/162399 objects misplaced (10.686%); 43 MiB/s, 692 objects/s recovering 2024-03-26T05:21:43.424 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:43.447 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:43.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:43.549 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:43.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:43.590 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:43.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:43.691 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:43.738 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:43.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:43.840 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:43.884 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:43.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:43.986 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:44.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:44.015 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:44.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:44.116 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:44.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:44.144 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:44.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:44.245 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:44.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:43 smithi132 ceph-mon[33500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:21:44.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:43 smithi132 ceph-mon[33500]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 203/174915 objects degraded (0.116%), 2 pgs degraded) 2024-03-26T05:21:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:44.292 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:44.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:44.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:43 smithi080 ceph-mon[29691]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:21:44.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:43 smithi080 ceph-mon[29691]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 203/174915 objects degraded (0.116%), 2 pgs degraded) 2024-03-26T05:21:44.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:21:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:21:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:21:44.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:43 smithi080 ceph-mon[40404]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:21:44.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:43 smithi080 ceph-mon[40404]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 203/174915 objects degraded (0.116%), 2 pgs degraded) 2024-03-26T05:21:44.393 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:44.418 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:44.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:44.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:44.520 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:44.560 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:44.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:44.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:44.662 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:44.701 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:44.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:44.803 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:44.844 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:44.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:44.945 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:44.971 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:44.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:44.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:45.073 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:45.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:45.102 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:45.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:45.203 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:45.237 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:45.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:45.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:44 smithi132 ceph-mon[33500]: pgmap v1006: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 2 peering, 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 157 active+clean; 3.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 964 B/s rd, 0 B/s wr, 879 op/s; 17354/162399 objects misplaced (10.686%); 33 MiB/s, 521 objects/s recovering 2024-03-26T05:21:45.339 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:45.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:45 smithi080 ceph-mon[40404]: pgmap v1006: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 2 peering, 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 157 active+clean; 3.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 964 B/s rd, 0 B/s wr, 879 op/s; 17354/162399 objects misplaced (10.686%); 33 MiB/s, 521 objects/s recovering 2024-03-26T05:21:45.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:44 smithi080 ceph-mon[29691]: pgmap v1006: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 2 peering, 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 157 active+clean; 3.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 964 B/s rd, 0 B/s wr, 879 op/s; 17354/162399 objects misplaced (10.686%); 33 MiB/s, 521 objects/s recovering 2024-03-26T05:21:45.367 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:45.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:45.469 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:45.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:45.502 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:45.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:45.603 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:45.632 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:45.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:45.734 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:45.774 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:45.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:45.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:45.876 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:45.916 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:45.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:45.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:46.018 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:46.047 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:46.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:46.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:46.149 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:46.189 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:46.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:46.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:45 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:21:46.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:45 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:21:46.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:45 smithi132 ceph-mon[33500]: osdmap e472: 8 total, 7 up, 6 in 2024-03-26T05:21:46.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:45 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:21:46.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:45 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:21:46.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:21:46 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:21:46.292 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:46.317 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:46.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:46.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:45 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:21:46.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:45 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:21:46.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:45 smithi080 ceph-mon[40404]: osdmap e472: 8 total, 7 up, 6 in 2024-03-26T05:21:46.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:45 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:21:46.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:45 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:21:46.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:45 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:21:46.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:45 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:21:46.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:45 smithi080 ceph-mon[29691]: osdmap e472: 8 total, 7 up, 6 in 2024-03-26T05:21:46.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:45 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:21:46.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:45 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:21:46.419 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:46.447 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:46.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:46.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:46.549 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:46.581 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:46.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:46.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:46.683 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:46.712 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:46.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:46.814 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:46.841 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:46.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:46.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:46.943 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:46.970 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:46.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:46.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:47.072 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:47.105 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:47.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:47.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:47.207 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:47.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:47.232 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:47.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:47.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:47 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:47.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:47 smithi132 ceph-mon[33500]: osdmap e473: 8 total, 7 up, 6 in 2024-03-26T05:21:47.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:47 smithi132 ceph-mon[33500]: pgmap v1009: 168 pgs: 1 peering, 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 159 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 970 B/s rd, 0 B/s wr, 920 op/s; 16678/154458 objects misplaced (10.798%); 23 MiB/s, 362 objects/s recovering 2024-03-26T05:21:47.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:47 smithi132 ceph-mon[33500]: osdmap e474: 8 total, 7 up, 6 in 2024-03-26T05:21:47.333 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:47.359 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:47.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:47.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:47 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:47.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:47 smithi080 ceph-mon[29691]: osdmap e473: 8 total, 7 up, 6 in 2024-03-26T05:21:47.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:47 smithi080 ceph-mon[29691]: pgmap v1009: 168 pgs: 1 peering, 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 159 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 970 B/s rd, 0 B/s wr, 920 op/s; 16678/154458 objects misplaced (10.798%); 23 MiB/s, 362 objects/s recovering 2024-03-26T05:21:47.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:47 smithi080 ceph-mon[29691]: osdmap e474: 8 total, 7 up, 6 in 2024-03-26T05:21:47.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:47 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:47.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:47 smithi080 ceph-mon[40404]: osdmap e473: 8 total, 7 up, 6 in 2024-03-26T05:21:47.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:47 smithi080 ceph-mon[40404]: pgmap v1009: 168 pgs: 1 peering, 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 159 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 970 B/s rd, 0 B/s wr, 920 op/s; 16678/154458 objects misplaced (10.798%); 23 MiB/s, 362 objects/s recovering 2024-03-26T05:21:47.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:47 smithi080 ceph-mon[40404]: osdmap e474: 8 total, 7 up, 6 in 2024-03-26T05:21:47.461 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:47.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:47.486 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:47.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:47.519 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:47.587 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:47.612 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:47.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:47.715 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:47.747 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:47.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:47.849 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:47.886 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:47.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:47.988 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:48.025 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:48.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:48.127 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:48.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:48.153 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:48.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:48.254 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:48.257 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:21:48.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:48.279 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:48.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:48.286 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:21:48.314 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:21:48.337 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:21:48.360 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:21:48.380 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:48.385 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:21:48.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:48.409 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:48.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:48.411 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:21:48.412 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:21:48.412 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:21:48.412 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:21:48.412 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:21:48.412 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:21:48.412 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:21:48.412 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:21:48.412 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:21:48.511 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:48.534 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:48.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:48.636 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:48.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:48.667 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:48.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:48.768 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:48.798 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:48.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:48.900 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:48.927 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:48.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:49.029 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:48 smithi132 ceph-mon[33500]: osdmap e475: 8 total, 7 up, 6 in 2024-03-26T05:21:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:48 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:21:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:48 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:21:49.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:48 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2797113131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:21:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:49.055 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:49.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:49.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:48 smithi080 ceph-mon[40404]: osdmap e475: 8 total, 7 up, 6 in 2024-03-26T05:21:49.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:48 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:21:49.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:48 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:21:49.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:48 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2797113131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:21:49.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:48 smithi080 ceph-mon[29691]: osdmap e475: 8 total, 7 up, 6 in 2024-03-26T05:21:49.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:48 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:21:49.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:48 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:21:49.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:48 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2797113131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:21:49.156 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:49.196 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:49.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:49.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:49.344 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:49.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:49.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:49.446 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:49.482 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:49.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:49.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:49.535 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:21:49.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:49.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:49.633 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:49.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:49.734 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:49.788 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:49.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:49.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:49 smithi080 ceph-mon[29691]: pgmap v1012: 168 pgs: 1 peering, 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 159 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 13723/149337 objects misplaced (9.189%); 23 MiB/s, 361 objects/s recovering 2024-03-26T05:21:49.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:49 smithi080 ceph-mon[29691]: osdmap e476: 8 total, 7 up, 6 in 2024-03-26T05:21:49.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:49 smithi080 ceph-mon[40404]: pgmap v1012: 168 pgs: 1 peering, 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 159 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 13723/149337 objects misplaced (9.189%); 23 MiB/s, 361 objects/s recovering 2024-03-26T05:21:49.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:49 smithi080 ceph-mon[40404]: osdmap e476: 8 total, 7 up, 6 in 2024-03-26T05:21:49.890 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:49.919 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:49.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:50.021 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:49 smithi132 ceph-mon[33500]: pgmap v1012: 168 pgs: 1 peering, 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 159 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 13723/149337 objects misplaced (9.189%); 23 MiB/s, 361 objects/s recovering 2024-03-26T05:21:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:49 smithi132 ceph-mon[33500]: osdmap e476: 8 total, 7 up, 6 in 2024-03-26T05:21:50.050 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:50.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:50.152 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:50.196 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:50.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:50.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:50.330 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:50.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:50.432 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:50.460 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:50.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:50.562 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:50.585 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:50.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:50.686 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:50.716 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:50.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:50.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:50.818 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:50.859 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:50.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:50.962 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:50.999 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:51.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:51.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:51.101 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:51.115 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:21:50 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:21:50.975+0000 7f2fcc605700 -1 mgr get_metadata_python Requested missing service osd.4 2024-03-26T05:21:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:51.126 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:51.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:51.227 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:51.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:51.252 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:51.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:51.352 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:51.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:51.377 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:51.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:51.478 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:51.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:51.502 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:51.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:51.603 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:51.627 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:51.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:51.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:51.729 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:51.754 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:51.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:51.855 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:51.880 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:51.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:51.980 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:52.022 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:52.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:51 smithi132 ceph-mon[33500]: pgmap v1014: 168 pgs: 1 peering, 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 159 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 637 op/s; 13601/148221 objects misplaced (9.176%); 13 MiB/s, 208 objects/s recovering 2024-03-26T05:21:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:51 smithi132 ceph-mon[33500]: osdmap e477: 8 total, 7 up, 6 in 2024-03-26T05:21:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:21:52.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:21:52.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:51 smithi080 ceph-mon[29691]: pgmap v1014: 168 pgs: 1 peering, 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 159 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 637 op/s; 13601/148221 objects misplaced (9.176%); 13 MiB/s, 208 objects/s recovering 2024-03-26T05:21:52.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:51 smithi080 ceph-mon[29691]: osdmap e477: 8 total, 7 up, 6 in 2024-03-26T05:21:52.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:21:52.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:21:52.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:51 smithi080 ceph-mon[40404]: pgmap v1014: 168 pgs: 1 peering, 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 159 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 637 op/s; 13601/148221 objects misplaced (9.176%); 13 MiB/s, 208 objects/s recovering 2024-03-26T05:21:52.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:51 smithi080 ceph-mon[40404]: osdmap e477: 8 total, 7 up, 6 in 2024-03-26T05:21:52.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:21:52.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:21:52.124 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:52.150 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:52.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:52.255 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:52.291 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:52.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:52.392 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:52.427 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:52.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:52.528 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:52.553 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:52.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:52.655 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:21:52.671 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:21:52.707 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:52.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:52.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:52.809 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:52.839 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:52.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:52.940 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:52.965 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:52.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:53.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:52 smithi132 ceph-mon[33500]: osdmap e478: 8 total, 7 up, 6 in 2024-03-26T05:21:53.066 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:53.105 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:53.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:53.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:52 smithi080 ceph-mon[29691]: osdmap e478: 8 total, 7 up, 6 in 2024-03-26T05:21:53.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:52 smithi080 ceph-mon[40404]: osdmap e478: 8 total, 7 up, 6 in 2024-03-26T05:21:53.208 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:53.245 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:53.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:53.347 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:53.386 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:53.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:53.487 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:53.530 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:53.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:53.631 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:53.668 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:53.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:53.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:53.770 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:53.811 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:53.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:53.913 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:53.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:53.955 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:53.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:54.016 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:53 smithi080 ceph-mon[40404]: pgmap v1017: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 163 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 901 op/s; 9923/137178 objects misplaced (7.234%); 20 MiB/s, 324 objects/s recovering 2024-03-26T05:21:54.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:53 smithi080 ceph-mon[29691]: pgmap v1017: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 163 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 901 op/s; 9923/137178 objects misplaced (7.234%); 20 MiB/s, 324 objects/s recovering 2024-03-26T05:21:54.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:53 smithi132 ceph-mon[33500]: pgmap v1017: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 163 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 901 op/s; 9923/137178 objects misplaced (7.234%); 20 MiB/s, 324 objects/s recovering 2024-03-26T05:21:54.056 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:54.081 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:54.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:54.182 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:54.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:54.220 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:54.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:54.321 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:54.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:54.345 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:54.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:54.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:21:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:21:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:21:54.447 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:54.471 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:54.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:54.572 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:54.605 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:54.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:54.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:54.672 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:21:54.708 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:54.734 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:54.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:54.835 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:54.887 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:54.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:54.988 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:55.012 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:55.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:55.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:54 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:55.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:55.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:54 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:55.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:54 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:55.166 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:55.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:55.268 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:55.309 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:55.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:55.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:55.449 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:55.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:55.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:55.580 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:55.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:55.681 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:55.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:55.713 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:55.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:55.815 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:55.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:55.843 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:55.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:55.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:55.970 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:55.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:56.074 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:56.101 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:56.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:56.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:56.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:55 smithi080 ceph-mon[29691]: pgmap v1018: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 163 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 676 op/s; 9923/137178 objects misplaced (7.234%); 15 MiB/s, 243 objects/s recovering 2024-03-26T05:21:56.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:55 smithi080 ceph-mon[40404]: pgmap v1018: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 163 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 676 op/s; 9923/137178 objects misplaced (7.234%); 15 MiB/s, 243 objects/s recovering 2024-03-26T05:21:56.203 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:56.232 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:56.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:56.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:55 smithi132 ceph-mon[33500]: pgmap v1018: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 163 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 676 op/s; 9923/137178 objects misplaced (7.234%); 15 MiB/s, 243 objects/s recovering 2024-03-26T05:21:56.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:21:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:21:56.333 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:56.359 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:56.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:56.460 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:56.485 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:56.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:56.509 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:56.587 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:56.611 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:56.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:56.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:56.712 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:56.737 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:56.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:56.838 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:56.865 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:56.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:56.967 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:56.998 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:56.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:57.099 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:21:57.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:56 smithi080 ceph-mon[29691]: osdmap e479: 8 total, 7 up, 6 in 2024-03-26T05:21:57.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:56 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:57.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:56 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:57.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:56 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:57.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:56 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:21:57.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:56 smithi080 ceph-mon[29691]: pgmap v1020: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 163 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.07k op/s; 9771/128892 objects misplaced (7.581%); 15 MiB/s, 243 objects/s recovering 2024-03-26T05:21:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:56 smithi080 ceph-mon[40404]: osdmap e479: 8 total, 7 up, 6 in 2024-03-26T05:21:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:56 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:56 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:56 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:56 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:21:57.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:56 smithi080 ceph-mon[40404]: pgmap v1020: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 163 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.07k op/s; 9771/128892 objects misplaced (7.581%); 15 MiB/s, 243 objects/s recovering 2024-03-26T05:21:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:57.140 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:57.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:21:57.242 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:57.281 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:57.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:57.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:56 smithi132 ceph-mon[33500]: osdmap e479: 8 total, 7 up, 6 in 2024-03-26T05:21:57.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:56 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:56 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:56 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:56 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:21:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:56 smithi132 ceph-mon[33500]: pgmap v1020: 168 pgs: 2 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 163 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.07k op/s; 9771/128892 objects misplaced (7.581%); 15 MiB/s, 243 objects/s recovering 2024-03-26T05:21:57.382 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:57.411 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:57.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:57.513 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:57.553 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:57.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:57.655 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:57.683 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:57.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:57.786 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:57.813 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:57.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:57.869 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:57 smithi080 ceph-mon[40404]: osdmap e480: 8 total, 7 up, 6 in 2024-03-26T05:21:57.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:57 smithi080 ceph-mon[29691]: osdmap e480: 8 total, 7 up, 6 in 2024-03-26T05:21:57.914 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:21:57.944 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:57.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:21:58.046 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:58.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:58.076 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:58.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:58.177 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:58.206 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:58.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:58.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:58.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:57 smithi132 ceph-mon[33500]: osdmap e480: 8 total, 7 up, 6 in 2024-03-26T05:21:58.308 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:58.351 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:58.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:58.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:58.453 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:58.481 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:58.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:58.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:21:58.612 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:58.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:58.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:21:58.714 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:58.743 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:58.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:58.844 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:58.882 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:58.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:58.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:58.995 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:59.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:59.011 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:59.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:59.112 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:21:59.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:58 smithi080 ceph-mon[29691]: pgmap v1022: 168 pgs: 1 active+undersized+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 163 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 790 B/s rd, 0 B/s wr, 769 op/s; 8659/123678 objects misplaced (7.001%); 12 MiB/s, 195 objects/s recovering 2024-03-26T05:21:59.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:58 smithi080 ceph-mon[40404]: pgmap v1022: 168 pgs: 1 active+undersized+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 163 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 790 B/s rd, 0 B/s wr, 769 op/s; 8659/123678 objects misplaced (7.001%); 12 MiB/s, 195 objects/s recovering 2024-03-26T05:21:59.142 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:21:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:59.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:21:59.244 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:59.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:58 smithi132 ceph-mon[33500]: pgmap v1022: 168 pgs: 1 active+undersized+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 163 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 790 B/s rd, 0 B/s wr, 769 op/s; 8659/123678 objects misplaced (7.001%); 12 MiB/s, 195 objects/s recovering 2024-03-26T05:21:59.293 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:59.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:59.395 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:21:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:59.436 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:59.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:21:59.537 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:21:59.579 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:59.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:59.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:21:59.598 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:59.681 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:59.726 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:59.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:21:59.827 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:21:59.900 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:21:59.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:21:59.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:00.002 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:00.042 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:00.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:00.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:00.054 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:22:00.058 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:22:00.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:21:59 smithi080 ceph-mon[29691]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:22:00.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:21:59 smithi080 ceph-mon[40404]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:22:00.144 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:00.186 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:00.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:00.288 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:00.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:21:59 smithi132 ceph-mon[33500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:22:00.318 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:00.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:00.420 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:00.461 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:00.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:00.563 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:00.605 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:00.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:00.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:00.707 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:00.747 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:00.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:00.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:00.849 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:00.891 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:00.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:00.992 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:01.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:01.035 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:01.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:01.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:00 smithi080 ceph-mon[29691]: from='client.16086 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:22:01.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:00 smithi080 ceph-mon[29691]: pgmap v1023: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 163 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 620 op/s; 8154/122292 objects misplaced (6.668%); 4.2 MiB/s, 67 objects/s recovering 2024-03-26T05:22:01.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:00 smithi080 ceph-mon[40404]: from='client.16086 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:22:01.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:00 smithi080 ceph-mon[40404]: pgmap v1023: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 163 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 620 op/s; 8154/122292 objects misplaced (6.668%); 4.2 MiB/s, 67 objects/s recovering 2024-03-26T05:22:01.136 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:01.175 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:01.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:01.276 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:01.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:00 smithi132 ceph-mon[33500]: from='client.16086 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:22:01.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:00 smithi132 ceph-mon[33500]: pgmap v1023: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 163 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 620 op/s; 8154/122292 objects misplaced (6.668%); 4.2 MiB/s, 67 objects/s recovering 2024-03-26T05:22:01.316 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:01.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:01.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:01.418 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:01.455 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:01.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:01.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:01.506 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":1022,"stamp":"2024-03-26T05:21:58.254452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2687220930,"num_objects":41226,"num_object_clones":0,"num_object_copies":123678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8659,"num_objects_unfound":0,"num_objects_dirty":41226,"num_whiteouts":0,"num_read":2341,"num_read_kb":2079,"num_write":202456,"num_write_kb":7751334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44635,"num_bytes_recovered":2907506245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117097,"ondisk_log_size":117097,"up":504,"acting":503,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":513,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":12641180,"kb_used_data":8051872,"kb_used_omap":12,"kb_used_meta":4589171,"kb_avail":549829732,"statfs":{"total":575970213888,"available":563025645568,"internally_reserved":0,"allocated":8245116928,"data_stored":8238054854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12412,"internal_metadata":4699312004},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,2,2,5,3,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-326107136,"num_objects":-4976,"num_object_clones":0,"num_object_copies":-14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3748,"num_objects_unfound":0,"num_objects_dirty":-4976,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4976,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1266,"num_bytes_recovered":82968576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.475556"},"pg_stats":[{"pgid":"11.3","version":"478'14735","reported_seq":14999,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:54.837214+0000","last_change":"2024-03-26T05:21:23.803264+0000","last_active":"2024-03-26T05:21:54.837214+0000","last_peered":"2024-03-26T05:21:54.837214+0000","last_clean":"2024-03-26T05:21:54.837214+0000","last_became_active":"2024-03-26T05:21:23.798630+0000","last_became_peered":"2024-03-26T05:21:23.798630+0000","last_unstale":"2024-03-26T05:21:54.837214+0000","last_undegraded":"2024-03-26T05:21:54.837214+0000","last_fullsized":"2024-03-26T05:21:54.837214+0000","mapping_epoch":408,"log_start":"453'10138","ondisk_log_start":"453'10138","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":3086,"log_size":4597,"log_dups_size":0,"ondisk_log_size":4597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:57.548834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86999040,"num_objects":1327,"num_object_clones":0,"num_object_copies":3981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":240704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"443'30","reported_seq":757,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.781204+0000","last_change":"2024-03-26T05:20:15.461803+0000","last_active":"2024-03-26T05:21:51.781204+0000","last_peered":"2024-03-26T05:21:51.781204+0000","last_clean":"2024-03-26T05:21:51.781204+0000","last_became_active":"2024-03-26T05:20:15.369000+0000","last_became_peered":"2024-03-26T05:20:15.369000+0000","last_unstale":"2024-03-26T05:21:51.781204+0000","last_undegraded":"2024-03-26T05:21:51.781204+0000","last_fullsized":"2024-03-26T05:21:51.781204+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'20","last_scrub_stamp":"2024-03-26T05:14:56.989928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:56.989928+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":916,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780685+0000","last_change":"2024-03-26T05:19:52.350482+0000","last_active":"2024-03-26T05:21:51.780685+0000","last_peered":"2024-03-26T05:21:51.780685+0000","last_clean":"2024-03-26T05:21:51.780685+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:21:51.780685+0000","last_undegraded":"2024-03-26T05:21:51.780685+0000","last_fullsized":"2024-03-26T05:21:51.780685+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:52.350328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:19:52.350328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"443'14","reported_seq":1207,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.823314+0000","last_change":"2024-03-26T05:20:30.368570+0000","last_active":"2024-03-26T05:21:55.823314+0000","last_peered":"2024-03-26T05:21:55.823314+0000","last_clean":"2024-03-26T05:21:55.823314+0000","last_became_active":"2024-03-26T05:20:30.348391+0000","last_became_peered":"2024-03-26T05:20:30.348391+0000","last_unstale":"2024-03-26T05:21:55.823314+0000","last_undegraded":"2024-03-26T05:21:55.823314+0000","last_fullsized":"2024-03-26T05:21:55.823314+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'13","last_scrub_stamp":"2024-03-26T05:18:56.306025+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:18:56.306025+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":296,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.815114+0000","last_change":"2024-03-26T05:20:30.342597+0000","last_active":"2024-03-26T05:21:55.815114+0000","last_peered":"2024-03-26T05:21:55.815114+0000","last_clean":"2024-03-26T05:21:55.815114+0000","last_became_active":"2024-03-26T05:20:30.342267+0000","last_became_peered":"2024-03-26T05:20:30.342267+0000","last_unstale":"2024-03-26T05:21:55.815114+0000","last_undegraded":"2024-03-26T05:21:55.815114+0000","last_fullsized":"2024-03-26T05:21:55.815114+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:54.067506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:54.067506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"88'128","reported_seq":1085,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817541+0000","last_change":"2024-03-26T05:19:59.233794+0000","last_active":"2024-03-26T05:21:55.817541+0000","last_peered":"2024-03-26T05:21:55.817541+0000","last_clean":"2024-03-26T05:21:55.817541+0000","last_became_active":"2024-03-26T05:15:33.616505+0000","last_became_peered":"2024-03-26T05:15:33.616505+0000","last_unstale":"2024-03-26T05:21:55.817541+0000","last_undegraded":"2024-03-26T05:21:55.817541+0000","last_fullsized":"2024-03-26T05:21:55.817541+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:19:59.233670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:19:59.233670+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"479'11510","reported_seq":11719,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.044261+0000","last_change":"2024-03-26T05:21:23.825656+0000","last_active":"2024-03-26T05:21:56.044261+0000","last_peered":"2024-03-26T05:21:56.044261+0000","last_clean":"2024-03-26T05:21:56.044261+0000","last_became_active":"2024-03-26T05:21:23.802417+0000","last_became_peered":"2024-03-26T05:21:23.802417+0000","last_unstale":"2024-03-26T05:21:56.044261+0000","last_undegraded":"2024-03-26T05:21:56.044261+0000","last_fullsized":"2024-03-26T05:21:56.044261+0000","mapping_epoch":408,"log_start":"431'6160","ondisk_log_start":"431'6160","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":4299,"log_size":5350,"log_dups_size":0,"ondisk_log_size":5350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.482906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82378752,"num_objects":1257,"num_object_clones":0,"num_object_copies":3771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6155,"num_write_kb":237184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"443'45","reported_seq":798,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.823021+0000","last_change":"2024-03-26T05:20:39.141432+0000","last_active":"2024-03-26T05:21:55.823021+0000","last_peered":"2024-03-26T05:21:55.823021+0000","last_clean":"2024-03-26T05:21:55.823021+0000","last_became_active":"2024-03-26T05:20:15.381623+0000","last_became_peered":"2024-03-26T05:20:15.381623+0000","last_unstale":"2024-03-26T05:21:55.823021+0000","last_undegraded":"2024-03-26T05:21:55.823021+0000","last_fullsized":"2024-03-26T05:21:55.823021+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'45","last_scrub_stamp":"2024-03-26T05:20:39.141285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:39.141285+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514402+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":945,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.808766+0000","last_change":"2024-03-26T05:20:15.361409+0000","last_active":"2024-03-26T05:21:51.808766+0000","last_peered":"2024-03-26T05:21:51.808766+0000","last_clean":"2024-03-26T05:21:51.808766+0000","last_became_active":"2024-03-26T05:20:15.361072+0000","last_became_peered":"2024-03-26T05:20:15.361072+0000","last_unstale":"2024-03-26T05:21:51.808766+0000","last_undegraded":"2024-03-26T05:21:51.808766+0000","last_fullsized":"2024-03-26T05:21:51.808766+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:47.849136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:19:47.849136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:52.486291+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":897,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.813740+0000","last_change":"2024-03-26T05:20:15.373542+0000","last_active":"2024-03-26T05:21:55.813740+0000","last_peered":"2024-03-26T05:21:55.813740+0000","last_clean":"2024-03-26T05:21:55.813740+0000","last_became_active":"2024-03-26T05:20:15.373227+0000","last_became_peered":"2024-03-26T05:20:15.373227+0000","last_unstale":"2024-03-26T05:21:55.813740+0000","last_undegraded":"2024-03-26T05:21:55.813740+0000","last_fullsized":"2024-03-26T05:21:55.813740+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:21.059907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:14:21.059907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":901,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.824235+0000","last_change":"2024-03-26T05:20:38.300338+0000","last_active":"2024-03-26T05:21:55.824235+0000","last_peered":"2024-03-26T05:21:55.824235+0000","last_clean":"2024-03-26T05:21:55.824235+0000","last_became_active":"2024-03-26T05:20:15.380687+0000","last_became_peered":"2024-03-26T05:20:15.380687+0000","last_unstale":"2024-03-26T05:21:55.824235+0000","last_undegraded":"2024-03-26T05:21:55.824235+0000","last_fullsized":"2024-03-26T05:21:55.824235+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:38.300223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:20:38.300223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514481+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"88'128","reported_seq":479,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.822219+0000","last_change":"2024-03-26T05:20:34.153749+0000","last_active":"2024-03-26T05:21:55.822219+0000","last_peered":"2024-03-26T05:21:55.822219+0000","last_clean":"2024-03-26T05:21:55.822219+0000","last_became_active":"2024-03-26T05:20:32.754455+0000","last_became_peered":"2024-03-26T05:20:32.754455+0000","last_unstale":"2024-03-26T05:21:55.822219+0000","last_undegraded":"2024-03-26T05:21:55.822219+0000","last_fullsized":"2024-03-26T05:21:55.822219+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:20:34.153595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:20:34.153595+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"478'11877","reported_seq":12104,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:54.875484+0000","last_change":"2024-03-26T05:21:23.797531+0000","last_active":"2024-03-26T05:21:54.875484+0000","last_peered":"2024-03-26T05:21:54.875484+0000","last_clean":"2024-03-26T05:21:54.875484+0000","last_became_active":"2024-03-26T05:21:23.782264+0000","last_became_peered":"2024-03-26T05:21:23.782264+0000","last_unstale":"2024-03-26T05:21:54.875484+0000","last_undegraded":"2024-03-26T05:21:54.875484+0000","last_fullsized":"2024-03-26T05:21:54.875484+0000","mapping_epoch":408,"log_start":"433'7135","ondisk_log_start":"433'7135","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'5729","last_scrub_stamp":"2024-03-26T05:19:50.459009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:19:50.459009+0000","objects_scrubbed":5667,"log_size":4742,"log_dups_size":0,"ondisk_log_size":4742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:57.548793+0000","scrub_duration":1694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86081536,"num_objects":1313,"num_object_clones":0,"num_object_copies":3939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6344,"num_write_kb":245056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"443'33","reported_seq":777,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.810080+0000","last_change":"2024-03-26T05:20:39.704389+0000","last_active":"2024-03-26T05:21:51.810080+0000","last_peered":"2024-03-26T05:21:51.810080+0000","last_clean":"2024-03-26T05:21:51.810080+0000","last_became_active":"2024-03-26T05:20:15.366493+0000","last_became_peered":"2024-03-26T05:20:15.366493+0000","last_unstale":"2024-03-26T05:21:51.810080+0000","last_undegraded":"2024-03-26T05:21:51.810080+0000","last_fullsized":"2024-03-26T05:21:51.810080+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'33","last_scrub_stamp":"2024-03-26T05:20:39.704184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:39.704184+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:52.486398+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":890,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.825057+0000","last_change":"2024-03-26T05:20:30.098403+0000","last_active":"2024-03-26T05:21:55.825057+0000","last_peered":"2024-03-26T05:21:55.825057+0000","last_clean":"2024-03-26T05:21:55.825057+0000","last_became_active":"2024-03-26T05:20:15.374939+0000","last_became_peered":"2024-03-26T05:20:15.374939+0000","last_unstale":"2024-03-26T05:21:55.825057+0000","last_undegraded":"2024-03-26T05:21:55.825057+0000","last_fullsized":"2024-03-26T05:21:55.825057+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:30.098264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:20:30.098264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":573,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.810199+0000","last_change":"2024-03-26T05:20:58.499533+0000","last_active":"2024-03-26T05:21:51.810199+0000","last_peered":"2024-03-26T05:21:51.810199+0000","last_clean":"2024-03-26T05:21:51.810199+0000","last_became_active":"2024-03-26T05:20:30.335018+0000","last_became_peered":"2024-03-26T05:20:30.335018+0000","last_unstale":"2024-03-26T05:21:51.810199+0000","last_undegraded":"2024-03-26T05:21:51.810199+0000","last_fullsized":"2024-03-26T05:21:51.810199+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:58.499385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:13:07.883082+0000","last_clean_scrub_stamp":"2024-03-26T05:20:58.499385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:17.654605+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":908,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780824+0000","last_change":"2024-03-26T05:20:30.327361+0000","last_active":"2024-03-26T05:21:51.780824+0000","last_peered":"2024-03-26T05:21:51.780824+0000","last_clean":"2024-03-26T05:21:51.780824+0000","last_became_active":"2024-03-26T05:20:30.327060+0000","last_became_peered":"2024-03-26T05:20:30.327060+0000","last_unstale":"2024-03-26T05:21:51.780824+0000","last_undegraded":"2024-03-26T05:21:51.780824+0000","last_fullsized":"2024-03-26T05:21:51.780824+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:46.337028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:46.337028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"479'11880","reported_seq":12115,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.189735+0000","last_change":"2024-03-26T05:21:23.808153+0000","last_active":"2024-03-26T05:21:56.189735+0000","last_peered":"2024-03-26T05:21:56.189735+0000","last_clean":"2024-03-26T05:21:56.189735+0000","last_became_active":"2024-03-26T05:21:23.788293+0000","last_became_peered":"2024-03-26T05:21:23.788293+0000","last_unstale":"2024-03-26T05:21:56.189735+0000","last_undegraded":"2024-03-26T05:21:56.189735+0000","last_fullsized":"2024-03-26T05:21:56.189735+0000","mapping_epoch":408,"log_start":"434'7269","ondisk_log_start":"434'7269","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'6421","last_scrub_stamp":"2024-03-26T05:20:00.971796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:20:00.971796+0000","objects_scrubbed":6351,"log_size":4611,"log_dups_size":0,"ondisk_log_size":4611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83034112,"num_objects":1267,"num_object_clones":0,"num_object_copies":3801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6359,"num_write_kb":244032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"443'46","reported_seq":993,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.825540+0000","last_change":"2024-03-26T05:20:15.465505+0000","last_active":"2024-03-26T05:21:55.825540+0000","last_peered":"2024-03-26T05:21:55.825540+0000","last_clean":"2024-03-26T05:21:55.825540+0000","last_became_active":"2024-03-26T05:20:15.378070+0000","last_became_peered":"2024-03-26T05:20:15.378070+0000","last_unstale":"2024-03-26T05:21:55.825540+0000","last_undegraded":"2024-03-26T05:21:55.825540+0000","last_fullsized":"2024-03-26T05:21:55.825540+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'34","last_scrub_stamp":"2024-03-26T05:15:43.090549+0000","last_deep_scrub":"292'34","last_deep_scrub_stamp":"2024-03-26T05:12:23.778630+0000","last_clean_scrub_stamp":"2024-03-26T05:15:43.090549+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":903,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780872+0000","last_change":"2024-03-26T05:19:54.430984+0000","last_active":"2024-03-26T05:21:51.780872+0000","last_peered":"2024-03-26T05:21:51.780872+0000","last_clean":"2024-03-26T05:21:51.780872+0000","last_became_active":"2024-03-26T05:15:32.626265+0000","last_became_peered":"2024-03-26T05:15:32.626265+0000","last_unstale":"2024-03-26T05:21:51.780872+0000","last_undegraded":"2024-03-26T05:21:51.780872+0000","last_fullsized":"2024-03-26T05:21:51.780872+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:54.430734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.430734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"443'13","reported_seq":133,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819608+0000","last_change":"2024-03-26T05:20:30.358019+0000","last_active":"2024-03-26T05:21:56.819608+0000","last_peered":"2024-03-26T05:21:56.819608+0000","last_clean":"2024-03-26T05:21:56.819608+0000","last_became_active":"2024-03-26T05:20:30.350797+0000","last_became_peered":"2024-03-26T05:20:30.350797+0000","last_unstale":"2024-03-26T05:21:56.819608+0000","last_undegraded":"2024-03-26T05:21:56.819608+0000","last_fullsized":"2024-03-26T05:21:56.819608+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:14:03.822423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:03.822423+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":206,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.820421+0000","last_change":"2024-03-26T05:20:30.342843+0000","last_active":"2024-03-26T05:21:55.820421+0000","last_peered":"2024-03-26T05:21:55.820421+0000","last_clean":"2024-03-26T05:21:55.820421+0000","last_became_active":"2024-03-26T05:20:30.342410+0000","last_became_peered":"2024-03-26T05:20:30.342410+0000","last_unstale":"2024-03-26T05:21:55.820421+0000","last_undegraded":"2024-03-26T05:21:55.820421+0000","last_fullsized":"2024-03-26T05:21:55.820421+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:49.287596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_clean_scrub_stamp":"2024-03-26T05:18:49.287596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"88'128","reported_seq":475,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.821543+0000","last_change":"2024-03-26T05:20:33.139347+0000","last_active":"2024-03-26T05:21:55.821543+0000","last_peered":"2024-03-26T05:21:55.821543+0000","last_clean":"2024-03-26T05:21:55.821543+0000","last_became_active":"2024-03-26T05:20:32.753941+0000","last_became_peered":"2024-03-26T05:20:32.753941+0000","last_unstale":"2024-03-26T05:21:55.821543+0000","last_undegraded":"2024-03-26T05:21:55.821543+0000","last_fullsized":"2024-03-26T05:21:55.821543+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:20:33.139232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:20:33.139232+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:59.863137+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"479'12650","reported_seq":12875,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.912149+0000","last_change":"2024-03-26T05:19:57.196304+0000","last_active":"2024-03-26T05:21:55.912149+0000","last_peered":"2024-03-26T05:21:55.912149+0000","last_clean":"2024-03-26T05:21:55.912149+0000","last_became_active":"2024-03-26T05:18:36.691270+0000","last_became_peered":"2024-03-26T05:18:36.691270+0000","last_unstale":"2024-03-26T05:21:55.912149+0000","last_undegraded":"2024-03-26T05:21:55.912149+0000","last_fullsized":"2024-03-26T05:21:55.912149+0000","mapping_epoch":408,"log_start":"449'9148","ondisk_log_start":"449'9148","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'6148","last_scrub_stamp":"2024-03-26T05:19:57.195991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.195991+0000","objects_scrubbed":6082,"log_size":3502,"log_dups_size":0,"ondisk_log_size":3502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444301+0000","scrub_duration":1951,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165019648,"num_objects":2518,"num_object_clones":0,"num_object_copies":7554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12650,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":865,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.781036+0000","last_change":"2024-03-26T05:20:15.376611+0000","last_active":"2024-03-26T05:21:51.781036+0000","last_peered":"2024-03-26T05:21:51.781036+0000","last_clean":"2024-03-26T05:21:51.781036+0000","last_became_active":"2024-03-26T05:20:15.376307+0000","last_became_peered":"2024-03-26T05:20:15.376307+0000","last_unstale":"2024-03-26T05:21:51.781036+0000","last_undegraded":"2024-03-26T05:21:51.781036+0000","last_fullsized":"2024-03-26T05:21:51.781036+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.411139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.411139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"443'64","reported_seq":1010,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.781034+0000","last_change":"2024-03-26T05:20:05.275701+0000","last_active":"2024-03-26T05:21:51.781034+0000","last_peered":"2024-03-26T05:21:51.781034+0000","last_clean":"2024-03-26T05:21:51.781034+0000","last_became_active":"2024-03-26T05:15:32.636466+0000","last_became_peered":"2024-03-26T05:15:32.636466+0000","last_unstale":"2024-03-26T05:21:51.781034+0000","last_undegraded":"2024-03-26T05:21:51.781034+0000","last_fullsized":"2024-03-26T05:21:51.781034+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'52","last_scrub_stamp":"2024-03-26T05:20:05.274985+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:20:05.274985+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":74,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":895,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.782270+0000","last_change":"2024-03-26T05:18:36.798166+0000","last_active":"2024-03-26T05:21:51.782270+0000","last_peered":"2024-03-26T05:21:51.782270+0000","last_clean":"2024-03-26T05:21:51.782270+0000","last_became_active":"2024-03-26T05:15:32.636983+0000","last_became_peered":"2024-03-26T05:15:32.636983+0000","last_unstale":"2024-03-26T05:21:51.782270+0000","last_undegraded":"2024-03-26T05:21:51.782270+0000","last_fullsized":"2024-03-26T05:21:51.782270+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.798083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.798083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":288,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.781168+0000","last_change":"2024-03-26T05:20:15.377248+0000","last_active":"2024-03-26T05:21:51.781168+0000","last_peered":"2024-03-26T05:21:51.781168+0000","last_clean":"2024-03-26T05:21:51.781168+0000","last_became_active":"2024-03-26T05:20:15.375300+0000","last_became_peered":"2024-03-26T05:20:15.375300+0000","last_unstale":"2024-03-26T05:21:51.781168+0000","last_undegraded":"2024-03-26T05:21:51.781168+0000","last_fullsized":"2024-03-26T05:21:51.781168+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:54.923744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:13:54.923744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"88'128","reported_seq":689,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.823400+0000","last_change":"2024-03-26T05:20:32.753218+0000","last_active":"2024-03-26T05:21:55.823400+0000","last_peered":"2024-03-26T05:21:55.823400+0000","last_clean":"2024-03-26T05:21:55.823400+0000","last_became_active":"2024-03-26T05:20:32.752804+0000","last_became_peered":"2024-03-26T05:20:32.752804+0000","last_unstale":"2024-03-26T05:21:55.823400+0000","last_undegraded":"2024-03-26T05:21:55.823400+0000","last_fullsized":"2024-03-26T05:21:55.823400+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:20:28.011935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:20:28.011935+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514456+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"479'12608","reported_seq":24695,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.139300+0000","last_change":"2024-03-26T05:21:23.036131+0000","last_active":"2024-03-26T05:21:56.139300+0000","last_peered":"2024-03-26T05:21:56.139300+0000","last_clean":"2024-03-26T05:21:56.139300+0000","last_became_active":"2024-03-26T05:21:22.979195+0000","last_became_peered":"2024-03-26T05:21:22.979195+0000","last_unstale":"2024-03-26T05:21:56.139300+0000","last_undegraded":"2024-03-26T05:21:56.139300+0000","last_fullsized":"2024-03-26T05:21:56.139300+0000","mapping_epoch":459,"log_start":"449'9053","ondisk_log_start":"449'9053","created":408,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3555,"log_dups_size":0,"ondisk_log_size":3555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169213952,"num_objects":2582,"num_object_clones":0,"num_object_copies":7746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12608,"num_write_kb":486080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5959,"num_bytes_recovered":390529024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"439'10","reported_seq":1152,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.782491+0000","last_change":"2024-03-26T05:20:15.470722+0000","last_active":"2024-03-26T05:21:51.782491+0000","last_peered":"2024-03-26T05:21:51.782491+0000","last_clean":"2024-03-26T05:21:51.782491+0000","last_became_active":"2024-03-26T05:20:15.376570+0000","last_became_peered":"2024-03-26T05:20:15.376570+0000","last_unstale":"2024-03-26T05:21:51.782491+0000","last_undegraded":"2024-03-26T05:21:51.782491+0000","last_fullsized":"2024-03-26T05:21:51.782491+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:13:56.721250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:13:56.721250+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.625502+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"443'48","reported_seq":644,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.808980+0000","last_change":"2024-03-26T05:20:55.489656+0000","last_active":"2024-03-26T05:21:51.808980+0000","last_peered":"2024-03-26T05:21:51.808980+0000","last_clean":"2024-03-26T05:21:51.808980+0000","last_became_active":"2024-03-26T05:20:30.343622+0000","last_became_peered":"2024-03-26T05:20:30.343622+0000","last_unstale":"2024-03-26T05:21:51.808980+0000","last_undegraded":"2024-03-26T05:21:51.808980+0000","last_fullsized":"2024-03-26T05:21:51.808980+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'48","last_scrub_stamp":"2024-03-26T05:20:55.489555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:55.489555+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:13.261547+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":63,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":921,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819399+0000","last_change":"2024-03-26T05:20:15.464557+0000","last_active":"2024-03-26T05:21:56.819399+0000","last_peered":"2024-03-26T05:21:56.819399+0000","last_clean":"2024-03-26T05:21:56.819399+0000","last_became_active":"2024-03-26T05:20:15.379302+0000","last_became_peered":"2024-03-26T05:20:15.379302+0000","last_unstale":"2024-03-26T05:21:56.819399+0000","last_undegraded":"2024-03-26T05:21:56.819399+0000","last_fullsized":"2024-03-26T05:21:56.819399+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:12.113993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:12.113993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657804+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":812,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817767+0000","last_change":"2024-03-26T05:19:49.427510+0000","last_active":"2024-03-26T05:21:55.817767+0000","last_peered":"2024-03-26T05:21:55.817767+0000","last_clean":"2024-03-26T05:21:55.817767+0000","last_became_active":"2024-03-26T05:15:32.630135+0000","last_became_peered":"2024-03-26T05:15:32.630135+0000","last_unstale":"2024-03-26T05:21:55.817767+0000","last_undegraded":"2024-03-26T05:21:55.817767+0000","last_fullsized":"2024-03-26T05:21:55.817767+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:49.427405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:01.721323+0000","last_clean_scrub_stamp":"2024-03-26T05:19:49.427405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444249+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"396'129","reported_seq":727,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780633+0000","last_change":"2024-03-26T05:20:32.747492+0000","last_active":"2024-03-26T05:21:51.780633+0000","last_peered":"2024-03-26T05:21:51.780633+0000","last_clean":"2024-03-26T05:21:51.780633+0000","last_became_active":"2024-03-26T05:20:32.747103+0000","last_became_peered":"2024-03-26T05:20:32.747103+0000","last_unstale":"2024-03-26T05:21:51.780633+0000","last_undegraded":"2024-03-26T05:21:51.780633+0000","last_fullsized":"2024-03-26T05:21:51.780633+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"396'129","last_scrub_stamp":"2024-03-26T05:17:59.038611+0000","last_deep_scrub":"396'129","last_deep_scrub_stamp":"2024-03-26T05:17:59.038611+0000","last_clean_scrub_stamp":"2024-03-26T05:17:59.038611+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"479'12568","reported_seq":22511,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.060246+0000","last_change":"2024-03-26T05:21:25.934696+0000","last_active":"2024-03-26T05:21:56.060246+0000","last_peered":"2024-03-26T05:21:56.060246+0000","last_clean":"2024-03-26T05:21:56.060246+0000","last_became_active":"2024-03-26T05:21:25.915527+0000","last_became_peered":"2024-03-26T05:21:25.915527+0000","last_unstale":"2024-03-26T05:21:56.060246+0000","last_undegraded":"2024-03-26T05:21:56.060246+0000","last_fullsized":"2024-03-26T05:21:56.060246+0000","mapping_epoch":462,"log_start":"447'8423","ondisk_log_start":"447'8423","created":408,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4145,"log_dups_size":0,"ondisk_log_size":4145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172163096,"num_objects":2628,"num_object_clones":0,"num_object_copies":7884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12568,"num_write_kb":486209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4886,"num_bytes_recovered":320143384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"443'12","reported_seq":1208,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819912+0000","last_change":"2024-03-26T05:20:30.357087+0000","last_active":"2024-03-26T05:21:56.819912+0000","last_peered":"2024-03-26T05:21:56.819912+0000","last_clean":"2024-03-26T05:21:56.819912+0000","last_became_active":"2024-03-26T05:20:30.350209+0000","last_became_peered":"2024-03-26T05:20:30.350209+0000","last_unstale":"2024-03-26T05:21:56.819912+0000","last_undegraded":"2024-03-26T05:21:56.819912+0000","last_fullsized":"2024-03-26T05:21:56.819912+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:18:36.306612+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.306612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657938+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"443'36","reported_seq":112,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.814095+0000","last_change":"2024-03-26T05:20:30.389164+0000","last_active":"2024-03-26T05:21:55.814095+0000","last_peered":"2024-03-26T05:21:55.814095+0000","last_clean":"2024-03-26T05:21:55.814095+0000","last_became_active":"2024-03-26T05:20:30.345634+0000","last_became_peered":"2024-03-26T05:20:30.345634+0000","last_unstale":"2024-03-26T05:21:55.814095+0000","last_undegraded":"2024-03-26T05:21:55.814095+0000","last_fullsized":"2024-03-26T05:21:55.814095+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:14:05.219767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:05.219767+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":197,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.781624+0000","last_change":"2024-03-26T05:20:30.342087+0000","last_active":"2024-03-26T05:21:51.781624+0000","last_peered":"2024-03-26T05:21:51.781624+0000","last_clean":"2024-03-26T05:21:51.781624+0000","last_became_active":"2024-03-26T05:20:30.341758+0000","last_became_peered":"2024-03-26T05:20:30.341758+0000","last_unstale":"2024-03-26T05:21:51.781624+0000","last_undegraded":"2024-03-26T05:21:51.781624+0000","last_fullsized":"2024-03-26T05:21:51.781624+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:35.144207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:35.144207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":904,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.818984+0000","last_change":"2024-03-26T05:20:04.436906+0000","last_active":"2024-03-26T05:21:56.818984+0000","last_peered":"2024-03-26T05:21:56.818984+0000","last_clean":"2024-03-26T05:21:56.818984+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:21:56.818984+0000","last_undegraded":"2024-03-26T05:21:56.818984+0000","last_fullsized":"2024-03-26T05:21:56.818984+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:04.436614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:20:04.436614+0000","last_clean_scrub_stamp":"2024-03-26T05:20:04.436614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.658090+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"88'128","reported_seq":653,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.809577+0000","last_change":"2024-03-26T05:19:43.824458+0000","last_active":"2024-03-26T05:21:51.809577+0000","last_peered":"2024-03-26T05:21:51.809577+0000","last_clean":"2024-03-26T05:21:51.809577+0000","last_became_active":"2024-03-26T05:15:33.617538+0000","last_became_peered":"2024-03-26T05:15:33.617538+0000","last_unstale":"2024-03-26T05:21:51.809577+0000","last_undegraded":"2024-03-26T05:21:51.809577+0000","last_fullsized":"2024-03-26T05:21:51.809577+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:19:43.824375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.824375+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:52.486309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"479'12853","reported_seq":26452,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.218201+0000","last_change":"2024-03-26T05:21:04.398592+0000","last_active":"2024-03-26T05:21:56.218201+0000","last_peered":"2024-03-26T05:21:56.218201+0000","last_clean":"2024-03-26T05:21:56.218201+0000","last_became_active":"2024-03-26T05:21:04.373554+0000","last_became_peered":"2024-03-26T05:21:04.373554+0000","last_unstale":"2024-03-26T05:21:56.218201+0000","last_undegraded":"2024-03-26T05:21:56.218201+0000","last_fullsized":"2024-03-26T05:21:56.218201+0000","mapping_epoch":452,"log_start":"450'9305","ondisk_log_start":"450'9305","created":408,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167182336,"num_objects":2551,"num_object_clones":0,"num_object_copies":7653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12853,"num_write_kb":492928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6714,"num_bytes_recovered":440008704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":952,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.809993+0000","last_change":"2024-03-26T05:21:09.515461+0000","last_active":"2024-03-26T05:21:51.809993+0000","last_peered":"2024-03-26T05:21:51.809993+0000","last_clean":"2024-03-26T05:21:51.809993+0000","last_became_active":"2024-03-26T05:20:30.343345+0000","last_became_peered":"2024-03-26T05:20:30.343345+0000","last_unstale":"2024-03-26T05:21:51.809993+0000","last_undegraded":"2024-03-26T05:21:51.809993+0000","last_fullsized":"2024-03-26T05:21:51.809993+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:09.515351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:21:09.515351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:36.619500+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"443'54","reported_seq":1078,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819559+0000","last_change":"2024-03-26T05:20:30.369614+0000","last_active":"2024-03-26T05:21:56.819559+0000","last_peered":"2024-03-26T05:21:56.819559+0000","last_clean":"2024-03-26T05:21:56.819559+0000","last_became_active":"2024-03-26T05:20:30.347061+0000","last_became_peered":"2024-03-26T05:20:30.347061+0000","last_unstale":"2024-03-26T05:21:56.819559+0000","last_undegraded":"2024-03-26T05:21:56.819559+0000","last_fullsized":"2024-03-26T05:21:56.819559+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'44","last_scrub_stamp":"2024-03-26T05:18:43.255284+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:18:43.255284+0000","objects_scrubbed":9,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":939,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.813493+0000","last_change":"2024-03-26T05:20:30.342347+0000","last_active":"2024-03-26T05:21:55.813493+0000","last_peered":"2024-03-26T05:21:55.813493+0000","last_clean":"2024-03-26T05:21:55.813493+0000","last_became_active":"2024-03-26T05:20:30.341954+0000","last_became_peered":"2024-03-26T05:20:30.341954+0000","last_unstale":"2024-03-26T05:21:55.813493+0000","last_undegraded":"2024-03-26T05:21:55.813493+0000","last_fullsized":"2024-03-26T05:21:55.813493+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:48.584746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:11:29.514915+0000","last_clean_scrub_stamp":"2024-03-26T05:19:48.584746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":901,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.812526+0000","last_change":"2024-03-26T05:19:44.571275+0000","last_active":"2024-03-26T05:21:55.812526+0000","last_peered":"2024-03-26T05:21:55.812526+0000","last_clean":"2024-03-26T05:21:55.812526+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:21:55.812526+0000","last_undegraded":"2024-03-26T05:21:55.812526+0000","last_fullsized":"2024-03-26T05:21:55.812526+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:44.571153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:19:44.571153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"88'128","reported_seq":655,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.809336+0000","last_change":"2024-03-26T05:19:56.753537+0000","last_active":"2024-03-26T05:21:51.809336+0000","last_peered":"2024-03-26T05:21:51.809336+0000","last_clean":"2024-03-26T05:21:51.809336+0000","last_became_active":"2024-03-26T05:15:22.624234+0000","last_became_peered":"2024-03-26T05:15:22.624234+0000","last_unstale":"2024-03-26T05:21:51.809336+0000","last_undegraded":"2024-03-26T05:21:51.809336+0000","last_fullsized":"2024-03-26T05:21:51.809336+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:19:56.753458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:19:56.753458+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:52.486381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"478'11557","reported_seq":11718,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.265609+0000","last_change":"2024-03-26T05:21:23.808141+0000","last_active":"2024-03-26T05:21:55.265609+0000","last_peered":"2024-03-26T05:21:55.265609+0000","last_clean":"2024-03-26T05:21:55.265609+0000","last_became_active":"2024-03-26T05:21:23.782748+0000","last_became_peered":"2024-03-26T05:21:23.782748+0000","last_unstale":"2024-03-26T05:21:55.265609+0000","last_undegraded":"2024-03-26T05:21:55.265609+0000","last_fullsized":"2024-03-26T05:21:55.265609+0000","mapping_epoch":408,"log_start":"438'7197","ondisk_log_start":"438'7197","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4360,"log_dups_size":0,"ondisk_log_size":4360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:57.643984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84672512,"num_objects":1292,"num_object_clones":0,"num_object_copies":3876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":239264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":206,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.826041+0000","last_change":"2024-03-26T05:20:30.339317+0000","last_active":"2024-03-26T05:21:55.826041+0000","last_peered":"2024-03-26T05:21:55.826041+0000","last_clean":"2024-03-26T05:21:55.826041+0000","last_became_active":"2024-03-26T05:20:30.339002+0000","last_became_peered":"2024-03-26T05:20:30.339002+0000","last_unstale":"2024-03-26T05:21:55.826041+0000","last_undegraded":"2024-03-26T05:21:55.826041+0000","last_fullsized":"2024-03-26T05:21:55.826041+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:57.304687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.304687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"443'42","reported_seq":1011,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819241+0000","last_change":"2024-03-26T05:20:05.431539+0000","last_active":"2024-03-26T05:21:56.819241+0000","last_peered":"2024-03-26T05:21:56.819241+0000","last_clean":"2024-03-26T05:21:56.819241+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:21:56.819241+0000","last_undegraded":"2024-03-26T05:21:56.819241+0000","last_fullsized":"2024-03-26T05:21:56.819241+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'30","last_scrub_stamp":"2024-03-26T05:20:05.430706+0000","last_deep_scrub":"254'30","last_deep_scrub_stamp":"2024-03-26T05:20:05.430706+0000","last_clean_scrub_stamp":"2024-03-26T05:20:05.430706+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.658113+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":1906,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817158+0000","last_change":"2024-03-26T05:19:47.348573+0000","last_active":"2024-03-26T05:21:55.817158+0000","last_peered":"2024-03-26T05:21:55.817158+0000","last_clean":"2024-03-26T05:21:55.817158+0000","last_became_active":"2024-03-26T05:15:32.633061+0000","last_became_peered":"2024-03-26T05:15:32.633061+0000","last_unstale":"2024-03-26T05:21:55.817158+0000","last_undegraded":"2024-03-26T05:21:55.817158+0000","last_fullsized":"2024-03-26T05:21:55.817158+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:19:47.348445+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:19:47.348445+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444283+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":448,"num_read_kb":443,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":629,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.820027+0000","last_change":"2024-03-26T05:20:30.349671+0000","last_active":"2024-03-26T05:21:56.820027+0000","last_peered":"2024-03-26T05:21:56.820027+0000","last_clean":"2024-03-26T05:21:56.820027+0000","last_became_active":"2024-03-26T05:20:30.349459+0000","last_became_peered":"2024-03-26T05:20:30.349459+0000","last_unstale":"2024-03-26T05:21:56.820027+0000","last_undegraded":"2024-03-26T05:21:56.820027+0000","last_fullsized":"2024-03-26T05:21:56.820027+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:16.863389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.863389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":927,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.813099+0000","last_change":"2024-03-26T05:20:30.332076+0000","last_active":"2024-03-26T05:21:55.813099+0000","last_peered":"2024-03-26T05:21:55.813099+0000","last_clean":"2024-03-26T05:21:55.813099+0000","last_became_active":"2024-03-26T05:20:30.331524+0000","last_became_peered":"2024-03-26T05:20:30.331524+0000","last_unstale":"2024-03-26T05:21:55.813099+0000","last_undegraded":"2024-03-26T05:21:55.813099+0000","last_fullsized":"2024-03-26T05:21:55.813099+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:52.594486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:52.594486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"88'128","reported_seq":723,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.811728+0000","last_change":"2024-03-26T05:20:32.758080+0000","last_active":"2024-03-26T05:21:55.811728+0000","last_peered":"2024-03-26T05:21:55.811728+0000","last_clean":"2024-03-26T05:21:55.811728+0000","last_became_active":"2024-03-26T05:20:32.754918+0000","last_became_peered":"2024-03-26T05:20:32.754918+0000","last_unstale":"2024-03-26T05:21:55.811728+0000","last_undegraded":"2024-03-26T05:21:55.811728+0000","last_fullsized":"2024-03-26T05:21:55.811728+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:15:44.054471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:15:44.054471+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"478'12644","reported_seq":23432,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.288253+0000","last_change":"2024-03-26T05:21:24.809434+0000","last_active":"2024-03-26T05:21:55.288253+0000","last_peered":"2024-03-26T05:21:55.288253+0000","last_clean":"2024-03-26T05:21:55.288253+0000","last_became_active":"2024-03-26T05:21:24.792538+0000","last_became_peered":"2024-03-26T05:21:24.792538+0000","last_unstale":"2024-03-26T05:21:55.288253+0000","last_undegraded":"2024-03-26T05:21:55.288253+0000","last_fullsized":"2024-03-26T05:21:55.288253+0000","mapping_epoch":461,"log_start":"449'9193","ondisk_log_start":"449'9193","created":408,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3451,"log_dups_size":0,"ondisk_log_size":3451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.625931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171835392,"num_objects":2622,"num_object_clones":0,"num_object_copies":7866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12644,"num_write_kb":488512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5310,"num_bytes_recovered":347996160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":839,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.818464+0000","last_change":"2024-03-26T05:20:30.342594+0000","last_active":"2024-03-26T05:21:55.818464+0000","last_peered":"2024-03-26T05:21:55.818464+0000","last_clean":"2024-03-26T05:21:55.818464+0000","last_became_active":"2024-03-26T05:20:30.342259+0000","last_became_peered":"2024-03-26T05:20:30.342259+0000","last_unstale":"2024-03-26T05:21:55.818464+0000","last_undegraded":"2024-03-26T05:21:55.818464+0000","last_fullsized":"2024-03-26T05:21:55.818464+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:43.408113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:14.948209+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.408113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444335+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"443'52","reported_seq":1270,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.809735+0000","last_change":"2024-03-26T05:21:03.528679+0000","last_active":"2024-03-26T05:21:51.809735+0000","last_peered":"2024-03-26T05:21:51.809735+0000","last_clean":"2024-03-26T05:21:51.809735+0000","last_became_active":"2024-03-26T05:20:15.372700+0000","last_became_peered":"2024-03-26T05:20:15.372700+0000","last_unstale":"2024-03-26T05:21:51.809735+0000","last_undegraded":"2024-03-26T05:21:51.809735+0000","last_fullsized":"2024-03-26T05:21:51.809735+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'52","last_scrub_stamp":"2024-03-26T05:21:03.528517+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:21:03.528517+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:15.476604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":921,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780736+0000","last_change":"2024-03-26T05:20:30.350444+0000","last_active":"2024-03-26T05:21:51.780736+0000","last_peered":"2024-03-26T05:21:51.780736+0000","last_clean":"2024-03-26T05:21:51.780736+0000","last_became_active":"2024-03-26T05:20:30.349656+0000","last_became_peered":"2024-03-26T05:20:30.349656+0000","last_unstale":"2024-03-26T05:21:51.780736+0000","last_undegraded":"2024-03-26T05:21:51.780736+0000","last_fullsized":"2024-03-26T05:21:51.780736+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:46.747932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:46.747932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":1080,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817890+0000","last_change":"2024-03-26T05:18:58.577783+0000","last_active":"2024-03-26T05:21:55.817890+0000","last_peered":"2024-03-26T05:21:55.817890+0000","last_clean":"2024-03-26T05:21:55.817890+0000","last_became_active":"2024-03-26T05:15:32.632929+0000","last_became_peered":"2024-03-26T05:15:32.632929+0000","last_unstale":"2024-03-26T05:21:55.817890+0000","last_undegraded":"2024-03-26T05:21:55.817890+0000","last_fullsized":"2024-03-26T05:21:55.817890+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:18:58.577676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:18:58.577676+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":573,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.782398+0000","last_change":"2024-03-26T05:20:30.347796+0000","last_active":"2024-03-26T05:21:51.782398+0000","last_peered":"2024-03-26T05:21:51.782398+0000","last_clean":"2024-03-26T05:21:51.782398+0000","last_became_active":"2024-03-26T05:20:30.346881+0000","last_became_peered":"2024-03-26T05:20:30.346881+0000","last_unstale":"2024-03-26T05:21:51.782398+0000","last_undegraded":"2024-03-26T05:21:51.782398+0000","last_fullsized":"2024-03-26T05:21:51.782398+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:14.528992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:14.528992+0000","last_clean_scrub_stamp":"2024-03-26T05:15:14.528992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"478'11839","reported_seq":17003,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.275538+0000","last_change":"2024-03-26T05:21:25.908260+0000","last_active":"2024-03-26T05:21:55.275538+0000","last_peered":"2024-03-26T05:21:55.275538+0000","last_clean":"2024-03-26T05:21:55.275538+0000","last_became_active":"2024-03-26T05:21:25.895183+0000","last_became_peered":"2024-03-26T05:21:25.895183+0000","last_unstale":"2024-03-26T05:21:55.275538+0000","last_undegraded":"2024-03-26T05:21:55.275538+0000","last_fullsized":"2024-03-26T05:21:55.275538+0000","mapping_epoch":462,"log_start":"439'7535","ondisk_log_start":"439'7535","created":408,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:59.642712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86343680,"num_objects":1317,"num_object_clones":0,"num_object_copies":3951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":244448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":81657856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":931,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.822764+0000","last_change":"2024-03-26T05:20:30.336844+0000","last_active":"2024-03-26T05:21:55.822764+0000","last_peered":"2024-03-26T05:21:55.822764+0000","last_clean":"2024-03-26T05:21:55.822764+0000","last_became_active":"2024-03-26T05:20:30.335654+0000","last_became_peered":"2024-03-26T05:20:30.335654+0000","last_unstale":"2024-03-26T05:21:55.822764+0000","last_undegraded":"2024-03-26T05:21:55.822764+0000","last_fullsized":"2024-03-26T05:21:55.822764+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:54.964851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.964851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514277+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"443'27","reported_seq":753,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.824483+0000","last_change":"2024-03-26T05:20:30.365782+0000","last_active":"2024-03-26T05:21:55.824483+0000","last_peered":"2024-03-26T05:21:55.824483+0000","last_clean":"2024-03-26T05:21:55.824483+0000","last_became_active":"2024-03-26T05:20:30.345752+0000","last_became_peered":"2024-03-26T05:20:30.345752+0000","last_unstale":"2024-03-26T05:21:55.824483+0000","last_undegraded":"2024-03-26T05:21:55.824483+0000","last_fullsized":"2024-03-26T05:21:55.824483+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'23","last_scrub_stamp":"2024-03-26T05:19:53.957953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:19:53.957953+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514247+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":930,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.825115+0000","last_change":"2024-03-26T05:19:50.912733+0000","last_active":"2024-03-26T05:21:55.825115+0000","last_peered":"2024-03-26T05:21:55.825115+0000","last_clean":"2024-03-26T05:21:55.825115+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:21:55.825115+0000","last_undegraded":"2024-03-26T05:21:55.825115+0000","last_fullsized":"2024-03-26T05:21:55.825115+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:50.912594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:03.193105+0000","last_clean_scrub_stamp":"2024-03-26T05:19:50.912594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514210+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":913,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.812580+0000","last_change":"2024-03-26T05:19:46.522718+0000","last_active":"2024-03-26T05:21:55.812580+0000","last_peered":"2024-03-26T05:21:55.812580+0000","last_clean":"2024-03-26T05:21:55.812580+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:21:55.812580+0000","last_undegraded":"2024-03-26T05:21:55.812580+0000","last_fullsized":"2024-03-26T05:21:55.812580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:46.522602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:19:46.522602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":773,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817982+0000","last_change":"2024-03-26T05:18:41.624630+0000","last_active":"2024-03-26T05:21:55.817982+0000","last_peered":"2024-03-26T05:21:55.817982+0000","last_clean":"2024-03-26T05:21:55.817982+0000","last_became_active":"2024-03-26T05:15:32.633633+0000","last_became_peered":"2024-03-26T05:15:32.633633+0000","last_unstale":"2024-03-26T05:21:55.817982+0000","last_undegraded":"2024-03-26T05:21:55.817982+0000","last_fullsized":"2024-03-26T05:21:55.817982+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:41.624502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:18:41.624502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444139+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"88'128","reported_seq":1074,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817526+0000","last_change":"2024-03-26T05:20:17.750124+0000","last_active":"2024-03-26T05:21:55.817526+0000","last_peered":"2024-03-26T05:21:55.817526+0000","last_clean":"2024-03-26T05:21:55.817526+0000","last_became_active":"2024-03-26T05:20:17.749765+0000","last_became_peered":"2024-03-26T05:20:17.749765+0000","last_unstale":"2024-03-26T05:21:55.817526+0000","last_undegraded":"2024-03-26T05:21:55.817526+0000","last_fullsized":"2024-03-26T05:21:55.817526+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:12:21.868614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:12:21.868614+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:59.461148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"478'12433","reported_seq":12588,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.254456+0000","last_change":"2024-03-26T05:18:36.696360+0000","last_active":"2024-03-26T05:21:55.254456+0000","last_peered":"2024-03-26T05:21:55.254456+0000","last_clean":"2024-03-26T05:21:55.254456+0000","last_became_active":"2024-03-26T05:18:36.695687+0000","last_became_peered":"2024-03-26T05:18:36.695687+0000","last_unstale":"2024-03-26T05:21:55.254456+0000","last_undegraded":"2024-03-26T05:21:55.254456+0000","last_fullsized":"2024-03-26T05:21:55.254456+0000","mapping_epoch":408,"log_start":"449'8988","ondisk_log_start":"449'8988","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3445,"log_dups_size":0,"ondisk_log_size":3445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.625811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168493056,"num_objects":2571,"num_object_clones":0,"num_object_copies":7713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12433,"num_write_kb":480128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":901,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.818841+0000","last_change":"2024-03-26T05:20:15.377798+0000","last_active":"2024-03-26T05:21:56.818841+0000","last_peered":"2024-03-26T05:21:56.818841+0000","last_clean":"2024-03-26T05:21:56.818841+0000","last_became_active":"2024-03-26T05:20:15.377492+0000","last_became_peered":"2024-03-26T05:20:15.377492+0000","last_unstale":"2024-03-26T05:21:56.818841+0000","last_undegraded":"2024-03-26T05:21:56.818841+0000","last_fullsized":"2024-03-26T05:21:56.818841+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:14.131427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:14.131427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657856+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"443'58","reported_seq":1275,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.813582+0000","last_change":"2024-03-26T05:20:00.666872+0000","last_active":"2024-03-26T05:21:55.813582+0000","last_peered":"2024-03-26T05:21:55.813582+0000","last_clean":"2024-03-26T05:21:55.813582+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:21:55.813582+0000","last_undegraded":"2024-03-26T05:21:55.813582+0000","last_fullsized":"2024-03-26T05:21:55.813582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'46","last_scrub_stamp":"2024-03-26T05:20:00.666615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:00.666615+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":290,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819376+0000","last_change":"2024-03-26T05:20:15.457605+0000","last_active":"2024-03-26T05:21:56.819376+0000","last_peered":"2024-03-26T05:21:56.819376+0000","last_clean":"2024-03-26T05:21:56.819376+0000","last_became_active":"2024-03-26T05:20:15.457163+0000","last_became_peered":"2024-03-26T05:20:15.457163+0000","last_unstale":"2024-03-26T05:21:56.819376+0000","last_undegraded":"2024-03-26T05:21:56.819376+0000","last_fullsized":"2024-03-26T05:21:56.819376+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:02.881800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:02.881800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":240,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.813859+0000","last_change":"2024-03-26T05:20:30.355675+0000","last_active":"2024-03-26T05:21:55.813859+0000","last_peered":"2024-03-26T05:21:55.813859+0000","last_clean":"2024-03-26T05:21:55.813859+0000","last_became_active":"2024-03-26T05:20:30.339519+0000","last_became_peered":"2024-03-26T05:20:30.339519+0000","last_unstale":"2024-03-26T05:21:55.813859+0000","last_undegraded":"2024-03-26T05:21:55.813859+0000","last_fullsized":"2024-03-26T05:21:55.813859+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:18:41.097424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:18:41.097424+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":876,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.782529+0000","last_change":"2024-03-26T05:20:15.360008+0000","last_active":"2024-03-26T05:21:51.782529+0000","last_peered":"2024-03-26T05:21:51.782529+0000","last_clean":"2024-03-26T05:21:51.782529+0000","last_became_active":"2024-03-26T05:20:15.359520+0000","last_became_peered":"2024-03-26T05:20:15.359520+0000","last_unstale":"2024-03-26T05:21:51.782529+0000","last_undegraded":"2024-03-26T05:21:51.782529+0000","last_fullsized":"2024-03-26T05:21:51.782529+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:31.518660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:31.518660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"88'128","reported_seq":475,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819877+0000","last_change":"2024-03-26T05:20:32.759374+0000","last_active":"2024-03-26T05:21:56.819877+0000","last_peered":"2024-03-26T05:21:56.819877+0000","last_clean":"2024-03-26T05:21:56.819877+0000","last_became_active":"2024-03-26T05:20:32.758743+0000","last_became_peered":"2024-03-26T05:20:32.758743+0000","last_unstale":"2024-03-26T05:21:56.819877+0000","last_undegraded":"2024-03-26T05:21:56.819877+0000","last_fullsized":"2024-03-26T05:21:56.819877+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:19:45.389633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:19:45.389633+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:57.616948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"479'11909","reported_seq":12148,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.904166+0000","last_change":"2024-03-26T05:21:23.807081+0000","last_active":"2024-03-26T05:21:55.904166+0000","last_peered":"2024-03-26T05:21:55.904166+0000","last_clean":"2024-03-26T05:21:55.904166+0000","last_became_active":"2024-03-26T05:21:23.792443+0000","last_became_peered":"2024-03-26T05:21:23.792443+0000","last_unstale":"2024-03-26T05:21:55.904166+0000","last_undegraded":"2024-03-26T05:21:55.904166+0000","last_fullsized":"2024-03-26T05:21:55.904166+0000","mapping_epoch":408,"log_start":"434'7321","ondisk_log_start":"434'7321","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'6714","last_scrub_stamp":"2024-03-26T05:20:05.667571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:20:05.667571+0000","objects_scrubbed":6680,"log_size":4588,"log_dups_size":0,"ondisk_log_size":4588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:57.438063+0000","scrub_duration":1569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85590016,"num_objects":1306,"num_object_clones":0,"num_object_copies":3918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":246240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"443'27","reported_seq":986,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819796+0000","last_change":"2024-03-26T05:20:16.386890+0000","last_active":"2024-03-26T05:21:56.819796+0000","last_peered":"2024-03-26T05:21:56.819796+0000","last_clean":"2024-03-26T05:21:56.819796+0000","last_became_active":"2024-03-26T05:20:16.378745+0000","last_became_peered":"2024-03-26T05:20:16.378745+0000","last_unstale":"2024-03-26T05:21:56.819796+0000","last_undegraded":"2024-03-26T05:21:56.819796+0000","last_fullsized":"2024-03-26T05:21:56.819796+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:18:31.264666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:18:31.264666+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":197,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.781073+0000","last_change":"2024-03-26T05:20:30.327239+0000","last_active":"2024-03-26T05:21:51.781073+0000","last_peered":"2024-03-26T05:21:51.781073+0000","last_clean":"2024-03-26T05:21:51.781073+0000","last_became_active":"2024-03-26T05:20:30.326917+0000","last_became_peered":"2024-03-26T05:20:30.326917+0000","last_unstale":"2024-03-26T05:21:51.781073+0000","last_undegraded":"2024-03-26T05:21:51.781073+0000","last_fullsized":"2024-03-26T05:21:51.781073+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:58.239217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:19:58.239217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":905,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.821697+0000","last_change":"2024-03-26T05:20:15.381990+0000","last_active":"2024-03-26T05:21:55.821697+0000","last_peered":"2024-03-26T05:21:55.821697+0000","last_clean":"2024-03-26T05:21:55.821697+0000","last_became_active":"2024-03-26T05:20:15.380969+0000","last_became_peered":"2024-03-26T05:20:15.380969+0000","last_unstale":"2024-03-26T05:21:55.821697+0000","last_undegraded":"2024-03-26T05:21:55.821697+0000","last_fullsized":"2024-03-26T05:21:55.821697+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:52.371799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:52.371799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.509183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":896,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.820219+0000","last_change":"2024-03-26T05:19:50.540208+0000","last_active":"2024-03-26T05:21:56.820219+0000","last_peered":"2024-03-26T05:21:56.820219+0000","last_clean":"2024-03-26T05:21:56.820219+0000","last_became_active":"2024-03-26T05:15:32.639943+0000","last_became_peered":"2024-03-26T05:15:32.639943+0000","last_unstale":"2024-03-26T05:21:56.820219+0000","last_undegraded":"2024-03-26T05:21:56.820219+0000","last_fullsized":"2024-03-26T05:21:56.820219+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:50.540100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:50.540100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.658032+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"479'11708","reported_seq":19935,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.908707+0000","last_change":"2024-03-26T05:21:25.896869+0000","last_active":"2024-03-26T05:21:55.908707+0000","last_peered":"2024-03-26T05:21:55.908707+0000","last_clean":"2024-03-26T05:21:55.908707+0000","last_became_active":"2024-03-26T05:21:25.887661+0000","last_became_peered":"2024-03-26T05:21:25.887661+0000","last_unstale":"2024-03-26T05:21:55.908707+0000","last_undegraded":"2024-03-26T05:21:55.908707+0000","last_fullsized":"2024-03-26T05:21:55.908707+0000","mapping_epoch":462,"log_start":"433'7126","ondisk_log_start":"433'7126","created":408,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"427'5138","last_scrub_stamp":"2024-03-26T05:19:43.145684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.145684+0000","objects_scrubbed":5110,"log_size":4582,"log_dups_size":0,"ondisk_log_size":4582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.419055+0000","scrub_duration":796,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82935808,"num_objects":1265,"num_object_clones":0,"num_object_copies":3795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6255,"num_write_kb":240672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1996,"num_bytes_recovered":130842624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"443'67","reported_seq":1023,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.809366+0000","last_change":"2024-03-26T05:21:08.573480+0000","last_active":"2024-03-26T05:21:51.809366+0000","last_peered":"2024-03-26T05:21:51.809366+0000","last_clean":"2024-03-26T05:21:51.809366+0000","last_became_active":"2024-03-26T05:15:32.626640+0000","last_became_peered":"2024-03-26T05:15:32.626640+0000","last_unstale":"2024-03-26T05:21:51.809366+0000","last_undegraded":"2024-03-26T05:21:51.809366+0000","last_fullsized":"2024-03-26T05:21:51.809366+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'67","last_scrub_stamp":"2024-03-26T05:21:08.573331+0000","last_deep_scrub":"343'51","last_deep_scrub_stamp":"2024-03-26T05:17:45.197613+0000","last_clean_scrub_stamp":"2024-03-26T05:21:08.573331+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:29.772598+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":799,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.818380+0000","last_change":"2024-03-26T05:20:30.330168+0000","last_active":"2024-03-26T05:21:55.818380+0000","last_peered":"2024-03-26T05:21:55.818380+0000","last_clean":"2024-03-26T05:21:55.818380+0000","last_became_active":"2024-03-26T05:20:30.329736+0000","last_became_peered":"2024-03-26T05:20:30.329736+0000","last_unstale":"2024-03-26T05:21:55.818380+0000","last_undegraded":"2024-03-26T05:21:55.818380+0000","last_fullsized":"2024-03-26T05:21:55.818380+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:46.539478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:46.539478+0000","last_clean_scrub_stamp":"2024-03-26T05:18:46.539478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":930,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.821603+0000","last_change":"2024-03-26T05:20:30.344308+0000","last_active":"2024-03-26T05:21:55.821603+0000","last_peered":"2024-03-26T05:21:55.821603+0000","last_clean":"2024-03-26T05:21:55.821603+0000","last_became_active":"2024-03-26T05:20:30.343819+0000","last_became_peered":"2024-03-26T05:20:30.343819+0000","last_unstale":"2024-03-26T05:21:55.821603+0000","last_undegraded":"2024-03-26T05:21:55.821603+0000","last_fullsized":"2024-03-26T05:21:55.821603+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:57.997324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.997324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514344+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":804,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817053+0000","last_change":"2024-03-26T05:19:48.333412+0000","last_active":"2024-03-26T05:21:55.817053+0000","last_peered":"2024-03-26T05:21:55.817053+0000","last_clean":"2024-03-26T05:21:55.817053+0000","last_became_active":"2024-03-26T05:15:32.633481+0000","last_became_peered":"2024-03-26T05:15:32.633481+0000","last_unstale":"2024-03-26T05:21:55.817053+0000","last_undegraded":"2024-03-26T05:21:55.817053+0000","last_fullsized":"2024-03-26T05:21:55.817053+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:48.333336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_clean_scrub_stamp":"2024-03-26T05:19:48.333336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444201+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"479'11792","reported_seq":11957,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.818730+0000","last_change":"2024-03-26T05:21:23.813371+0000","last_active":"2024-03-26T05:21:56.818730+0000","last_peered":"2024-03-26T05:21:56.818730+0000","last_clean":"2024-03-26T05:21:56.818730+0000","last_became_active":"2024-03-26T05:21:23.796343+0000","last_became_peered":"2024-03-26T05:21:23.796343+0000","last_unstale":"2024-03-26T05:21:56.818730+0000","last_undegraded":"2024-03-26T05:21:56.818730+0000","last_fullsized":"2024-03-26T05:21:56.818730+0000","mapping_epoch":408,"log_start":"437'7227","ondisk_log_start":"437'7227","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4565,"log_dups_size":0,"ondisk_log_size":4565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:57.616981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81952768,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6310,"num_write_kb":241952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":904,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.821956+0000","last_change":"2024-03-26T05:18:52.271198+0000","last_active":"2024-03-26T05:21:55.821956+0000","last_peered":"2024-03-26T05:21:55.821956+0000","last_clean":"2024-03-26T05:21:55.821956+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:21:55.821956+0000","last_undegraded":"2024-03-26T05:21:55.821956+0000","last_fullsized":"2024-03-26T05:21:55.821956+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:52.271117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:52.271117+0000","last_clean_scrub_stamp":"2024-03-26T05:18:52.271117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514179+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"443'43","reported_seq":790,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.812075+0000","last_change":"2024-03-26T05:20:15.454067+0000","last_active":"2024-03-26T05:21:55.812075+0000","last_peered":"2024-03-26T05:21:55.812075+0000","last_clean":"2024-03-26T05:21:55.812075+0000","last_became_active":"2024-03-26T05:20:15.361123+0000","last_became_peered":"2024-03-26T05:20:15.361123+0000","last_unstale":"2024-03-26T05:21:55.812075+0000","last_undegraded":"2024-03-26T05:21:55.812075+0000","last_fullsized":"2024-03-26T05:21:55.812075+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-26T05:14:55.963773+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_clean_scrub_stamp":"2024-03-26T05:14:55.963773+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":816,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.818440+0000","last_change":"2024-03-26T05:20:15.374221+0000","last_active":"2024-03-26T05:21:55.818440+0000","last_peered":"2024-03-26T05:21:55.818440+0000","last_clean":"2024-03-26T05:21:55.818440+0000","last_became_active":"2024-03-26T05:20:15.373910+0000","last_became_peered":"2024-03-26T05:20:15.373910+0000","last_unstale":"2024-03-26T05:21:55.818440+0000","last_undegraded":"2024-03-26T05:21:55.818440+0000","last_fullsized":"2024-03-26T05:21:55.818440+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:20.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:20.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":889,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780900+0000","last_change":"2024-03-26T05:20:15.377202+0000","last_active":"2024-03-26T05:21:51.780900+0000","last_peered":"2024-03-26T05:21:51.780900+0000","last_clean":"2024-03-26T05:21:51.780900+0000","last_became_active":"2024-03-26T05:20:15.376729+0000","last_became_peered":"2024-03-26T05:20:15.376729+0000","last_unstale":"2024-03-26T05:21:51.780900+0000","last_undegraded":"2024-03-26T05:21:51.780900+0000","last_fullsized":"2024-03-26T05:21:51.780900+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:51.030982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:51.030982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"478'11529","reported_seq":24190,"reported_epoch":478,"state":"peering","last_fresh":"2024-03-26T05:21:55.795814+0000","last_change":"2024-03-26T05:21:55.816234+0000","last_active":"2024-03-26T05:21:55.816234+0000","last_peered":"2024-03-26T05:21:55.795814+0000","last_clean":"2024-03-26T05:18:36.613885+0000","last_became_active":"2024-03-26T05:21:23.782892+0000","last_became_peered":"2024-03-26T05:21:23.782892+0000","last_unstale":"2024-03-26T05:21:55.795814+0000","last_undegraded":"2024-03-26T05:21:55.795814+0000","last_fullsized":"2024-03-26T05:21:23.772355+0000","mapping_epoch":479,"log_start":"434'6999","ondisk_log_start":"434'6999","created":408,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79331328,"num_objects":1210,"num_object_clones":0,"num_object_copies":3630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6172,"num_write_kb":236256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2081,"num_bytes_recovered":136380416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"439'12","reported_seq":1032,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817695+0000","last_change":"2024-03-26T05:20:15.466421+0000","last_active":"2024-03-26T05:21:55.817695+0000","last_peered":"2024-03-26T05:21:55.817695+0000","last_clean":"2024-03-26T05:21:55.817695+0000","last_became_active":"2024-03-26T05:20:15.379548+0000","last_became_peered":"2024-03-26T05:20:15.379548+0000","last_unstale":"2024-03-26T05:21:55.817695+0000","last_undegraded":"2024-03-26T05:21:55.817695+0000","last_fullsized":"2024-03-26T05:21:55.817695+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:12:03.961391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:03.961391+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"443'60","reported_seq":267,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817596+0000","last_change":"2024-03-26T05:20:30.363622+0000","last_active":"2024-03-26T05:21:55.817596+0000","last_peered":"2024-03-26T05:21:55.817596+0000","last_clean":"2024-03-26T05:21:55.817596+0000","last_became_active":"2024-03-26T05:20:30.341167+0000","last_became_peered":"2024-03-26T05:20:30.341167+0000","last_unstale":"2024-03-26T05:21:55.817596+0000","last_undegraded":"2024-03-26T05:21:55.817596+0000","last_fullsized":"2024-03-26T05:21:55.817596+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'48","last_scrub_stamp":"2024-03-26T05:19:54.208469+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.208469+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444234+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":813,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.818847+0000","last_change":"2024-03-26T05:20:15.470838+0000","last_active":"2024-03-26T05:21:55.818847+0000","last_peered":"2024-03-26T05:21:55.818847+0000","last_clean":"2024-03-26T05:21:55.818847+0000","last_became_active":"2024-03-26T05:20:15.453507+0000","last_became_peered":"2024-03-26T05:20:15.453507+0000","last_unstale":"2024-03-26T05:21:55.818847+0000","last_undegraded":"2024-03-26T05:21:55.818847+0000","last_fullsized":"2024-03-26T05:21:55.818847+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.959120+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444036+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":274,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.808658+0000","last_change":"2024-03-26T05:20:30.368240+0000","last_active":"2024-03-26T05:21:51.808658+0000","last_peered":"2024-03-26T05:21:51.808658+0000","last_clean":"2024-03-26T05:21:51.808658+0000","last_became_active":"2024-03-26T05:20:30.343599+0000","last_became_peered":"2024-03-26T05:20:30.343599+0000","last_unstale":"2024-03-26T05:21:51.808658+0000","last_undegraded":"2024-03-26T05:21:51.808658+0000","last_fullsized":"2024-03-26T05:21:51.808658+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:20:04.267766+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:20:04.267766+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:52.486345+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"479'11877","reported_seq":22548,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.012919+0000","last_change":"2024-03-26T05:21:46.784920+0000","last_active":"2024-03-26T05:21:56.012919+0000","last_peered":"2024-03-26T05:21:56.012919+0000","last_clean":"2024-03-26T05:21:56.012919+0000","last_became_active":"2024-03-26T05:21:46.757565+0000","last_became_peered":"2024-03-26T05:21:46.757565+0000","last_unstale":"2024-03-26T05:21:56.012919+0000","last_undegraded":"2024-03-26T05:21:56.012919+0000","last_fullsized":"2024-03-26T05:21:56.012919+0000","mapping_epoch":473,"log_start":"439'7535","ondisk_log_start":"439'7535","created":460,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:57.526066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83853312,"num_objects":1280,"num_object_clones":0,"num_object_copies":3840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6360,"num_write_kb":244448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3139,"num_bytes_recovered":205717504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"443'24","reported_seq":937,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.782557+0000","last_change":"2024-03-26T05:20:04.274955+0000","last_active":"2024-03-26T05:21:51.782557+0000","last_peered":"2024-03-26T05:21:51.782557+0000","last_clean":"2024-03-26T05:21:51.782557+0000","last_became_active":"2024-03-26T05:15:32.640871+0000","last_became_peered":"2024-03-26T05:15:32.640871+0000","last_unstale":"2024-03-26T05:21:51.782557+0000","last_undegraded":"2024-03-26T05:21:51.782557+0000","last_fullsized":"2024-03-26T05:21:51.782557+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'18","last_scrub_stamp":"2024-03-26T05:20:04.273413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:04.273413+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":32,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":962,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.808446+0000","last_change":"2024-03-26T05:20:30.370885+0000","last_active":"2024-03-26T05:21:51.808446+0000","last_peered":"2024-03-26T05:21:51.808446+0000","last_clean":"2024-03-26T05:21:51.808446+0000","last_became_active":"2024-03-26T05:20:30.343651+0000","last_became_peered":"2024-03-26T05:20:30.343651+0000","last_unstale":"2024-03-26T05:21:51.808446+0000","last_undegraded":"2024-03-26T05:21:51.808446+0000","last_fullsized":"2024-03-26T05:21:51.808446+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:19:45.829142+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:17:50.147911+0000","last_clean_scrub_stamp":"2024-03-26T05:19:45.829142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:52.486327+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":195,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.782352+0000","last_change":"2024-03-26T05:20:30.340915+0000","last_active":"2024-03-26T05:21:51.782352+0000","last_peered":"2024-03-26T05:21:51.782352+0000","last_clean":"2024-03-26T05:21:51.782352+0000","last_became_active":"2024-03-26T05:20:30.330443+0000","last_became_peered":"2024-03-26T05:20:30.330443+0000","last_unstale":"2024-03-26T05:21:51.782352+0000","last_undegraded":"2024-03-26T05:21:51.782352+0000","last_fullsized":"2024-03-26T05:21:51.782352+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.146286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.146286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":804,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.818078+0000","last_change":"2024-03-26T05:20:30.321663+0000","last_active":"2024-03-26T05:21:55.818078+0000","last_peered":"2024-03-26T05:21:55.818078+0000","last_clean":"2024-03-26T05:21:55.818078+0000","last_became_active":"2024-03-26T05:20:30.321301+0000","last_became_peered":"2024-03-26T05:20:30.321301+0000","last_unstale":"2024-03-26T05:21:55.818078+0000","last_undegraded":"2024-03-26T05:21:55.818078+0000","last_fullsized":"2024-03-26T05:21:55.818078+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:07.006139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.006139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.19","version":"479'11558","reported_seq":16281,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.818869+0000","last_change":"2024-03-26T05:21:51.803900+0000","last_active":"2024-03-26T05:21:56.818869+0000","last_peered":"2024-03-26T05:21:56.818869+0000","last_clean":"2024-03-26T05:21:56.818869+0000","last_became_active":"2024-03-26T05:21:51.789764+0000","last_became_peered":"2024-03-26T05:21:51.789764+0000","last_unstale":"2024-03-26T05:21:56.818869+0000","last_undegraded":"2024-03-26T05:21:56.818869+0000","last_fullsized":"2024-03-26T05:21:56.818869+0000","mapping_epoch":477,"log_start":"438'7197","ondisk_log_start":"438'7197","created":460,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4361,"log_dups_size":0,"ondisk_log_size":4361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:57.617011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6186,"num_write_kb":239264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1515,"num_bytes_recovered":99287040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"443'39","reported_seq":976,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819509+0000","last_change":"2024-03-26T05:19:54.421287+0000","last_active":"2024-03-26T05:21:56.819509+0000","last_peered":"2024-03-26T05:21:56.819509+0000","last_clean":"2024-03-26T05:21:56.819509+0000","last_became_active":"2024-03-26T05:15:32.634434+0000","last_became_peered":"2024-03-26T05:15:32.634434+0000","last_unstale":"2024-03-26T05:21:56.819509+0000","last_undegraded":"2024-03-26T05:21:56.819509+0000","last_fullsized":"2024-03-26T05:21:56.819509+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'33","last_scrub_stamp":"2024-03-26T05:19:54.421104+0000","last_deep_scrub":"429'33","last_deep_scrub_stamp":"2024-03-26T05:19:54.421104+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.421104+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.658002+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":772,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817978+0000","last_change":"2024-03-26T05:20:15.376147+0000","last_active":"2024-03-26T05:21:55.817978+0000","last_peered":"2024-03-26T05:21:55.817978+0000","last_clean":"2024-03-26T05:21:55.817978+0000","last_became_active":"2024-03-26T05:20:15.375661+0000","last_became_peered":"2024-03-26T05:20:15.375661+0000","last_unstale":"2024-03-26T05:21:55.817978+0000","last_undegraded":"2024-03-26T05:21:55.817978+0000","last_fullsized":"2024-03-26T05:21:55.817978+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:25.939576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:25.939576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.443972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":886,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.782456+0000","last_change":"2024-03-26T05:18:03.064748+0000","last_active":"2024-03-26T05:21:51.782456+0000","last_peered":"2024-03-26T05:21:51.782456+0000","last_clean":"2024-03-26T05:21:51.782456+0000","last_became_active":"2024-03-26T05:15:32.641031+0000","last_became_peered":"2024-03-26T05:15:32.641031+0000","last_unstale":"2024-03-26T05:21:51.782456+0000","last_undegraded":"2024-03-26T05:21:51.782456+0000","last_fullsized":"2024-03-26T05:21:51.782456+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:03.064638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:18:03.064638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":910,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.822822+0000","last_change":"2024-03-26T05:20:32.095405+0000","last_active":"2024-03-26T05:21:55.822822+0000","last_peered":"2024-03-26T05:21:55.822822+0000","last_clean":"2024-03-26T05:21:55.822822+0000","last_became_active":"2024-03-26T05:20:15.379306+0000","last_became_peered":"2024-03-26T05:20:15.379306+0000","last_unstale":"2024-03-26T05:21:55.822822+0000","last_undegraded":"2024-03-26T05:21:55.822822+0000","last_fullsized":"2024-03-26T05:21:55.822822+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:32.094955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:20:32.094955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514372+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"443'32","reported_seq":216,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817874+0000","last_change":"2024-03-26T05:20:30.356411+0000","last_active":"2024-03-26T05:21:55.817874+0000","last_peered":"2024-03-26T05:21:55.817874+0000","last_clean":"2024-03-26T05:21:55.817874+0000","last_became_active":"2024-03-26T05:20:30.329795+0000","last_became_peered":"2024-03-26T05:20:30.329795+0000","last_unstale":"2024-03-26T05:21:55.817874+0000","last_undegraded":"2024-03-26T05:21:55.817874+0000","last_fullsized":"2024-03-26T05:21:55.817874+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:19:53.221647+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:19:53.221647+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":916,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819488+0000","last_change":"2024-03-26T05:18:51.276677+0000","last_active":"2024-03-26T05:21:56.819488+0000","last_peered":"2024-03-26T05:21:56.819488+0000","last_clean":"2024-03-26T05:21:56.819488+0000","last_became_active":"2024-03-26T05:15:32.632387+0000","last_became_peered":"2024-03-26T05:15:32.632387+0000","last_unstale":"2024-03-26T05:21:56.819488+0000","last_undegraded":"2024-03-26T05:21:56.819488+0000","last_fullsized":"2024-03-26T05:21:56.819488+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:51.276565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:45.352678+0000","last_clean_scrub_stamp":"2024-03-26T05:18:51.276565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"336'11","reported_seq":1168,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.781436+0000","last_change":"2024-03-26T05:19:56.382330+0000","last_active":"2024-03-26T05:21:51.781436+0000","last_peered":"2024-03-26T05:21:51.781436+0000","last_clean":"2024-03-26T05:21:51.781436+0000","last_became_active":"2024-03-26T05:15:32.636392+0000","last_became_peered":"2024-03-26T05:15:32.636392+0000","last_unstale":"2024-03-26T05:21:51.781436+0000","last_undegraded":"2024-03-26T05:21:51.781436+0000","last_fullsized":"2024-03-26T05:21:51.781436+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-03-26T05:19:56.382205+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:19:56.382205+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.626445+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":616,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.824372+0000","last_change":"2024-03-26T05:20:30.347802+0000","last_active":"2024-03-26T05:21:55.824372+0000","last_peered":"2024-03-26T05:21:55.824372+0000","last_clean":"2024-03-26T05:21:55.824372+0000","last_became_active":"2024-03-26T05:20:30.344267+0000","last_became_peered":"2024-03-26T05:20:30.344267+0000","last_unstale":"2024-03-26T05:21:55.824372+0000","last_undegraded":"2024-03-26T05:21:55.824372+0000","last_fullsized":"2024-03-26T05:21:55.824372+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:06.096138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:06.096138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.509154+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"443'49","reported_seq":983,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780937+0000","last_change":"2024-03-26T05:19:58.389112+0000","last_active":"2024-03-26T05:21:51.780937+0000","last_peered":"2024-03-26T05:21:51.780937+0000","last_clean":"2024-03-26T05:21:51.780937+0000","last_became_active":"2024-03-26T05:15:32.626448+0000","last_became_peered":"2024-03-26T05:15:32.626448+0000","last_unstale":"2024-03-26T05:21:51.780937+0000","last_undegraded":"2024-03-26T05:21:51.780937+0000","last_fullsized":"2024-03-26T05:21:51.780937+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'39","last_scrub_stamp":"2024-03-26T05:19:58.388933+0000","last_deep_scrub":"345'39","last_deep_scrub_stamp":"2024-03-26T05:15:56.674423+0000","last_clean_scrub_stamp":"2024-03-26T05:19:58.388933+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":806,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817994+0000","last_change":"2024-03-26T05:20:15.364695+0000","last_active":"2024-03-26T05:21:55.817994+0000","last_peered":"2024-03-26T05:21:55.817994+0000","last_clean":"2024-03-26T05:21:55.817994+0000","last_became_active":"2024-03-26T05:20:15.364437+0000","last_became_peered":"2024-03-26T05:20:15.364437+0000","last_unstale":"2024-03-26T05:21:55.817994+0000","last_undegraded":"2024-03-26T05:21:55.817994+0000","last_fullsized":"2024-03-26T05:21:55.817994+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:01.962048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:01.962048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"443'11","reported_seq":286,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.820641+0000","last_change":"2024-03-26T05:20:30.356968+0000","last_active":"2024-03-26T05:21:55.820641+0000","last_peered":"2024-03-26T05:21:55.820641+0000","last_clean":"2024-03-26T05:21:55.820641+0000","last_became_active":"2024-03-26T05:20:30.338470+0000","last_became_peered":"2024-03-26T05:20:30.338470+0000","last_unstale":"2024-03-26T05:21:55.820641+0000","last_undegraded":"2024-03-26T05:21:55.820641+0000","last_fullsized":"2024-03-26T05:21:55.820641+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:18:40.082750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:18:40.082750+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":903,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.818982+0000","last_change":"2024-03-26T05:19:53.535068+0000","last_active":"2024-03-26T05:21:56.818982+0000","last_peered":"2024-03-26T05:21:56.818982+0000","last_clean":"2024-03-26T05:21:56.818982+0000","last_became_active":"2024-03-26T05:15:32.631444+0000","last_became_peered":"2024-03-26T05:15:32.631444+0000","last_unstale":"2024-03-26T05:21:56.818982+0000","last_undegraded":"2024-03-26T05:21:56.818982+0000","last_fullsized":"2024-03-26T05:21:56.818982+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:53.534717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:59.508745+0000","last_clean_scrub_stamp":"2024-03-26T05:19:53.534717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.658047+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"443'32","reported_seq":786,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.808879+0000","last_change":"2024-03-26T05:20:40.672261+0000","last_active":"2024-03-26T05:21:51.808879+0000","last_peered":"2024-03-26T05:21:51.808879+0000","last_clean":"2024-03-26T05:21:51.808879+0000","last_became_active":"2024-03-26T05:20:30.323810+0000","last_became_peered":"2024-03-26T05:20:30.323810+0000","last_unstale":"2024-03-26T05:21:51.808879+0000","last_undegraded":"2024-03-26T05:21:51.808879+0000","last_fullsized":"2024-03-26T05:21:51.808879+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'32","last_scrub_stamp":"2024-03-26T05:20:40.672162+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:20:40.672162+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.557112+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":874,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.818587+0000","last_change":"2024-03-26T05:20:15.372145+0000","last_active":"2024-03-26T05:21:55.818587+0000","last_peered":"2024-03-26T05:21:55.818587+0000","last_clean":"2024-03-26T05:21:55.818587+0000","last_became_active":"2024-03-26T05:20:15.371796+0000","last_became_peered":"2024-03-26T05:20:15.371796+0000","last_unstale":"2024-03-26T05:21:55.818587+0000","last_undegraded":"2024-03-26T05:21:55.818587+0000","last_fullsized":"2024-03-26T05:21:55.818587+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:21.515294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:21.515294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444004+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":938,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.814181+0000","last_change":"2024-03-26T05:20:30.332976+0000","last_active":"2024-03-26T05:21:55.814181+0000","last_peered":"2024-03-26T05:21:55.814181+0000","last_clean":"2024-03-26T05:21:55.814181+0000","last_became_active":"2024-03-26T05:20:30.332245+0000","last_became_peered":"2024-03-26T05:20:30.332245+0000","last_unstale":"2024-03-26T05:21:55.814181+0000","last_undegraded":"2024-03-26T05:21:55.814181+0000","last_fullsized":"2024-03-26T05:21:55.814181+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:05.539590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:55.496800+0000","last_clean_scrub_stamp":"2024-03-26T05:20:05.539590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":900,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780725+0000","last_change":"2024-03-26T05:20:30.346728+0000","last_active":"2024-03-26T05:21:51.780725+0000","last_peered":"2024-03-26T05:21:51.780725+0000","last_clean":"2024-03-26T05:21:51.780725+0000","last_became_active":"2024-03-26T05:20:30.346313+0000","last_became_peered":"2024-03-26T05:20:30.346313+0000","last_unstale":"2024-03-26T05:21:51.780725+0000","last_undegraded":"2024-03-26T05:21:51.780725+0000","last_fullsized":"2024-03-26T05:21:51.780725+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:38.817507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:18:38.817507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"443'21","reported_seq":950,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.814259+0000","last_change":"2024-03-26T05:20:01.620894+0000","last_active":"2024-03-26T05:21:55.814259+0000","last_peered":"2024-03-26T05:21:55.814259+0000","last_clean":"2024-03-26T05:21:55.814259+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:21:55.814259+0000","last_undegraded":"2024-03-26T05:21:55.814259+0000","last_fullsized":"2024-03-26T05:21:55.814259+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'15","last_scrub_stamp":"2024-03-26T05:20:01.620742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:01.620742+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":778,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.818174+0000","last_change":"2024-03-26T05:20:15.375970+0000","last_active":"2024-03-26T05:21:55.818174+0000","last_peered":"2024-03-26T05:21:55.818174+0000","last_clean":"2024-03-26T05:21:55.818174+0000","last_became_active":"2024-03-26T05:20:15.375609+0000","last_became_peered":"2024-03-26T05:20:15.375609+0000","last_unstale":"2024-03-26T05:21:55.818174+0000","last_undegraded":"2024-03-26T05:21:55.818174+0000","last_fullsized":"2024-03-26T05:21:55.818174+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:33.016072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:11:33.016072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":894,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780733+0000","last_change":"2024-03-26T05:19:55.403001+0000","last_active":"2024-03-26T05:21:51.780733+0000","last_peered":"2024-03-26T05:21:51.780733+0000","last_clean":"2024-03-26T05:21:51.780733+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:21:51.780733+0000","last_undegraded":"2024-03-26T05:21:51.780733+0000","last_fullsized":"2024-03-26T05:21:51.780733+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:55.402737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:19:55.402737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":921,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.818785+0000","last_change":"2024-03-26T05:20:30.344287+0000","last_active":"2024-03-26T05:21:56.818785+0000","last_peered":"2024-03-26T05:21:56.818785+0000","last_clean":"2024-03-26T05:21:56.818785+0000","last_became_active":"2024-03-26T05:20:30.343972+0000","last_became_peered":"2024-03-26T05:20:30.343972+0000","last_unstale":"2024-03-26T05:21:56.818785+0000","last_undegraded":"2024-03-26T05:21:56.818785+0000","last_fullsized":"2024-03-26T05:21:56.818785+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:38.324541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:18:38.324541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657872+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"443'33","reported_seq":954,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.781064+0000","last_change":"2024-03-26T05:19:57.364173+0000","last_active":"2024-03-26T05:21:51.781064+0000","last_peered":"2024-03-26T05:21:51.781064+0000","last_clean":"2024-03-26T05:21:51.781064+0000","last_became_active":"2024-03-26T05:15:32.636359+0000","last_became_peered":"2024-03-26T05:15:32.636359+0000","last_unstale":"2024-03-26T05:21:51.781064+0000","last_undegraded":"2024-03-26T05:21:51.781064+0000","last_fullsized":"2024-03-26T05:21:51.781064+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:19:57.363571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.363571+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":349,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.818233+0000","last_change":"2024-03-26T05:20:30.351327+0000","last_active":"2024-03-26T05:21:55.818233+0000","last_peered":"2024-03-26T05:21:55.818233+0000","last_clean":"2024-03-26T05:21:55.818233+0000","last_became_active":"2024-03-26T05:20:30.350957+0000","last_became_peered":"2024-03-26T05:20:30.350957+0000","last_unstale":"2024-03-26T05:21:55.818233+0000","last_undegraded":"2024-03-26T05:21:55.818233+0000","last_fullsized":"2024-03-26T05:21:55.818233+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:27.232872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:20:27.232872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:57.438099+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":815,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817767+0000","last_change":"2024-03-26T05:20:15.377844+0000","last_active":"2024-03-26T05:21:55.817767+0000","last_peered":"2024-03-26T05:21:55.817767+0000","last_clean":"2024-03-26T05:21:55.817767+0000","last_became_active":"2024-03-26T05:20:15.377506+0000","last_became_peered":"2024-03-26T05:20:15.377506+0000","last_unstale":"2024-03-26T05:21:55.817767+0000","last_undegraded":"2024-03-26T05:21:55.817767+0000","last_fullsized":"2024-03-26T05:21:55.817767+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:23.957542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:11:23.957542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.443989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":898,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780941+0000","last_change":"2024-03-26T05:20:28.593855+0000","last_active":"2024-03-26T05:21:51.780941+0000","last_peered":"2024-03-26T05:21:51.780941+0000","last_clean":"2024-03-26T05:21:51.780941+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:21:51.780941+0000","last_undegraded":"2024-03-26T05:21:51.780941+0000","last_fullsized":"2024-03-26T05:21:51.780941+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:28.593685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:53.017168+0000","last_clean_scrub_stamp":"2024-03-26T05:20:28.593685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"443'14","reported_seq":918,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780556+0000","last_change":"2024-03-26T05:18:56.877975+0000","last_active":"2024-03-26T05:21:51.780556+0000","last_peered":"2024-03-26T05:21:51.780556+0000","last_clean":"2024-03-26T05:21:51.780556+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:21:51.780556+0000","last_undegraded":"2024-03-26T05:21:51.780556+0000","last_fullsized":"2024-03-26T05:21:51.780556+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'10","last_scrub_stamp":"2024-03-26T05:18:56.877852+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:18:56.877852+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":932,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819334+0000","last_change":"2024-03-26T05:20:30.351206+0000","last_active":"2024-03-26T05:21:56.819334+0000","last_peered":"2024-03-26T05:21:56.819334+0000","last_clean":"2024-03-26T05:21:56.819334+0000","last_became_active":"2024-03-26T05:20:30.350521+0000","last_became_peered":"2024-03-26T05:20:30.350521+0000","last_unstale":"2024-03-26T05:21:56.819334+0000","last_undegraded":"2024-03-26T05:21:56.819334+0000","last_fullsized":"2024-03-26T05:21:56.819334+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:40.333981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:11.070341+0000","last_clean_scrub_stamp":"2024-03-26T05:18:40.333981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":903,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.809552+0000","last_change":"2024-03-26T05:20:37.670090+0000","last_active":"2024-03-26T05:21:51.809552+0000","last_peered":"2024-03-26T05:21:51.809552+0000","last_clean":"2024-03-26T05:21:51.809552+0000","last_became_active":"2024-03-26T05:20:15.372788+0000","last_became_peered":"2024-03-26T05:20:15.372788+0000","last_unstale":"2024-03-26T05:21:51.809552+0000","last_undegraded":"2024-03-26T05:21:51.809552+0000","last_fullsized":"2024-03-26T05:21:51.809552+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:37.669979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:20:37.669979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.557092+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":799,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817423+0000","last_change":"2024-03-26T05:20:15.364726+0000","last_active":"2024-03-26T05:21:55.817423+0000","last_peered":"2024-03-26T05:21:55.817423+0000","last_clean":"2024-03-26T05:21:55.817423+0000","last_became_active":"2024-03-26T05:20:15.364379+0000","last_became_peered":"2024-03-26T05:20:15.364379+0000","last_unstale":"2024-03-26T05:21:55.817423+0000","last_undegraded":"2024-03-26T05:21:55.817423+0000","last_fullsized":"2024-03-26T05:21:55.817423+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:10.020572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:10.020572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"479'11926","reported_seq":12168,"reported_epoch":479,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T05:21:56.125757+0000","last_change":"2024-03-26T05:21:40.866171+0000","last_active":"2024-03-26T05:21:56.125757+0000","last_peered":"2024-03-26T05:21:56.125757+0000","last_clean":"2024-03-26T05:21:24.457072+0000","last_became_active":"2024-03-26T05:21:26.961051+0000","last_became_peered":"2024-03-26T05:21:26.961051+0000","last_unstale":"2024-03-26T05:21:56.125757+0000","last_undegraded":"2024-03-26T05:21:56.125757+0000","last_fullsized":"2024-03-26T05:21:56.125757+0000","mapping_epoch":463,"log_start":"434'7269","ondisk_log_start":"434'7269","created":460,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":5,"last_scrub":"431'6421","last_scrub_stamp":"2024-03-26T05:20:00.971796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:20:00.971796+0000","objects_scrubbed":6351,"log_size":4657,"log_dups_size":0,"ondisk_log_size":4657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80019456,"num_objects":1221,"num_object_clones":0,"num_object_copies":3663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3663,"num_objects_unfound":0,"num_objects_dirty":1221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6405,"num_write_kb":244032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[1,3,7],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":1221}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"443'39","reported_seq":1071,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.809815+0000","last_change":"2024-03-26T05:20:30.366464+0000","last_active":"2024-03-26T05:21:51.809815+0000","last_peered":"2024-03-26T05:21:51.809815+0000","last_clean":"2024-03-26T05:21:51.809815+0000","last_became_active":"2024-03-26T05:20:30.343540+0000","last_became_peered":"2024-03-26T05:20:30.343540+0000","last_unstale":"2024-03-26T05:21:51.809815+0000","last_undegraded":"2024-03-26T05:21:51.809815+0000","last_fullsized":"2024-03-26T05:21:51.809815+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'31","last_scrub_stamp":"2024-03-26T05:19:57.767398+0000","last_deep_scrub":"413'31","last_deep_scrub_stamp":"2024-03-26T05:18:46.881054+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.767398+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:52.486363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":901,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780420+0000","last_change":"2024-03-26T05:18:47.776529+0000","last_active":"2024-03-26T05:21:51.780420+0000","last_peered":"2024-03-26T05:21:51.780420+0000","last_clean":"2024-03-26T05:21:51.780420+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:21:51.780420+0000","last_undegraded":"2024-03-26T05:21:51.780420+0000","last_fullsized":"2024-03-26T05:21:51.780420+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:47.776444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:47.776444+0000","last_clean_scrub_stamp":"2024-03-26T05:18:47.776444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":901,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819616+0000","last_change":"2024-03-26T05:19:48.378592+0000","last_active":"2024-03-26T05:21:56.819616+0000","last_peered":"2024-03-26T05:21:56.819616+0000","last_clean":"2024-03-26T05:21:56.819616+0000","last_became_active":"2024-03-26T05:15:32.634716+0000","last_became_peered":"2024-03-26T05:15:32.634716+0000","last_unstale":"2024-03-26T05:21:56.819616+0000","last_undegraded":"2024-03-26T05:21:56.819616+0000","last_fullsized":"2024-03-26T05:21:56.819616+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:48.378485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:48.378485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.658017+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":910,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.820920+0000","last_change":"2024-03-26T05:20:30.342850+0000","last_active":"2024-03-26T05:21:55.820920+0000","last_peered":"2024-03-26T05:21:55.820920+0000","last_clean":"2024-03-26T05:21:55.820920+0000","last_became_active":"2024-03-26T05:20:30.342550+0000","last_became_peered":"2024-03-26T05:20:30.342550+0000","last_unstale":"2024-03-26T05:21:55.820920+0000","last_undegraded":"2024-03-26T05:21:55.820920+0000","last_fullsized":"2024-03-26T05:21:55.820920+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:00.151355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.151355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"479'11900","reported_seq":12135,"reported_epoch":480,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T05:21:56.818031+0000","last_change":"2024-03-26T05:21:27.038051+0000","last_active":"2024-03-26T05:21:56.818031+0000","last_peered":"2024-03-26T05:21:56.818031+0000","last_clean":"2024-03-26T05:21:23.856457+0000","last_became_active":"2024-03-26T05:21:26.962137+0000","last_became_peered":"2024-03-26T05:21:26.962137+0000","last_unstale":"2024-03-26T05:21:56.818031+0000","last_undegraded":"2024-03-26T05:21:56.818031+0000","last_fullsized":"2024-03-26T05:21:56.818031+0000","mapping_epoch":463,"log_start":"433'7135","ondisk_log_start":"433'7135","created":460,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":5,"last_scrub":"428'5729","last_scrub_stamp":"2024-03-26T05:19:50.459009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:19:50.459009+0000","objects_scrubbed":5667,"log_size":4765,"log_dups_size":0,"ondisk_log_size":4765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84574208,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2582,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6368,"num_write_kb":245056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[0,3,6],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3,6","objects":1291}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"443'42","reported_seq":976,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819513+0000","last_change":"2024-03-26T05:18:57.256490+0000","last_active":"2024-03-26T05:21:56.819513+0000","last_peered":"2024-03-26T05:21:56.819513+0000","last_clean":"2024-03-26T05:21:56.819513+0000","last_became_active":"2024-03-26T05:15:32.634216+0000","last_became_peered":"2024-03-26T05:15:32.634216+0000","last_unstale":"2024-03-26T05:21:56.819513+0000","last_undegraded":"2024-03-26T05:21:56.819513+0000","last_fullsized":"2024-03-26T05:21:56.819513+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'32","last_scrub_stamp":"2024-03-26T05:18:57.255727+0000","last_deep_scrub":"290'32","last_deep_scrub_stamp":"2024-03-26T05:12:15.970313+0000","last_clean_scrub_stamp":"2024-03-26T05:18:57.255727+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657970+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":909,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.812216+0000","last_change":"2024-03-26T05:20:15.367466+0000","last_active":"2024-03-26T05:21:55.812216+0000","last_peered":"2024-03-26T05:21:55.812216+0000","last_clean":"2024-03-26T05:21:55.812216+0000","last_became_active":"2024-03-26T05:20:15.361116+0000","last_became_peered":"2024-03-26T05:20:15.361116+0000","last_unstale":"2024-03-26T05:21:55.812216+0000","last_undegraded":"2024-03-26T05:21:55.812216+0000","last_fullsized":"2024-03-26T05:21:55.812216+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:49.948818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:49.948818+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"439'12","reported_seq":1197,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.809758+0000","last_change":"2024-03-26T05:21:02.540243+0000","last_active":"2024-03-26T05:21:51.809758+0000","last_peered":"2024-03-26T05:21:51.809758+0000","last_clean":"2024-03-26T05:21:51.809758+0000","last_became_active":"2024-03-26T05:20:15.375030+0000","last_became_peered":"2024-03-26T05:20:15.375030+0000","last_unstale":"2024-03-26T05:21:51.809758+0000","last_undegraded":"2024-03-26T05:21:51.809758+0000","last_fullsized":"2024-03-26T05:21:51.809758+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'12","last_scrub_stamp":"2024-03-26T05:21:02.540128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:21:02.540128+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:05.850824+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"429'2","reported_seq":839,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.818526+0000","last_change":"2024-03-26T05:20:30.352085+0000","last_active":"2024-03-26T05:21:55.818526+0000","last_peered":"2024-03-26T05:21:55.818526+0000","last_clean":"2024-03-26T05:21:55.818526+0000","last_became_active":"2024-03-26T05:20:30.341958+0000","last_became_peered":"2024-03-26T05:20:30.341958+0000","last_unstale":"2024-03-26T05:21:55.818526+0000","last_undegraded":"2024-03-26T05:21:55.818526+0000","last_fullsized":"2024-03-26T05:21:55.818526+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'2","last_scrub_stamp":"2024-03-26T05:19:54.275264+0000","last_deep_scrub":"429'2","last_deep_scrub_stamp":"2024-03-26T05:19:54.275264+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.275264+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444266+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.12","version":"479'11517","reported_seq":15542,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.916047+0000","last_change":"2024-03-26T05:21:41.001182+0000","last_active":"2024-03-26T05:21:55.916047+0000","last_peered":"2024-03-26T05:21:55.916047+0000","last_clean":"2024-03-26T05:21:55.916047+0000","last_became_active":"2024-03-26T05:21:40.985985+0000","last_became_peered":"2024-03-26T05:21:40.985985+0000","last_unstale":"2024-03-26T05:21:55.916047+0000","last_undegraded":"2024-03-26T05:21:55.916047+0000","last_fullsized":"2024-03-26T05:21:55.916047+0000","mapping_epoch":469,"log_start":"431'6160","ondisk_log_start":"431'6160","created":460,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":4299,"log_size":5357,"log_dups_size":0,"ondisk_log_size":5357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:57.438083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":237184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1902,"num_bytes_recovered":124649472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"443'38","reported_seq":965,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.781715+0000","last_change":"2024-03-26T05:20:15.462061+0000","last_active":"2024-03-26T05:21:51.781715+0000","last_peered":"2024-03-26T05:21:51.781715+0000","last_clean":"2024-03-26T05:21:51.781715+0000","last_became_active":"2024-03-26T05:20:15.362085+0000","last_became_peered":"2024-03-26T05:20:15.362085+0000","last_unstale":"2024-03-26T05:21:51.781715+0000","last_undegraded":"2024-03-26T05:21:51.781715+0000","last_fullsized":"2024-03-26T05:21:51.781715+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'32","last_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_deep_scrub":"322'32","last_deep_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_clean_scrub_stamp":"2024-03-26T05:15:10.715786+0000","objects_scrubbed":6,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":289,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.813771+0000","last_change":"2024-03-26T05:20:15.379698+0000","last_active":"2024-03-26T05:21:55.813771+0000","last_peered":"2024-03-26T05:21:55.813771+0000","last_clean":"2024-03-26T05:21:55.813771+0000","last_became_active":"2024-03-26T05:20:15.379345+0000","last_became_peered":"2024-03-26T05:20:15.379345+0000","last_unstale":"2024-03-26T05:21:55.813771+0000","last_undegraded":"2024-03-26T05:21:55.813771+0000","last_fullsized":"2024-03-26T05:21:55.813771+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:03.047441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:15:03.047441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":287,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.781116+0000","last_change":"2024-03-26T05:20:15.372365+0000","last_active":"2024-03-26T05:21:51.781116+0000","last_peered":"2024-03-26T05:21:51.781116+0000","last_clean":"2024-03-26T05:21:51.781116+0000","last_became_active":"2024-03-26T05:20:15.372060+0000","last_became_peered":"2024-03-26T05:20:15.372060+0000","last_unstale":"2024-03-26T05:21:51.781116+0000","last_undegraded":"2024-03-26T05:21:51.781116+0000","last_fullsized":"2024-03-26T05:21:51.781116+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:04.830989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:04.830989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":917,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.782515+0000","last_change":"2024-03-26T05:20:30.348058+0000","last_active":"2024-03-26T05:21:51.782515+0000","last_peered":"2024-03-26T05:21:51.782515+0000","last_clean":"2024-03-26T05:21:51.782515+0000","last_became_active":"2024-03-26T05:20:30.347326+0000","last_became_peered":"2024-03-26T05:20:30.347326+0000","last_unstale":"2024-03-26T05:21:51.782515+0000","last_undegraded":"2024-03-26T05:21:51.782515+0000","last_fullsized":"2024-03-26T05:21:51.782515+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:01.576615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:20:01.576615+0000","last_clean_scrub_stamp":"2024-03-26T05:20:01.576615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"479'14809","reported_seq":20260,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819465+0000","last_change":"2024-03-26T05:21:48.775309+0000","last_active":"2024-03-26T05:21:56.819465+0000","last_peered":"2024-03-26T05:21:56.819465+0000","last_clean":"2024-03-26T05:21:56.819465+0000","last_became_active":"2024-03-26T05:21:48.762442+0000","last_became_peered":"2024-03-26T05:21:48.762442+0000","last_unstale":"2024-03-26T05:21:56.819465+0000","last_undegraded":"2024-03-26T05:21:56.819465+0000","last_fullsized":"2024-03-26T05:21:56.819465+0000","mapping_epoch":475,"log_start":"453'10038","ondisk_log_start":"453'10038","created":460,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":3086,"log_size":4771,"log_dups_size":0,"ondisk_log_size":4771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:59.703782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82149376,"num_objects":1254,"num_object_clones":0,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6269,"num_write_kb":240704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1724,"num_bytes_recovered":112132096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"443'39","reported_seq":976,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.822536+0000","last_change":"2024-03-26T05:19:59.984067+0000","last_active":"2024-03-26T05:21:55.822536+0000","last_peered":"2024-03-26T05:21:55.822536+0000","last_clean":"2024-03-26T05:21:55.822536+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:21:55.822536+0000","last_undegraded":"2024-03-26T05:21:55.822536+0000","last_fullsized":"2024-03-26T05:21:55.822536+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:19:59.983973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:19:59.983973+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.514313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":908,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819647+0000","last_change":"2024-03-26T05:20:15.457277+0000","last_active":"2024-03-26T05:21:56.819647+0000","last_peered":"2024-03-26T05:21:56.819647+0000","last_clean":"2024-03-26T05:21:56.819647+0000","last_became_active":"2024-03-26T05:20:15.456698+0000","last_became_peered":"2024-03-26T05:20:15.456698+0000","last_unstale":"2024-03-26T05:21:56.819647+0000","last_undegraded":"2024-03-26T05:21:56.819647+0000","last_fullsized":"2024-03-26T05:21:56.819647+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:04.980730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:12:04.980730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657767+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":902,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.820784+0000","last_change":"2024-03-26T05:20:30.339668+0000","last_active":"2024-03-26T05:21:55.820784+0000","last_peered":"2024-03-26T05:21:55.820784+0000","last_clean":"2024-03-26T05:21:55.820784+0000","last_became_active":"2024-03-26T05:20:30.338293+0000","last_became_peered":"2024-03-26T05:20:30.338293+0000","last_unstale":"2024-03-26T05:21:55.820784+0000","last_undegraded":"2024-03-26T05:21:55.820784+0000","last_fullsized":"2024-03-26T05:21:55.820784+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:11.912203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:15:11.912203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":631,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.810070+0000","last_change":"2024-03-26T05:20:50.576887+0000","last_active":"2024-03-26T05:21:51.810070+0000","last_peered":"2024-03-26T05:21:51.810070+0000","last_clean":"2024-03-26T05:21:51.810070+0000","last_became_active":"2024-03-26T05:20:30.346249+0000","last_became_peered":"2024-03-26T05:20:30.346249+0000","last_unstale":"2024-03-26T05:21:51.810070+0000","last_undegraded":"2024-03-26T05:21:51.810070+0000","last_fullsized":"2024-03-26T05:21:51.810070+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:50.576761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:20:50.576761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:16.470321+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"478'11834","reported_seq":12076,"reported_epoch":478,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T05:21:54.825889+0000","last_change":"2024-03-26T05:21:26.975333+0000","last_active":"2024-03-26T05:21:54.825889+0000","last_peered":"2024-03-26T05:21:54.825889+0000","last_clean":"2024-03-26T05:21:23.848570+0000","last_became_active":"2024-03-26T05:21:26.958296+0000","last_became_peered":"2024-03-26T05:21:26.958296+0000","last_unstale":"2024-03-26T05:21:54.825889+0000","last_undegraded":"2024-03-26T05:21:54.825889+0000","last_fullsized":"2024-03-26T05:21:54.825889+0000","mapping_epoch":463,"log_start":"434'7321","ondisk_log_start":"434'7321","created":460,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":5,"last_scrub":"431'6714","last_scrub_stamp":"2024-03-26T05:20:05.667571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:20:05.667571+0000","objects_scrubbed":6680,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90505216,"num_objects":1381,"num_object_clones":0,"num_object_copies":4143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1381,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":246240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,2,7],"avail_no_missing":["6","2","7"],"object_location_counts":[{"shards":"2,6,7","objects":1381}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"443'18","reported_seq":884,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780619+0000","last_change":"2024-03-26T05:20:15.460323+0000","last_active":"2024-03-26T05:21:51.780619+0000","last_peered":"2024-03-26T05:21:51.780619+0000","last_clean":"2024-03-26T05:21:51.780619+0000","last_became_active":"2024-03-26T05:20:15.374306+0000","last_became_peered":"2024-03-26T05:20:15.374306+0000","last_unstale":"2024-03-26T05:21:51.780619+0000","last_undegraded":"2024-03-26T05:21:51.780619+0000","last_fullsized":"2024-03-26T05:21:51.780619+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'16","last_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_deep_scrub":"321'16","last_deep_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.995735+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":571,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.813254+0000","last_change":"2024-03-26T05:20:30.329198+0000","last_active":"2024-03-26T05:21:55.813254+0000","last_peered":"2024-03-26T05:21:55.813254+0000","last_clean":"2024-03-26T05:21:55.813254+0000","last_became_active":"2024-03-26T05:20:30.328801+0000","last_became_peered":"2024-03-26T05:20:30.328801+0000","last_unstale":"2024-03-26T05:21:55.813254+0000","last_undegraded":"2024-03-26T05:21:55.813254+0000","last_fullsized":"2024-03-26T05:21:55.813254+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:50.218114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:00.635308+0000","last_clean_scrub_stamp":"2024-03-26T05:18:50.218114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":814,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.817631+0000","last_change":"2024-03-26T05:20:15.374216+0000","last_active":"2024-03-26T05:21:55.817631+0000","last_peered":"2024-03-26T05:21:55.817631+0000","last_clean":"2024-03-26T05:21:55.817631+0000","last_became_active":"2024-03-26T05:20:15.373862+0000","last_became_peered":"2024-03-26T05:20:15.373862+0000","last_unstale":"2024-03-26T05:21:55.817631+0000","last_undegraded":"2024-03-26T05:21:55.817631+0000","last_fullsized":"2024-03-26T05:21:55.817631+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_clean_scrub_stamp":"2024-03-26T05:12:18.959687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:56.444109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":197,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.813396+0000","last_change":"2024-03-26T05:20:30.339972+0000","last_active":"2024-03-26T05:21:55.813396+0000","last_peered":"2024-03-26T05:21:55.813396+0000","last_clean":"2024-03-26T05:21:55.813396+0000","last_became_active":"2024-03-26T05:20:30.337386+0000","last_became_peered":"2024-03-26T05:20:30.337386+0000","last_unstale":"2024-03-26T05:21:55.813396+0000","last_undegraded":"2024-03-26T05:21:55.813396+0000","last_fullsized":"2024-03-26T05:21:55.813396+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:06.363747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:20:06.363747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.15","version":"479'11681","reported_seq":25458,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.074732+0000","last_change":"2024-03-26T05:21:41.788950+0000","last_active":"2024-03-26T05:21:56.074732+0000","last_peered":"2024-03-26T05:21:56.074732+0000","last_clean":"2024-03-26T05:21:56.074732+0000","last_became_active":"2024-03-26T05:21:41.755579+0000","last_became_peered":"2024-03-26T05:21:41.755579+0000","last_unstale":"2024-03-26T05:21:56.074732+0000","last_undegraded":"2024-03-26T05:21:56.074732+0000","last_fullsized":"2024-03-26T05:21:56.074732+0000","mapping_epoch":470,"log_start":"433'7126","ondisk_log_start":"433'7126","created":460,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":5,"last_scrub":"427'5138","last_scrub_stamp":"2024-03-26T05:19:43.145684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.145684+0000","objects_scrubbed":5110,"log_size":4555,"log_dups_size":0,"ondisk_log_size":4555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:57.526033+0000","scrub_duration":796,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84705280,"num_objects":1293,"num_object_clones":0,"num_object_copies":3879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6229,"num_write_kb":240672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3848,"num_bytes_recovered":252149760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"443'33","reported_seq":772,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780907+0000","last_change":"2024-03-26T05:20:15.472237+0000","last_active":"2024-03-26T05:21:51.780907+0000","last_peered":"2024-03-26T05:21:51.780907+0000","last_clean":"2024-03-26T05:21:51.780907+0000","last_became_active":"2024-03-26T05:20:15.378661+0000","last_became_peered":"2024-03-26T05:20:15.378661+0000","last_unstale":"2024-03-26T05:21:51.780907+0000","last_undegraded":"2024-03-26T05:21:51.780907+0000","last_fullsized":"2024-03-26T05:21:51.780907+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:13:22.026354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:22.026354+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":572,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.809622+0000","last_change":"2024-03-26T05:20:52.465761+0000","last_active":"2024-03-26T05:21:51.809622+0000","last_peered":"2024-03-26T05:21:51.809622+0000","last_clean":"2024-03-26T05:21:51.809622+0000","last_became_active":"2024-03-26T05:20:30.331329+0000","last_became_peered":"2024-03-26T05:20:30.331329+0000","last_unstale":"2024-03-26T05:21:51.809622+0000","last_undegraded":"2024-03-26T05:21:51.809622+0000","last_fullsized":"2024-03-26T05:21:51.809622+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:52.465623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:20:52.465623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.189365+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":359,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.819226+0000","last_change":"2024-03-26T05:20:30.343688+0000","last_active":"2024-03-26T05:21:56.819226+0000","last_peered":"2024-03-26T05:21:56.819226+0000","last_clean":"2024-03-26T05:21:56.819226+0000","last_became_active":"2024-03-26T05:20:30.343356+0000","last_became_peered":"2024-03-26T05:20:30.343356+0000","last_unstale":"2024-03-26T05:21:56.819226+0000","last_undegraded":"2024-03-26T05:21:56.819226+0000","last_fullsized":"2024-03-26T05:21:56.819226+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:56.259964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:56.259964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.658076+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":903,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.820095+0000","last_change":"2024-03-26T05:19:43.520950+0000","last_active":"2024-03-26T05:21:56.820095+0000","last_peered":"2024-03-26T05:21:56.820095+0000","last_clean":"2024-03-26T05:21:56.820095+0000","last_became_active":"2024-03-26T05:15:32.640052+0000","last_became_peered":"2024-03-26T05:15:32.640052+0000","last_unstale":"2024-03-26T05:21:56.820095+0000","last_undegraded":"2024-03-26T05:21:56.820095+0000","last_fullsized":"2024-03-26T05:21:56.820095+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:43.520824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.520824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.658061+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.16","version":"479'11791","reported_seq":17353,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.912061+0000","last_change":"2024-03-26T05:21:47.777567+0000","last_active":"2024-03-26T05:21:55.912061+0000","last_peered":"2024-03-26T05:21:55.912061+0000","last_clean":"2024-03-26T05:21:55.912061+0000","last_became_active":"2024-03-26T05:21:47.760547+0000","last_became_peered":"2024-03-26T05:21:47.760547+0000","last_unstale":"2024-03-26T05:21:55.912061+0000","last_undegraded":"2024-03-26T05:21:55.912061+0000","last_fullsized":"2024-03-26T05:21:55.912061+0000","mapping_epoch":474,"log_start":"437'7227","ondisk_log_start":"437'7227","created":460,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:58.419091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82018304,"num_objects":1252,"num_object_clones":0,"num_object_copies":3756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6310,"num_write_kb":241952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1795,"num_bytes_recovered":117637120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"443'63","reported_seq":1006,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.781132+0000","last_change":"2024-03-26T05:19:48.357831+0000","last_active":"2024-03-26T05:21:51.781132+0000","last_peered":"2024-03-26T05:21:51.781132+0000","last_clean":"2024-03-26T05:21:51.781132+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:21:51.781132+0000","last_undegraded":"2024-03-26T05:21:51.781132+0000","last_fullsized":"2024-03-26T05:21:51.781132+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'51","last_scrub_stamp":"2024-03-26T05:19:48.356743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:19:48.356743+0000","objects_scrubbed":11,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":98,"num_read_kb":85,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":903,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.782278+0000","last_change":"2024-03-26T05:19:43.320734+0000","last_active":"2024-03-26T05:21:51.782278+0000","last_peered":"2024-03-26T05:21:51.782278+0000","last_clean":"2024-03-26T05:21:51.782278+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:21:51.782278+0000","last_undegraded":"2024-03-26T05:21:51.782278+0000","last_fullsized":"2024-03-26T05:21:51.782278+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:43.320603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.320603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":295,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.808516+0000","last_change":"2024-03-26T05:20:46.537730+0000","last_active":"2024-03-26T05:21:51.808516+0000","last_peered":"2024-03-26T05:21:51.808516+0000","last_clean":"2024-03-26T05:21:51.808516+0000","last_became_active":"2024-03-26T05:20:15.370449+0000","last_became_peered":"2024-03-26T05:20:15.370449+0000","last_unstale":"2024-03-26T05:21:51.808516+0000","last_undegraded":"2024-03-26T05:21:51.808516+0000","last_fullsized":"2024-03-26T05:21:51.808516+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:46.537613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:20:46.537613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:21:53.610982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":882,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.780824+0000","last_change":"2024-03-26T05:19:53.622022+0000","last_active":"2024-03-26T05:21:51.780824+0000","last_peered":"2024-03-26T05:21:51.780824+0000","last_clean":"2024-03-26T05:21:51.780824+0000","last_became_active":"2024-03-26T05:15:32.637717+0000","last_became_peered":"2024-03-26T05:15:32.637717+0000","last_unstale":"2024-03-26T05:21:51.780824+0000","last_undegraded":"2024-03-26T05:21:51.780824+0000","last_fullsized":"2024-03-26T05:21:51.780824+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:53.621892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:53.621892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"479'11535","reported_seq":24539,"reported_epoch":479,"state":"active+undersized+remapped+backfilling","last_fresh":"2024-03-26T05:21:56.488884+0000","last_change":"2024-03-26T05:21:55.938240+0000","last_active":"2024-03-26T05:21:56.488884+0000","last_peered":"2024-03-26T05:21:56.488884+0000","last_clean":"2024-03-26T05:18:36.613885+0000","last_became_active":"2024-03-26T05:21:25.910751+0000","last_became_peered":"2024-03-26T05:21:25.910751+0000","last_unstale":"2024-03-26T05:21:56.488884+0000","last_undegraded":"2024-03-26T05:21:56.488884+0000","last_fullsized":"2024-03-26T05:21:25.875743+0000","mapping_epoch":462,"log_start":"434'6999","ondisk_log_start":"434'6999","created":460,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4536,"log_dups_size":0,"ondisk_log_size":4536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78938112,"num_objects":1205,"num_object_clones":0,"num_object_copies":3615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1033,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6179,"num_write_kb":236256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2254,"num_bytes_recovered":147718144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[1,5],"avail_no_missing":["1","0","5","7"],"object_location_counts":[{"shards":"1,5","objects":1205}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"443'27","reported_seq":766,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-26T05:21:51.810029+0000","last_change":"2024-03-26T05:21:01.539850+0000","last_active":"2024-03-26T05:21:51.810029+0000","last_peered":"2024-03-26T05:21:51.810029+0000","last_clean":"2024-03-26T05:21:51.810029+0000","last_became_active":"2024-03-26T05:20:15.377639+0000","last_became_peered":"2024-03-26T05:20:15.377639+0000","last_unstale":"2024-03-26T05:21:51.810029+0000","last_undegraded":"2024-03-26T05:21:51.810029+0000","last_fullsized":"2024-03-26T05:21:51.810029+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'27","last_scrub_stamp":"2024-03-26T05:21:01.539708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:21:01.539708+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:15.954464+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":926,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T05:21:56.818531+0000","last_change":"2024-03-26T05:20:30.349760+0000","last_active":"2024-03-26T05:21:56.818531+0000","last_peered":"2024-03-26T05:21:56.818531+0000","last_clean":"2024-03-26T05:21:56.818531+0000","last_became_active":"2024-03-26T05:20:30.349429+0000","last_became_peered":"2024-03-26T05:20:30.349429+0000","last_unstale":"2024-03-26T05:21:56.818531+0000","last_undegraded":"2024-03-26T05:21:56.818531+0000","last_fullsized":"2024-03-26T05:21:56.818531+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:34.300011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:34.300011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:01.657953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":922,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.812445+0000","last_change":"2024-03-26T05:20:30.337232+0000","last_active":"2024-03-26T05:21:55.812445+0000","last_peered":"2024-03-26T05:21:55.812445+0000","last_clean":"2024-03-26T05:21:55.812445+0000","last_became_active":"2024-03-26T05:20:30.336902+0000","last_became_peered":"2024-03-26T05:20:30.336902+0000","last_unstale":"2024-03-26T05:21:55.812445+0000","last_undegraded":"2024-03-26T05:21:55.812445+0000","last_fullsized":"2024-03-26T05:21:55.812445+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:51.618208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:51.618208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":598,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T05:21:55.813681+0000","last_change":"2024-03-26T05:20:30.331664+0000","last_active":"2024-03-26T05:21:55.813681+0000","last_peered":"2024-03-26T05:21:55.813681+0000","last_clean":"2024-03-26T05:21:55.813681+0000","last_became_active":"2024-03-26T05:20:30.327890+0000","last_became_peered":"2024-03-26T05:20:30.327890+0000","last_unstale":"2024-03-26T05:21:55.813681+0000","last_undegraded":"2024-03-26T05:21:55.813681+0000","last_fullsized":"2024-03-26T05:21:55.813681+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:53.055153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:53.055153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":26,"stat_sum":{"num_bytes":2686517272,"num_objects":40994,"num_object_clones":0,"num_object_copies":122982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8659,"num_objects_unfound":0,"num_objects_dirty":40994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201222,"num_write_kb":7750849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44369,"num_bytes_recovered":2906849304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8857202688,"data_stored":8857190472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114296,"ondisk_log_size":114296,"up":78,"acting":77,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":448,"num_read_kb":443,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697708,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":54,"num_write":34,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":649700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1832,"num_read_kb":1557,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":4554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":13020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1275,"ondisk_log_size":1275,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":335,"seq":1438814044238,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1919564,"kb_used_data":1415108,"kb_used_omap":4,"kb_used_meta":504443,"kb_avail":91825588,"statfs":{"total":95995035648,"available":94029402112,"internally_reserved":0,"allocated":1449070592,"data_stored":1447903290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":516550312},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332364,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1994128,"kb_used_data":1286852,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":91751024,"statfs":{"total":95995035648,"available":93953048576,"internally_reserved":0,"allocated":1317736448,"data_stored":1316568418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":724236742},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575142,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2517884,"kb_used_data":1624560,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":91227268,"statfs":{"total":95995035648,"available":93416722432,"internally_reserved":0,"allocated":1663549440,"data_stored":1662345689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":914749893},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,2,4,3,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738667,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2302752,"kb_used_data":1378900,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":91442400,"statfs":{"total":95995035648,"available":93637017600,"internally_reserved":0,"allocated":1411993600,"data_stored":1410818218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411617,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1952964,"kb_used_data":1249788,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":91792188,"statfs":{"total":95995035648,"available":93995200512,"internally_reserved":0,"allocated":1279782912,"data_stored":1278622855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117275,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1953888,"kb_used_data":1096664,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":91791264,"statfs":{"total":95995035648,"available":93994254336,"internally_reserved":0,"allocated":1122983936,"data_stored":1121796384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":877787591},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":176093659409,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1406861312,"data_stored":1406861312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1651970048,"data_stored":1651965976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1253969920,"data_stored":1253965848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1110310912,"data_stored":1110310912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":699858944,"data_stored":699858944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1310396416,"data_stored":1310392344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1423835136,"data_stored":1423835136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:22:01.514 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-03-26T05:22:01.557 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:01.600 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:01.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:01.702 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:01.731 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:01.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:01.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:01.833 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:01.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:01.874 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:01.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:01.975 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:02.010 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:02.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:02.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:02.112 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:02.151 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:02.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:02.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:02.281 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:02.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:02.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:02.383 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:02.425 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:02.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:02.526 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:02.555 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:02.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:02.657 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:02.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:02.757 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:02.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:02.858 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:02.907 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:02.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:02.929 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:02 smithi080 ceph-mon[40404]: osdmap e481: 8 total, 7 up, 6 in 2024-03-26T05:22:02.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:02 smithi080 ceph-mon[29691]: osdmap e481: 8 total, 7 up, 6 in 2024-03-26T05:22:03.009 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:03.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:02 smithi132 ceph-mon[33500]: osdmap e481: 8 total, 7 up, 6 in 2024-03-26T05:22:03.048 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:03.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:03.151 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:03.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:03.190 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:03.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:03.291 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:03.318 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:03.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:03.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:03.419 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:03.469 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:03.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:03.570 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:03.599 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:03.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:03.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:03.701 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:03.729 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:03.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:03.831 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:03.873 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:03.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:03.975 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:04.002 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:04.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:04.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:04.023 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:03 smithi080 ceph-mon[29691]: pgmap v1025: 168 pgs: 1 active+undersized+remapped+backfilling, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.34k op/s; 6409/107472 objects misplaced (5.963%); 12 MiB/s, 187 objects/s recovering 2024-03-26T05:22:04.023 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:03 smithi080 ceph-mon[40404]: pgmap v1025: 168 pgs: 1 active+undersized+remapped+backfilling, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.34k op/s; 6409/107472 objects misplaced (5.963%); 12 MiB/s, 187 objects/s recovering 2024-03-26T05:22:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:03 smithi132 ceph-mon[33500]: pgmap v1025: 168 pgs: 1 active+undersized+remapped+backfilling, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.34k op/s; 6409/107472 objects misplaced (5.963%); 12 MiB/s, 187 objects/s recovering 2024-03-26T05:22:04.104 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:04.133 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:04.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:04.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:04.235 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:04.277 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:04.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:04.314 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:22:04.315 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:22:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:22:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:22:04.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:04.380 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:22:04.406 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:04.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:04.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:04.438 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:22:04.475 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:22:04.496 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:22:04.508 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:04.536 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:04.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:04.546 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:22:04.584 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:22:04.585 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:22:04.585 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:22:04.585 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:22:04.585 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:22:04.585 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:22:04.585 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:22:04.585 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:22:04.585 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:22:04.638 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:04.681 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:04.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:04.773 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:22:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a[63840]: logger=cleanup t=2024-03-26T05:22:04.358824294Z level=info msg="Completed cleanup jobs" duration=3.351299ms 2024-03-26T05:22:04.783 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:04.788 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:22:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:04.810 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:04.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:04.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:04 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4264535429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:22:04.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:04 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2835441595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:22:04.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:04 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:22:04.838 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:04 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4264535429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:22:04.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:04 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2835441595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:22:04.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:04 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:22:04.911 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:04.951 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:04.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:05.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:04 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4264535429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:22:05.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:04 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2835441595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:22:05.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:04 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:22:05.052 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:05.079 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:05.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:05.181 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:05.206 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:05.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:05.308 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:05.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:05.332 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:05.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:05.433 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:05.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:05.458 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:05.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:05.559 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:05.583 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:05.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:05.684 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:05.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:05.708 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:05.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:05.748 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:22:05.748 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:22:05.809 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:05.839 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:05.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:05.940 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:05.980 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:05.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:05.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:05 smithi080 ceph-mon[40404]: pgmap v1026: 168 pgs: 1 active+undersized+remapped+backfilling, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 892 op/s; 6409/107472 objects misplaced (5.963%); 9.4 MiB/s, 151 objects/s recovering 2024-03-26T05:22:05.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:05 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:22:05.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:05 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:22:05.988 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:05 smithi080 ceph-mon[40404]: osdmap e482: 8 total, 7 up, 6 in 2024-03-26T05:22:05.988 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:05 smithi080 ceph-mon[29691]: pgmap v1026: 168 pgs: 1 active+undersized+remapped+backfilling, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 892 op/s; 6409/107472 objects misplaced (5.963%); 9.4 MiB/s, 151 objects/s recovering 2024-03-26T05:22:05.988 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:05 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:22:05.988 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:05 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:22:05.988 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:05 smithi080 ceph-mon[29691]: osdmap e482: 8 total, 7 up, 6 in 2024-03-26T05:22:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:05 smithi132 ceph-mon[33500]: pgmap v1026: 168 pgs: 1 active+undersized+remapped+backfilling, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 892 op/s; 6409/107472 objects misplaced (5.963%); 9.4 MiB/s, 151 objects/s recovering 2024-03-26T05:22:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:05 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:22:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:05 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:22:06.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:05 smithi132 ceph-mon[33500]: osdmap e482: 8 total, 7 up, 6 in 2024-03-26T05:22:06.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:22:06 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:22:06.083 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:06.110 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:06.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:06.212 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:06.238 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:06.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:06.267 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:22:05 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:22:05.996+0000 7f2fcc605700 -1 mgr get_metadata_python Requested missing service osd.4 2024-03-26T05:22:06.339 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:06.363 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:06.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:06.465 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:06.489 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:06.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:06.515 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7] out_osds: [4, 5] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T05:22:06.515 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:22:06.516 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-03-26T05:22:06.516 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:22:06.591 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:06.614 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:06.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:06.716 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:06.741 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:06.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:06.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:06.843 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:06.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[29691]: 1.2 scrub starts 2024-03-26T05:22:06.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[29691]: 1.2 scrub ok 2024-03-26T05:22:06.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[29691]: 11.8 scrub starts 2024-03-26T05:22:06.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[29691]: 11.14 scrub starts 2024-03-26T05:22:06.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[29691]: osdmap e483: 8 total, 7 up, 6 in 2024-03-26T05:22:06.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[29691]: 11.8 scrub ok 2024-03-26T05:22:06.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:06.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[29691]: 11.13 scrub starts 2024-03-26T05:22:06.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[29691]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:22:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[40404]: 1.2 scrub starts 2024-03-26T05:22:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[40404]: 1.2 scrub ok 2024-03-26T05:22:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[40404]: 11.8 scrub starts 2024-03-26T05:22:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[40404]: 11.14 scrub starts 2024-03-26T05:22:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[40404]: osdmap e483: 8 total, 7 up, 6 in 2024-03-26T05:22:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[40404]: 11.8 scrub ok 2024-03-26T05:22:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[40404]: 11.13 scrub starts 2024-03-26T05:22:06.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:06 smithi080 ceph-mon[40404]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:22:06.872 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:06.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:06.974 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:07.030 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:07.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:06 smithi132 ceph-mon[33500]: 1.2 scrub starts 2024-03-26T05:22:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:06 smithi132 ceph-mon[33500]: 1.2 scrub ok 2024-03-26T05:22:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:06 smithi132 ceph-mon[33500]: 11.8 scrub starts 2024-03-26T05:22:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:06 smithi132 ceph-mon[33500]: 11.14 scrub starts 2024-03-26T05:22:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:06 smithi132 ceph-mon[33500]: osdmap e483: 8 total, 7 up, 6 in 2024-03-26T05:22:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:06 smithi132 ceph-mon[33500]: 11.8 scrub ok 2024-03-26T05:22:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:06 smithi132 ceph-mon[33500]: 11.13 scrub starts 2024-03-26T05:22:07.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:06 smithi132 ceph-mon[33500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T05:22:07.050 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:07.119 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:22:07.120 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:22:07.132 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:07.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:07.182 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:07.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:07.283 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:07.320 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:07.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:07.422 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:07.460 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:07.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:07.562 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:07.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:07.588 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:07.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:07.689 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:07.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:07.713 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:07.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:07.749 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:22:07.815 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:07.841 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:07.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:07.943 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:07.972 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:07.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:07 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:07 smithi132 ceph-mon[33500]: pgmap v1029: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.43k op/s; 5468/96573 objects misplaced (5.662%); 16 MiB/s, 250 objects/s recovering 2024-03-26T05:22:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:07 smithi132 ceph-mon[33500]: 11.17 scrub starts 2024-03-26T05:22:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:07 smithi132 ceph-mon[33500]: 11.17 scrub ok 2024-03-26T05:22:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:07 smithi132 ceph-mon[33500]: 11.13 scrub ok 2024-03-26T05:22:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:07 smithi132 ceph-mon[33500]: osdmap e484: 8 total, 7 up, 6 in 2024-03-26T05:22:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:07 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:22:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:07 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:22:08.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:07 smithi132 ceph-mon[33500]: 11.2 scrub starts 2024-03-26T05:22:08.074 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:08.108 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:08.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:08.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:08.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[29691]: pgmap v1029: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.43k op/s; 5468/96573 objects misplaced (5.662%); 16 MiB/s, 250 objects/s recovering 2024-03-26T05:22:08.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[29691]: 11.17 scrub starts 2024-03-26T05:22:08.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[29691]: 11.17 scrub ok 2024-03-26T05:22:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[29691]: 11.13 scrub ok 2024-03-26T05:22:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[29691]: osdmap e484: 8 total, 7 up, 6 in 2024-03-26T05:22:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:22:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:22:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[29691]: 11.2 scrub starts 2024-03-26T05:22:08.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:08.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[40404]: pgmap v1029: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.43k op/s; 5468/96573 objects misplaced (5.662%); 16 MiB/s, 250 objects/s recovering 2024-03-26T05:22:08.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[40404]: 11.17 scrub starts 2024-03-26T05:22:08.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[40404]: 11.17 scrub ok 2024-03-26T05:22:08.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[40404]: 11.13 scrub ok 2024-03-26T05:22:08.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[40404]: osdmap e484: 8 total, 7 up, 6 in 2024-03-26T05:22:08.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:22:08.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:22:08.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:07 smithi080 ceph-mon[40404]: 11.2 scrub starts 2024-03-26T05:22:08.208 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:08.256 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:08.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:08.358 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:08.397 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:08.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:08.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:08.500 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:08.530 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:08.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:08.635 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:08.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:08.659 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:08.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:08.734 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:22:08.761 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:08.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:08.789 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:08.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:08.890 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:08.915 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:08.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:09.017 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:09.067 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:09.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:09.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:09.084 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:08 smithi080 ceph-mon[29691]: 11.9 scrub starts 2024-03-26T05:22:09.084 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:08 smithi080 ceph-mon[29691]: 11.1 scrub starts 2024-03-26T05:22:09.085 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:08 smithi080 ceph-mon[29691]: 11.2 scrub ok 2024-03-26T05:22:09.085 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:08 smithi080 ceph-mon[29691]: 11.0 scrub starts 2024-03-26T05:22:09.085 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:08 smithi080 ceph-mon[29691]: 11.1 scrub ok 2024-03-26T05:22:09.085 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:08 smithi080 ceph-mon[40404]: 11.9 scrub starts 2024-03-26T05:22:09.085 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:08 smithi080 ceph-mon[40404]: 11.1 scrub starts 2024-03-26T05:22:09.085 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:08 smithi080 ceph-mon[40404]: 11.2 scrub ok 2024-03-26T05:22:09.085 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:08 smithi080 ceph-mon[40404]: 11.0 scrub starts 2024-03-26T05:22:09.085 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:08 smithi080 ceph-mon[40404]: 11.1 scrub ok 2024-03-26T05:22:09.169 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:09.202 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:09.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:09.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:08 smithi132 ceph-mon[33500]: 11.9 scrub starts 2024-03-26T05:22:09.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:08 smithi132 ceph-mon[33500]: 11.1 scrub starts 2024-03-26T05:22:09.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:08 smithi132 ceph-mon[33500]: 11.2 scrub ok 2024-03-26T05:22:09.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:08 smithi132 ceph-mon[33500]: 11.0 scrub starts 2024-03-26T05:22:09.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:08 smithi132 ceph-mon[33500]: 11.1 scrub ok 2024-03-26T05:22:09.303 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:09.355 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:09.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:09.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:09.457 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:09.481 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:09.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:09.582 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:09.623 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:09.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:09.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:09.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:09.754 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:09.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:09.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:09.859 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:09.903 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:09.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:10.005 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:10.031 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:10.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:10.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:10.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[29691]: 11.9 scrub ok 2024-03-26T05:22:10.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[29691]: pgmap v1031: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 4825/92061 objects misplaced (5.241%) 2024-03-26T05:22:10.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[29691]: 11.4 scrub starts 2024-03-26T05:22:10.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[29691]: 11.0 scrub ok 2024-03-26T05:22:10.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[29691]: 11.4 scrub ok 2024-03-26T05:22:10.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[29691]: 11.3 scrub starts 2024-03-26T05:22:10.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[29691]: 11.3 scrub ok 2024-03-26T05:22:10.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[29691]: 11.7 scrub starts 2024-03-26T05:22:10.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[29691]: 11.7 scrub ok 2024-03-26T05:22:10.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[29691]: 11.15 scrub starts 2024-03-26T05:22:10.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[29691]: 1.1 scrub starts 2024-03-26T05:22:10.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[29691]: 1.1 scrub ok 2024-03-26T05:22:10.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[40404]: 11.9 scrub ok 2024-03-26T05:22:10.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[40404]: pgmap v1031: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 4825/92061 objects misplaced (5.241%) 2024-03-26T05:22:10.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[40404]: 11.4 scrub starts 2024-03-26T05:22:10.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[40404]: 11.0 scrub ok 2024-03-26T05:22:10.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[40404]: 11.4 scrub ok 2024-03-26T05:22:10.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[40404]: 11.3 scrub starts 2024-03-26T05:22:10.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[40404]: 11.3 scrub ok 2024-03-26T05:22:10.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[40404]: 11.7 scrub starts 2024-03-26T05:22:10.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[40404]: 11.7 scrub ok 2024-03-26T05:22:10.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[40404]: 11.15 scrub starts 2024-03-26T05:22:10.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[40404]: 1.1 scrub starts 2024-03-26T05:22:10.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:09 smithi080 ceph-mon[40404]: 1.1 scrub ok 2024-03-26T05:22:10.132 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:10.173 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:10.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:10.274 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:10.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:09 smithi132 ceph-mon[33500]: 11.9 scrub ok 2024-03-26T05:22:10.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:09 smithi132 ceph-mon[33500]: pgmap v1031: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 4825/92061 objects misplaced (5.241%) 2024-03-26T05:22:10.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:09 smithi132 ceph-mon[33500]: 11.4 scrub starts 2024-03-26T05:22:10.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:09 smithi132 ceph-mon[33500]: 11.0 scrub ok 2024-03-26T05:22:10.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:09 smithi132 ceph-mon[33500]: 11.4 scrub ok 2024-03-26T05:22:10.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:09 smithi132 ceph-mon[33500]: 11.3 scrub starts 2024-03-26T05:22:10.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:09 smithi132 ceph-mon[33500]: 11.3 scrub ok 2024-03-26T05:22:10.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:09 smithi132 ceph-mon[33500]: 11.7 scrub starts 2024-03-26T05:22:10.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:09 smithi132 ceph-mon[33500]: 11.7 scrub ok 2024-03-26T05:22:10.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:09 smithi132 ceph-mon[33500]: 11.15 scrub starts 2024-03-26T05:22:10.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:09 smithi132 ceph-mon[33500]: 1.1 scrub starts 2024-03-26T05:22:10.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:09 smithi132 ceph-mon[33500]: 1.1 scrub ok 2024-03-26T05:22:10.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:10.298 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:10.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:10.400 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:10.443 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:10.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:10.545 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:10.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:10.586 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:10.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:10.687 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:10.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:10.711 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:10.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:10.812 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:10.856 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:10.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:10.957 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:10.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:10.989 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:10.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:11.091 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:11.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[29691]: 11.12 scrub starts 2024-03-26T05:22:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[29691]: 11.12 scrub ok 2024-03-26T05:22:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[29691]: 11.14 scrub starts 2024-03-26T05:22:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[29691]: 11.15 scrub ok 2024-03-26T05:22:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[29691]: 4.2 scrub starts 2024-03-26T05:22:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[29691]: 4.2 scrub ok 2024-03-26T05:22:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[29691]: 11.14 scrub ok 2024-03-26T05:22:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[29691]: pgmap v1032: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 4616/90879 objects misplaced (5.079%) 2024-03-26T05:22:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[29691]: 11.18 scrub starts 2024-03-26T05:22:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[29691]: 11.6 scrub starts 2024-03-26T05:22:11.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[40404]: 11.12 scrub starts 2024-03-26T05:22:11.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[40404]: 11.12 scrub ok 2024-03-26T05:22:11.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[40404]: 11.14 scrub starts 2024-03-26T05:22:11.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[40404]: 11.15 scrub ok 2024-03-26T05:22:11.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[40404]: 4.2 scrub starts 2024-03-26T05:22:11.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[40404]: 4.2 scrub ok 2024-03-26T05:22:11.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[40404]: 11.14 scrub ok 2024-03-26T05:22:11.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[40404]: pgmap v1032: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 4616/90879 objects misplaced (5.079%) 2024-03-26T05:22:11.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[40404]: 11.18 scrub starts 2024-03-26T05:22:11.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:10 smithi080 ceph-mon[40404]: 11.6 scrub starts 2024-03-26T05:22:11.129 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:11.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:11.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:11.231 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:11.257 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:11.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:11.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:10 smithi132 ceph-mon[33500]: 11.12 scrub starts 2024-03-26T05:22:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:10 smithi132 ceph-mon[33500]: 11.12 scrub ok 2024-03-26T05:22:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:10 smithi132 ceph-mon[33500]: 11.14 scrub starts 2024-03-26T05:22:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:10 smithi132 ceph-mon[33500]: 11.15 scrub ok 2024-03-26T05:22:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:10 smithi132 ceph-mon[33500]: 4.2 scrub starts 2024-03-26T05:22:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:10 smithi132 ceph-mon[33500]: 4.2 scrub ok 2024-03-26T05:22:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:10 smithi132 ceph-mon[33500]: 11.14 scrub ok 2024-03-26T05:22:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:10 smithi132 ceph-mon[33500]: pgmap v1032: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 4616/90879 objects misplaced (5.079%) 2024-03-26T05:22:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:10 smithi132 ceph-mon[33500]: 11.18 scrub starts 2024-03-26T05:22:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:10 smithi132 ceph-mon[33500]: 11.6 scrub starts 2024-03-26T05:22:11.359 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:11.402 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:11.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:11.504 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:11.545 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:11.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:11.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:11.647 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:11.674 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:11.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:11.776 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:11.823 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:11.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:11.925 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:11.955 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:11.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:12.056 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:12.091 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:12.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:12.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[29691]: 11.5 scrub starts 2024-03-26T05:22:12.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[29691]: 11.5 scrub ok 2024-03-26T05:22:12.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[29691]: 1.7 scrub starts 2024-03-26T05:22:12.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[29691]: 1.7 scrub ok 2024-03-26T05:22:12.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[29691]: 11.18 scrub ok 2024-03-26T05:22:12.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[29691]: 11.6 scrub ok 2024-03-26T05:22:12.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[29691]: 5.c scrub starts 2024-03-26T05:22:12.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[29691]: 5.c scrub ok 2024-03-26T05:22:12.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[29691]: 11.19 scrub starts 2024-03-26T05:22:12.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[29691]: osdmap e485: 8 total, 7 up, 6 in 2024-03-26T05:22:12.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[40404]: 11.5 scrub starts 2024-03-26T05:22:12.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[40404]: 11.5 scrub ok 2024-03-26T05:22:12.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[40404]: 1.7 scrub starts 2024-03-26T05:22:12.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[40404]: 1.7 scrub ok 2024-03-26T05:22:12.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[40404]: 11.18 scrub ok 2024-03-26T05:22:12.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[40404]: 11.6 scrub ok 2024-03-26T05:22:12.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[40404]: 5.c scrub starts 2024-03-26T05:22:12.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[40404]: 5.c scrub ok 2024-03-26T05:22:12.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[40404]: 11.19 scrub starts 2024-03-26T05:22:12.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:11 smithi080 ceph-mon[40404]: osdmap e485: 8 total, 7 up, 6 in 2024-03-26T05:22:12.192 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:12.227 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:12.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:12.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:11 smithi132 ceph-mon[33500]: 11.5 scrub starts 2024-03-26T05:22:12.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:11 smithi132 ceph-mon[33500]: 11.5 scrub ok 2024-03-26T05:22:12.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:11 smithi132 ceph-mon[33500]: 1.7 scrub starts 2024-03-26T05:22:12.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:11 smithi132 ceph-mon[33500]: 1.7 scrub ok 2024-03-26T05:22:12.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:11 smithi132 ceph-mon[33500]: 11.18 scrub ok 2024-03-26T05:22:12.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:11 smithi132 ceph-mon[33500]: 11.6 scrub ok 2024-03-26T05:22:12.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:11 smithi132 ceph-mon[33500]: 5.c scrub starts 2024-03-26T05:22:12.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:11 smithi132 ceph-mon[33500]: 5.c scrub ok 2024-03-26T05:22:12.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:11 smithi132 ceph-mon[33500]: 11.19 scrub starts 2024-03-26T05:22:12.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:11 smithi132 ceph-mon[33500]: osdmap e485: 8 total, 7 up, 6 in 2024-03-26T05:22:12.329 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:12.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:12.370 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:12.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:12.471 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:12.494 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:12.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:12.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:12.596 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:12.640 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:12.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:12.742 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:12.778 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:12.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:12.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:12.880 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:12.920 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:12.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:13.021 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:13.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:13.058 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:13.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:13.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[29691]: 11.16 scrub starts 2024-03-26T05:22:13.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[29691]: 11.16 scrub ok 2024-03-26T05:22:13.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[29691]: 11.19 scrub ok 2024-03-26T05:22:13.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[29691]: 3.10 scrub starts 2024-03-26T05:22:13.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[29691]: 3.10 scrub ok 2024-03-26T05:22:13.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[29691]: pgmap v1034: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 3785/77472 objects misplaced (4.886%) 2024-03-26T05:22:13.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[29691]: 3.1b scrub starts 2024-03-26T05:22:13.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[29691]: 3.1b scrub ok 2024-03-26T05:22:13.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[40404]: 11.16 scrub starts 2024-03-26T05:22:13.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[40404]: 11.16 scrub ok 2024-03-26T05:22:13.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[40404]: 11.19 scrub ok 2024-03-26T05:22:13.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[40404]: 3.10 scrub starts 2024-03-26T05:22:13.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[40404]: 3.10 scrub ok 2024-03-26T05:22:13.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[40404]: pgmap v1034: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 3785/77472 objects misplaced (4.886%) 2024-03-26T05:22:13.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[40404]: 3.1b scrub starts 2024-03-26T05:22:13.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:12 smithi080 ceph-mon[40404]: 3.1b scrub ok 2024-03-26T05:22:13.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:13.200 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:13.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:13.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:13.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:12 smithi132 ceph-mon[33500]: 11.16 scrub starts 2024-03-26T05:22:13.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:12 smithi132 ceph-mon[33500]: 11.16 scrub ok 2024-03-26T05:22:13.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:12 smithi132 ceph-mon[33500]: 11.19 scrub ok 2024-03-26T05:22:13.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:12 smithi132 ceph-mon[33500]: 3.10 scrub starts 2024-03-26T05:22:13.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:12 smithi132 ceph-mon[33500]: 3.10 scrub ok 2024-03-26T05:22:13.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:12 smithi132 ceph-mon[33500]: pgmap v1034: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 3785/77472 objects misplaced (4.886%) 2024-03-26T05:22:13.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:12 smithi132 ceph-mon[33500]: 3.1b scrub starts 2024-03-26T05:22:13.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:12 smithi132 ceph-mon[33500]: 3.1b scrub ok 2024-03-26T05:22:13.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:13.327 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:13.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:13.429 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:13.475 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:13.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:13.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:13.577 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:13.605 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:13.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:13.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:13.707 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:13.749 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:13.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:13.854 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:13.882 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:13.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:13.983 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:14.011 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:14.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:14.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:14.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:13 smithi080 ceph-mon[29691]: 3.19 scrub starts 2024-03-26T05:22:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:13 smithi080 ceph-mon[29691]: 3.19 scrub ok 2024-03-26T05:22:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:13 smithi080 ceph-mon[29691]: 5.e scrub starts 2024-03-26T05:22:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:13 smithi080 ceph-mon[29691]: 5.e scrub ok 2024-03-26T05:22:14.117 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:22:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:22:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:22:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:13 smithi080 ceph-mon[40404]: 3.19 scrub starts 2024-03-26T05:22:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:13 smithi080 ceph-mon[40404]: 3.19 scrub ok 2024-03-26T05:22:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:13 smithi080 ceph-mon[40404]: 5.e scrub starts 2024-03-26T05:22:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:13 smithi080 ceph-mon[40404]: 5.e scrub ok 2024-03-26T05:22:14.154 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:14.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:14.256 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:14.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:13 smithi132 ceph-mon[33500]: 3.19 scrub starts 2024-03-26T05:22:14.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:13 smithi132 ceph-mon[33500]: 3.19 scrub ok 2024-03-26T05:22:14.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:13 smithi132 ceph-mon[33500]: 5.e scrub starts 2024-03-26T05:22:14.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:13 smithi132 ceph-mon[33500]: 5.e scrub ok 2024-03-26T05:22:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:14.296 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:14.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:14.397 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:14.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:14.426 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:14.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:14.527 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:14.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:14.567 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:14.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:14.668 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:14.695 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:14.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:14.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:14.797 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:14.832 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:14.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:14.934 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:14.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:14.982 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:14.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:15.083 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:15.111 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:15.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:15.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:14 smithi080 ceph-mon[29691]: 6.a scrub starts 2024-03-26T05:22:15.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:14 smithi080 ceph-mon[29691]: 6.a scrub ok 2024-03-26T05:22:15.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:14 smithi080 ceph-mon[29691]: pgmap v1035: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 3785/77472 objects misplaced (4.886%) 2024-03-26T05:22:15.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:14 smithi080 ceph-mon[29691]: 6.17 scrub starts 2024-03-26T05:22:15.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:14 smithi080 ceph-mon[40404]: 6.a scrub starts 2024-03-26T05:22:15.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:14 smithi080 ceph-mon[40404]: 6.a scrub ok 2024-03-26T05:22:15.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:14 smithi080 ceph-mon[40404]: pgmap v1035: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 3785/77472 objects misplaced (4.886%) 2024-03-26T05:22:15.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:14 smithi080 ceph-mon[40404]: 6.17 scrub starts 2024-03-26T05:22:15.213 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:15.249 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:15.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:15.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:14 smithi132 ceph-mon[33500]: 6.a scrub starts 2024-03-26T05:22:15.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:14 smithi132 ceph-mon[33500]: 6.a scrub ok 2024-03-26T05:22:15.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:14 smithi132 ceph-mon[33500]: pgmap v1035: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 3785/77472 objects misplaced (4.886%) 2024-03-26T05:22:15.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:14 smithi132 ceph-mon[33500]: 6.17 scrub starts 2024-03-26T05:22:15.352 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:15.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:15.375 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:15.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:15.476 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:15.517 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:15.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:15.619 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:15.642 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:15.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:15.744 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:15.787 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:15.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:15.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:15.889 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:15.918 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:15.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:16.019 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:16.046 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:16.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:16.069 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T05:22:16.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:15 smithi080 ceph-mon[29691]: 3.11 scrub starts 2024-03-26T05:22:16.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:15 smithi080 ceph-mon[29691]: 3.11 scrub ok 2024-03-26T05:22:16.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:15 smithi080 ceph-mon[29691]: 6.17 scrub ok 2024-03-26T05:22:16.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:15 smithi080 ceph-mon[29691]: 3.1a scrub starts 2024-03-26T05:22:16.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:15 smithi080 ceph-mon[29691]: 3.1a scrub ok 2024-03-26T05:22:16.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:15 smithi080 ceph-mon[40404]: 3.11 scrub starts 2024-03-26T05:22:16.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:15 smithi080 ceph-mon[40404]: 3.11 scrub ok 2024-03-26T05:22:16.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:15 smithi080 ceph-mon[40404]: 6.17 scrub ok 2024-03-26T05:22:16.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:15 smithi080 ceph-mon[40404]: 3.1a scrub starts 2024-03-26T05:22:16.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:15 smithi080 ceph-mon[40404]: 3.1a scrub ok 2024-03-26T05:22:16.148 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:16.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:16.215 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:16.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:16.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:15 smithi132 ceph-mon[33500]: 3.11 scrub starts 2024-03-26T05:22:16.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:15 smithi132 ceph-mon[33500]: 3.11 scrub ok 2024-03-26T05:22:16.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:15 smithi132 ceph-mon[33500]: 6.17 scrub ok 2024-03-26T05:22:16.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:15 smithi132 ceph-mon[33500]: 3.1a scrub starts 2024-03-26T05:22:16.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:15 smithi132 ceph-mon[33500]: 3.1a scrub ok 2024-03-26T05:22:16.292 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:22:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:22:16.316 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:16.342 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:16.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:16.444 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:16.467 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:16.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:16.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:16.568 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:16.597 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:16.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:16.698 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:16.735 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:16.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:16.836 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:16.875 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:16.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:16.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:16.976 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:17.003 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:17.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:17.105 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:17.147 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:17.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:17.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:17.249 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:17.276 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:17.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:17.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:16 smithi132 ceph-mon[33500]: 5.1 scrub starts 2024-03-26T05:22:17.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:16 smithi132 ceph-mon[33500]: 5.1 scrub ok 2024-03-26T05:22:17.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:16 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:17.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:16 smithi132 ceph-mon[33500]: 6.1a scrub starts 2024-03-26T05:22:17.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:16 smithi132 ceph-mon[33500]: 6.1a scrub ok 2024-03-26T05:22:17.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:16 smithi132 ceph-mon[33500]: pgmap v1036: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.3 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 3785/66084 objects misplaced (5.728%) 2024-03-26T05:22:17.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:16 smithi080 ceph-mon[29691]: 5.1 scrub starts 2024-03-26T05:22:17.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:16 smithi080 ceph-mon[29691]: 5.1 scrub ok 2024-03-26T05:22:17.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:16 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:17.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:16 smithi080 ceph-mon[29691]: 6.1a scrub starts 2024-03-26T05:22:17.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:16 smithi080 ceph-mon[29691]: 6.1a scrub ok 2024-03-26T05:22:17.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:16 smithi080 ceph-mon[29691]: pgmap v1036: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.3 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 3785/66084 objects misplaced (5.728%) 2024-03-26T05:22:17.299 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:16 smithi080 ceph-mon[40404]: 5.1 scrub starts 2024-03-26T05:22:17.299 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:16 smithi080 ceph-mon[40404]: 5.1 scrub ok 2024-03-26T05:22:17.300 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:16 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:17.300 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:16 smithi080 ceph-mon[40404]: 6.1a scrub starts 2024-03-26T05:22:17.300 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:16 smithi080 ceph-mon[40404]: 6.1a scrub ok 2024-03-26T05:22:17.300 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:16 smithi080 ceph-mon[40404]: pgmap v1036: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.3 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 3785/66084 objects misplaced (5.728%) 2024-03-26T05:22:17.375 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:17.378 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:17.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:17.467 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:17.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:17.569 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:17.596 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:17.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:17.698 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:17.735 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:17.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:17.837 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:17.867 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:17.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:17.970 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:18.010 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:18.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:18.112 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:18.139 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:18.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:18.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:18.194 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:18.241 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:18.248 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:18.267 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:18.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:18.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:17 smithi132 ceph-mon[33500]: 6.e scrub starts 2024-03-26T05:22:18.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:17 smithi132 ceph-mon[33500]: 6.e scrub ok 2024-03-26T05:22:18.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:17 smithi132 ceph-mon[33500]: 5.a scrub starts 2024-03-26T05:22:18.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:17 smithi132 ceph-mon[33500]: 5.a scrub ok 2024-03-26T05:22:18.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:17 smithi132 ceph-mon[33500]: 3.6 scrub starts 2024-03-26T05:22:18.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:17 smithi132 ceph-mon[33500]: 3.6 scrub ok 2024-03-26T05:22:18.352 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:17 smithi080 ceph-mon[29691]: 6.e scrub starts 2024-03-26T05:22:18.352 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:17 smithi080 ceph-mon[29691]: 6.e scrub ok 2024-03-26T05:22:18.352 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:17 smithi080 ceph-mon[29691]: 5.a scrub starts 2024-03-26T05:22:18.352 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:17 smithi080 ceph-mon[29691]: 5.a scrub ok 2024-03-26T05:22:18.352 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:17 smithi080 ceph-mon[29691]: 3.6 scrub starts 2024-03-26T05:22:18.353 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:17 smithi080 ceph-mon[29691]: 3.6 scrub ok 2024-03-26T05:22:18.353 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:17 smithi080 ceph-mon[40404]: 6.e scrub starts 2024-03-26T05:22:18.353 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:17 smithi080 ceph-mon[40404]: 6.e scrub ok 2024-03-26T05:22:18.353 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:17 smithi080 ceph-mon[40404]: 5.a scrub starts 2024-03-26T05:22:18.353 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:17 smithi080 ceph-mon[40404]: 5.a scrub ok 2024-03-26T05:22:18.353 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:17 smithi080 ceph-mon[40404]: 3.6 scrub starts 2024-03-26T05:22:18.353 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:17 smithi080 ceph-mon[40404]: 3.6 scrub ok 2024-03-26T05:22:18.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:18.399 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:18.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:18.500 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:18.528 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:18.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:18.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:18.657 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:18.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:18.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:18.759 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:18.809 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:18.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:18.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:18.937 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:18.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:18.958 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:18 smithi080 ceph-mon[29691]: 5.10 scrub starts 2024-03-26T05:22:18.958 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:18 smithi080 ceph-mon[29691]: 5.10 scrub ok 2024-03-26T05:22:18.958 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:18 smithi080 ceph-mon[29691]: 6.6 scrub starts 2024-03-26T05:22:18.958 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:18 smithi080 ceph-mon[29691]: 6.6 scrub ok 2024-03-26T05:22:18.958 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:18 smithi080 ceph-mon[29691]: pgmap v1037: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.2 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 2831/60000 objects misplaced (4.718%) 2024-03-26T05:22:18.959 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:18 smithi080 ceph-mon[40404]: 5.10 scrub starts 2024-03-26T05:22:18.959 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:18 smithi080 ceph-mon[40404]: 5.10 scrub ok 2024-03-26T05:22:18.959 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:18 smithi080 ceph-mon[40404]: 6.6 scrub starts 2024-03-26T05:22:18.959 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:18 smithi080 ceph-mon[40404]: 6.6 scrub ok 2024-03-26T05:22:18.959 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:18 smithi080 ceph-mon[40404]: pgmap v1037: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.2 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 2831/60000 objects misplaced (4.718%) 2024-03-26T05:22:19.039 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:19.066 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:19.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:19.168 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:19.205 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:19.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:19.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:18 smithi132 ceph-mon[33500]: 5.10 scrub starts 2024-03-26T05:22:19.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:18 smithi132 ceph-mon[33500]: 5.10 scrub ok 2024-03-26T05:22:19.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:18 smithi132 ceph-mon[33500]: 6.6 scrub starts 2024-03-26T05:22:19.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:18 smithi132 ceph-mon[33500]: 6.6 scrub ok 2024-03-26T05:22:19.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:18 smithi132 ceph-mon[33500]: pgmap v1037: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.2 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 2831/60000 objects misplaced (4.718%) 2024-03-26T05:22:19.306 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:19.333 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:19.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:19.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:19.436 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:19.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:19.492 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:19.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:19.593 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:19.639 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:19.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:19.741 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:19.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:19.794 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:19.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:19.895 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:19.941 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:19.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:20.042 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:20.068 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:20.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:20.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:19 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:22:20.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:19 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:22:20.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:19 smithi080 ceph-mon[40404]: 11.d scrub starts 2024-03-26T05:22:20.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:19 smithi080 ceph-mon[40404]: 3.3 scrub starts 2024-03-26T05:22:20.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:19 smithi080 ceph-mon[40404]: 3.3 scrub ok 2024-03-26T05:22:20.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:19 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:22:20.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:19 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:22:20.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:19 smithi080 ceph-mon[29691]: 11.d scrub starts 2024-03-26T05:22:20.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:19 smithi080 ceph-mon[29691]: 3.3 scrub starts 2024-03-26T05:22:20.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:19 smithi080 ceph-mon[29691]: 3.3 scrub ok 2024-03-26T05:22:20.170 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:20.196 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:20.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:20.243 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:22:20.248 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:22:20.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:19 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:22:20.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:19 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:22:20.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:19 smithi132 ceph-mon[33500]: 11.d scrub starts 2024-03-26T05:22:20.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:19 smithi132 ceph-mon[33500]: 3.3 scrub starts 2024-03-26T05:22:20.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:19 smithi132 ceph-mon[33500]: 3.3 scrub ok 2024-03-26T05:22:20.298 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:20.325 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:20.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:20.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:20.427 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:20.468 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:20.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:20.569 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:20.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:20.598 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:20.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:20.699 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:20.741 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:20.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:20.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:20.843 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:20.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:20.885 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:20.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:20.986 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:21.011 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:21.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:21.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:21.149 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:21.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:21.190 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:20 smithi080 ceph-mon[29691]: 1.6 scrub starts 2024-03-26T05:22:21.190 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:20 smithi080 ceph-mon[29691]: 1.6 scrub ok 2024-03-26T05:22:21.190 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:20 smithi080 ceph-mon[29691]: 11.d scrub ok 2024-03-26T05:22:21.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:20 smithi080 ceph-mon[29691]: from='client.16098 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:22:21.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:20 smithi080 ceph-mon[29691]: pgmap v1038: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.2 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 2831/58152 objects misplaced (4.868%) 2024-03-26T05:22:21.191 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:20 smithi080 ceph-mon[40404]: 1.6 scrub starts 2024-03-26T05:22:21.191 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:20 smithi080 ceph-mon[40404]: 1.6 scrub ok 2024-03-26T05:22:21.191 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:20 smithi080 ceph-mon[40404]: 11.d scrub ok 2024-03-26T05:22:21.191 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:20 smithi080 ceph-mon[40404]: from='client.16098 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:22:21.191 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:20 smithi080 ceph-mon[40404]: pgmap v1038: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.2 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 2831/58152 objects misplaced (4.868%) 2024-03-26T05:22:21.191 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:22:21.015+0000 7f2fcc605700 -1 mgr get_metadata_python Requested missing service osd.4 2024-03-26T05:22:21.251 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:21.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:20 smithi132 ceph-mon[33500]: 1.6 scrub starts 2024-03-26T05:22:21.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:20 smithi132 ceph-mon[33500]: 1.6 scrub ok 2024-03-26T05:22:21.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:20 smithi132 ceph-mon[33500]: 11.d scrub ok 2024-03-26T05:22:21.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:20 smithi132 ceph-mon[33500]: from='client.16098 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:22:21.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:20 smithi132 ceph-mon[33500]: pgmap v1038: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.2 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 2831/58152 objects misplaced (4.868%) 2024-03-26T05:22:21.294 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:21.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:21.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:21.396 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:21.422 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:21.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:21.524 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:21.563 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:21.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:21.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:21.665 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:21.693 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:21.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:21.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:21.795 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:21.833 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:21.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:21.935 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:21.964 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:21.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:21.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:22.066 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:22.128 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:22.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:22.230 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:22.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:22.273 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:22.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:22.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:21 smithi132 ceph-mon[33500]: 3.14 scrub starts 2024-03-26T05:22:22.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:21 smithi132 ceph-mon[33500]: 3.14 scrub ok 2024-03-26T05:22:22.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:22.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:21 smithi132 ceph-mon[33500]: 4.9 scrub starts 2024-03-26T05:22:22.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:21 smithi132 ceph-mon[33500]: 4.9 scrub ok 2024-03-26T05:22:22.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:21 smithi132 ceph-mon[33500]: 3.1 scrub starts 2024-03-26T05:22:22.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:21 smithi132 ceph-mon[33500]: 3.1 scrub ok 2024-03-26T05:22:22.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[40404]: 3.14 scrub starts 2024-03-26T05:22:22.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[40404]: 3.14 scrub ok 2024-03-26T05:22:22.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:22.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[40404]: 4.9 scrub starts 2024-03-26T05:22:22.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[40404]: 4.9 scrub ok 2024-03-26T05:22:22.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[40404]: 3.1 scrub starts 2024-03-26T05:22:22.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[40404]: 3.1 scrub ok 2024-03-26T05:22:22.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[29691]: 3.14 scrub starts 2024-03-26T05:22:22.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[29691]: 3.14 scrub ok 2024-03-26T05:22:22.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:22.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[29691]: 4.9 scrub starts 2024-03-26T05:22:22.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[29691]: 4.9 scrub ok 2024-03-26T05:22:22.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[29691]: 3.1 scrub starts 2024-03-26T05:22:22.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:21 smithi080 ceph-mon[29691]: 3.1 scrub ok 2024-03-26T05:22:22.374 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:22.401 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:22.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:22.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:22.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:22.528 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:22.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:22.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:22.657 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:22.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:22.759 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:22.805 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:22.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:22.906 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:22.943 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:22.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:23.045 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:23.051 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":1037,"stamp":"2024-03-26T05:22:18.264266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1296144758,"num_objects":20000,"num_object_clones":0,"num_object_copies":60000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2831,"num_objects_unfound":0,"num_objects_dirty":20000,"num_whiteouts":0,"num_read":2325,"num_read_kb":2045,"num_write":103519,"num_write_kb":3408366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23133,"num_bytes_recovered":1499012193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128731,"ondisk_log_size":128731,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":511,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":8735984,"kb_used_data":4020352,"kb_used_omap":12,"kb_used_meta":4715507,"kb_avail":553734928,"statfs":{"total":575970213888,"available":567024566272,"internally_reserved":0,"allocated":4116840448,"data_stored":4109619172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12412,"internal_metadata":4828680068},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-700436902,"num_objects":-10687,"num_object_clones":0,"num_object_copies":-32061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1994,"num_objects_unfound":0,"num_objects_dirty":-10687,"num_whiteouts":0,"num_read":-8,"num_read_kb":-17,"num_write":-102908,"num_write_kb":-4102044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-21645,"num_bytes_recovered":-1418556658,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002853"},"pg_stats":[{"pgid":"11.3","version":"485'15379","reported_seq":15674,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:14.961608+0000","last_change":"2024-03-26T05:22:09.010961+0000","last_active":"2024-03-26T05:22:14.961608+0000","last_peered":"2024-03-26T05:22:14.961608+0000","last_clean":"2024-03-26T05:22:14.961608+0000","last_became_active":"2024-03-26T05:21:23.798630+0000","last_became_peered":"2024-03-26T05:21:23.798630+0000","last_unstale":"2024-03-26T05:22:14.961608+0000","last_undegraded":"2024-03-26T05:22:14.961608+0000","last_fullsized":"2024-03-26T05:22:14.961608+0000","mapping_epoch":408,"log_start":"453'10677","ondisk_log_start":"453'10677","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'15166","last_scrub_stamp":"2024-03-26T05:22:09.010783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:09.010783+0000","objects_scrubbed":869,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:14.748046+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42795008,"num_objects":653,"num_object_clones":0,"num_object_copies":1959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"443'30","reported_seq":771,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765499+0000","last_change":"2024-03-26T05:20:15.461803+0000","last_active":"2024-03-26T05:22:11.765499+0000","last_peered":"2024-03-26T05:22:11.765499+0000","last_clean":"2024-03-26T05:22:11.765499+0000","last_became_active":"2024-03-26T05:20:15.369000+0000","last_became_peered":"2024-03-26T05:20:15.369000+0000","last_unstale":"2024-03-26T05:22:11.765499+0000","last_undegraded":"2024-03-26T05:22:11.765499+0000","last_fullsized":"2024-03-26T05:22:11.765499+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'20","last_scrub_stamp":"2024-03-26T05:14:56.989928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:14:56.989928+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":930,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764105+0000","last_change":"2024-03-26T05:19:52.350482+0000","last_active":"2024-03-26T05:22:11.764105+0000","last_peered":"2024-03-26T05:22:11.764105+0000","last_clean":"2024-03-26T05:22:11.764105+0000","last_became_active":"2024-03-26T05:06:06.521931+0000","last_became_peered":"2024-03-26T05:06:06.521931+0000","last_unstale":"2024-03-26T05:22:11.764105+0000","last_undegraded":"2024-03-26T05:22:11.764105+0000","last_fullsized":"2024-03-26T05:22:11.764105+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:52.350328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:19:52.350328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"443'14","reported_seq":1223,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.780543+0000","last_change":"2024-03-26T05:20:30.368570+0000","last_active":"2024-03-26T05:22:11.780543+0000","last_peered":"2024-03-26T05:22:11.780543+0000","last_clean":"2024-03-26T05:22:11.780543+0000","last_became_active":"2024-03-26T05:20:30.348391+0000","last_became_peered":"2024-03-26T05:20:30.348391+0000","last_unstale":"2024-03-26T05:22:11.780543+0000","last_undegraded":"2024-03-26T05:22:11.780543+0000","last_fullsized":"2024-03-26T05:22:11.780543+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'13","last_scrub_stamp":"2024-03-26T05:18:56.306025+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:18:56.306025+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":308,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.780484+0000","last_change":"2024-03-26T05:20:30.342597+0000","last_active":"2024-03-26T05:22:11.780484+0000","last_peered":"2024-03-26T05:22:11.780484+0000","last_clean":"2024-03-26T05:22:11.780484+0000","last_became_active":"2024-03-26T05:20:30.342267+0000","last_became_peered":"2024-03-26T05:20:30.342267+0000","last_unstale":"2024-03-26T05:22:11.780484+0000","last_undegraded":"2024-03-26T05:22:11.780484+0000","last_fullsized":"2024-03-26T05:22:11.780484+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:54.067506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:54.067506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"88'128","reported_seq":1098,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.768838+0000","last_change":"2024-03-26T05:19:59.233794+0000","last_active":"2024-03-26T05:22:11.768838+0000","last_peered":"2024-03-26T05:22:11.768838+0000","last_clean":"2024-03-26T05:22:11.768838+0000","last_became_active":"2024-03-26T05:15:33.616505+0000","last_became_peered":"2024-03-26T05:15:33.616505+0000","last_unstale":"2024-03-26T05:22:11.768838+0000","last_undegraded":"2024-03-26T05:22:11.768838+0000","last_fullsized":"2024-03-26T05:22:11.768838+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:19:59.233670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:19:59.233670+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"485'12124","reported_seq":12362,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:16.055803+0000","last_change":"2024-03-26T05:22:07.971102+0000","last_active":"2024-03-26T05:22:16.055803+0000","last_peered":"2024-03-26T05:22:16.055803+0000","last_clean":"2024-03-26T05:22:16.055803+0000","last_became_active":"2024-03-26T05:21:23.802417+0000","last_became_peered":"2024-03-26T05:21:23.802417+0000","last_unstale":"2024-03-26T05:22:16.055803+0000","last_undegraded":"2024-03-26T05:22:16.055803+0000","last_fullsized":"2024-03-26T05:22:16.055803+0000","mapping_epoch":408,"log_start":"431'6160","ondisk_log_start":"431'6160","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'11855","last_scrub_stamp":"2024-03-26T05:22:07.970758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:07.970758+0000","objects_scrubbed":889,"log_size":5964,"log_dups_size":0,"ondisk_log_size":5964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:13.241793+0000","scrub_duration":476,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40042496,"num_objects":611,"num_object_clones":0,"num_object_copies":1833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":278,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"443'45","reported_seq":810,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.780248+0000","last_change":"2024-03-26T05:20:39.141432+0000","last_active":"2024-03-26T05:22:11.780248+0000","last_peered":"2024-03-26T05:22:11.780248+0000","last_clean":"2024-03-26T05:22:11.780248+0000","last_became_active":"2024-03-26T05:20:15.381623+0000","last_became_peered":"2024-03-26T05:20:15.381623+0000","last_unstale":"2024-03-26T05:22:11.780248+0000","last_undegraded":"2024-03-26T05:22:11.780248+0000","last_fullsized":"2024-03-26T05:22:11.780248+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'45","last_scrub_stamp":"2024-03-26T05:20:39.141285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:39.141285+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":968,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764719+0000","last_change":"2024-03-26T05:22:06.816625+0000","last_active":"2024-03-26T05:22:11.764719+0000","last_peered":"2024-03-26T05:22:11.764719+0000","last_clean":"2024-03-26T05:22:11.764719+0000","last_became_active":"2024-03-26T05:20:15.361072+0000","last_became_peered":"2024-03-26T05:20:15.361072+0000","last_unstale":"2024-03-26T05:22:11.764719+0000","last_undegraded":"2024-03-26T05:22:11.764719+0000","last_fullsized":"2024-03-26T05:22:11.764719+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:22:06.816507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:22:06.816507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:14.375386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":917,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.777151+0000","last_change":"2024-03-26T05:22:11.070622+0000","last_active":"2024-03-26T05:22:11.777151+0000","last_peered":"2024-03-26T05:22:11.777151+0000","last_clean":"2024-03-26T05:22:11.777151+0000","last_became_active":"2024-03-26T05:20:15.373227+0000","last_became_peered":"2024-03-26T05:20:15.373227+0000","last_unstale":"2024-03-26T05:22:11.777151+0000","last_undegraded":"2024-03-26T05:22:11.777151+0000","last_fullsized":"2024-03-26T05:22:11.777151+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:22:11.070375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:22:11.070375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:15.847171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":913,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.779938+0000","last_change":"2024-03-26T05:20:38.300338+0000","last_active":"2024-03-26T05:22:11.779938+0000","last_peered":"2024-03-26T05:22:11.779938+0000","last_clean":"2024-03-26T05:22:11.779938+0000","last_became_active":"2024-03-26T05:20:15.380687+0000","last_became_peered":"2024-03-26T05:20:15.380687+0000","last_unstale":"2024-03-26T05:22:11.779938+0000","last_undegraded":"2024-03-26T05:22:11.779938+0000","last_fullsized":"2024-03-26T05:22:11.779938+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:38.300223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:20:38.300223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"88'128","reported_seq":491,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.782988+0000","last_change":"2024-03-26T05:20:34.153749+0000","last_active":"2024-03-26T05:22:11.782988+0000","last_peered":"2024-03-26T05:22:11.782988+0000","last_clean":"2024-03-26T05:22:11.782988+0000","last_became_active":"2024-03-26T05:20:32.754455+0000","last_became_peered":"2024-03-26T05:20:32.754455+0000","last_unstale":"2024-03-26T05:22:11.782988+0000","last_undegraded":"2024-03-26T05:22:11.782988+0000","last_fullsized":"2024-03-26T05:22:11.782988+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:20:34.153595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:20:34.153595+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"485'12511","reported_seq":12769,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:14.915084+0000","last_change":"2024-03-26T05:22:08.085656+0000","last_active":"2024-03-26T05:22:14.915084+0000","last_peered":"2024-03-26T05:22:14.915084+0000","last_clean":"2024-03-26T05:22:14.915084+0000","last_became_active":"2024-03-26T05:21:23.782264+0000","last_became_peered":"2024-03-26T05:21:23.782264+0000","last_unstale":"2024-03-26T05:22:14.915084+0000","last_undegraded":"2024-03-26T05:22:14.915084+0000","last_fullsized":"2024-03-26T05:22:14.915084+0000","mapping_epoch":408,"log_start":"433'7135","ondisk_log_start":"433'7135","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'12277","last_scrub_stamp":"2024-03-26T05:22:08.085499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:08.085499+0000","objects_scrubbed":920,"log_size":5376,"log_dups_size":0,"ondisk_log_size":5376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:30.635209+0000","scrub_duration":254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44892160,"num_objects":685,"num_object_clones":0,"num_object_copies":2055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":235,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"443'33","reported_seq":791,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765456+0000","last_change":"2024-03-26T05:20:39.704389+0000","last_active":"2024-03-26T05:22:11.765456+0000","last_peered":"2024-03-26T05:22:11.765456+0000","last_clean":"2024-03-26T05:22:11.765456+0000","last_became_active":"2024-03-26T05:20:15.366493+0000","last_became_peered":"2024-03-26T05:20:15.366493+0000","last_unstale":"2024-03-26T05:22:11.765456+0000","last_undegraded":"2024-03-26T05:22:11.765456+0000","last_fullsized":"2024-03-26T05:22:11.765456+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'33","last_scrub_stamp":"2024-03-26T05:20:39.704184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:39.704184+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":902,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.782339+0000","last_change":"2024-03-26T05:20:30.098403+0000","last_active":"2024-03-26T05:22:11.782339+0000","last_peered":"2024-03-26T05:22:11.782339+0000","last_clean":"2024-03-26T05:22:11.782339+0000","last_became_active":"2024-03-26T05:20:15.374939+0000","last_became_peered":"2024-03-26T05:20:15.374939+0000","last_unstale":"2024-03-26T05:22:11.782339+0000","last_undegraded":"2024-03-26T05:22:11.782339+0000","last_fullsized":"2024-03-26T05:22:11.782339+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:30.098264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:20:30.098264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":587,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765653+0000","last_change":"2024-03-26T05:20:58.499533+0000","last_active":"2024-03-26T05:22:11.765653+0000","last_peered":"2024-03-26T05:22:11.765653+0000","last_clean":"2024-03-26T05:22:11.765653+0000","last_became_active":"2024-03-26T05:20:30.335018+0000","last_became_peered":"2024-03-26T05:20:30.335018+0000","last_unstale":"2024-03-26T05:22:11.765653+0000","last_undegraded":"2024-03-26T05:22:11.765653+0000","last_fullsized":"2024-03-26T05:22:11.765653+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:58.499385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:13:07.883082+0000","last_clean_scrub_stamp":"2024-03-26T05:20:58.499385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:17.654605+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":922,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765174+0000","last_change":"2024-03-26T05:20:30.327361+0000","last_active":"2024-03-26T05:22:11.765174+0000","last_peered":"2024-03-26T05:22:11.765174+0000","last_clean":"2024-03-26T05:22:11.765174+0000","last_became_active":"2024-03-26T05:20:30.327060+0000","last_became_peered":"2024-03-26T05:20:30.327060+0000","last_unstale":"2024-03-26T05:22:11.765174+0000","last_undegraded":"2024-03-26T05:22:11.765174+0000","last_fullsized":"2024-03-26T05:22:11.765174+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:46.337028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:46.337028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"485'12521","reported_seq":12785,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:16.496210+0000","last_change":"2024-03-26T05:22:08.511959+0000","last_active":"2024-03-26T05:22:16.496210+0000","last_peered":"2024-03-26T05:22:16.496210+0000","last_clean":"2024-03-26T05:22:16.496210+0000","last_became_active":"2024-03-26T05:21:23.788293+0000","last_became_peered":"2024-03-26T05:21:23.788293+0000","last_unstale":"2024-03-26T05:22:16.496210+0000","last_undegraded":"2024-03-26T05:22:16.496210+0000","last_fullsized":"2024-03-26T05:22:16.496210+0000","mapping_epoch":408,"log_start":"434'7269","ondisk_log_start":"434'7269","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'12248","last_scrub_stamp":"2024-03-26T05:22:08.511765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:08.511765+0000","objects_scrubbed":877,"log_size":5252,"log_dups_size":0,"ondisk_log_size":5252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:23.438232+0000","scrub_duration":474,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38666240,"num_objects":590,"num_object_clones":0,"num_object_copies":1770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"443'46","reported_seq":1005,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.785027+0000","last_change":"2024-03-26T05:20:15.465505+0000","last_active":"2024-03-26T05:22:11.785027+0000","last_peered":"2024-03-26T05:22:11.785027+0000","last_clean":"2024-03-26T05:22:11.785027+0000","last_became_active":"2024-03-26T05:20:15.378070+0000","last_became_peered":"2024-03-26T05:20:15.378070+0000","last_unstale":"2024-03-26T05:22:11.785027+0000","last_undegraded":"2024-03-26T05:22:11.785027+0000","last_fullsized":"2024-03-26T05:22:11.785027+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'34","last_scrub_stamp":"2024-03-26T05:15:43.090549+0000","last_deep_scrub":"292'34","last_deep_scrub_stamp":"2024-03-26T05:12:23.778630+0000","last_clean_scrub_stamp":"2024-03-26T05:15:43.090549+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":918,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764406+0000","last_change":"2024-03-26T05:19:54.430984+0000","last_active":"2024-03-26T05:22:11.764406+0000","last_peered":"2024-03-26T05:22:11.764406+0000","last_clean":"2024-03-26T05:22:11.764406+0000","last_became_active":"2024-03-26T05:15:32.626265+0000","last_became_peered":"2024-03-26T05:15:32.626265+0000","last_unstale":"2024-03-26T05:22:11.764406+0000","last_undegraded":"2024-03-26T05:22:11.764406+0000","last_fullsized":"2024-03-26T05:22:11.764406+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:54.430734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.430734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"443'13","reported_seq":155,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:13.592719+0000","last_change":"2024-03-26T05:22:13.592719+0000","last_active":"2024-03-26T05:22:13.592719+0000","last_peered":"2024-03-26T05:22:13.592719+0000","last_clean":"2024-03-26T05:22:13.592719+0000","last_became_active":"2024-03-26T05:20:30.350797+0000","last_became_peered":"2024-03-26T05:20:30.350797+0000","last_unstale":"2024-03-26T05:22:13.592719+0000","last_undegraded":"2024-03-26T05:22:13.592719+0000","last_fullsized":"2024-03-26T05:22:13.592719+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'13","last_scrub_stamp":"2024-03-26T05:22:13.592574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:22:13.592574+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:19.574068+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":218,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.777441+0000","last_change":"2024-03-26T05:20:30.342843+0000","last_active":"2024-03-26T05:22:11.777441+0000","last_peered":"2024-03-26T05:22:11.777441+0000","last_clean":"2024-03-26T05:22:11.777441+0000","last_became_active":"2024-03-26T05:20:30.342410+0000","last_became_peered":"2024-03-26T05:20:30.342410+0000","last_unstale":"2024-03-26T05:22:11.777441+0000","last_undegraded":"2024-03-26T05:22:11.777441+0000","last_fullsized":"2024-03-26T05:22:11.777441+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:49.287596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:52.465619+0000","last_clean_scrub_stamp":"2024-03-26T05:18:49.287596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"88'128","reported_seq":487,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.779499+0000","last_change":"2024-03-26T05:20:33.139347+0000","last_active":"2024-03-26T05:22:11.779499+0000","last_peered":"2024-03-26T05:22:11.779499+0000","last_clean":"2024-03-26T05:22:11.779499+0000","last_became_active":"2024-03-26T05:20:32.753941+0000","last_became_peered":"2024-03-26T05:20:32.753941+0000","last_unstale":"2024-03-26T05:22:11.779499+0000","last_undegraded":"2024-03-26T05:22:11.779499+0000","last_fullsized":"2024-03-26T05:22:11.779499+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:20:33.139232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:20:33.139232+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"485'13965","reported_seq":14203,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:15.901639+0000","last_change":"2024-03-26T05:19:57.196304+0000","last_active":"2024-03-26T05:22:15.901639+0000","last_peered":"2024-03-26T05:22:15.901639+0000","last_clean":"2024-03-26T05:22:15.901639+0000","last_became_active":"2024-03-26T05:18:36.691270+0000","last_became_peered":"2024-03-26T05:18:36.691270+0000","last_unstale":"2024-03-26T05:22:15.901639+0000","last_undegraded":"2024-03-26T05:22:15.901639+0000","last_fullsized":"2024-03-26T05:22:15.901639+0000","mapping_epoch":408,"log_start":"455'10448","ondisk_log_start":"455'10448","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'6148","last_scrub_stamp":"2024-03-26T05:19:57.195991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.195991+0000","objects_scrubbed":6082,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1951,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78839808,"num_objects":1203,"num_object_clones":0,"num_object_copies":3609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13965,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":880,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.772879+0000","last_change":"2024-03-26T05:20:15.376611+0000","last_active":"2024-03-26T05:22:11.772879+0000","last_peered":"2024-03-26T05:22:11.772879+0000","last_clean":"2024-03-26T05:22:11.772879+0000","last_became_active":"2024-03-26T05:20:15.376307+0000","last_became_peered":"2024-03-26T05:20:15.376307+0000","last_unstale":"2024-03-26T05:22:11.772879+0000","last_undegraded":"2024-03-26T05:22:11.772879+0000","last_fullsized":"2024-03-26T05:22:11.772879+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:10:33.411139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:10:33.411139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"443'64","reported_seq":1024,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764585+0000","last_change":"2024-03-26T05:20:05.275701+0000","last_active":"2024-03-26T05:22:11.764585+0000","last_peered":"2024-03-26T05:22:11.764585+0000","last_clean":"2024-03-26T05:22:11.764585+0000","last_became_active":"2024-03-26T05:15:32.636466+0000","last_became_peered":"2024-03-26T05:15:32.636466+0000","last_unstale":"2024-03-26T05:22:11.764585+0000","last_undegraded":"2024-03-26T05:22:11.764585+0000","last_fullsized":"2024-03-26T05:22:11.764585+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'52","last_scrub_stamp":"2024-03-26T05:20:05.274985+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:20:05.274985+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":74,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":910,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764414+0000","last_change":"2024-03-26T05:18:36.798166+0000","last_active":"2024-03-26T05:22:11.764414+0000","last_peered":"2024-03-26T05:22:11.764414+0000","last_clean":"2024-03-26T05:22:11.764414+0000","last_became_active":"2024-03-26T05:15:32.636983+0000","last_became_peered":"2024-03-26T05:15:32.636983+0000","last_unstale":"2024-03-26T05:22:11.764414+0000","last_undegraded":"2024-03-26T05:22:11.764414+0000","last_fullsized":"2024-03-26T05:22:11.764414+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.798083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.798083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":303,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.767444+0000","last_change":"2024-03-26T05:20:15.377248+0000","last_active":"2024-03-26T05:22:11.767444+0000","last_peered":"2024-03-26T05:22:11.767444+0000","last_clean":"2024-03-26T05:22:11.767444+0000","last_became_active":"2024-03-26T05:20:15.375300+0000","last_became_peered":"2024-03-26T05:20:15.375300+0000","last_unstale":"2024-03-26T05:22:11.767444+0000","last_undegraded":"2024-03-26T05:22:11.767444+0000","last_fullsized":"2024-03-26T05:22:11.767444+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:13:54.923744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:13:54.923744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"88'128","reported_seq":702,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.780598+0000","last_change":"2024-03-26T05:20:32.753218+0000","last_active":"2024-03-26T05:22:11.780598+0000","last_peered":"2024-03-26T05:22:11.780598+0000","last_clean":"2024-03-26T05:22:11.780598+0000","last_became_active":"2024-03-26T05:20:32.752804+0000","last_became_peered":"2024-03-26T05:20:32.752804+0000","last_unstale":"2024-03-26T05:22:11.780598+0000","last_undegraded":"2024-03-26T05:22:11.780598+0000","last_fullsized":"2024-03-26T05:22:11.780598+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:20:28.011935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:20:28.011935+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"485'13948","reported_seq":26046,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:16.523502+0000","last_change":"2024-03-26T05:21:23.036131+0000","last_active":"2024-03-26T05:22:16.523502+0000","last_peered":"2024-03-26T05:22:16.523502+0000","last_clean":"2024-03-26T05:22:16.523502+0000","last_became_active":"2024-03-26T05:21:22.979195+0000","last_became_peered":"2024-03-26T05:21:22.979195+0000","last_unstale":"2024-03-26T05:22:16.523502+0000","last_undegraded":"2024-03-26T05:22:16.523502+0000","last_fullsized":"2024-03-26T05:22:16.523502+0000","mapping_epoch":459,"log_start":"455'10453","ondisk_log_start":"455'10453","created":408,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81395712,"num_objects":1242,"num_object_clones":0,"num_object_copies":3726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13948,"num_write_kb":486080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5959,"num_bytes_recovered":390529024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"439'10","reported_seq":1170,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.768451+0000","last_change":"2024-03-26T05:20:15.470722+0000","last_active":"2024-03-26T05:22:11.768451+0000","last_peered":"2024-03-26T05:22:11.768451+0000","last_clean":"2024-03-26T05:22:11.768451+0000","last_became_active":"2024-03-26T05:20:15.376570+0000","last_became_peered":"2024-03-26T05:20:15.376570+0000","last_unstale":"2024-03-26T05:22:11.768451+0000","last_undegraded":"2024-03-26T05:22:11.768451+0000","last_fullsized":"2024-03-26T05:22:11.768451+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:13:56.721250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:13:56.721250+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"443'48","reported_seq":658,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764760+0000","last_change":"2024-03-26T05:20:55.489656+0000","last_active":"2024-03-26T05:22:11.764760+0000","last_peered":"2024-03-26T05:22:11.764760+0000","last_clean":"2024-03-26T05:22:11.764760+0000","last_became_active":"2024-03-26T05:20:30.343622+0000","last_became_peered":"2024-03-26T05:20:30.343622+0000","last_unstale":"2024-03-26T05:22:11.764760+0000","last_undegraded":"2024-03-26T05:22:11.764760+0000","last_fullsized":"2024-03-26T05:22:11.764760+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'48","last_scrub_stamp":"2024-03-26T05:20:55.489555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:55.489555+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:13.261547+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":63,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":932,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.772849+0000","last_change":"2024-03-26T05:20:15.464557+0000","last_active":"2024-03-26T05:22:11.772849+0000","last_peered":"2024-03-26T05:22:11.772849+0000","last_clean":"2024-03-26T05:22:11.772849+0000","last_became_active":"2024-03-26T05:20:15.379302+0000","last_became_peered":"2024-03-26T05:20:15.379302+0000","last_unstale":"2024-03-26T05:22:11.772849+0000","last_undegraded":"2024-03-26T05:22:11.772849+0000","last_fullsized":"2024-03-26T05:22:11.772849+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:14:12.113993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:12.113993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":824,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769058+0000","last_change":"2024-03-26T05:19:49.427510+0000","last_active":"2024-03-26T05:22:11.769058+0000","last_peered":"2024-03-26T05:22:11.769058+0000","last_clean":"2024-03-26T05:22:11.769058+0000","last_became_active":"2024-03-26T05:15:32.630135+0000","last_became_peered":"2024-03-26T05:15:32.630135+0000","last_unstale":"2024-03-26T05:22:11.769058+0000","last_undegraded":"2024-03-26T05:22:11.769058+0000","last_fullsized":"2024-03-26T05:22:11.769058+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:49.427405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:01.721323+0000","last_clean_scrub_stamp":"2024-03-26T05:19:49.427405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"396'129","reported_seq":741,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.763869+0000","last_change":"2024-03-26T05:20:32.747492+0000","last_active":"2024-03-26T05:22:11.763869+0000","last_peered":"2024-03-26T05:22:11.763869+0000","last_clean":"2024-03-26T05:22:11.763869+0000","last_became_active":"2024-03-26T05:20:32.747103+0000","last_became_peered":"2024-03-26T05:20:32.747103+0000","last_unstale":"2024-03-26T05:22:11.763869+0000","last_undegraded":"2024-03-26T05:22:11.763869+0000","last_fullsized":"2024-03-26T05:22:11.763869+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"396'129","last_scrub_stamp":"2024-03-26T05:17:59.038611+0000","last_deep_scrub":"396'129","last_deep_scrub_stamp":"2024-03-26T05:17:59.038611+0000","last_clean_scrub_stamp":"2024-03-26T05:17:59.038611+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"485'13906","reported_seq":23861,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:16.040249+0000","last_change":"2024-03-26T05:21:25.934696+0000","last_active":"2024-03-26T05:22:16.040249+0000","last_peered":"2024-03-26T05:22:16.040249+0000","last_clean":"2024-03-26T05:22:16.040249+0000","last_became_active":"2024-03-26T05:21:25.915527+0000","last_became_peered":"2024-03-26T05:21:25.915527+0000","last_unstale":"2024-03-26T05:22:16.040249+0000","last_undegraded":"2024-03-26T05:22:16.040249+0000","last_fullsized":"2024-03-26T05:22:16.040249+0000","mapping_epoch":462,"log_start":"451'9823","ondisk_log_start":"451'9823","created":408,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84475928,"num_objects":1290,"num_object_clones":0,"num_object_copies":3870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13906,"num_write_kb":486209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4886,"num_bytes_recovered":320143384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"443'12","reported_seq":1222,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769636+0000","last_change":"2024-03-26T05:20:30.357087+0000","last_active":"2024-03-26T05:22:11.769636+0000","last_peered":"2024-03-26T05:22:11.769636+0000","last_clean":"2024-03-26T05:22:11.769636+0000","last_became_active":"2024-03-26T05:20:30.350209+0000","last_became_peered":"2024-03-26T05:20:30.350209+0000","last_unstale":"2024-03-26T05:22:11.769636+0000","last_undegraded":"2024-03-26T05:22:11.769636+0000","last_fullsized":"2024-03-26T05:22:11.769636+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11","last_scrub_stamp":"2024-03-26T05:18:36.306612+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.306612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"443'36","reported_seq":132,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.775740+0000","last_change":"2024-03-26T05:22:10.070963+0000","last_active":"2024-03-26T05:22:11.775740+0000","last_peered":"2024-03-26T05:22:11.775740+0000","last_clean":"2024-03-26T05:22:11.775740+0000","last_became_active":"2024-03-26T05:20:30.345634+0000","last_became_peered":"2024-03-26T05:20:30.345634+0000","last_unstale":"2024-03-26T05:22:11.775740+0000","last_undegraded":"2024-03-26T05:22:11.775740+0000","last_fullsized":"2024-03-26T05:22:11.775740+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'36","last_scrub_stamp":"2024-03-26T05:22:10.070715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:22:10.070715+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:27.499286+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":211,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764509+0000","last_change":"2024-03-26T05:20:30.342087+0000","last_active":"2024-03-26T05:22:11.764509+0000","last_peered":"2024-03-26T05:22:11.764509+0000","last_clean":"2024-03-26T05:22:11.764509+0000","last_became_active":"2024-03-26T05:20:30.341758+0000","last_became_peered":"2024-03-26T05:20:30.341758+0000","last_unstale":"2024-03-26T05:22:11.764509+0000","last_undegraded":"2024-03-26T05:22:11.764509+0000","last_fullsized":"2024-03-26T05:22:11.764509+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:35.144207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:35.144207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":914,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.767386+0000","last_change":"2024-03-26T05:20:04.436906+0000","last_active":"2024-03-26T05:22:11.767386+0000","last_peered":"2024-03-26T05:22:11.767386+0000","last_clean":"2024-03-26T05:22:11.767386+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:22:11.767386+0000","last_undegraded":"2024-03-26T05:22:11.767386+0000","last_fullsized":"2024-03-26T05:22:11.767386+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:04.436614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:20:04.436614+0000","last_clean_scrub_stamp":"2024-03-26T05:20:04.436614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"88'128","reported_seq":675,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765400+0000","last_change":"2024-03-26T05:22:10.865923+0000","last_active":"2024-03-26T05:22:11.765400+0000","last_peered":"2024-03-26T05:22:11.765400+0000","last_clean":"2024-03-26T05:22:11.765400+0000","last_became_active":"2024-03-26T05:15:33.617538+0000","last_became_peered":"2024-03-26T05:15:33.617538+0000","last_unstale":"2024-03-26T05:22:11.765400+0000","last_undegraded":"2024-03-26T05:22:11.765400+0000","last_fullsized":"2024-03-26T05:22:11.765400+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:22:10.865834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:22:10.865834+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:25.183900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"485'14192","reported_seq":27803,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:16.490957+0000","last_change":"2024-03-26T05:21:04.398592+0000","last_active":"2024-03-26T05:22:16.490957+0000","last_peered":"2024-03-26T05:22:16.490957+0000","last_clean":"2024-03-26T05:22:16.490957+0000","last_became_active":"2024-03-26T05:21:04.373554+0000","last_became_peered":"2024-03-26T05:21:04.373554+0000","last_unstale":"2024-03-26T05:22:16.490957+0000","last_undegraded":"2024-03-26T05:22:16.490957+0000","last_fullsized":"2024-03-26T05:22:16.490957+0000","mapping_epoch":452,"log_start":"455'10605","ondisk_log_start":"455'10605","created":408,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3587,"log_dups_size":0,"ondisk_log_size":3587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79429632,"num_objects":1212,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14192,"num_write_kb":492928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6714,"num_bytes_recovered":440008704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":966,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764849+0000","last_change":"2024-03-26T05:21:09.515461+0000","last_active":"2024-03-26T05:22:11.764849+0000","last_peered":"2024-03-26T05:22:11.764849+0000","last_clean":"2024-03-26T05:22:11.764849+0000","last_became_active":"2024-03-26T05:20:30.343345+0000","last_became_peered":"2024-03-26T05:20:30.343345+0000","last_unstale":"2024-03-26T05:22:11.764849+0000","last_undegraded":"2024-03-26T05:22:11.764849+0000","last_fullsized":"2024-03-26T05:22:11.764849+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:09.515351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:21:09.515351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:36.619500+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"443'54","reported_seq":1088,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769268+0000","last_change":"2024-03-26T05:20:30.369614+0000","last_active":"2024-03-26T05:22:11.769268+0000","last_peered":"2024-03-26T05:22:11.769268+0000","last_clean":"2024-03-26T05:22:11.769268+0000","last_became_active":"2024-03-26T05:20:30.347061+0000","last_became_peered":"2024-03-26T05:20:30.347061+0000","last_unstale":"2024-03-26T05:22:11.769268+0000","last_undegraded":"2024-03-26T05:22:11.769268+0000","last_fullsized":"2024-03-26T05:22:11.769268+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'44","last_scrub_stamp":"2024-03-26T05:18:43.255284+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:18:43.255284+0000","objects_scrubbed":9,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":951,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.776935+0000","last_change":"2024-03-26T05:20:30.342347+0000","last_active":"2024-03-26T05:22:11.776935+0000","last_peered":"2024-03-26T05:22:11.776935+0000","last_clean":"2024-03-26T05:22:11.776935+0000","last_became_active":"2024-03-26T05:20:30.341954+0000","last_became_peered":"2024-03-26T05:20:30.341954+0000","last_unstale":"2024-03-26T05:22:11.776935+0000","last_undegraded":"2024-03-26T05:22:11.776935+0000","last_fullsized":"2024-03-26T05:22:11.776935+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:48.584746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:11:29.514915+0000","last_clean_scrub_stamp":"2024-03-26T05:19:48.584746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":913,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.772124+0000","last_change":"2024-03-26T05:19:44.571275+0000","last_active":"2024-03-26T05:22:11.772124+0000","last_peered":"2024-03-26T05:22:11.772124+0000","last_clean":"2024-03-26T05:22:11.772124+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:22:11.772124+0000","last_undegraded":"2024-03-26T05:22:11.772124+0000","last_fullsized":"2024-03-26T05:22:11.772124+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:44.571153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:19:44.571153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"88'128","reported_seq":669,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765256+0000","last_change":"2024-03-26T05:19:56.753537+0000","last_active":"2024-03-26T05:22:11.765256+0000","last_peered":"2024-03-26T05:22:11.765256+0000","last_clean":"2024-03-26T05:22:11.765256+0000","last_became_active":"2024-03-26T05:15:22.624234+0000","last_became_peered":"2024-03-26T05:15:22.624234+0000","last_unstale":"2024-03-26T05:22:11.765256+0000","last_undegraded":"2024-03-26T05:22:11.765256+0000","last_fullsized":"2024-03-26T05:22:11.765256+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:19:56.753458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:19:56.753458+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"485'12224","reported_seq":12417,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:15.355552+0000","last_change":"2024-03-26T05:22:08.211416+0000","last_active":"2024-03-26T05:22:15.355552+0000","last_peered":"2024-03-26T05:22:15.355552+0000","last_clean":"2024-03-26T05:22:15.355552+0000","last_became_active":"2024-03-26T05:21:23.782748+0000","last_became_peered":"2024-03-26T05:21:23.782748+0000","last_unstale":"2024-03-26T05:22:15.355552+0000","last_undegraded":"2024-03-26T05:22:15.355552+0000","last_fullsized":"2024-03-26T05:22:15.355552+0000","mapping_epoch":408,"log_start":"438'7197","ondisk_log_start":"438'7197","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'11947","last_scrub_stamp":"2024-03-26T05:22:08.211276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:08.211276+0000","objects_scrubbed":964,"log_size":5027,"log_dups_size":0,"ondisk_log_size":5027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:23.971456+0000","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44367872,"num_objects":677,"num_object_clones":0,"num_object_copies":2031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":218,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.785329+0000","last_change":"2024-03-26T05:20:30.339317+0000","last_active":"2024-03-26T05:22:11.785329+0000","last_peered":"2024-03-26T05:22:11.785329+0000","last_clean":"2024-03-26T05:22:11.785329+0000","last_became_active":"2024-03-26T05:20:30.339002+0000","last_became_peered":"2024-03-26T05:20:30.339002+0000","last_unstale":"2024-03-26T05:22:11.785329+0000","last_undegraded":"2024-03-26T05:22:11.785329+0000","last_fullsized":"2024-03-26T05:22:11.785329+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:57.304687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.304687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"443'42","reported_seq":1022,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.772687+0000","last_change":"2024-03-26T05:20:05.431539+0000","last_active":"2024-03-26T05:22:11.772687+0000","last_peered":"2024-03-26T05:22:11.772687+0000","last_clean":"2024-03-26T05:22:11.772687+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:22:11.772687+0000","last_undegraded":"2024-03-26T05:22:11.772687+0000","last_fullsized":"2024-03-26T05:22:11.772687+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'30","last_scrub_stamp":"2024-03-26T05:20:05.430706+0000","last_deep_scrub":"254'30","last_deep_scrub_stamp":"2024-03-26T05:20:05.430706+0000","last_clean_scrub_stamp":"2024-03-26T05:20:05.430706+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":1938,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:14.412210+0000","last_change":"2024-03-26T05:19:47.348573+0000","last_active":"2024-03-26T05:22:14.412210+0000","last_peered":"2024-03-26T05:22:14.412210+0000","last_clean":"2024-03-26T05:22:14.412210+0000","last_became_active":"2024-03-26T05:15:32.633061+0000","last_became_peered":"2024-03-26T05:15:32.633061+0000","last_unstale":"2024-03-26T05:22:14.412210+0000","last_undegraded":"2024-03-26T05:22:14.412210+0000","last_fullsized":"2024-03-26T05:22:14.412210+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:19:47.348445+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:19:47.348445+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":468,"num_read_kb":463,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":640,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.775292+0000","last_change":"2024-03-26T05:20:30.349671+0000","last_active":"2024-03-26T05:22:11.775292+0000","last_peered":"2024-03-26T05:22:11.775292+0000","last_clean":"2024-03-26T05:22:11.775292+0000","last_became_active":"2024-03-26T05:20:30.349459+0000","last_became_peered":"2024-03-26T05:20:30.349459+0000","last_unstale":"2024-03-26T05:22:11.775292+0000","last_undegraded":"2024-03-26T05:22:11.775292+0000","last_fullsized":"2024-03-26T05:22:11.775292+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:16.863389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:48.473160+0000","last_clean_scrub_stamp":"2024-03-26T05:14:16.863389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":939,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.776539+0000","last_change":"2024-03-26T05:20:30.332076+0000","last_active":"2024-03-26T05:22:11.776539+0000","last_peered":"2024-03-26T05:22:11.776539+0000","last_clean":"2024-03-26T05:22:11.776539+0000","last_became_active":"2024-03-26T05:20:30.331524+0000","last_became_peered":"2024-03-26T05:20:30.331524+0000","last_unstale":"2024-03-26T05:22:11.776539+0000","last_undegraded":"2024-03-26T05:22:11.776539+0000","last_fullsized":"2024-03-26T05:22:11.776539+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:52.594486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:52.594486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"88'128","reported_seq":735,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.776309+0000","last_change":"2024-03-26T05:20:32.758080+0000","last_active":"2024-03-26T05:22:11.776309+0000","last_peered":"2024-03-26T05:22:11.776309+0000","last_clean":"2024-03-26T05:22:11.776309+0000","last_became_active":"2024-03-26T05:20:32.754918+0000","last_became_peered":"2024-03-26T05:20:32.754918+0000","last_unstale":"2024-03-26T05:22:11.776309+0000","last_undegraded":"2024-03-26T05:22:11.776309+0000","last_fullsized":"2024-03-26T05:22:11.776309+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:15:44.054471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:15:44.054471+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"485'13992","reported_seq":24795,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:15.347500+0000","last_change":"2024-03-26T05:21:24.809434+0000","last_active":"2024-03-26T05:22:15.347500+0000","last_peered":"2024-03-26T05:22:15.347500+0000","last_clean":"2024-03-26T05:22:15.347500+0000","last_became_active":"2024-03-26T05:21:24.792538+0000","last_became_peered":"2024-03-26T05:21:24.792538+0000","last_unstale":"2024-03-26T05:22:15.347500+0000","last_undegraded":"2024-03-26T05:22:15.347500+0000","last_fullsized":"2024-03-26T05:22:15.347500+0000","mapping_epoch":461,"log_start":"455'10593","ondisk_log_start":"455'10593","created":408,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3399,"log_dups_size":0,"ondisk_log_size":3399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83492864,"num_objects":1274,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13992,"num_write_kb":488512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5310,"num_bytes_recovered":347996160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":851,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769848+0000","last_change":"2024-03-26T05:20:30.342594+0000","last_active":"2024-03-26T05:22:11.769848+0000","last_peered":"2024-03-26T05:22:11.769848+0000","last_clean":"2024-03-26T05:22:11.769848+0000","last_became_active":"2024-03-26T05:20:30.342259+0000","last_became_peered":"2024-03-26T05:20:30.342259+0000","last_unstale":"2024-03-26T05:22:11.769848+0000","last_undegraded":"2024-03-26T05:22:11.769848+0000","last_fullsized":"2024-03-26T05:22:11.769848+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:43.408113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:14.948209+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.408113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"443'52","reported_seq":1288,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765755+0000","last_change":"2024-03-26T05:21:03.528679+0000","last_active":"2024-03-26T05:22:11.765755+0000","last_peered":"2024-03-26T05:22:11.765755+0000","last_clean":"2024-03-26T05:22:11.765755+0000","last_became_active":"2024-03-26T05:20:15.372700+0000","last_became_peered":"2024-03-26T05:20:15.372700+0000","last_unstale":"2024-03-26T05:22:11.765755+0000","last_undegraded":"2024-03-26T05:22:11.765755+0000","last_fullsized":"2024-03-26T05:22:11.765755+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'52","last_scrub_stamp":"2024-03-26T05:21:03.528517+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:21:03.528517+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:15.476604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":935,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.766298+0000","last_change":"2024-03-26T05:20:30.350444+0000","last_active":"2024-03-26T05:22:11.766298+0000","last_peered":"2024-03-26T05:22:11.766298+0000","last_clean":"2024-03-26T05:22:11.766298+0000","last_became_active":"2024-03-26T05:20:30.349656+0000","last_became_peered":"2024-03-26T05:20:30.349656+0000","last_unstale":"2024-03-26T05:22:11.766298+0000","last_undegraded":"2024-03-26T05:22:11.766298+0000","last_fullsized":"2024-03-26T05:22:11.766298+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:46.747932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:46.747932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":1092,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769156+0000","last_change":"2024-03-26T05:18:58.577783+0000","last_active":"2024-03-26T05:22:11.769156+0000","last_peered":"2024-03-26T05:22:11.769156+0000","last_clean":"2024-03-26T05:22:11.769156+0000","last_became_active":"2024-03-26T05:15:32.632929+0000","last_became_peered":"2024-03-26T05:15:32.632929+0000","last_unstale":"2024-03-26T05:22:11.769156+0000","last_undegraded":"2024-03-26T05:22:11.769156+0000","last_fullsized":"2024-03-26T05:22:11.769156+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:18:58.577676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:18:58.577676+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":587,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764562+0000","last_change":"2024-03-26T05:20:30.347796+0000","last_active":"2024-03-26T05:22:11.764562+0000","last_peered":"2024-03-26T05:22:11.764562+0000","last_clean":"2024-03-26T05:22:11.764562+0000","last_became_active":"2024-03-26T05:20:30.346881+0000","last_became_peered":"2024-03-26T05:20:30.346881+0000","last_unstale":"2024-03-26T05:22:11.764562+0000","last_undegraded":"2024-03-26T05:22:11.764562+0000","last_fullsized":"2024-03-26T05:22:11.764562+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:14.528992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:14.528992+0000","last_clean_scrub_stamp":"2024-03-26T05:15:14.528992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"485'12499","reported_seq":17694,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:15.331134+0000","last_change":"2024-03-26T05:22:05.876347+0000","last_active":"2024-03-26T05:22:15.331134+0000","last_peered":"2024-03-26T05:22:15.331134+0000","last_clean":"2024-03-26T05:22:15.331134+0000","last_became_active":"2024-03-26T05:21:25.895183+0000","last_became_peered":"2024-03-26T05:21:25.895183+0000","last_unstale":"2024-03-26T05:22:15.331134+0000","last_undegraded":"2024-03-26T05:22:15.331134+0000","last_fullsized":"2024-03-26T05:22:15.331134+0000","mapping_epoch":462,"log_start":"439'7535","ondisk_log_start":"439'7535","created":408,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'12167","last_scrub_stamp":"2024-03-26T05:22:05.875888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:05.875888+0000","objects_scrubbed":972,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:31.181891+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41811968,"num_objects":638,"num_object_clones":0,"num_object_copies":1914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":334,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":944,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.779082+0000","last_change":"2024-03-26T05:20:30.336844+0000","last_active":"2024-03-26T05:22:11.779082+0000","last_peered":"2024-03-26T05:22:11.779082+0000","last_clean":"2024-03-26T05:22:11.779082+0000","last_became_active":"2024-03-26T05:20:30.335654+0000","last_became_peered":"2024-03-26T05:20:30.335654+0000","last_unstale":"2024-03-26T05:22:11.779082+0000","last_undegraded":"2024-03-26T05:22:11.779082+0000","last_fullsized":"2024-03-26T05:22:11.779082+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:54.964851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.964851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"443'27","reported_seq":765,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.780162+0000","last_change":"2024-03-26T05:20:30.365782+0000","last_active":"2024-03-26T05:22:11.780162+0000","last_peered":"2024-03-26T05:22:11.780162+0000","last_clean":"2024-03-26T05:22:11.780162+0000","last_became_active":"2024-03-26T05:20:30.345752+0000","last_became_peered":"2024-03-26T05:20:30.345752+0000","last_unstale":"2024-03-26T05:22:11.780162+0000","last_undegraded":"2024-03-26T05:22:11.780162+0000","last_fullsized":"2024-03-26T05:22:11.780162+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'23","last_scrub_stamp":"2024-03-26T05:19:53.957953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:19:53.957953+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":942,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.782419+0000","last_change":"2024-03-26T05:19:50.912733+0000","last_active":"2024-03-26T05:22:11.782419+0000","last_peered":"2024-03-26T05:22:11.782419+0000","last_clean":"2024-03-26T05:22:11.782419+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:22:11.782419+0000","last_undegraded":"2024-03-26T05:22:11.782419+0000","last_fullsized":"2024-03-26T05:22:11.782419+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:50.912594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:03.193105+0000","last_clean_scrub_stamp":"2024-03-26T05:19:50.912594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":925,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.775148+0000","last_change":"2024-03-26T05:19:46.522718+0000","last_active":"2024-03-26T05:22:11.775148+0000","last_peered":"2024-03-26T05:22:11.775148+0000","last_clean":"2024-03-26T05:22:11.775148+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:22:11.775148+0000","last_undegraded":"2024-03-26T05:22:11.775148+0000","last_fullsized":"2024-03-26T05:22:11.775148+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:46.522602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:19:46.522602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":785,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769106+0000","last_change":"2024-03-26T05:18:41.624630+0000","last_active":"2024-03-26T05:22:11.769106+0000","last_peered":"2024-03-26T05:22:11.769106+0000","last_clean":"2024-03-26T05:22:11.769106+0000","last_became_active":"2024-03-26T05:15:32.633633+0000","last_became_peered":"2024-03-26T05:15:32.633633+0000","last_unstale":"2024-03-26T05:22:11.769106+0000","last_undegraded":"2024-03-26T05:22:11.769106+0000","last_fullsized":"2024-03-26T05:22:11.769106+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:41.624502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:18:41.624502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"88'128","reported_seq":1094,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.768996+0000","last_change":"2024-03-26T05:22:05.450629+0000","last_active":"2024-03-26T05:22:11.768996+0000","last_peered":"2024-03-26T05:22:11.768996+0000","last_clean":"2024-03-26T05:22:11.768996+0000","last_became_active":"2024-03-26T05:20:17.749765+0000","last_became_peered":"2024-03-26T05:20:17.749765+0000","last_unstale":"2024-03-26T05:22:11.768996+0000","last_undegraded":"2024-03-26T05:22:11.768996+0000","last_fullsized":"2024-03-26T05:22:11.768996+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:22:05.450548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:22:05.450548+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:29.657049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"485'13731","reported_seq":13900,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:15.339075+0000","last_change":"2024-03-26T05:18:36.696360+0000","last_active":"2024-03-26T05:22:15.339075+0000","last_peered":"2024-03-26T05:22:15.339075+0000","last_clean":"2024-03-26T05:22:15.339075+0000","last_became_active":"2024-03-26T05:18:36.695687+0000","last_became_peered":"2024-03-26T05:18:36.695687+0000","last_unstale":"2024-03-26T05:22:15.339075+0000","last_undegraded":"2024-03-26T05:22:15.339075+0000","last_fullsized":"2024-03-26T05:22:15.339075+0000","mapping_epoch":408,"log_start":"454'10288","ondisk_log_start":"454'10288","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.613885+0000","objects_scrubbed":0,"log_size":3443,"log_dups_size":0,"ondisk_log_size":3443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83427328,"num_objects":1273,"num_object_clones":0,"num_object_copies":3819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13731,"num_write_kb":480128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":911,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.768665+0000","last_change":"2024-03-26T05:20:15.377798+0000","last_active":"2024-03-26T05:22:11.768665+0000","last_peered":"2024-03-26T05:22:11.768665+0000","last_clean":"2024-03-26T05:22:11.768665+0000","last_became_active":"2024-03-26T05:20:15.377492+0000","last_became_peered":"2024-03-26T05:20:15.377492+0000","last_unstale":"2024-03-26T05:22:11.768665+0000","last_undegraded":"2024-03-26T05:22:11.768665+0000","last_fullsized":"2024-03-26T05:22:11.768665+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:14.131427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:14.131427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"443'58","reported_seq":1291,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.777051+0000","last_change":"2024-03-26T05:20:00.666872+0000","last_active":"2024-03-26T05:22:11.777051+0000","last_peered":"2024-03-26T05:22:11.777051+0000","last_clean":"2024-03-26T05:22:11.777051+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:22:11.777051+0000","last_undegraded":"2024-03-26T05:22:11.777051+0000","last_fullsized":"2024-03-26T05:22:11.777051+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'46","last_scrub_stamp":"2024-03-26T05:20:00.666615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:00.666615+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":301,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769085+0000","last_change":"2024-03-26T05:20:15.457605+0000","last_active":"2024-03-26T05:22:11.769085+0000","last_peered":"2024-03-26T05:22:11.769085+0000","last_clean":"2024-03-26T05:22:11.769085+0000","last_became_active":"2024-03-26T05:20:15.457163+0000","last_became_peered":"2024-03-26T05:20:15.457163+0000","last_unstale":"2024-03-26T05:22:11.769085+0000","last_undegraded":"2024-03-26T05:22:11.769085+0000","last_fullsized":"2024-03-26T05:22:11.769085+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:02.881800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:14:02.881800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":252,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.772182+0000","last_change":"2024-03-26T05:20:30.355675+0000","last_active":"2024-03-26T05:22:11.772182+0000","last_peered":"2024-03-26T05:22:11.772182+0000","last_clean":"2024-03-26T05:22:11.772182+0000","last_became_active":"2024-03-26T05:20:30.339519+0000","last_became_peered":"2024-03-26T05:20:30.339519+0000","last_unstale":"2024-03-26T05:22:11.772182+0000","last_undegraded":"2024-03-26T05:22:11.772182+0000","last_fullsized":"2024-03-26T05:22:11.772182+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T05:18:41.097424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:20.741332+0000","last_clean_scrub_stamp":"2024-03-26T05:18:41.097424+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":891,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.766580+0000","last_change":"2024-03-26T05:20:15.360008+0000","last_active":"2024-03-26T05:22:11.766580+0000","last_peered":"2024-03-26T05:22:11.766580+0000","last_clean":"2024-03-26T05:22:11.766580+0000","last_became_active":"2024-03-26T05:20:15.359520+0000","last_became_peered":"2024-03-26T05:20:15.359520+0000","last_unstale":"2024-03-26T05:22:11.766580+0000","last_undegraded":"2024-03-26T05:22:11.766580+0000","last_fullsized":"2024-03-26T05:22:11.766580+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:31.518660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:11:31.518660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"88'128","reported_seq":493,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.775130+0000","last_change":"2024-03-26T05:22:09.550098+0000","last_active":"2024-03-26T05:22:11.775130+0000","last_peered":"2024-03-26T05:22:11.775130+0000","last_clean":"2024-03-26T05:22:11.775130+0000","last_became_active":"2024-03-26T05:20:32.758743+0000","last_became_peered":"2024-03-26T05:20:32.758743+0000","last_unstale":"2024-03-26T05:22:11.775130+0000","last_undegraded":"2024-03-26T05:22:11.775130+0000","last_fullsized":"2024-03-26T05:22:11.775130+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:22:09.549826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:22:09.549826+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:23.041580+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"485'12570","reported_seq":12838,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:15.884176+0000","last_change":"2024-03-26T05:22:08.730432+0000","last_active":"2024-03-26T05:22:15.884176+0000","last_peered":"2024-03-26T05:22:15.884176+0000","last_clean":"2024-03-26T05:22:15.884176+0000","last_became_active":"2024-03-26T05:21:23.792443+0000","last_became_peered":"2024-03-26T05:21:23.792443+0000","last_unstale":"2024-03-26T05:22:15.884176+0000","last_undegraded":"2024-03-26T05:22:15.884176+0000","last_fullsized":"2024-03-26T05:22:15.884176+0000","mapping_epoch":408,"log_start":"434'7321","ondisk_log_start":"434'7321","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'12297","last_scrub_stamp":"2024-03-26T05:22:08.730080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:08.730080+0000","objects_scrubbed":939,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:27.526458+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43515904,"num_objects":664,"num_object_clones":0,"num_object_copies":1992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":275,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"443'27","reported_seq":996,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769493+0000","last_change":"2024-03-26T05:20:16.386890+0000","last_active":"2024-03-26T05:22:11.769493+0000","last_peered":"2024-03-26T05:22:11.769493+0000","last_clean":"2024-03-26T05:22:11.769493+0000","last_became_active":"2024-03-26T05:20:16.378745+0000","last_became_peered":"2024-03-26T05:20:16.378745+0000","last_unstale":"2024-03-26T05:22:11.769493+0000","last_undegraded":"2024-03-26T05:22:11.769493+0000","last_fullsized":"2024-03-26T05:22:11.769493+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'21","last_scrub_stamp":"2024-03-26T05:18:31.264666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:18:31.264666+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":211,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765335+0000","last_change":"2024-03-26T05:20:30.327239+0000","last_active":"2024-03-26T05:22:11.765335+0000","last_peered":"2024-03-26T05:22:11.765335+0000","last_clean":"2024-03-26T05:22:11.765335+0000","last_became_active":"2024-03-26T05:20:30.326917+0000","last_became_peered":"2024-03-26T05:20:30.326917+0000","last_unstale":"2024-03-26T05:22:11.765335+0000","last_undegraded":"2024-03-26T05:22:11.765335+0000","last_fullsized":"2024-03-26T05:22:11.765335+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:58.239217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:19:58.239217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":918,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.779683+0000","last_change":"2024-03-26T05:20:15.381990+0000","last_active":"2024-03-26T05:22:11.779683+0000","last_peered":"2024-03-26T05:22:11.779683+0000","last_clean":"2024-03-26T05:22:11.779683+0000","last_became_active":"2024-03-26T05:20:15.380969+0000","last_became_peered":"2024-03-26T05:20:15.380969+0000","last_unstale":"2024-03-26T05:22:11.779683+0000","last_undegraded":"2024-03-26T05:22:11.779683+0000","last_fullsized":"2024-03-26T05:22:11.779683+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:52.371799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:52.371799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":907,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.776444+0000","last_change":"2024-03-26T05:19:50.540208+0000","last_active":"2024-03-26T05:22:11.776444+0000","last_peered":"2024-03-26T05:22:11.776444+0000","last_clean":"2024-03-26T05:22:11.776444+0000","last_became_active":"2024-03-26T05:15:32.639943+0000","last_became_peered":"2024-03-26T05:15:32.639943+0000","last_unstale":"2024-03-26T05:22:11.776444+0000","last_undegraded":"2024-03-26T05:22:11.776444+0000","last_fullsized":"2024-03-26T05:22:11.776444+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:50.540100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:50.540100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"485'12414","reported_seq":20669,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:15.880521+0000","last_change":"2024-03-26T05:22:10.650829+0000","last_active":"2024-03-26T05:22:15.880521+0000","last_peered":"2024-03-26T05:22:15.880521+0000","last_clean":"2024-03-26T05:22:15.880521+0000","last_became_active":"2024-03-26T05:21:25.887661+0000","last_became_peered":"2024-03-26T05:21:25.887661+0000","last_unstale":"2024-03-26T05:22:15.880521+0000","last_undegraded":"2024-03-26T05:22:15.880521+0000","last_fullsized":"2024-03-26T05:22:15.880521+0000","mapping_epoch":462,"log_start":"433'7126","ondisk_log_start":"433'7126","created":408,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'12200","last_scrub_stamp":"2024-03-26T05:22:10.650549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:10.650549+0000","objects_scrubbed":821,"log_size":5288,"log_dups_size":0,"ondisk_log_size":5288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:14.499841+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39649280,"num_objects":605,"num_object_clones":0,"num_object_copies":1815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"443'67","reported_seq":1038,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764999+0000","last_change":"2024-03-26T05:21:08.573480+0000","last_active":"2024-03-26T05:22:11.764999+0000","last_peered":"2024-03-26T05:22:11.764999+0000","last_clean":"2024-03-26T05:22:11.764999+0000","last_became_active":"2024-03-26T05:15:32.626640+0000","last_became_peered":"2024-03-26T05:15:32.626640+0000","last_unstale":"2024-03-26T05:22:11.764999+0000","last_undegraded":"2024-03-26T05:22:11.764999+0000","last_fullsized":"2024-03-26T05:22:11.764999+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'67","last_scrub_stamp":"2024-03-26T05:21:08.573331+0000","last_deep_scrub":"343'51","last_deep_scrub_stamp":"2024-03-26T05:17:45.197613+0000","last_clean_scrub_stamp":"2024-03-26T05:21:08.573331+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:29.772598+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":812,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769521+0000","last_change":"2024-03-26T05:20:30.330168+0000","last_active":"2024-03-26T05:22:11.769521+0000","last_peered":"2024-03-26T05:22:11.769521+0000","last_clean":"2024-03-26T05:22:11.769521+0000","last_became_active":"2024-03-26T05:20:30.329736+0000","last_became_peered":"2024-03-26T05:20:30.329736+0000","last_unstale":"2024-03-26T05:22:11.769521+0000","last_undegraded":"2024-03-26T05:22:11.769521+0000","last_fullsized":"2024-03-26T05:22:11.769521+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:46.539478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:46.539478+0000","last_clean_scrub_stamp":"2024-03-26T05:18:46.539478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":942,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.778714+0000","last_change":"2024-03-26T05:20:30.344308+0000","last_active":"2024-03-26T05:22:11.778714+0000","last_peered":"2024-03-26T05:22:11.778714+0000","last_clean":"2024-03-26T05:22:11.778714+0000","last_became_active":"2024-03-26T05:20:30.343819+0000","last_became_peered":"2024-03-26T05:20:30.343819+0000","last_unstale":"2024-03-26T05:22:11.778714+0000","last_undegraded":"2024-03-26T05:22:11.778714+0000","last_fullsized":"2024-03-26T05:22:11.778714+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:57.997324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.997324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":817,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.768562+0000","last_change":"2024-03-26T05:19:48.333412+0000","last_active":"2024-03-26T05:22:11.768562+0000","last_peered":"2024-03-26T05:22:11.768562+0000","last_clean":"2024-03-26T05:22:11.768562+0000","last_became_active":"2024-03-26T05:15:32.633481+0000","last_became_peered":"2024-03-26T05:15:32.633481+0000","last_unstale":"2024-03-26T05:22:11.768562+0000","last_undegraded":"2024-03-26T05:22:11.768562+0000","last_fullsized":"2024-03-26T05:22:11.768562+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:48.333336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:08:39.681463+0000","last_clean_scrub_stamp":"2024-03-26T05:19:48.333336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"485'12434","reported_seq":12625,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:16.791664+0000","last_change":"2024-03-26T05:22:10.937498+0000","last_active":"2024-03-26T05:22:16.791664+0000","last_peered":"2024-03-26T05:22:16.791664+0000","last_clean":"2024-03-26T05:22:16.791664+0000","last_became_active":"2024-03-26T05:21:23.796343+0000","last_became_peered":"2024-03-26T05:21:23.796343+0000","last_unstale":"2024-03-26T05:22:16.791664+0000","last_undegraded":"2024-03-26T05:22:16.791664+0000","last_fullsized":"2024-03-26T05:22:16.791664+0000","mapping_epoch":408,"log_start":"439'7339","ondisk_log_start":"439'7339","created":408,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'12217","last_scrub_stamp":"2024-03-26T05:22:10.937244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:10.937244+0000","objects_scrubbed":788,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:30.789792+0000","scrub_duration":382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37027840,"num_objects":565,"num_object_clones":0,"num_object_copies":1695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":223,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":917,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.782742+0000","last_change":"2024-03-26T05:18:52.271198+0000","last_active":"2024-03-26T05:22:11.782742+0000","last_peered":"2024-03-26T05:22:11.782742+0000","last_clean":"2024-03-26T05:22:11.782742+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:22:11.782742+0000","last_undegraded":"2024-03-26T05:22:11.782742+0000","last_fullsized":"2024-03-26T05:22:11.782742+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:52.271117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:52.271117+0000","last_clean_scrub_stamp":"2024-03-26T05:18:52.271117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"443'43","reported_seq":802,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.771812+0000","last_change":"2024-03-26T05:20:15.454067+0000","last_active":"2024-03-26T05:22:11.771812+0000","last_peered":"2024-03-26T05:22:11.771812+0000","last_clean":"2024-03-26T05:22:11.771812+0000","last_became_active":"2024-03-26T05:20:15.361123+0000","last_became_peered":"2024-03-26T05:20:15.361123+0000","last_unstale":"2024-03-26T05:22:11.771812+0000","last_undegraded":"2024-03-26T05:22:11.771812+0000","last_fullsized":"2024-03-26T05:22:11.771812+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-26T05:14:55.963773+0000","last_deep_scrub":"188'31","last_deep_scrub_stamp":"2024-03-26T05:09:53.374102+0000","last_clean_scrub_stamp":"2024-03-26T05:14:55.963773+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":829,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769967+0000","last_change":"2024-03-26T05:20:15.374221+0000","last_active":"2024-03-26T05:22:11.769967+0000","last_peered":"2024-03-26T05:22:11.769967+0000","last_clean":"2024-03-26T05:22:11.769967+0000","last_became_active":"2024-03-26T05:20:15.373910+0000","last_became_peered":"2024-03-26T05:20:15.373910+0000","last_unstale":"2024-03-26T05:22:11.769967+0000","last_undegraded":"2024-03-26T05:22:11.769967+0000","last_fullsized":"2024-03-26T05:22:11.769967+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:20.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:20.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":903,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764370+0000","last_change":"2024-03-26T05:20:15.377202+0000","last_active":"2024-03-26T05:22:11.764370+0000","last_peered":"2024-03-26T05:22:11.764370+0000","last_clean":"2024-03-26T05:22:11.764370+0000","last_became_active":"2024-03-26T05:20:15.376729+0000","last_became_peered":"2024-03-26T05:20:15.376729+0000","last_unstale":"2024-03-26T05:22:11.764370+0000","last_undegraded":"2024-03-26T05:22:11.764370+0000","last_fullsized":"2024-03-26T05:22:11.764370+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:51.030982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:14:51.030982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"485'12187","reported_seq":24884,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:16.354170+0000","last_change":"2024-03-26T05:22:09.405309+0000","last_active":"2024-03-26T05:22:16.354170+0000","last_peered":"2024-03-26T05:22:16.354170+0000","last_clean":"2024-03-26T05:22:16.354170+0000","last_became_active":"2024-03-26T05:21:56.820341+0000","last_became_peered":"2024-03-26T05:21:56.820341+0000","last_unstale":"2024-03-26T05:22:16.354170+0000","last_undegraded":"2024-03-26T05:22:16.354170+0000","last_fullsized":"2024-03-26T05:22:16.354170+0000","mapping_epoch":479,"log_start":"434'6999","ondisk_log_start":"434'6999","created":408,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'11919","last_scrub_stamp":"2024-03-26T05:22:09.405170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:09.405170+0000","objects_scrubbed":821,"log_size":5188,"log_dups_size":0,"ondisk_log_size":5188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:10.051892+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36110336,"num_objects":551,"num_object_clones":0,"num_object_copies":1653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":270,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"439'12","reported_seq":1049,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769730+0000","last_change":"2024-03-26T05:20:15.466421+0000","last_active":"2024-03-26T05:22:11.769730+0000","last_peered":"2024-03-26T05:22:11.769730+0000","last_clean":"2024-03-26T05:22:11.769730+0000","last_became_active":"2024-03-26T05:20:15.379548+0000","last_became_peered":"2024-03-26T05:20:15.379548+0000","last_unstale":"2024-03-26T05:22:11.769730+0000","last_undegraded":"2024-03-26T05:22:11.769730+0000","last_fullsized":"2024-03-26T05:22:11.769730+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-26T05:12:03.961391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:12:03.961391+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"443'60","reported_seq":279,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.768865+0000","last_change":"2024-03-26T05:20:30.363622+0000","last_active":"2024-03-26T05:22:11.768865+0000","last_peered":"2024-03-26T05:22:11.768865+0000","last_clean":"2024-03-26T05:22:11.768865+0000","last_became_active":"2024-03-26T05:20:30.341167+0000","last_became_peered":"2024-03-26T05:20:30.341167+0000","last_unstale":"2024-03-26T05:22:11.768865+0000","last_undegraded":"2024-03-26T05:22:11.768865+0000","last_fullsized":"2024-03-26T05:22:11.768865+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'48","last_scrub_stamp":"2024-03-26T05:19:54.208469+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.208469+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":826,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769787+0000","last_change":"2024-03-26T05:20:15.470838+0000","last_active":"2024-03-26T05:22:11.769787+0000","last_peered":"2024-03-26T05:22:11.769787+0000","last_clean":"2024-03-26T05:22:11.769787+0000","last_became_active":"2024-03-26T05:20:15.453507+0000","last_became_peered":"2024-03-26T05:20:15.453507+0000","last_unstale":"2024-03-26T05:22:11.769787+0000","last_undegraded":"2024-03-26T05:22:11.769787+0000","last_fullsized":"2024-03-26T05:22:11.769787+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_clean_scrub_stamp":"2024-03-26T05:11:29.959120+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":296,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:13.831912+0000","last_change":"2024-03-26T05:22:13.831949+0000","last_active":"2024-03-26T05:22:13.831912+0000","last_peered":"2024-03-26T05:22:13.831912+0000","last_clean":"2024-03-26T05:22:13.831912+0000","last_became_active":"2024-03-26T05:20:30.343599+0000","last_became_peered":"2024-03-26T05:20:30.343599+0000","last_unstale":"2024-03-26T05:22:13.831912+0000","last_undegraded":"2024-03-26T05:22:13.831912+0000","last_fullsized":"2024-03-26T05:22:13.831912+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:22:13.831855+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:22:13.831855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:33.159672+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"485'12566","reported_seq":23264,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:16.072766+0000","last_change":"2024-03-26T05:22:10.882572+0000","last_active":"2024-03-26T05:22:16.072766+0000","last_peered":"2024-03-26T05:22:16.072766+0000","last_clean":"2024-03-26T05:22:16.072766+0000","last_became_active":"2024-03-26T05:21:46.757565+0000","last_became_peered":"2024-03-26T05:21:46.757565+0000","last_unstale":"2024-03-26T05:22:16.072766+0000","last_undegraded":"2024-03-26T05:22:16.072766+0000","last_fullsized":"2024-03-26T05:22:16.072766+0000","mapping_epoch":473,"log_start":"439'7535","ondisk_log_start":"439'7535","created":460,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":5,"last_scrub":"484'12372","last_scrub_stamp":"2024-03-26T05:22:10.882327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:10.882327+0000","objects_scrubbed":810,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:19.368312+0000","scrub_duration":398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39976960,"num_objects":610,"num_object_clones":0,"num_object_copies":1830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"443'24","reported_seq":952,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765033+0000","last_change":"2024-03-26T05:20:04.274955+0000","last_active":"2024-03-26T05:22:11.765033+0000","last_peered":"2024-03-26T05:22:11.765033+0000","last_clean":"2024-03-26T05:22:11.765033+0000","last_became_active":"2024-03-26T05:15:32.640871+0000","last_became_peered":"2024-03-26T05:15:32.640871+0000","last_unstale":"2024-03-26T05:22:11.765033+0000","last_undegraded":"2024-03-26T05:22:11.765033+0000","last_fullsized":"2024-03-26T05:22:11.765033+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'18","last_scrub_stamp":"2024-03-26T05:20:04.273413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:04.273413+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":32,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":984,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.883082+0000","last_change":"2024-03-26T05:22:11.883125+0000","last_active":"2024-03-26T05:22:11.883082+0000","last_peered":"2024-03-26T05:22:11.883082+0000","last_clean":"2024-03-26T05:22:11.883082+0000","last_became_active":"2024-03-26T05:20:30.343651+0000","last_became_peered":"2024-03-26T05:20:30.343651+0000","last_unstale":"2024-03-26T05:22:11.883082+0000","last_undegraded":"2024-03-26T05:22:11.883082+0000","last_fullsized":"2024-03-26T05:22:11.883082+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:22:11.882991+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:17:50.147911+0000","last_clean_scrub_stamp":"2024-03-26T05:22:11.882991+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:18.277915+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":209,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.766422+0000","last_change":"2024-03-26T05:20:30.340915+0000","last_active":"2024-03-26T05:22:11.766422+0000","last_peered":"2024-03-26T05:22:11.766422+0000","last_clean":"2024-03-26T05:22:11.766422+0000","last_became_active":"2024-03-26T05:20:30.330443+0000","last_became_peered":"2024-03-26T05:20:30.330443+0000","last_unstale":"2024-03-26T05:22:11.766422+0000","last_undegraded":"2024-03-26T05:22:11.766422+0000","last_fullsized":"2024-03-26T05:22:11.766422+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:36.146286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:18:36.146286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":816,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769237+0000","last_change":"2024-03-26T05:20:30.321663+0000","last_active":"2024-03-26T05:22:11.769237+0000","last_peered":"2024-03-26T05:22:11.769237+0000","last_clean":"2024-03-26T05:22:11.769237+0000","last_became_active":"2024-03-26T05:20:30.321301+0000","last_became_peered":"2024-03-26T05:20:30.321301+0000","last_unstale":"2024-03-26T05:22:11.769237+0000","last_undegraded":"2024-03-26T05:22:11.769237+0000","last_fullsized":"2024-03-26T05:22:11.769237+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:07.006139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:07.006139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.19","version":"485'12246","reported_seq":16994,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:16.764137+0000","last_change":"2024-03-26T05:22:11.825639+0000","last_active":"2024-03-26T05:22:16.764137+0000","last_peered":"2024-03-26T05:22:16.764137+0000","last_clean":"2024-03-26T05:22:16.764137+0000","last_became_active":"2024-03-26T05:21:51.789764+0000","last_became_peered":"2024-03-26T05:21:51.789764+0000","last_unstale":"2024-03-26T05:22:16.764137+0000","last_undegraded":"2024-03-26T05:22:16.764137+0000","last_fullsized":"2024-03-26T05:22:16.764137+0000","mapping_epoch":477,"log_start":"438'7197","ondisk_log_start":"438'7197","created":460,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"485'12073","last_scrub_stamp":"2024-03-26T05:22:11.825329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:11.825329+0000","objects_scrubbed":729,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:23.935057+0000","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36110336,"num_objects":551,"num_object_clones":0,"num_object_copies":1653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":178,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"443'39","reported_seq":986,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.773041+0000","last_change":"2024-03-26T05:19:54.421287+0000","last_active":"2024-03-26T05:22:11.773041+0000","last_peered":"2024-03-26T05:22:11.773041+0000","last_clean":"2024-03-26T05:22:11.773041+0000","last_became_active":"2024-03-26T05:15:32.634434+0000","last_became_peered":"2024-03-26T05:15:32.634434+0000","last_unstale":"2024-03-26T05:22:11.773041+0000","last_undegraded":"2024-03-26T05:22:11.773041+0000","last_fullsized":"2024-03-26T05:22:11.773041+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'33","last_scrub_stamp":"2024-03-26T05:19:54.421104+0000","last_deep_scrub":"429'33","last_deep_scrub_stamp":"2024-03-26T05:19:54.421104+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.421104+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":792,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:14.381369+0000","last_change":"2024-03-26T05:22:14.381389+0000","last_active":"2024-03-26T05:22:14.381369+0000","last_peered":"2024-03-26T05:22:14.381369+0000","last_clean":"2024-03-26T05:22:14.381369+0000","last_became_active":"2024-03-26T05:20:15.375661+0000","last_became_peered":"2024-03-26T05:20:15.375661+0000","last_unstale":"2024-03-26T05:22:14.381369+0000","last_undegraded":"2024-03-26T05:22:14.381369+0000","last_fullsized":"2024-03-26T05:22:14.381369+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:22:14.381309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:22:14.381309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:29.507604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":901,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764900+0000","last_change":"2024-03-26T05:18:03.064748+0000","last_active":"2024-03-26T05:22:11.764900+0000","last_peered":"2024-03-26T05:22:11.764900+0000","last_clean":"2024-03-26T05:22:11.764900+0000","last_became_active":"2024-03-26T05:15:32.641031+0000","last_became_peered":"2024-03-26T05:15:32.641031+0000","last_unstale":"2024-03-26T05:22:11.764900+0000","last_undegraded":"2024-03-26T05:22:11.764900+0000","last_fullsized":"2024-03-26T05:22:11.764900+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:03.064638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:18:03.064638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":922,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.779775+0000","last_change":"2024-03-26T05:20:32.095405+0000","last_active":"2024-03-26T05:22:11.779775+0000","last_peered":"2024-03-26T05:22:11.779775+0000","last_clean":"2024-03-26T05:22:11.779775+0000","last_became_active":"2024-03-26T05:20:15.379306+0000","last_became_peered":"2024-03-26T05:20:15.379306+0000","last_unstale":"2024-03-26T05:22:11.779775+0000","last_undegraded":"2024-03-26T05:22:11.779775+0000","last_fullsized":"2024-03-26T05:22:11.779775+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:32.094955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:20:32.094955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"443'32","reported_seq":228,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769000+0000","last_change":"2024-03-26T05:20:30.356411+0000","last_active":"2024-03-26T05:22:11.769000+0000","last_peered":"2024-03-26T05:22:11.769000+0000","last_clean":"2024-03-26T05:22:11.769000+0000","last_became_active":"2024-03-26T05:20:30.329795+0000","last_became_peered":"2024-03-26T05:20:30.329795+0000","last_unstale":"2024-03-26T05:22:11.769000+0000","last_undegraded":"2024-03-26T05:22:11.769000+0000","last_fullsized":"2024-03-26T05:22:11.769000+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'26","last_scrub_stamp":"2024-03-26T05:19:53.221647+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:19:53.221647+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":926,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.767542+0000","last_change":"2024-03-26T05:18:51.276677+0000","last_active":"2024-03-26T05:22:11.767542+0000","last_peered":"2024-03-26T05:22:11.767542+0000","last_clean":"2024-03-26T05:22:11.767542+0000","last_became_active":"2024-03-26T05:15:32.632387+0000","last_became_peered":"2024-03-26T05:15:32.632387+0000","last_unstale":"2024-03-26T05:22:11.767542+0000","last_undegraded":"2024-03-26T05:22:11.767542+0000","last_fullsized":"2024-03-26T05:22:11.767542+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:51.276565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:45.352678+0000","last_clean_scrub_stamp":"2024-03-26T05:18:51.276565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"336'11","reported_seq":1186,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765742+0000","last_change":"2024-03-26T05:19:56.382330+0000","last_active":"2024-03-26T05:22:11.765742+0000","last_peered":"2024-03-26T05:22:11.765742+0000","last_clean":"2024-03-26T05:22:11.765742+0000","last_became_active":"2024-03-26T05:15:32.636392+0000","last_became_peered":"2024-03-26T05:15:32.636392+0000","last_unstale":"2024-03-26T05:22:11.765742+0000","last_undegraded":"2024-03-26T05:22:11.765742+0000","last_fullsized":"2024-03-26T05:22:11.765742+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-03-26T05:19:56.382205+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:19:56.382205+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":636,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:14.800183+0000","last_change":"2024-03-26T05:22:14.800222+0000","last_active":"2024-03-26T05:22:14.800183+0000","last_peered":"2024-03-26T05:22:14.800183+0000","last_clean":"2024-03-26T05:22:14.800183+0000","last_became_active":"2024-03-26T05:20:30.344267+0000","last_became_peered":"2024-03-26T05:20:30.344267+0000","last_unstale":"2024-03-26T05:22:14.800183+0000","last_undegraded":"2024-03-26T05:22:14.800183+0000","last_fullsized":"2024-03-26T05:22:14.800183+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:22:14.800084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:22:14.800084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:28.864561+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"443'49","reported_seq":997,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765251+0000","last_change":"2024-03-26T05:19:58.389112+0000","last_active":"2024-03-26T05:22:11.765251+0000","last_peered":"2024-03-26T05:22:11.765251+0000","last_clean":"2024-03-26T05:22:11.765251+0000","last_became_active":"2024-03-26T05:15:32.626448+0000","last_became_peered":"2024-03-26T05:15:32.626448+0000","last_unstale":"2024-03-26T05:22:11.765251+0000","last_undegraded":"2024-03-26T05:22:11.765251+0000","last_fullsized":"2024-03-26T05:22:11.765251+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'39","last_scrub_stamp":"2024-03-26T05:19:58.388933+0000","last_deep_scrub":"345'39","last_deep_scrub_stamp":"2024-03-26T05:15:56.674423+0000","last_clean_scrub_stamp":"2024-03-26T05:19:58.388933+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":818,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769165+0000","last_change":"2024-03-26T05:20:15.364695+0000","last_active":"2024-03-26T05:22:11.769165+0000","last_peered":"2024-03-26T05:22:11.769165+0000","last_clean":"2024-03-26T05:22:11.769165+0000","last_became_active":"2024-03-26T05:20:15.364437+0000","last_became_peered":"2024-03-26T05:20:15.364437+0000","last_unstale":"2024-03-26T05:22:11.769165+0000","last_undegraded":"2024-03-26T05:22:11.769165+0000","last_fullsized":"2024-03-26T05:22:11.769165+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:01.962048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:12:01.962048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"443'11","reported_seq":302,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.777651+0000","last_change":"2024-03-26T05:20:30.356968+0000","last_active":"2024-03-26T05:22:11.777651+0000","last_peered":"2024-03-26T05:22:11.777651+0000","last_clean":"2024-03-26T05:22:11.777651+0000","last_became_active":"2024-03-26T05:20:30.338470+0000","last_became_peered":"2024-03-26T05:20:30.338470+0000","last_unstale":"2024-03-26T05:22:11.777651+0000","last_undegraded":"2024-03-26T05:22:11.777651+0000","last_fullsized":"2024-03-26T05:22:11.777651+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-26T05:18:40.082750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:18:40.082750+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":914,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.772639+0000","last_change":"2024-03-26T05:19:53.535068+0000","last_active":"2024-03-26T05:22:11.772639+0000","last_peered":"2024-03-26T05:22:11.772639+0000","last_clean":"2024-03-26T05:22:11.772639+0000","last_became_active":"2024-03-26T05:15:32.631444+0000","last_became_peered":"2024-03-26T05:15:32.631444+0000","last_unstale":"2024-03-26T05:22:11.772639+0000","last_undegraded":"2024-03-26T05:22:11.772639+0000","last_fullsized":"2024-03-26T05:22:11.772639+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:53.534717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:59.508745+0000","last_clean_scrub_stamp":"2024-03-26T05:19:53.534717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"443'32","reported_seq":800,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764965+0000","last_change":"2024-03-26T05:20:40.672261+0000","last_active":"2024-03-26T05:22:11.764965+0000","last_peered":"2024-03-26T05:22:11.764965+0000","last_clean":"2024-03-26T05:22:11.764965+0000","last_became_active":"2024-03-26T05:20:30.323810+0000","last_became_peered":"2024-03-26T05:20:30.323810+0000","last_unstale":"2024-03-26T05:22:11.764965+0000","last_undegraded":"2024-03-26T05:22:11.764965+0000","last_fullsized":"2024-03-26T05:22:11.764965+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'32","last_scrub_stamp":"2024-03-26T05:20:40.672162+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:20:40.672162+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":886,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769923+0000","last_change":"2024-03-26T05:20:15.372145+0000","last_active":"2024-03-26T05:22:11.769923+0000","last_peered":"2024-03-26T05:22:11.769923+0000","last_clean":"2024-03-26T05:22:11.769923+0000","last_became_active":"2024-03-26T05:20:15.371796+0000","last_became_peered":"2024-03-26T05:20:15.371796+0000","last_unstale":"2024-03-26T05:22:11.769923+0000","last_undegraded":"2024-03-26T05:22:11.769923+0000","last_fullsized":"2024-03-26T05:22:11.769923+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:21.515294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:11:21.515294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":950,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.775854+0000","last_change":"2024-03-26T05:20:30.332976+0000","last_active":"2024-03-26T05:22:11.775854+0000","last_peered":"2024-03-26T05:22:11.775854+0000","last_clean":"2024-03-26T05:22:11.775854+0000","last_became_active":"2024-03-26T05:20:30.332245+0000","last_became_peered":"2024-03-26T05:20:30.332245+0000","last_unstale":"2024-03-26T05:22:11.775854+0000","last_undegraded":"2024-03-26T05:22:11.775854+0000","last_fullsized":"2024-03-26T05:22:11.775854+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:05.539590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:55.496800+0000","last_clean_scrub_stamp":"2024-03-26T05:20:05.539590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":914,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.772610+0000","last_change":"2024-03-26T05:20:30.346728+0000","last_active":"2024-03-26T05:22:11.772610+0000","last_peered":"2024-03-26T05:22:11.772610+0000","last_clean":"2024-03-26T05:22:11.772610+0000","last_became_active":"2024-03-26T05:20:30.346313+0000","last_became_peered":"2024-03-26T05:20:30.346313+0000","last_unstale":"2024-03-26T05:22:11.772610+0000","last_undegraded":"2024-03-26T05:22:11.772610+0000","last_fullsized":"2024-03-26T05:22:11.772610+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:38.817507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:18:38.817507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"443'21","reported_seq":962,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.775943+0000","last_change":"2024-03-26T05:20:01.620894+0000","last_active":"2024-03-26T05:22:11.775943+0000","last_peered":"2024-03-26T05:22:11.775943+0000","last_clean":"2024-03-26T05:22:11.775943+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:22:11.775943+0000","last_undegraded":"2024-03-26T05:22:11.775943+0000","last_fullsized":"2024-03-26T05:22:11.775943+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'15","last_scrub_stamp":"2024-03-26T05:20:01.620742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:20:01.620742+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":790,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769272+0000","last_change":"2024-03-26T05:20:15.375970+0000","last_active":"2024-03-26T05:22:11.769272+0000","last_peered":"2024-03-26T05:22:11.769272+0000","last_clean":"2024-03-26T05:22:11.769272+0000","last_became_active":"2024-03-26T05:20:15.375609+0000","last_became_peered":"2024-03-26T05:20:15.375609+0000","last_unstale":"2024-03-26T05:22:11.769272+0000","last_undegraded":"2024-03-26T05:22:11.769272+0000","last_fullsized":"2024-03-26T05:22:11.769272+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:33.016072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:11:33.016072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":908,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764193+0000","last_change":"2024-03-26T05:19:55.403001+0000","last_active":"2024-03-26T05:22:11.764193+0000","last_peered":"2024-03-26T05:22:11.764193+0000","last_clean":"2024-03-26T05:22:11.764193+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:22:11.764193+0000","last_undegraded":"2024-03-26T05:22:11.764193+0000","last_fullsized":"2024-03-26T05:22:11.764193+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:55.402737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:19:55.402737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":931,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.767293+0000","last_change":"2024-03-26T05:20:30.344287+0000","last_active":"2024-03-26T05:22:11.767293+0000","last_peered":"2024-03-26T05:22:11.767293+0000","last_clean":"2024-03-26T05:22:11.767293+0000","last_became_active":"2024-03-26T05:20:30.343972+0000","last_became_peered":"2024-03-26T05:20:30.343972+0000","last_unstale":"2024-03-26T05:22:11.767293+0000","last_undegraded":"2024-03-26T05:22:11.767293+0000","last_fullsized":"2024-03-26T05:22:11.767293+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:38.324541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:18:38.324541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"443'33","reported_seq":968,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.772965+0000","last_change":"2024-03-26T05:19:57.364173+0000","last_active":"2024-03-26T05:22:11.772965+0000","last_peered":"2024-03-26T05:22:11.772965+0000","last_clean":"2024-03-26T05:22:11.772965+0000","last_became_active":"2024-03-26T05:15:32.636359+0000","last_became_peered":"2024-03-26T05:15:32.636359+0000","last_unstale":"2024-03-26T05:22:11.772965+0000","last_undegraded":"2024-03-26T05:22:11.772965+0000","last_fullsized":"2024-03-26T05:22:11.772965+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'25","last_scrub_stamp":"2024-03-26T05:19:57.363571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.363571+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":361,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769583+0000","last_change":"2024-03-26T05:20:30.351327+0000","last_active":"2024-03-26T05:22:11.769583+0000","last_peered":"2024-03-26T05:22:11.769583+0000","last_clean":"2024-03-26T05:22:11.769583+0000","last_became_active":"2024-03-26T05:20:30.350957+0000","last_became_peered":"2024-03-26T05:20:30.350957+0000","last_unstale":"2024-03-26T05:22:11.769583+0000","last_undegraded":"2024-03-26T05:22:11.769583+0000","last_fullsized":"2024-03-26T05:22:11.769583+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:27.232872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:06:56.886632+0000","last_clean_scrub_stamp":"2024-03-26T05:20:27.232872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":827,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769048+0000","last_change":"2024-03-26T05:20:15.377844+0000","last_active":"2024-03-26T05:22:11.769048+0000","last_peered":"2024-03-26T05:22:11.769048+0000","last_clean":"2024-03-26T05:22:11.769048+0000","last_became_active":"2024-03-26T05:20:15.377506+0000","last_became_peered":"2024-03-26T05:20:15.377506+0000","last_unstale":"2024-03-26T05:22:11.769048+0000","last_undegraded":"2024-03-26T05:22:11.769048+0000","last_fullsized":"2024-03-26T05:22:11.769048+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:11:23.957542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:11:23.957542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":912,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765146+0000","last_change":"2024-03-26T05:20:28.593855+0000","last_active":"2024-03-26T05:22:11.765146+0000","last_peered":"2024-03-26T05:22:11.765146+0000","last_clean":"2024-03-26T05:22:11.765146+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:22:11.765146+0000","last_undegraded":"2024-03-26T05:22:11.765146+0000","last_fullsized":"2024-03-26T05:22:11.765146+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:28.593685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:53.017168+0000","last_clean_scrub_stamp":"2024-03-26T05:20:28.593685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"443'14","reported_seq":932,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764182+0000","last_change":"2024-03-26T05:18:56.877975+0000","last_active":"2024-03-26T05:22:11.764182+0000","last_peered":"2024-03-26T05:22:11.764182+0000","last_clean":"2024-03-26T05:22:11.764182+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:22:11.764182+0000","last_undegraded":"2024-03-26T05:22:11.764182+0000","last_fullsized":"2024-03-26T05:22:11.764182+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'10","last_scrub_stamp":"2024-03-26T05:18:56.877852+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:18:56.877852+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":943,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.772493+0000","last_change":"2024-03-26T05:20:30.351206+0000","last_active":"2024-03-26T05:22:11.772493+0000","last_peered":"2024-03-26T05:22:11.772493+0000","last_clean":"2024-03-26T05:22:11.772493+0000","last_became_active":"2024-03-26T05:20:30.350521+0000","last_became_peered":"2024-03-26T05:20:30.350521+0000","last_unstale":"2024-03-26T05:22:11.772493+0000","last_undegraded":"2024-03-26T05:22:11.772493+0000","last_fullsized":"2024-03-26T05:22:11.772493+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:40.333981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:11.070341+0000","last_clean_scrub_stamp":"2024-03-26T05:18:40.333981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":917,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764978+0000","last_change":"2024-03-26T05:20:37.670090+0000","last_active":"2024-03-26T05:22:11.764978+0000","last_peered":"2024-03-26T05:22:11.764978+0000","last_clean":"2024-03-26T05:22:11.764978+0000","last_became_active":"2024-03-26T05:20:15.372788+0000","last_became_peered":"2024-03-26T05:20:15.372788+0000","last_unstale":"2024-03-26T05:22:11.764978+0000","last_undegraded":"2024-03-26T05:22:11.764978+0000","last_fullsized":"2024-03-26T05:22:11.764978+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:37.669979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:20:37.669979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":811,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.768893+0000","last_change":"2024-03-26T05:20:15.364726+0000","last_active":"2024-03-26T05:22:11.768893+0000","last_peered":"2024-03-26T05:22:11.768893+0000","last_clean":"2024-03-26T05:22:11.768893+0000","last_became_active":"2024-03-26T05:20:15.364379+0000","last_became_peered":"2024-03-26T05:20:15.364379+0000","last_unstale":"2024-03-26T05:22:11.768893+0000","last_undegraded":"2024-03-26T05:22:11.768893+0000","last_fullsized":"2024-03-26T05:22:11.768893+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:10.020572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:12:10.020572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"485'12606","reported_seq":12864,"reported_epoch":485,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T05:22:16.422990+0000","last_change":"2024-03-26T05:22:10.887472+0000","last_active":"2024-03-26T05:22:16.422990+0000","last_peered":"2024-03-26T05:22:16.422990+0000","last_clean":"2024-03-26T05:21:24.457072+0000","last_became_active":"2024-03-26T05:21:26.961051+0000","last_became_peered":"2024-03-26T05:21:26.961051+0000","last_unstale":"2024-03-26T05:22:16.422990+0000","last_undegraded":"2024-03-26T05:22:16.422990+0000","last_fullsized":"2024-03-26T05:22:16.422990+0000","mapping_epoch":463,"log_start":"434'7269","ondisk_log_start":"434'7269","created":460,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":5,"last_scrub":"431'6421","last_scrub_stamp":"2024-03-26T05:20:00.971796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:20:00.971796+0000","objects_scrubbed":6351,"log_size":5337,"log_dups_size":0,"ondisk_log_size":5337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35454976,"num_objects":541,"num_object_clones":0,"num_object_copies":1623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1623,"num_objects_unfound":0,"num_objects_dirty":541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7085,"num_write_kb":244032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[1,3,7],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":541}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"443'39","reported_seq":1085,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765533+0000","last_change":"2024-03-26T05:20:30.366464+0000","last_active":"2024-03-26T05:22:11.765533+0000","last_peered":"2024-03-26T05:22:11.765533+0000","last_clean":"2024-03-26T05:22:11.765533+0000","last_became_active":"2024-03-26T05:20:30.343540+0000","last_became_peered":"2024-03-26T05:20:30.343540+0000","last_unstale":"2024-03-26T05:22:11.765533+0000","last_undegraded":"2024-03-26T05:22:11.765533+0000","last_fullsized":"2024-03-26T05:22:11.765533+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'31","last_scrub_stamp":"2024-03-26T05:19:57.767398+0000","last_deep_scrub":"413'31","last_deep_scrub_stamp":"2024-03-26T05:18:46.881054+0000","last_clean_scrub_stamp":"2024-03-26T05:19:57.767398+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":915,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.763980+0000","last_change":"2024-03-26T05:18:47.776529+0000","last_active":"2024-03-26T05:22:11.763980+0000","last_peered":"2024-03-26T05:22:11.763980+0000","last_clean":"2024-03-26T05:22:11.763980+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:22:11.763980+0000","last_undegraded":"2024-03-26T05:22:11.763980+0000","last_fullsized":"2024-03-26T05:22:11.763980+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:47.776444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:47.776444+0000","last_clean_scrub_stamp":"2024-03-26T05:18:47.776444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":911,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.773221+0000","last_change":"2024-03-26T05:19:48.378592+0000","last_active":"2024-03-26T05:22:11.773221+0000","last_peered":"2024-03-26T05:22:11.773221+0000","last_clean":"2024-03-26T05:22:11.773221+0000","last_became_active":"2024-03-26T05:15:32.634716+0000","last_became_peered":"2024-03-26T05:15:32.634716+0000","last_unstale":"2024-03-26T05:22:11.773221+0000","last_undegraded":"2024-03-26T05:22:11.773221+0000","last_fullsized":"2024-03-26T05:22:11.773221+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:48.378485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:48.378485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":922,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.780088+0000","last_change":"2024-03-26T05:20:30.342850+0000","last_active":"2024-03-26T05:22:11.780088+0000","last_peered":"2024-03-26T05:22:11.780088+0000","last_clean":"2024-03-26T05:22:11.780088+0000","last_became_active":"2024-03-26T05:20:30.342550+0000","last_became_peered":"2024-03-26T05:20:30.342550+0000","last_unstale":"2024-03-26T05:22:11.780088+0000","last_undegraded":"2024-03-26T05:22:11.780088+0000","last_fullsized":"2024-03-26T05:22:11.780088+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:00.151355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.151355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"485'12587","reported_seq":12836,"reported_epoch":485,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T05:22:16.810936+0000","last_change":"2024-03-26T05:21:57.040702+0000","last_active":"2024-03-26T05:22:16.810936+0000","last_peered":"2024-03-26T05:22:16.810936+0000","last_clean":"2024-03-26T05:21:23.856457+0000","last_became_active":"2024-03-26T05:21:26.962137+0000","last_became_peered":"2024-03-26T05:21:26.962137+0000","last_unstale":"2024-03-26T05:22:16.810936+0000","last_undegraded":"2024-03-26T05:22:16.810936+0000","last_fullsized":"2024-03-26T05:22:16.810936+0000","mapping_epoch":463,"log_start":"439'7391","ondisk_log_start":"439'7391","created":460,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":5,"last_scrub":"428'5729","last_scrub_stamp":"2024-03-26T05:19:50.459009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:19:50.459009+0000","objects_scrubbed":5667,"log_size":5196,"log_dups_size":0,"ondisk_log_size":5196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39550976,"num_objects":604,"num_object_clones":0,"num_object_copies":1812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1208,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":245056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[0,3,6],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3,6","objects":604}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"443'42","reported_seq":986,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.772947+0000","last_change":"2024-03-26T05:18:57.256490+0000","last_active":"2024-03-26T05:22:11.772947+0000","last_peered":"2024-03-26T05:22:11.772947+0000","last_clean":"2024-03-26T05:22:11.772947+0000","last_became_active":"2024-03-26T05:15:32.634216+0000","last_became_peered":"2024-03-26T05:15:32.634216+0000","last_unstale":"2024-03-26T05:22:11.772947+0000","last_undegraded":"2024-03-26T05:22:11.772947+0000","last_fullsized":"2024-03-26T05:22:11.772947+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'32","last_scrub_stamp":"2024-03-26T05:18:57.255727+0000","last_deep_scrub":"290'32","last_deep_scrub_stamp":"2024-03-26T05:12:15.970313+0000","last_clean_scrub_stamp":"2024-03-26T05:18:57.255727+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":930,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:13.058300+0000","last_change":"2024-03-26T05:22:13.058300+0000","last_active":"2024-03-26T05:22:13.058300+0000","last_peered":"2024-03-26T05:22:13.058300+0000","last_clean":"2024-03-26T05:22:13.058300+0000","last_became_active":"2024-03-26T05:20:15.361116+0000","last_became_peered":"2024-03-26T05:20:15.361116+0000","last_unstale":"2024-03-26T05:22:13.058300+0000","last_undegraded":"2024-03-26T05:22:13.058300+0000","last_fullsized":"2024-03-26T05:22:13.058300+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:22:13.058181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:22:13.058181+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:36.813678+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"439'12","reported_seq":1215,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765526+0000","last_change":"2024-03-26T05:21:02.540243+0000","last_active":"2024-03-26T05:22:11.765526+0000","last_peered":"2024-03-26T05:22:11.765526+0000","last_clean":"2024-03-26T05:22:11.765526+0000","last_became_active":"2024-03-26T05:20:15.375030+0000","last_became_peered":"2024-03-26T05:20:15.375030+0000","last_unstale":"2024-03-26T05:22:11.765526+0000","last_undegraded":"2024-03-26T05:22:11.765526+0000","last_fullsized":"2024-03-26T05:22:11.765526+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'12","last_scrub_stamp":"2024-03-26T05:21:02.540128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:21:02.540128+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"429'2","reported_seq":851,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769719+0000","last_change":"2024-03-26T05:20:30.352085+0000","last_active":"2024-03-26T05:22:11.769719+0000","last_peered":"2024-03-26T05:22:11.769719+0000","last_clean":"2024-03-26T05:22:11.769719+0000","last_became_active":"2024-03-26T05:20:30.341958+0000","last_became_peered":"2024-03-26T05:20:30.341958+0000","last_unstale":"2024-03-26T05:22:11.769719+0000","last_undegraded":"2024-03-26T05:22:11.769719+0000","last_fullsized":"2024-03-26T05:22:11.769719+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'2","last_scrub_stamp":"2024-03-26T05:19:54.275264+0000","last_deep_scrub":"429'2","last_deep_scrub_stamp":"2024-03-26T05:19:54.275264+0000","last_clean_scrub_stamp":"2024-03-26T05:19:54.275264+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.12","version":"485'12185","reported_seq":16238,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:15.857078+0000","last_change":"2024-03-26T05:22:09.645260+0000","last_active":"2024-03-26T05:22:15.857078+0000","last_peered":"2024-03-26T05:22:15.857078+0000","last_clean":"2024-03-26T05:22:15.857078+0000","last_became_active":"2024-03-26T05:21:40.985985+0000","last_became_peered":"2024-03-26T05:21:40.985985+0000","last_unstale":"2024-03-26T05:22:15.857078+0000","last_undegraded":"2024-03-26T05:22:15.857078+0000","last_fullsized":"2024-03-26T05:22:15.857078+0000","mapping_epoch":469,"log_start":"433'6713","ondisk_log_start":"433'6713","created":460,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":5,"last_scrub":"484'11960","last_scrub_stamp":"2024-03-26T05:22:09.644943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:09.644943+0000","objects_scrubbed":846,"log_size":5472,"log_dups_size":0,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:17.719079+0000","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40239104,"num_objects":614,"num_object_clones":0,"num_object_copies":1842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"443'38","reported_seq":979,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764595+0000","last_change":"2024-03-26T05:20:15.462061+0000","last_active":"2024-03-26T05:22:11.764595+0000","last_peered":"2024-03-26T05:22:11.764595+0000","last_clean":"2024-03-26T05:22:11.764595+0000","last_became_active":"2024-03-26T05:20:15.362085+0000","last_became_peered":"2024-03-26T05:20:15.362085+0000","last_unstale":"2024-03-26T05:22:11.764595+0000","last_undegraded":"2024-03-26T05:22:11.764595+0000","last_fullsized":"2024-03-26T05:22:11.764595+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'32","last_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_deep_scrub":"322'32","last_deep_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_clean_scrub_stamp":"2024-03-26T05:15:10.715786+0000","objects_scrubbed":6,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":309,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:15.091578+0000","last_change":"2024-03-26T05:22:15.091617+0000","last_active":"2024-03-26T05:22:15.091578+0000","last_peered":"2024-03-26T05:22:15.091578+0000","last_clean":"2024-03-26T05:22:15.091578+0000","last_became_active":"2024-03-26T05:20:15.379345+0000","last_became_peered":"2024-03-26T05:20:15.379345+0000","last_unstale":"2024-03-26T05:22:15.091578+0000","last_undegraded":"2024-03-26T05:22:15.091578+0000","last_fullsized":"2024-03-26T05:22:15.091578+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:22:15.091498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:22:15.091498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:29.321931+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":301,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765415+0000","last_change":"2024-03-26T05:20:15.372365+0000","last_active":"2024-03-26T05:22:11.765415+0000","last_peered":"2024-03-26T05:22:11.765415+0000","last_clean":"2024-03-26T05:22:11.765415+0000","last_became_active":"2024-03-26T05:20:15.372060+0000","last_became_peered":"2024-03-26T05:20:15.372060+0000","last_unstale":"2024-03-26T05:22:11.765415+0000","last_undegraded":"2024-03-26T05:22:11.765415+0000","last_fullsized":"2024-03-26T05:22:11.765415+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:14:04.830989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:14:04.830989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":931,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764955+0000","last_change":"2024-03-26T05:20:30.348058+0000","last_active":"2024-03-26T05:22:11.764955+0000","last_peered":"2024-03-26T05:22:11.764955+0000","last_clean":"2024-03-26T05:22:11.764955+0000","last_became_active":"2024-03-26T05:20:30.347326+0000","last_became_peered":"2024-03-26T05:20:30.347326+0000","last_unstale":"2024-03-26T05:22:11.764955+0000","last_undegraded":"2024-03-26T05:22:11.764955+0000","last_fullsized":"2024-03-26T05:22:11.764955+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:01.576615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:20:01.576615+0000","last_clean_scrub_stamp":"2024-03-26T05:20:01.576615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"485'15501","reported_seq":20979,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:16.805240+0000","last_change":"2024-03-26T05:22:06.714850+0000","last_active":"2024-03-26T05:22:16.805240+0000","last_peered":"2024-03-26T05:22:16.805240+0000","last_clean":"2024-03-26T05:22:16.805240+0000","last_became_active":"2024-03-26T05:21:48.762442+0000","last_became_peered":"2024-03-26T05:21:48.762442+0000","last_unstale":"2024-03-26T05:22:16.805240+0000","last_undegraded":"2024-03-26T05:22:16.805240+0000","last_fullsized":"2024-03-26T05:22:16.805240+0000","mapping_epoch":475,"log_start":"453'10038","ondisk_log_start":"453'10038","created":460,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":5,"last_scrub":"483'15120","last_scrub_stamp":"2024-03-26T05:22:06.714572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:06.714572+0000","objects_scrubbed":973,"log_size":5463,"log_dups_size":0,"ondisk_log_size":5463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:20.428751+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38797312,"num_objects":592,"num_object_clones":0,"num_object_copies":1776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":381,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"443'39","reported_seq":988,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.779530+0000","last_change":"2024-03-26T05:19:59.984067+0000","last_active":"2024-03-26T05:22:11.779530+0000","last_peered":"2024-03-26T05:22:11.779530+0000","last_clean":"2024-03-26T05:22:11.779530+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:22:11.779530+0000","last_undegraded":"2024-03-26T05:22:11.779530+0000","last_fullsized":"2024-03-26T05:22:11.779530+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:19:59.983973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:19:59.983973+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":926,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:12.589515+0000","last_change":"2024-03-26T05:22:12.589537+0000","last_active":"2024-03-26T05:22:12.589515+0000","last_peered":"2024-03-26T05:22:12.589515+0000","last_clean":"2024-03-26T05:22:12.589515+0000","last_became_active":"2024-03-26T05:20:15.456698+0000","last_became_peered":"2024-03-26T05:20:15.456698+0000","last_unstale":"2024-03-26T05:22:12.589515+0000","last_undegraded":"2024-03-26T05:22:12.589515+0000","last_fullsized":"2024-03-26T05:22:12.589515+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:22:12.589464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:22:12.589464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:15.538457+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":914,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.777766+0000","last_change":"2024-03-26T05:20:30.339668+0000","last_active":"2024-03-26T05:22:11.777766+0000","last_peered":"2024-03-26T05:22:11.777766+0000","last_clean":"2024-03-26T05:22:11.777766+0000","last_became_active":"2024-03-26T05:20:30.338293+0000","last_became_peered":"2024-03-26T05:20:30.338293+0000","last_unstale":"2024-03-26T05:22:11.777766+0000","last_undegraded":"2024-03-26T05:22:11.777766+0000","last_fullsized":"2024-03-26T05:22:11.777766+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:15:11.912203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:15:11.912203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":645,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765433+0000","last_change":"2024-03-26T05:20:50.576887+0000","last_active":"2024-03-26T05:22:11.765433+0000","last_peered":"2024-03-26T05:22:11.765433+0000","last_clean":"2024-03-26T05:22:11.765433+0000","last_became_active":"2024-03-26T05:20:30.346249+0000","last_became_peered":"2024-03-26T05:20:30.346249+0000","last_unstale":"2024-03-26T05:22:11.765433+0000","last_undegraded":"2024-03-26T05:22:11.765433+0000","last_fullsized":"2024-03-26T05:22:11.765433+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:50.576761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:20:50.576761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:16.470321+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"485'12497","reported_seq":15026,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:14.944186+0000","last_change":"2024-03-26T05:22:10.101836+0000","last_active":"2024-03-26T05:22:14.944186+0000","last_peered":"2024-03-26T05:22:14.944186+0000","last_clean":"2024-03-26T05:22:14.944186+0000","last_became_active":"2024-03-26T05:22:06.801658+0000","last_became_peered":"2024-03-26T05:22:06.801658+0000","last_unstale":"2024-03-26T05:22:14.944186+0000","last_undegraded":"2024-03-26T05:22:14.944186+0000","last_fullsized":"2024-03-26T05:22:14.944186+0000","mapping_epoch":483,"log_start":"434'7321","ondisk_log_start":"434'7321","created":460,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"484'12325","last_scrub_stamp":"2024-03-26T05:22:10.101668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:10.101668+0000","objects_scrubbed":874,"log_size":5176,"log_dups_size":0,"ondisk_log_size":5176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:23.850646+0000","scrub_duration":264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45809664,"num_objects":699,"num_object_clones":0,"num_object_copies":2097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":175,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"443'18","reported_seq":898,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.763999+0000","last_change":"2024-03-26T05:20:15.460323+0000","last_active":"2024-03-26T05:22:11.763999+0000","last_peered":"2024-03-26T05:22:11.763999+0000","last_clean":"2024-03-26T05:22:11.763999+0000","last_became_active":"2024-03-26T05:20:15.374306+0000","last_became_peered":"2024-03-26T05:20:15.374306+0000","last_unstale":"2024-03-26T05:22:11.763999+0000","last_undegraded":"2024-03-26T05:22:11.763999+0000","last_fullsized":"2024-03-26T05:22:11.763999+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'16","last_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_deep_scrub":"321'16","last_deep_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_clean_scrub_stamp":"2024-03-26T05:15:00.995735+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":583,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.776696+0000","last_change":"2024-03-26T05:20:30.329198+0000","last_active":"2024-03-26T05:22:11.776696+0000","last_peered":"2024-03-26T05:22:11.776696+0000","last_clean":"2024-03-26T05:22:11.776696+0000","last_became_active":"2024-03-26T05:20:30.328801+0000","last_became_peered":"2024-03-26T05:20:30.328801+0000","last_unstale":"2024-03-26T05:22:11.776696+0000","last_undegraded":"2024-03-26T05:22:11.776696+0000","last_fullsized":"2024-03-26T05:22:11.776696+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:50.218114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:00.635308+0000","last_clean_scrub_stamp":"2024-03-26T05:18:50.218114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":826,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.769094+0000","last_change":"2024-03-26T05:20:15.374216+0000","last_active":"2024-03-26T05:22:11.769094+0000","last_peered":"2024-03-26T05:22:11.769094+0000","last_clean":"2024-03-26T05:22:11.769094+0000","last_became_active":"2024-03-26T05:20:15.373862+0000","last_became_peered":"2024-03-26T05:20:15.373862+0000","last_unstale":"2024-03-26T05:22:11.769094+0000","last_undegraded":"2024-03-26T05:22:11.769094+0000","last_fullsized":"2024-03-26T05:22:11.769094+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_clean_scrub_stamp":"2024-03-26T05:12:18.959687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":209,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.775374+0000","last_change":"2024-03-26T05:20:30.339972+0000","last_active":"2024-03-26T05:22:11.775374+0000","last_peered":"2024-03-26T05:22:11.775374+0000","last_clean":"2024-03-26T05:22:11.775374+0000","last_became_active":"2024-03-26T05:20:30.337386+0000","last_became_peered":"2024-03-26T05:20:30.337386+0000","last_unstale":"2024-03-26T05:22:11.775374+0000","last_undegraded":"2024-03-26T05:22:11.775374+0000","last_fullsized":"2024-03-26T05:22:11.775374+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:06.363747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:20:06.363747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.15","version":"485'12312","reported_seq":26117,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:16.081575+0000","last_change":"2024-03-26T05:22:09.872744+0000","last_active":"2024-03-26T05:22:16.081575+0000","last_peered":"2024-03-26T05:22:16.081575+0000","last_clean":"2024-03-26T05:22:16.081575+0000","last_became_active":"2024-03-26T05:21:41.755579+0000","last_became_peered":"2024-03-26T05:21:41.755579+0000","last_unstale":"2024-03-26T05:22:16.081575+0000","last_undegraded":"2024-03-26T05:22:16.081575+0000","last_fullsized":"2024-03-26T05:22:16.081575+0000","mapping_epoch":470,"log_start":"433'7126","ondisk_log_start":"433'7126","created":460,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":5,"last_scrub":"484'12115","last_scrub_stamp":"2024-03-26T05:22:09.872444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:09.872444+0000","objects_scrubbed":817,"log_size":5186,"log_dups_size":0,"ondisk_log_size":5186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:38.260277+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40370176,"num_objects":616,"num_object_clones":0,"num_object_copies":1848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":201,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"443'33","reported_seq":786,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.766420+0000","last_change":"2024-03-26T05:20:15.472237+0000","last_active":"2024-03-26T05:22:11.766420+0000","last_peered":"2024-03-26T05:22:11.766420+0000","last_clean":"2024-03-26T05:22:11.766420+0000","last_became_active":"2024-03-26T05:20:15.378661+0000","last_became_peered":"2024-03-26T05:20:15.378661+0000","last_unstale":"2024-03-26T05:22:11.766420+0000","last_undegraded":"2024-03-26T05:22:11.766420+0000","last_fullsized":"2024-03-26T05:22:11.766420+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'27","last_scrub_stamp":"2024-03-26T05:13:22.026354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:13:22.026354+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":586,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765722+0000","last_change":"2024-03-26T05:20:52.465761+0000","last_active":"2024-03-26T05:22:11.765722+0000","last_peered":"2024-03-26T05:22:11.765722+0000","last_clean":"2024-03-26T05:22:11.765722+0000","last_became_active":"2024-03-26T05:20:30.331329+0000","last_became_peered":"2024-03-26T05:20:30.331329+0000","last_unstale":"2024-03-26T05:22:11.765722+0000","last_undegraded":"2024-03-26T05:22:11.765722+0000","last_fullsized":"2024-03-26T05:22:11.765722+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:52.465623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:20:52.465623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":369,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.768940+0000","last_change":"2024-03-26T05:20:30.343688+0000","last_active":"2024-03-26T05:22:11.768940+0000","last_peered":"2024-03-26T05:22:11.768940+0000","last_clean":"2024-03-26T05:22:11.768940+0000","last_became_active":"2024-03-26T05:20:30.343356+0000","last_became_peered":"2024-03-26T05:20:30.343356+0000","last_unstale":"2024-03-26T05:22:11.768940+0000","last_undegraded":"2024-03-26T05:22:11.768940+0000","last_fullsized":"2024-03-26T05:22:11.768940+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:56.259964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:56.259964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":914,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.767903+0000","last_change":"2024-03-26T05:19:43.520950+0000","last_active":"2024-03-26T05:22:11.767903+0000","last_peered":"2024-03-26T05:22:11.767903+0000","last_clean":"2024-03-26T05:22:11.767903+0000","last_became_active":"2024-03-26T05:15:32.640052+0000","last_became_peered":"2024-03-26T05:15:32.640052+0000","last_unstale":"2024-03-26T05:22:11.767903+0000","last_undegraded":"2024-03-26T05:22:11.767903+0000","last_fullsized":"2024-03-26T05:22:11.767903+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:43.520824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.520824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.16","version":"485'12478","reported_seq":18068,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:15.878324+0000","last_change":"2024-03-26T05:22:11.720743+0000","last_active":"2024-03-26T05:22:15.878324+0000","last_peered":"2024-03-26T05:22:15.878324+0000","last_clean":"2024-03-26T05:22:15.878324+0000","last_became_active":"2024-03-26T05:21:47.760547+0000","last_became_peered":"2024-03-26T05:21:47.760547+0000","last_unstale":"2024-03-26T05:22:15.878324+0000","last_undegraded":"2024-03-26T05:22:15.878324+0000","last_fullsized":"2024-03-26T05:22:15.878324+0000","mapping_epoch":474,"log_start":"437'7227","ondisk_log_start":"437'7227","created":460,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":5,"last_scrub":"484'12319","last_scrub_stamp":"2024-03-26T05:22:11.720539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:11.720539+0000","objects_scrubbed":770,"log_size":5251,"log_dups_size":0,"ondisk_log_size":5251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:38.772492+0000","scrub_duration":269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39845888,"num_objects":608,"num_object_clones":0,"num_object_copies":1824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"443'63","reported_seq":1020,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.773058+0000","last_change":"2024-03-26T05:19:48.357831+0000","last_active":"2024-03-26T05:22:11.773058+0000","last_peered":"2024-03-26T05:22:11.773058+0000","last_clean":"2024-03-26T05:22:11.773058+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:22:11.773058+0000","last_undegraded":"2024-03-26T05:22:11.773058+0000","last_fullsized":"2024-03-26T05:22:11.773058+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'51","last_scrub_stamp":"2024-03-26T05:19:48.356743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:19:48.356743+0000","objects_scrubbed":11,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":98,"num_read_kb":85,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":917,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.766262+0000","last_change":"2024-03-26T05:19:43.320734+0000","last_active":"2024-03-26T05:22:11.766262+0000","last_peered":"2024-03-26T05:22:11.766262+0000","last_clean":"2024-03-26T05:22:11.766262+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:22:11.766262+0000","last_undegraded":"2024-03-26T05:22:11.766262+0000","last_fullsized":"2024-03-26T05:22:11.766262+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:43.320603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:19:43.320603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":309,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764564+0000","last_change":"2024-03-26T05:20:46.537730+0000","last_active":"2024-03-26T05:22:11.764564+0000","last_peered":"2024-03-26T05:22:11.764564+0000","last_clean":"2024-03-26T05:22:11.764564+0000","last_became_active":"2024-03-26T05:20:15.370449+0000","last_became_peered":"2024-03-26T05:20:15.370449+0000","last_unstale":"2024-03-26T05:22:11.764564+0000","last_undegraded":"2024-03-26T05:22:11.764564+0000","last_fullsized":"2024-03-26T05:22:11.764564+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:20:46.537613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:20:46.537613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":896,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.764279+0000","last_change":"2024-03-26T05:19:53.622022+0000","last_active":"2024-03-26T05:22:11.764279+0000","last_peered":"2024-03-26T05:22:11.764279+0000","last_clean":"2024-03-26T05:22:11.764279+0000","last_became_active":"2024-03-26T05:15:32.637717+0000","last_became_peered":"2024-03-26T05:15:32.637717+0000","last_unstale":"2024-03-26T05:22:11.764279+0000","last_undegraded":"2024-03-26T05:22:11.764279+0000","last_fullsized":"2024-03-26T05:22:11.764279+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:53.621892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:19:53.621892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"485'12139","reported_seq":27431,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:15.891369+0000","last_change":"2024-03-26T05:22:06.572766+0000","last_active":"2024-03-26T05:22:15.891369+0000","last_peered":"2024-03-26T05:22:15.891369+0000","last_clean":"2024-03-26T05:22:15.891369+0000","last_became_active":"2024-03-26T05:22:05.804464+0000","last_became_peered":"2024-03-26T05:22:05.804464+0000","last_unstale":"2024-03-26T05:22:15.891369+0000","last_undegraded":"2024-03-26T05:22:15.891369+0000","last_fullsized":"2024-03-26T05:22:15.891369+0000","mapping_epoch":482,"log_start":"434'6999","ondisk_log_start":"434'6999","created":460,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":5,"last_scrub":"483'11844","last_scrub_stamp":"2024-03-26T05:22:06.572616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:36.613885+0000","last_clean_scrub_stamp":"2024-03-26T05:22:06.572616+0000","objects_scrubbed":897,"log_size":5140,"log_dups_size":0,"ondisk_log_size":5140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:34.736774+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39452672,"num_objects":602,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":295,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"443'27","reported_seq":780,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.765216+0000","last_change":"2024-03-26T05:21:01.539850+0000","last_active":"2024-03-26T05:22:11.765216+0000","last_peered":"2024-03-26T05:22:11.765216+0000","last_clean":"2024-03-26T05:22:11.765216+0000","last_became_active":"2024-03-26T05:20:15.377639+0000","last_became_peered":"2024-03-26T05:20:15.377639+0000","last_unstale":"2024-03-26T05:22:11.765216+0000","last_undegraded":"2024-03-26T05:22:11.765216+0000","last_fullsized":"2024-03-26T05:22:11.765216+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'27","last_scrub_stamp":"2024-03-26T05:21:01.539708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:21:01.539708+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:22:15.954464+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":936,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.770775+0000","last_change":"2024-03-26T05:20:30.349760+0000","last_active":"2024-03-26T05:22:11.770775+0000","last_peered":"2024-03-26T05:22:11.770775+0000","last_clean":"2024-03-26T05:22:11.770775+0000","last_became_active":"2024-03-26T05:20:30.349429+0000","last_became_peered":"2024-03-26T05:20:30.349429+0000","last_unstale":"2024-03-26T05:22:11.770775+0000","last_undegraded":"2024-03-26T05:22:11.770775+0000","last_fullsized":"2024-03-26T05:22:11.770775+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:18:34.300011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:18:34.300011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":934,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.772032+0000","last_change":"2024-03-26T05:20:30.337232+0000","last_active":"2024-03-26T05:22:11.772032+0000","last_peered":"2024-03-26T05:22:11.772032+0000","last_clean":"2024-03-26T05:22:11.772032+0000","last_became_active":"2024-03-26T05:20:30.336902+0000","last_became_peered":"2024-03-26T05:20:30.336902+0000","last_unstale":"2024-03-26T05:22:11.772032+0000","last_undegraded":"2024-03-26T05:22:11.772032+0000","last_fullsized":"2024-03-26T05:22:11.772032+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:51.618208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:19:51.618208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":618,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T05:22:16.213994+0000","last_change":"2024-03-26T05:22:16.214028+0000","last_active":"2024-03-26T05:22:16.213994+0000","last_peered":"2024-03-26T05:22:16.213994+0000","last_clean":"2024-03-26T05:22:16.213994+0000","last_became_active":"2024-03-26T05:20:30.327890+0000","last_became_peered":"2024-03-26T05:20:30.327890+0000","last_unstale":"2024-03-26T05:22:16.213994+0000","last_undegraded":"2024-03-26T05:22:16.213994+0000","last_fullsized":"2024-03-26T05:22:16.213994+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:22:16.213919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:22:16.213919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:23:30.844260+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":26,"stat_sum":{"num_bytes":1295548440,"num_objects":19770,"num_object_clones":0,"num_object_copies":59310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2831,"num_objects_unfound":0,"num_objects_dirty":19770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102319,"num_write_kb":3408321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22869,"num_bytes_recovered":1498677272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4425658368,"data_stored":4425646152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125930,"ondisk_log_size":125930,"up":78,"acting":78,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":468,"num_read_kb":463,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1832,"num_read_kb":1557,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":4554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":13020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1275,"ondisk_log_size":1275,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":335,"seq":1438814044242,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1177184,"kb_used_data":650520,"kb_used_omap":4,"kb_used_meta":526651,"kb_avail":92567968,"statfs":{"total":95995035648,"available":94789599232,"internally_reserved":0,"allocated":666132480,"data_stored":664937333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":539291304},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332368,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1404384,"kb_used_data":691544,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":92340768,"statfs":{"total":95995035648,"available":94556946432,"internally_reserved":0,"allocated":708141056,"data_stored":706945181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575146,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1737600,"kb_used_data":826232,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":92007552,"statfs":{"total":95995035648,"available":94215733248,"internally_reserved":0,"allocated":846061568,"data_stored":844833906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":933231045},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738671,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1590308,"kb_used_data":678940,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":92154844,"statfs":{"total":95995035648,"available":94366560256,"internally_reserved":0,"allocated":695234560,"data_stored":694035267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":933231045},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411621,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1359704,"kb_used_data":616272,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92385448,"statfs":{"total":95995035648,"available":94602698752,"internally_reserved":0,"allocated":631062528,"data_stored":629874626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117279,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1466804,"kb_used_data":556844,"kb_used_omap":1,"kb_used_meta":909950,"kb_avail":92278348,"statfs":{"total":95995035648,"available":94493028352,"internally_reserved":0,"allocated":570208256,"data_stored":568992859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":931789255},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":176093659413,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":683474944,"data_stored":683474944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":822480896,"data_stored":822476824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":606212096,"data_stored":606208024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":546045952,"data_stored":546045952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":426311680,"data_stored":426311680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":698421248,"data_stored":698417176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":642711552,"data_stored":642711552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:22:23.059 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool set unique_pool_4 pgp_num 26 2024-03-26T05:22:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:23.085 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:23.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:23.187 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:23.283 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:23.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:23.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:23.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:22 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:22:23.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:22 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:22:23.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:22 smithi132 ceph-mon[33500]: osdmap e486: 8 total, 7 up, 6 in 2024-03-26T05:22:23.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:22 smithi132 ceph-mon[33500]: pgmap v1040: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 876 MiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 1955/42735 objects misplaced (4.575%) 2024-03-26T05:22:23.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:22 smithi132 ceph-mon[33500]: 5.d scrub starts 2024-03-26T05:22:23.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:22 smithi132 ceph-mon[33500]: 5.d scrub ok 2024-03-26T05:22:23.348 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:22 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:22:23.348 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:22 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:22:23.348 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:22 smithi080 ceph-mon[29691]: osdmap e486: 8 total, 7 up, 6 in 2024-03-26T05:22:23.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:22 smithi080 ceph-mon[29691]: pgmap v1040: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 876 MiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 1955/42735 objects misplaced (4.575%) 2024-03-26T05:22:23.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:22 smithi080 ceph-mon[29691]: 5.d scrub starts 2024-03-26T05:22:23.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:22 smithi080 ceph-mon[29691]: 5.d scrub ok 2024-03-26T05:22:23.349 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:22 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:22:23.349 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:22 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:22:23.349 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:22 smithi080 ceph-mon[40404]: osdmap e486: 8 total, 7 up, 6 in 2024-03-26T05:22:23.349 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:22 smithi080 ceph-mon[40404]: pgmap v1040: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 876 MiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 1955/42735 objects misplaced (4.575%) 2024-03-26T05:22:23.349 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:22 smithi080 ceph-mon[40404]: 5.d scrub starts 2024-03-26T05:22:23.349 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:22 smithi080 ceph-mon[40404]: 5.d scrub ok 2024-03-26T05:22:23.385 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:23.411 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:23.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:23.512 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:23.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:23.553 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:23.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:23.654 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:23.680 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:23.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:23.781 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:23.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:23.810 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:23.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:23.911 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:23.962 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:22:23.973 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:23.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:23.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:24.075 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:24.102 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:24.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:24.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:24.204 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:24.247 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:24.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:24.251 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:22:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:22:24.251 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[40404]: 4.1a scrub starts 2024-03-26T05:22:24.251 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[40404]: 4.1a scrub ok 2024-03-26T05:22:24.251 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[40404]: 6.1d scrub starts 2024-03-26T05:22:24.251 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2615291460' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:22:24.251 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[40404]: 6.1d scrub ok 2024-03-26T05:22:24.251 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[40404]: 5.5 scrub starts 2024-03-26T05:22:24.251 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[40404]: 5.5 scrub ok 2024-03-26T05:22:24.252 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[29691]: 4.1a scrub starts 2024-03-26T05:22:24.252 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[29691]: 4.1a scrub ok 2024-03-26T05:22:24.252 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[29691]: 6.1d scrub starts 2024-03-26T05:22:24.252 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2615291460' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:22:24.252 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[29691]: 6.1d scrub ok 2024-03-26T05:22:24.252 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[29691]: 5.5 scrub starts 2024-03-26T05:22:24.252 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:23 smithi080 ceph-mon[29691]: 5.5 scrub ok 2024-03-26T05:22:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:23 smithi132 ceph-mon[33500]: 4.1a scrub starts 2024-03-26T05:22:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:23 smithi132 ceph-mon[33500]: 4.1a scrub ok 2024-03-26T05:22:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:23 smithi132 ceph-mon[33500]: 6.1d scrub starts 2024-03-26T05:22:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:23 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2615291460' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:22:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:23 smithi132 ceph-mon[33500]: 6.1d scrub ok 2024-03-26T05:22:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:23 smithi132 ceph-mon[33500]: 5.5 scrub starts 2024-03-26T05:22:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:23 smithi132 ceph-mon[33500]: 5.5 scrub ok 2024-03-26T05:22:24.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:24.388 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:22:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:24.392 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:24.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:24.439 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:22:24.483 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:22:24.494 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:24.525 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:22:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:24.535 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:24.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:24.568 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:22:24.611 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:22:24.636 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:24.647 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:22:24.647 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:22:24.647 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:22:24.647 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:22:24.647 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:22:24.647 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:22:24.647 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:22:24.647 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:22:24.647 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:22:24.679 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:24.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:24.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:24.781 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:24.810 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:24.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:24.902 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:24.911 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:24.965 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:24.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:25.066 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:25.112 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:25.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:25.214 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:25.257 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:25.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:25.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: 4.e scrub starts 2024-03-26T05:22:25.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: 4.e scrub ok 2024-03-26T05:22:25.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:22:25.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2615291460' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:22:25.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: osdmap e487: 8 total, 7 up, 6 in 2024-03-26T05:22:25.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: 5.1d scrub starts 2024-03-26T05:22:25.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: 5.1d scrub ok 2024-03-26T05:22:25.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: pgmap v1042: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 876 MiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 973 op/s; 1955/42735 objects misplaced (4.575%) 2024-03-26T05:22:25.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2050365771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:22:25.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: 5.8 scrub starts 2024-03-26T05:22:25.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: 5.8 scrub ok 2024-03-26T05:22:25.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: 6.10 scrub starts 2024-03-26T05:22:25.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:24 smithi132 ceph-mon[33500]: 6.10 scrub ok 2024-03-26T05:22:25.357 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:25.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: 4.e scrub starts 2024-03-26T05:22:25.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: 4.e scrub ok 2024-03-26T05:22:25.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:22:25.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2615291460' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:22:25.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: osdmap e487: 8 total, 7 up, 6 in 2024-03-26T05:22:25.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: 5.1d scrub starts 2024-03-26T05:22:25.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: 5.1d scrub ok 2024-03-26T05:22:25.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: pgmap v1042: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 876 MiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 973 op/s; 1955/42735 objects misplaced (4.575%) 2024-03-26T05:22:25.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2050365771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:22:25.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: 5.8 scrub starts 2024-03-26T05:22:25.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: 5.8 scrub ok 2024-03-26T05:22:25.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: 6.10 scrub starts 2024-03-26T05:22:25.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[29691]: 6.10 scrub ok 2024-03-26T05:22:25.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: 4.e scrub starts 2024-03-26T05:22:25.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: 4.e scrub ok 2024-03-26T05:22:25.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:22:25.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2615291460' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:22:25.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: osdmap e487: 8 total, 7 up, 6 in 2024-03-26T05:22:25.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: 5.1d scrub starts 2024-03-26T05:22:25.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: 5.1d scrub ok 2024-03-26T05:22:25.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: pgmap v1042: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 876 MiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 973 op/s; 1955/42735 objects misplaced (4.575%) 2024-03-26T05:22:25.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2050365771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:22:25.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: 5.8 scrub starts 2024-03-26T05:22:25.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: 5.8 scrub ok 2024-03-26T05:22:25.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: 6.10 scrub starts 2024-03-26T05:22:25.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:24 smithi080 ceph-mon[40404]: 6.10 scrub ok 2024-03-26T05:22:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:25.417 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:25.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:25.518 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:25.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:25.560 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:25.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:25.661 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:25.689 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:25.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:25.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:25.718 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:22:25.718 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:165606 10.19 10:9cdc6bfe:::benchmark_data_smithi080_161990_object53883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:17.696202+0000", 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 308.02450202599999, 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0290944799999999, 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 165606 2024-03-26T05:22:25.719 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696202+0000", 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696202+0000", 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.720 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696203+0000", 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.012e-06 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696208+0000", 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8380000000000001e-06 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696222+0000", 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3747000000000001e-05 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.721 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696230+0000", 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9340000000000006e-06 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696291+0000", 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1270000000000001e-05 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.696301+0000", 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0356999999999999e-05 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.722 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.720378+0000", 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.024076612 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.720507+0000", 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012966400000000001 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.720862+0000", 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00035415799999999998 2024-03-26T05:22:25.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.724880+0000", 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.004018527 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.724925+0000", 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5166000000000001e-05 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.725126+0000", 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020027700000000001 2024-03-26T05:22:25.724 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.725181+0000", 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5071999999999997e-05 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.725296+0000", 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115846 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.725 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:165608 10.19 10:9cef172c:::benchmark_data_smithi080_161990_object53885:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:17.698646+0000", 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 308.02205780000003, 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0308346859999999, 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 165608 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.726 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698646+0000", 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698646+0000", 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698647+0000", 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1459999999999999e-06 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.727 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698652+0000", 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9080000000000003e-06 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698663+0000", 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0736e-05 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698670+0000", 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1709999999999996e-06 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.728 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698744+0000", 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3672999999999997e-05 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:17.698754+0000", 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.679e-06 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.723226+0000", 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0244719899999999 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.729 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.730 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.730 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.723356+0000", 2024-03-26T05:22:25.730 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013032199999999999 2024-03-26T05:22:25.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.730 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.730 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,3", 2024-03-26T05:22:25.730 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.723607+0000", 2024-03-26T05:22:25.730 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025147 2024-03-26T05:22:25.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.730 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.730 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.730 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729207+0000", 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055995130000000004 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729272+0000", 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5174999999999999e-05 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729342+0000", 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9556999999999998e-05 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.731 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729380+0000", 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8303999999999998e-05 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:18.729481+0000", 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000101042 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168339 10.13 10:ccc6060e:::benchmark_data_smithi080_161990_object56616:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:22:25.732 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.683443+0000", 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 305.03726110700001, 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.401619631, 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168339 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683443+0000", 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.733 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683443+0000", 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683444+0000", 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.305e-06 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683450+0000", 2024-03-26T05:22:25.734 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9100000000000002e-06 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683461+0000", 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0479000000000001e-05 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683469+0000", 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3550000000000005e-06 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.735 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.911908+0000", 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.228439168 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.911916+0000", 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8699999999999992e-06 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.084959+0000", 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1730434750000001 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.736 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.085063+0000", 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103069 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168349 10.13 10:cb8495a4:::benchmark_data_smithi080_161990_object56626:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.683484+0000", 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 305.03722023, 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4120273919999999, 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.737 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168349 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683484+0000", 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683484+0000", 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.738 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683484+0000", 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8100000000000003e-07 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683487+0000", 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6869999999999999e-06 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683494+0000", 2024-03-26T05:22:25.739 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8519999999999998e-06 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.683499+0000", 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5809999999999996e-06 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.911937+0000", 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.22843730400000001 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.911942+0000", 2024-03-26T05:22:25.740 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4790000000000004e-06 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.085360+0000", 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.173417366 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for missing object", 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.085466+0000", 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000106228 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095470+0000", 2024-03-26T05:22:25.741 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010004503999999999 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095511+0000", 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.091e-05 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168383 10.13 10:ca51ce05:::benchmark_data_smithi080_161990_object56660:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.925543+0000", 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 304.79516111100003, 2024-03-26T05:22:25.742 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.165529024, 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168383 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925543+0000", 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.743 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925543+0000", 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925544+0000", 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.455e-06 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925550+0000", 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0410000000000003e-06 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.745 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.745 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925568+0000", 2024-03-26T05:22:25.745 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.73e-05 2024-03-26T05:22:25.745 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.745 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.745 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.745 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925580+0000", 2024-03-26T05:22:25.745 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1902999999999999e-05 2024-03-26T05:22:25.745 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.745 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.745 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.746 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925666+0000", 2024-03-26T05:22:25.746 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.6580000000000001e-05 2024-03-26T05:22:25.746 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.746 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.746 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.746 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.925686+0000", 2024-03-26T05:22:25.747 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0253000000000001e-05 2024-03-26T05:22:25.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.747 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.747 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.747 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.085520+0000", 2024-03-26T05:22:25.747 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1598332060000001 2024-03-26T05:22:25.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.747 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.747 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.747 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.085623+0000", 2024-03-26T05:22:25.747 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103026 2024-03-26T05:22:25.749 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.750 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.750 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:22:25.750 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.085887+0000", 2024-03-26T05:22:25.750 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00026435400000000001 2024-03-26T05:22:25.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.750 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.750 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.750 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.090700+0000", 2024-03-26T05:22:25.750 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0048128499999999996 2024-03-26T05:22:25.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.090754+0000", 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4143e-05 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.090969+0000", 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000215305 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.751 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091008+0000", 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8424999999999999e-05 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091072+0000", 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4183000000000005e-05 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168409 10.13 10:cf3fad42:::benchmark_data_smithi080_161990_object56686:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.941003+0000", 2024-03-26T05:22:25.752 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 304.77970144599999, 2024-03-26T05:22:25.753 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1502961890000001, 2024-03-26T05:22:25.753 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.753 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.753 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.753 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.753 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.753 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168409 2024-03-26T05:22:25.753 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.753 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.753 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.753 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.753 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.941003+0000", 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.941003+0000", 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.941004+0000", 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5319999999999999e-06 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.754 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.941010+0000", 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6239999999999999e-06 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.941023+0000", 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3076e-05 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.941031+0000", 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1259999999999998e-06 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.943036+0000", 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0020049070000000002 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.943045+0000", 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8729999999999996e-06 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.086959+0000", 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.143914691 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.756 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.087044+0000", 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4965999999999999e-05 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.087270+0000", 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000225629 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.090707+0000", 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0034370410000000001 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.757 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.090899+0000", 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019213499999999999 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091138+0000", 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00023836300000000001 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091191+0000", 2024-03-26T05:22:25.758 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.3118999999999997e-05 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091299+0000", 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000108107 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168440 10.13 10:cd651acd:::benchmark_data_smithi080_161990_object56717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.966300+0000", 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 304.75440407899998, 2024-03-26T05:22:25.759 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.128878874, 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168440 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966300+0000", 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.760 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966300+0000", 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966301+0000", 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.373e-06 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966307+0000", 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0569999999999998e-06 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.761 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.762 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.762 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966323+0000", 2024-03-26T05:22:25.762 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6018000000000001e-05 2024-03-26T05:22:25.762 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.762 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.762 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.762 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966333+0000", 2024-03-26T05:22:25.762 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7189999999999996e-06 2024-03-26T05:22:25.762 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.762 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.762 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.762 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966417+0000", 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3811000000000001e-05 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.966426+0000", 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.5130000000000002e-06 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.089436+0000", 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.12301001 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.763 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.089524+0000", 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.7381999999999998e-05 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.089697+0000", 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00017316 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091085+0000", 2024-03-26T05:22:25.764 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001387934 2024-03-26T05:22:25.765 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.765 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.765 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.765 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.094840+0000", 2024-03-26T05:22:25.765 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.003755525 2024-03-26T05:22:25.765 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.765 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.765 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.765 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095034+0000", 2024-03-26T05:22:25.765 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000193921 2024-03-26T05:22:25.765 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.765 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095081+0000", 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6910000000000003e-05 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095179+0000", 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7540999999999993e-05 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.766 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168459 10.13 10:cd1968e0:::benchmark_data_smithi080_161990_object56736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.977540+0000", 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 304.743163998, 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1180914230000001, 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168459 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.767 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977540+0000", 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977540+0000", 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977541+0000", 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1379999999999999e-06 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.768 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977545+0000", 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.7500000000000001e-06 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977556+0000", 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1479e-05 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977565+0000", 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0879999999999995e-06 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.769 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977626+0000", 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1005000000000003e-05 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.977635+0000", 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.713e-06 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091346+0000", 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1137104769999999 2024-03-26T05:22:25.770 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091467+0000", 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000120827 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.091660+0000", 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00019336900000000001 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.094855+0000", 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00319528 2024-03-26T05:22:25.771 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095531+0000", 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000676251 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095553+0000", 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.1467999999999998e-05 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095579+0000", 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.614e-05 2024-03-26T05:22:25.772 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095631+0000", 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2438000000000002e-05 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168464 10.13 10:cb7a0ce6:::benchmark_data_smithi080_161990_object56741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:20.981939+0000", 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 304.73876503700001, 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.125574021, 2024-03-26T05:22:25.773 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168464 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.981939+0000", 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.774 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.981939+0000", 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.981940+0000", 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.16e-07 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.981945+0000", 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2379999999999997e-06 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.775 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.981957+0000", 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1384e-05 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.981964+0000", 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5959999999999998e-06 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.982024+0000", 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.0302e-05 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.776 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:20.982033+0000", 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0669999999999992e-06 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.093107+0000", 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.111074559 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.093219+0000", 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011222000000000001 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.777 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.093439+0000", 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000219483 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095654+0000", 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0022155450000000002 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101179+0000", 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055248880000000004 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.778 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.107325+0000", 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0061455609999999999 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.107395+0000", 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.0432999999999995e-05 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.107513+0000", 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117829 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.779 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.780 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.780 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.780 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.780 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168500 10.13 10:ca1c5b2d:::benchmark_data_smithi080_161990_object56777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:22:25.780 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:21.010741+0000", 2024-03-26T05:22:25.780 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 304.70996317499998, 2024-03-26T05:22:25.780 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1041035909999999, 2024-03-26T05:22:25.780 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.780 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.780 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.780 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168500 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010741+0000", 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010741+0000", 2024-03-26T05:22:25.781 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010742+0000", 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4780000000000001e-06 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010747+0000", 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7729999999999999e-06 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010763+0000", 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.5953000000000001e-05 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010771+0000", 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6389999999999992e-06 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010830+0000", 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9478999999999998e-05 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.010848+0000", 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7520000000000002e-05 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095678+0000", 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.084830381 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.095804+0000", 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000126074 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.784 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.096062+0000", 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025742700000000001 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101217+0000", 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051549079999999997 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101228+0000", 2024-03-26T05:22:25.785 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1559999999999999e-05 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.114439+0000", 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.013210629 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.114719+0000", 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00028040600000000003 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.114844+0000", 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012536399999999999 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168515 10.13 10:cf8059c6:::benchmark_data_smithi080_161990_object56792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:21.022607+0000", 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 304.698096787, 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0925560329999999, 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168515 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022607+0000", 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022607+0000", 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022608+0000", 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1999999999999999e-06 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022613+0000", 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7389999999999999e-06 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022624+0000", 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0397000000000001e-05 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.790 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022631+0000", 2024-03-26T05:22:25.790 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9249999999999998e-06 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022918+0000", 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00028781 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.022926+0000", 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0430000000000008e-06 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.791 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.097531+0000", 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0746048880000001 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.097654+0000", 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012278899999999999 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.097906+0000", 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025170699999999999 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.792 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101233+0000", 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0033272610000000002 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101263+0000", 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.9913999999999999e-05 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.114892+0000", 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.013629535999999999 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.793 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115069+0000", 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000176121 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115163+0000", 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4703000000000005e-05 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168578 10.13 10:c8138cee:::benchmark_data_smithi080_161990_object56855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:22:25.794 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:21.072743+0000", 2024-03-26T05:22:25.795 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 304.64796127, 2024-03-26T05:22:25.795 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0427139839999999, 2024-03-26T05:22:25.795 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.795 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.795 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.795 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.795 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.795 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168578 2024-03-26T05:22:25.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.795 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072743+0000", 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072743+0000", 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072744+0000", 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1909999999999999e-06 2024-03-26T05:22:25.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072749+0000", 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8770000000000001e-06 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072762+0000", 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2911000000000001e-05 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072768+0000", 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.083e-06 2024-03-26T05:22:25.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072865+0000", 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6766000000000002e-05 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.072874+0000", 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.7860000000000008e-06 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.099335+0000", 2024-03-26T05:22:25.798 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0264605760000001 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.099469+0000", 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013444499999999999 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.099684+0000", 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021462799999999999 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.103173+0000", 2024-03-26T05:22:25.799 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.003488545 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.107277+0000", 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0041039509999999998 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115206+0000", 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0079292340000000003 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115365+0000", 2024-03-26T05:22:25.800 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000159073 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115457+0000", 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1917999999999996e-05 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:168588 10.13 10:c9ff2f28:::benchmark_data_smithi080_161990_object56865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:21.080490+0000", 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 304.64021433300002, 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.035269016, 2024-03-26T05:22:25.801 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 168588 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080490+0000", 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.802 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080490+0000", 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080491+0000", 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.562e-06 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080497+0000", 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8209999999999998e-06 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.803 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080510+0000", 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3188e-05 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080518+0000", 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9039999999999992e-06 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080581+0000", 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2546999999999996e-05 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.804 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.805 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.805 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:21.080591+0000", 2024-03-26T05:22:25.805 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.8689999999999999e-06 2024-03-26T05:22:25.805 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.805 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.805 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.805 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101289+0000", 2024-03-26T05:22:25.805 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.020698133 2024-03-26T05:22:25.805 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.805 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.806 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.806 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101398+0000", 2024-03-26T05:22:25.806 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000109347 2024-03-26T05:22:25.806 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.806 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.806 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 2,7", 2024-03-26T05:22:25.806 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.101607+0000", 2024-03-26T05:22:25.806 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00020852800000000001 2024-03-26T05:22:25.806 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.806 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.806 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.107256+0000", 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0056498429999999999 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.107283+0000", 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6378000000000001e-05 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115499+0000", 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0082165060000000002 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.807 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115666+0000", 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016688900000000001 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:22.115759+0000", 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2501000000000001e-05 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:129362 11.7 11:e02951c7:::benchmark_data_smithi080_189724_object8252:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e443)", 2024-03-26T05:22:25.808 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:30.313954+0000", 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 115.40675002, 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0923770610000001, 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 129362 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.313954+0000", 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.809 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.313954+0000", 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.313955+0000", 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.33e-06 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.313959+0000", 2024-03-26T05:22:25.810 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0749999999999999e-06 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.313970+0000", 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1018e-05 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.313981+0000", 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0402e-05 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.811 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314043+0000", 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2433000000000003e-05 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op must wait for map", 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314052+0000", 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3359999999999995e-06 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.319204+0000", 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051520589999999996 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.812 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.319214+0000", 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9890000000000004e-06 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.386043+0000", 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.066829603 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.389018+0000", 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0029745090000000002 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,5,7", 2024-03-26T05:22:25.813 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.389397+0000", 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00037883799999999999 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.399919+0000", 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010522666 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.406057+0000", 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.006137254 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.814 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.406074+0000", 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7394e-05 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.406123+0000", 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9255999999999997e-05 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.406194+0000", 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1055000000000003e-05 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.815 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.406331+0000", 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013684400000000001 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:129369 11.7 11:ec38f0e5:::benchmark_data_smithi080_189724_object8259:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e443)", 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:30.314000+0000", 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 115.40670426699999, 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.098530521, 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.816 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 129369 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314000+0000", 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314000+0000", 2024-03-26T05:22:25.817 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314000+0000", 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9099999999999999e-07 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314003+0000", 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2499999999999998e-06 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314068+0000", 2024-03-26T05:22:25.818 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4997000000000001e-05 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314077+0000", 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.191e-06 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314121+0000", 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.4777999999999997e-05 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting_for_map not empty", 2024-03-26T05:22:25.819 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314128+0000", 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8050000000000001e-06 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.319237+0000", 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051084499999999996 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.319243+0000", 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4989999999999999e-06 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.820 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.399973+0000", 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.080729786 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.400119+0000", 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014596300000000001 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,5,7", 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.400330+0000", 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000211141 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.821 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.406081+0000", 2024-03-26T05:22:25.822 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0057513290000000003 2024-03-26T05:22:25.822 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.822 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.822 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.822 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.406388+0000", 2024-03-26T05:22:25.822 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00030674100000000002 2024-03-26T05:22:25.822 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.822 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.822 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.822 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.406438+0000", 2024-03-26T05:22:25.822 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0195000000000003e-05 2024-03-26T05:22:25.822 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.412358+0000", 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0059195589999999996 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.412422+0000", 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3683000000000007e-05 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.412530+0000", 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000108763 2024-03-26T05:22:25.823 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:129382 11.7 11:e973e08c:::benchmark_data_smithi080_189724_object8272:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e443)", 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:30.314092+0000", 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 115.406611583, 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.102588911, 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:22:25.824 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 129382 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314092+0000", 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314092+0000", 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.825 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314093+0000", 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9100000000000004e-07 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314096+0000", 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.118e-06 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314103+0000", 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3119999999999996e-06 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.826 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314109+0000", 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7139999999999998e-06 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314155+0000", 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6269e-05 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting_for_map not empty", 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314161+0000", 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5500000000000002e-06 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.827 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.319266+0000", 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.005105119 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.319273+0000", 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9469999999999997e-06 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.403200+0000", 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0839272950000001 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.403333+0000", 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013268899999999999 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,5,7", 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.403465+0000", 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000131593 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.406343+0000", 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0028785569999999999 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.412573+0000", 2024-03-26T05:22:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0062294170000000001 2024-03-26T05:22:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.412975+0000", 2024-03-26T05:22:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000402385 2024-03-26T05:22:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.416471+0000", 2024-03-26T05:22:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0034956150000000001 2024-03-26T05:22:25.831 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:25.832 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.832 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.832 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.832 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.416572+0000", 2024-03-26T05:22:25.832 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010100800000000001 2024-03-26T05:22:25.832 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.832 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.832 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.832 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.416681+0000", 2024-03-26T05:22:25.832 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000109832 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:129386 11.7 11:e158ec3c:::benchmark_data_smithi080_189724_object8276:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e443)", 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:30.314116+0000", 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 115.40658789299999, 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1280657190000001, 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:22:25.833 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 129386 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314116+0000", 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314116+0000", 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.834 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314117+0000", 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8099999999999998e-07 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314119+0000", 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3140000000000002e-06 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314125+0000", 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.229e-06 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.835 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314131+0000", 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4380000000000003e-06 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314187+0000", 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6907999999999998e-05 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting_for_map not empty", 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314193+0000", 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4020000000000001e-06 2024-03-26T05:22:25.836 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.319296+0000", 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0051027900000000003 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.319302+0000", 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6760000000000001e-06 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.406475+0000", 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.087172542 2024-03-26T05:22:25.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.406615+0000", 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013990399999999999 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,5,7", 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.406771+0000", 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00015625 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.838 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.412723+0000", 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.005951886 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.419173+0000", 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0064504580000000001 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.432304+0000", 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.013131062000000001 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.839 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.441938+0000", 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0096338689999999998 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.442057+0000", 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118402 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.442182+0000", 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000125208 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.840 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:129428 11.7 11:e3d41562:::benchmark_data_smithi080_189724_object8318:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e443)", 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:30.314156+0000", 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 115.406547629, 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1282353679999999, 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 129428 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.841 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314156+0000", 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314156+0000", 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.842 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314157+0000", 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.2e-07 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314159+0000", 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.6290000000000002e-06 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314166+0000", 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.6669999999999999e-06 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.843 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314172+0000", 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.852e-06 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314222+0000", 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9947999999999997e-05 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting_for_map not empty", 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314228+0000", 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9309999999999996e-06 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.844 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.319327+0000", 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0050989939999999999 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.319333+0000", 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4910000000000002e-06 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.409460+0000", 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0901265769999999 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.845 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.409577+0000", 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000117261 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,5,7", 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.409720+0000", 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000142874 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.412766+0000", 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00304572 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.846 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.423781+0000", 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.011015097 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.432349+0000", 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0085680459999999993 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.442250+0000", 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0099007220000000007 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.847 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.442300+0000", 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0000000000000002e-05 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.442392+0000", 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2139000000000007e-05 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.848 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:129435 11.7 11:e5be881c:::benchmark_data_smithi080_189724_object8325:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e443)", 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:30.314179+0000", 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 115.406525229, 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1283957090000001, 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 129435 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.849 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314179+0000", 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314179+0000", 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314179+0000", 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.1399999999999998e-07 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.850 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314181+0000", 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.902e-06 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314187+0000", 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.891e-06 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.314192+0000", 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.383e-06 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.851 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.315213+0000", 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0010211650000000001 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting_for_map not empty", 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.315221+0000", 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9929999999999995e-06 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.319357+0000", 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0041351110000000003 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.852 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.319363+0000", 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2129999999999996e-06 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.413008+0000", 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.09364478 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.413126+0000", 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118226 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.853 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,5,7", 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.413259+0000", 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013301400000000001 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.417678+0000", 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0044194760000000003 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.423825+0000", 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0061471369999999996 2024-03-26T05:22:25.854 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.432392+0000", 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0085666089999999993 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.442403+0000", 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.010010554 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.442450+0000", 2024-03-26T05:22:25.855 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7608999999999998e-05 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.442575+0000", 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000124332 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:201226 11.7 11:e1d47afd:::benchmark_data_smithi080_189724_object80116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:21:55.798778+0000", 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 29.921926497000001, 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0340023359999999, 2024-03-26T05:22:25.856 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "reached pg", 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 201226 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.798778+0000", 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.857 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.798778+0000", 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.798778+0000", 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1399999999999996e-07 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.798783+0000", 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5310000000000003e-06 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.858 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.798794+0000", 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1409e-05 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.798802+0000", 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1159999999999999e-06 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.798864+0000", 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1983000000000006e-05 2024-03-26T05:22:25.859 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.798978+0000", 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000113478 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,5,7", 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.799290+0000", 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00031232800000000001 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.805000+0000", 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.005709558 2024-03-26T05:22:25.860 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.808618+0000", 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036183460000000001 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.825215+0000", 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.016597015 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:55.825227+0000", 2024-03-26T05:22:25.861 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2276000000000001e-05 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:56.832711+0000", 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0074835520000001 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:21:56.832780+0000", 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.8930000000000006e-05 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:25.863 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:25.863 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:22:25.904 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:25.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:25.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:26.006 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:26.033 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:26.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:26.135 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:26.162 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:26.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:26.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:26.264 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:26.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:25 smithi132 ceph-mon[33500]: 5.11 scrub starts 2024-03-26T05:22:26.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:25 smithi132 ceph-mon[33500]: 5.11 scrub ok 2024-03-26T05:22:26.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:25 smithi132 ceph-mon[33500]: 5.0 scrub starts 2024-03-26T05:22:26.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:25 smithi132 ceph-mon[33500]: 5.0 scrub ok 2024-03-26T05:22:26.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:22:26 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:22:26.304 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:26.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:26.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:25 smithi080 ceph-mon[29691]: 5.11 scrub starts 2024-03-26T05:22:26.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:25 smithi080 ceph-mon[29691]: 5.11 scrub ok 2024-03-26T05:22:26.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:25 smithi080 ceph-mon[29691]: 5.0 scrub starts 2024-03-26T05:22:26.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:25 smithi080 ceph-mon[29691]: 5.0 scrub ok 2024-03-26T05:22:26.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:25 smithi080 ceph-mon[40404]: 5.11 scrub starts 2024-03-26T05:22:26.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:25 smithi080 ceph-mon[40404]: 5.11 scrub ok 2024-03-26T05:22:26.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:25 smithi080 ceph-mon[40404]: 5.0 scrub starts 2024-03-26T05:22:26.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:25 smithi080 ceph-mon[40404]: 5.0 scrub ok 2024-03-26T05:22:26.407 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:26.435 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:26.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:26.536 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:26.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:26.567 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:26.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:26.668 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:26.709 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:26.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:26.810 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:26.850 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:26.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:26.951 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:26.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:26.981 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:26.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:27.082 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:27.108 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:27.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:27.211 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:27.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:27.237 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:27.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:27 smithi132 ceph-mon[33500]: 3.15 scrub starts 2024-03-26T05:22:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:27 smithi132 ceph-mon[33500]: 3.15 scrub ok 2024-03-26T05:22:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:27 smithi132 ceph-mon[33500]: 5.1c scrub starts 2024-03-26T05:22:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:27 smithi132 ceph-mon[33500]: 5.1c scrub ok 2024-03-26T05:22:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:27 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:27 smithi132 ceph-mon[33500]: pgmap v1043: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 631 MiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 1955/30951 objects misplaced (6.316%) 2024-03-26T05:22:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:27 smithi132 ceph-mon[33500]: 3.17 scrub starts 2024-03-26T05:22:27.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:27 smithi132 ceph-mon[33500]: 3.17 scrub ok 2024-03-26T05:22:27.339 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:27.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[29691]: 3.15 scrub starts 2024-03-26T05:22:27.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[29691]: 3.15 scrub ok 2024-03-26T05:22:27.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[29691]: 5.1c scrub starts 2024-03-26T05:22:27.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[29691]: 5.1c scrub ok 2024-03-26T05:22:27.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:27.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[29691]: pgmap v1043: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 631 MiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 1955/30951 objects misplaced (6.316%) 2024-03-26T05:22:27.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[29691]: 3.17 scrub starts 2024-03-26T05:22:27.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[29691]: 3.17 scrub ok 2024-03-26T05:22:27.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[40404]: 3.15 scrub starts 2024-03-26T05:22:27.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[40404]: 3.15 scrub ok 2024-03-26T05:22:27.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[40404]: 5.1c scrub starts 2024-03-26T05:22:27.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[40404]: 5.1c scrub ok 2024-03-26T05:22:27.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:27.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[40404]: pgmap v1043: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 631 MiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 1955/30951 objects misplaced (6.316%) 2024-03-26T05:22:27.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[40404]: 3.17 scrub starts 2024-03-26T05:22:27.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:27 smithi080 ceph-mon[40404]: 3.17 scrub ok 2024-03-26T05:22:27.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:27.368 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:27.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:27.422 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T05:22:27.468 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:27.494 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:27.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:27.596 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:27.647 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:27.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:27.748 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:27.792 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:27.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:27.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:27.894 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:27.922 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:27.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:28.024 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:22:28.024 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:22:28.025 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:28.062 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:28.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:28.164 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:28.193 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:28.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:28 smithi132 ceph-mon[33500]: 4.13 scrub starts 2024-03-26T05:22:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:28 smithi132 ceph-mon[33500]: 4.13 scrub ok 2024-03-26T05:22:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:28 smithi132 ceph-mon[33500]: 6.2 scrub starts 2024-03-26T05:22:28.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:28 smithi132 ceph-mon[33500]: 6.2 scrub ok 2024-03-26T05:22:28.294 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:28.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:28.336 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:28.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:28.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:28 smithi080 ceph-mon[29691]: 4.13 scrub starts 2024-03-26T05:22:28.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:28 smithi080 ceph-mon[29691]: 4.13 scrub ok 2024-03-26T05:22:28.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:28 smithi080 ceph-mon[29691]: 6.2 scrub starts 2024-03-26T05:22:28.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:28 smithi080 ceph-mon[29691]: 6.2 scrub ok 2024-03-26T05:22:28.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:28 smithi080 ceph-mon[40404]: 4.13 scrub starts 2024-03-26T05:22:28.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:28 smithi080 ceph-mon[40404]: 4.13 scrub ok 2024-03-26T05:22:28.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:28 smithi080 ceph-mon[40404]: 6.2 scrub starts 2024-03-26T05:22:28.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:28 smithi080 ceph-mon[40404]: 6.2 scrub ok 2024-03-26T05:22:28.438 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:28.477 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:28.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:28.579 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:28.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:28.620 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:28.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:28.721 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:28.752 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:28.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:28.855 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:28.903 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:28.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:29.004 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:29.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:29.042 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:29.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:29.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:29 smithi080 ceph-mon[29691]: 3.f scrub starts 2024-03-26T05:22:29.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:29 smithi080 ceph-mon[29691]: 3.f scrub ok 2024-03-26T05:22:29.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:29 smithi080 ceph-mon[29691]: pgmap v1044: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 505 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 1101/24891 objects misplaced (4.423%) 2024-03-26T05:22:29.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:29 smithi080 ceph-mon[40404]: 3.f scrub starts 2024-03-26T05:22:29.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:29 smithi080 ceph-mon[40404]: 3.f scrub ok 2024-03-26T05:22:29.112 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:29 smithi080 ceph-mon[40404]: pgmap v1044: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 505 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 1101/24891 objects misplaced (4.423%) 2024-03-26T05:22:29.143 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:29.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T05:22:29.169 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:29.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:29.271 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:29.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:29 smithi132 ceph-mon[33500]: 3.f scrub starts 2024-03-26T05:22:29.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:29 smithi132 ceph-mon[33500]: 3.f scrub ok 2024-03-26T05:22:29.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:29 smithi132 ceph-mon[33500]: pgmap v1044: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 505 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 1101/24891 objects misplaced (4.423%) 2024-03-26T05:22:29.301 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:29.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:29.404 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:29.457 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:29.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:29.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:29.559 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:29.590 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:29.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:29.692 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:29.743 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:29.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:29.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:29.764 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:29.817 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:29.845 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:29.871 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:29.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:29.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:29.973 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:29.999 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:30.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:30.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:30.025 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T05:22:30.101 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:30.130 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:30.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:30.231 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:30.287 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:30.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:30.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:30.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:30 smithi132 ceph-mon[33500]: 5.18 scrub starts 2024-03-26T05:22:30.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:30 smithi132 ceph-mon[33500]: 5.18 scrub ok 2024-03-26T05:22:30.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:30 smithi132 ceph-mon[33500]: 1.3 scrub starts 2024-03-26T05:22:30.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:30 smithi132 ceph-mon[33500]: 1.3 scrub ok 2024-03-26T05:22:30.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:30 smithi132 ceph-mon[33500]: 4.3 scrub starts 2024-03-26T05:22:30.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:30 smithi132 ceph-mon[33500]: 4.3 scrub ok 2024-03-26T05:22:30.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:30 smithi132 ceph-mon[33500]: 6.b scrub starts 2024-03-26T05:22:30.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[29691]: 5.18 scrub starts 2024-03-26T05:22:30.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[29691]: 5.18 scrub ok 2024-03-26T05:22:30.313 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[29691]: 1.3 scrub starts 2024-03-26T05:22:30.313 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[29691]: 1.3 scrub ok 2024-03-26T05:22:30.313 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[29691]: 4.3 scrub starts 2024-03-26T05:22:30.313 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[29691]: 4.3 scrub ok 2024-03-26T05:22:30.313 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[29691]: 6.b scrub starts 2024-03-26T05:22:30.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[40404]: 5.18 scrub starts 2024-03-26T05:22:30.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[40404]: 5.18 scrub ok 2024-03-26T05:22:30.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[40404]: 1.3 scrub starts 2024-03-26T05:22:30.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[40404]: 1.3 scrub ok 2024-03-26T05:22:30.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[40404]: 4.3 scrub starts 2024-03-26T05:22:30.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[40404]: 4.3 scrub ok 2024-03-26T05:22:30.314 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:30 smithi080 ceph-mon[40404]: 6.b scrub starts 2024-03-26T05:22:30.389 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:30.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:30.446 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:30.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:30.547 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:30.588 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:30.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:30.689 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:30.716 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:30.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:30.818 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:30.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:30.857 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:30.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:30.958 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:30.986 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:30.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:30.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:31.089 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:31.128 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:31.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:31.230 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:31.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:31.285 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:31.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:31.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:31 smithi080 ceph-mon[29691]: 3.13 scrub starts 2024-03-26T05:22:31.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:31 smithi080 ceph-mon[29691]: 3.13 scrub ok 2024-03-26T05:22:31.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:31 smithi080 ceph-mon[29691]: 6.b scrub ok 2024-03-26T05:22:31.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:31 smithi080 ceph-mon[29691]: pgmap v1045: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 469 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 613 B/s rd, 0 B/s wr, 781 op/s; 1101/23199 objects misplaced (4.746%) 2024-03-26T05:22:31.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:31 smithi080 ceph-mon[29691]: 4.b scrub starts 2024-03-26T05:22:31.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:31 smithi080 ceph-mon[29691]: 4.b scrub ok 2024-03-26T05:22:31.290 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:31 smithi080 ceph-mon[40404]: 3.13 scrub starts 2024-03-26T05:22:31.290 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:31 smithi080 ceph-mon[40404]: 3.13 scrub ok 2024-03-26T05:22:31.290 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:31 smithi080 ceph-mon[40404]: 6.b scrub ok 2024-03-26T05:22:31.290 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:31 smithi080 ceph-mon[40404]: pgmap v1045: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 469 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 613 B/s rd, 0 B/s wr, 781 op/s; 1101/23199 objects misplaced (4.746%) 2024-03-26T05:22:31.290 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:31 smithi080 ceph-mon[40404]: 4.b scrub starts 2024-03-26T05:22:31.290 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:31 smithi080 ceph-mon[40404]: 4.b scrub ok 2024-03-26T05:22:31.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:31 smithi132 ceph-mon[33500]: 3.13 scrub starts 2024-03-26T05:22:31.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:31 smithi132 ceph-mon[33500]: 3.13 scrub ok 2024-03-26T05:22:31.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:31 smithi132 ceph-mon[33500]: 6.b scrub ok 2024-03-26T05:22:31.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:31 smithi132 ceph-mon[33500]: pgmap v1045: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 469 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 613 B/s rd, 0 B/s wr, 781 op/s; 1101/23199 objects misplaced (4.746%) 2024-03-26T05:22:31.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:31 smithi132 ceph-mon[33500]: 4.b scrub starts 2024-03-26T05:22:31.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:31 smithi132 ceph-mon[33500]: 4.b scrub ok 2024-03-26T05:22:31.386 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:31.412 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:31.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:31.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:31.514 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:31.556 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:31.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:31.658 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:31.685 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:31.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:31.787 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:31.814 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:31.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:31.916 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:31.958 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:31.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:32.059 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:32.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:32.088 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:32.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:32.189 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:32.257 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:32.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:32.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:32 smithi132 ceph-mon[33500]: 5.3 scrub starts 2024-03-26T05:22:32.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:32 smithi132 ceph-mon[33500]: 5.3 scrub ok 2024-03-26T05:22:32.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:32 smithi132 ceph-mon[33500]: 4.c scrub starts 2024-03-26T05:22:32.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:32 smithi132 ceph-mon[33500]: 4.c scrub ok 2024-03-26T05:22:32.292 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:32 smithi080 ceph-mon[40404]: 5.3 scrub starts 2024-03-26T05:22:32.292 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:32 smithi080 ceph-mon[40404]: 5.3 scrub ok 2024-03-26T05:22:32.292 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:32 smithi080 ceph-mon[40404]: 4.c scrub starts 2024-03-26T05:22:32.292 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:32 smithi080 ceph-mon[40404]: 4.c scrub ok 2024-03-26T05:22:32.292 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:32 smithi080 ceph-mon[29691]: 5.3 scrub starts 2024-03-26T05:22:32.292 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:32 smithi080 ceph-mon[29691]: 5.3 scrub ok 2024-03-26T05:22:32.293 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:32 smithi080 ceph-mon[29691]: 4.c scrub starts 2024-03-26T05:22:32.293 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:32 smithi080 ceph-mon[29691]: 4.c scrub ok 2024-03-26T05:22:32.359 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:32.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:32.412 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:32.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:32.513 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:32.541 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:32.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:32.642 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:32.683 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:32.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:32.785 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:32.822 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Removed 121107 objects 2024-03-26T05:22:32.822 INFO:tasks.radosbench.radosbench.0.smithi080.stdout:Clean up completed and total clean up time :130.839 2024-03-26T05:22:32.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:32.823 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:32.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:32.925 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:32.950 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:32.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:33.029 INFO:teuthology.orchestra.run.smithi080.stderr:set pool 11 pgp_num to 26 2024-03-26T05:22:33.053 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:33.080 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:33.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:33.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:33.182 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:33.194 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[29691]: 6.15 scrub starts 2024-03-26T05:22:33.194 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[29691]: 6.15 scrub ok 2024-03-26T05:22:33.194 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[29691]: 3.1d scrub starts 2024-03-26T05:22:33.194 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[29691]: 3.1d scrub ok 2024-03-26T05:22:33.195 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[29691]: pgmap v1046: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 135 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 113/7170 objects misplaced (1.576%) 2024-03-26T05:22:33.195 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2855312296' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-26T05:22:33.195 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-26T05:22:33.195 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[40404]: 6.15 scrub starts 2024-03-26T05:22:33.195 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[40404]: 6.15 scrub ok 2024-03-26T05:22:33.195 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[40404]: 3.1d scrub starts 2024-03-26T05:22:33.195 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[40404]: 3.1d scrub ok 2024-03-26T05:22:33.195 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[40404]: pgmap v1046: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 135 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 113/7170 objects misplaced (1.576%) 2024-03-26T05:22:33.195 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2855312296' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-26T05:22:33.195 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:33 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-26T05:22:33.205 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:33.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:33.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:33.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:33 smithi132 ceph-mon[33500]: 6.15 scrub starts 2024-03-26T05:22:33.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:33 smithi132 ceph-mon[33500]: 6.15 scrub ok 2024-03-26T05:22:33.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:33 smithi132 ceph-mon[33500]: 3.1d scrub starts 2024-03-26T05:22:33.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:33 smithi132 ceph-mon[33500]: 3.1d scrub ok 2024-03-26T05:22:33.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:33 smithi132 ceph-mon[33500]: pgmap v1046: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 135 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 113/7170 objects misplaced (1.576%) 2024-03-26T05:22:33.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:33 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2855312296' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-26T05:22:33.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:33 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-26T05:22:33.307 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:33.331 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:33.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:33.353 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:22:33.353 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:22:33.353 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 0 2024-03-26T05:22:33.353 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:22:33.433 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:33.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:33.457 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:33.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:33.558 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:33.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:33.583 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:33.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:33.684 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:33.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:33.709 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:33.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:33.810 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:33.835 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:33.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:33.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:33.961 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:33.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:34.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:34.087 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:34.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:34.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:34.189 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:34.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:34.214 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:34.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:34.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:34 smithi132 ceph-mon[33500]: 5.1f scrub starts 2024-03-26T05:22:34.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:34 smithi132 ceph-mon[33500]: 5.1f scrub ok 2024-03-26T05:22:34.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:34 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-03-26T05:22:34.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:34 smithi132 ceph-mon[33500]: osdmap e488: 8 total, 7 up, 6 in 2024-03-26T05:22:34.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:34 smithi132 ceph-mon[33500]: 3.1f scrub starts 2024-03-26T05:22:34.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:34 smithi132 ceph-mon[33500]: 3.1f scrub ok 2024-03-26T05:22:34.315 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:34.340 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:34.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:34.356 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:22:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:22:34.357 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-mon[29691]: 5.1f scrub starts 2024-03-26T05:22:34.357 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-mon[29691]: 5.1f scrub ok 2024-03-26T05:22:34.357 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-03-26T05:22:34.357 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-mon[29691]: osdmap e488: 8 total, 7 up, 6 in 2024-03-26T05:22:34.357 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-mon[29691]: 3.1f scrub starts 2024-03-26T05:22:34.357 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-mon[29691]: 3.1f scrub ok 2024-03-26T05:22:34.357 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-mon[40404]: 5.1f scrub starts 2024-03-26T05:22:34.357 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-mon[40404]: 5.1f scrub ok 2024-03-26T05:22:34.357 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-03-26T05:22:34.357 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-mon[40404]: osdmap e488: 8 total, 7 up, 6 in 2024-03-26T05:22:34.357 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-mon[40404]: 3.1f scrub starts 2024-03-26T05:22:34.358 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:34 smithi080 ceph-mon[40404]: 3.1f scrub ok 2024-03-26T05:22:34.442 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:34.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:34.467 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:34.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:34.568 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:34.593 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:34.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:34.694 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:34.719 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:34.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:34.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:34.820 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:34.845 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:34.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:34.946 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:34.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:34.971 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:34.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:35.038 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is set 2024-03-26T05:22:35.072 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:35.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:35.101 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:35.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:35.202 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:35.226 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:35.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:35.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:35.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:35 smithi132 ceph-mon[33500]: 5.9 scrub starts 2024-03-26T05:22:35.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:35 smithi132 ceph-mon[33500]: 5.9 scrub ok 2024-03-26T05:22:35.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:35 smithi132 ceph-mon[33500]: 4.f scrub starts 2024-03-26T05:22:35.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:35 smithi132 ceph-mon[33500]: 4.f scrub ok 2024-03-26T05:22:35.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:35 smithi132 ceph-mon[33500]: pgmap v1048: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 135 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 113/7170 objects misplaced (1.576%) 2024-03-26T05:22:35.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:35 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1549327048' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:22:35.328 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:35.351 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:35.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:35.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:35.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:35 smithi080 ceph-mon[29691]: 5.9 scrub starts 2024-03-26T05:22:35.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:35 smithi080 ceph-mon[29691]: 5.9 scrub ok 2024-03-26T05:22:35.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:35 smithi080 ceph-mon[29691]: 4.f scrub starts 2024-03-26T05:22:35.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:35 smithi080 ceph-mon[29691]: 4.f scrub ok 2024-03-26T05:22:35.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:35 smithi080 ceph-mon[29691]: pgmap v1048: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 135 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 113/7170 objects misplaced (1.576%) 2024-03-26T05:22:35.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:35 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1549327048' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:22:35.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:35 smithi080 ceph-mon[40404]: 5.9 scrub starts 2024-03-26T05:22:35.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:35 smithi080 ceph-mon[40404]: 5.9 scrub ok 2024-03-26T05:22:35.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:35 smithi080 ceph-mon[40404]: 4.f scrub starts 2024-03-26T05:22:35.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:35 smithi080 ceph-mon[40404]: 4.f scrub ok 2024-03-26T05:22:35.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:35 smithi080 ceph-mon[40404]: pgmap v1048: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 135 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 113/7170 objects misplaced (1.576%) 2024-03-26T05:22:35.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:35 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1549327048' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T05:22:35.454 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:35.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:35.479 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:35.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:35.580 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:35.604 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:35.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:35.706 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:35.733 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:35.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:35.835 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:35.864 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:35.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:35.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:35.965 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:35.994 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:35.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:36.095 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:36.115 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:22:36 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:22:36.028+0000 7f2fcc605700 -1 mgr get_metadata_python Requested missing service osd.4 2024-03-26T05:22:36.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:36 smithi080 ceph-mon[29691]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:22:36.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:36 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1549327048' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:22:36.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:36 smithi080 ceph-mon[29691]: osdmap e489: 8 total, 7 up, 6 in 2024-03-26T05:22:36.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:36 smithi080 ceph-mon[29691]: 6.12 scrub starts 2024-03-26T05:22:36.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:36 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:36.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:36 smithi080 ceph-mon[40404]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:22:36.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:36 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1549327048' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:22:36.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:36 smithi080 ceph-mon[40404]: osdmap e489: 8 total, 7 up, 6 in 2024-03-26T05:22:36.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:36 smithi080 ceph-mon[40404]: 6.12 scrub starts 2024-03-26T05:22:36.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:36 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:36.120 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:36.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:36.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:36.222 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:36.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:36.247 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:36.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:36.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:22:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:22:36.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:36 smithi132 ceph-mon[33500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:22:36.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:36 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1549327048' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T05:22:36.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:36 smithi132 ceph-mon[33500]: osdmap e489: 8 total, 7 up, 6 in 2024-03-26T05:22:36.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:36 smithi132 ceph-mon[33500]: 6.12 scrub starts 2024-03-26T05:22:36.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:36 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:36.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:36.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:36.372 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:36.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:36.473 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:36.499 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:36.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:36.509 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T05:22:36.532 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:36.600 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:36.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:36.629 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:36.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:36.730 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:36.756 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:36.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:36.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:36.858 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:36.883 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:36.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:36.984 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:37.007 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:37.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:37.109 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:37.137 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:37.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:37.239 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:37.266 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:37.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:37.368 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:37.393 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:37.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:37.494 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:37.519 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-26T05:22:37.519 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-26T05:22:37.522 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:37.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:37.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:37.624 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:37.647 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:37.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:37.749 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:37.773 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:37.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:37.855 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T05:22:37.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:37 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:37.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:37 smithi080 ceph-mon[40404]: pgmap v1050: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 29 MiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 951 op/s; 113/2076 objects misplaced (5.443%) 2024-03-26T05:22:37.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:37 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:37.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:37 smithi080 ceph-mon[29691]: pgmap v1050: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 29 MiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 951 op/s; 113/2076 objects misplaced (5.443%) 2024-03-26T05:22:37.875 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:37.898 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:37.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:38.000 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:38.028 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:38.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:38.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:37 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:38.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:37 smithi132 ceph-mon[33500]: pgmap v1050: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 29 MiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 951 op/s; 113/2076 objects misplaced (5.443%) 2024-03-26T05:22:38.130 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:38.160 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:38.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:38.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:38.262 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:38.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:38.290 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:38.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:38.391 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:38.415 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:38.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:38.516 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:38.545 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:38.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:38.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:38.647 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:38.675 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:38.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:38.776 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:38.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:38.802 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:38.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:38.903 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:38.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:38.932 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:38.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:39.033 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:39.040 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:38 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:22:39.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:38 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:22:39.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:38 smithi132 ceph-mon[33500]: osdmap e490: 8 total, 7 up, 6 in 2024-03-26T05:22:39.060 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:39.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:39.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:38 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:22:39.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:38 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:22:39.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:38 smithi080 ceph-mon[29691]: osdmap e490: 8 total, 7 up, 6 in 2024-03-26T05:22:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:38 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:22:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:38 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:22:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:38 smithi080 ceph-mon[40404]: osdmap e490: 8 total, 7 up, 6 in 2024-03-26T05:22:39.162 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:39.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:39.190 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:39.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:39.291 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:39.319 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:39.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:39.421 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:39.448 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:39.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:39.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:39.550 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:39.576 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:39.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:39.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:39.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:39.706 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:39.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:39.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:39.835 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:39.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:39.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:39.937 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:39.965 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:39.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:40.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:39 smithi132 ceph-mon[33500]: pgmap v1052: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 365 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:40.067 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:40.089 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7] out_osds: [4, 5] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T05:22:40.089 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T05:22:40.089 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-26T05:22:40.090 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T05:22:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:40.097 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:40.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:40.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:39 smithi080 ceph-mon[40404]: pgmap v1052: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 365 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:40.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:39 smithi080 ceph-mon[29691]: pgmap v1052: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 365 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:40.198 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:40.223 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:40.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:40.325 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:40.351 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:40.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:40.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:40.369 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:40.408 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:40.412 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:40.453 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:40.481 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:40.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:40.583 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:40.607 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:40.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:40.708 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:40.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:40.738 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:40.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:40.748 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:40.839 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:40.868 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:40.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:40.969 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:40.997 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:40.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:41.098 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:41.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:41.126 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:41.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:41.227 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:41.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:41.252 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:41.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:41.353 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:41.377 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:41.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:41.478 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:41.508 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:41.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:41.609 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:41.637 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:41.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:41.720 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:22:41.738 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:41.740 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:22:41.765 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:22:41.766 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:41.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:41.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:41.793 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:22:41.814 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:22:41.833 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:22:41.853 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:22:41.854 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:22:41.854 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:22:41.854 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:22:41.854 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:22:41.854 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:22:41.854 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:22:41.854 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:22:41.854 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:22:41.855 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:41 smithi080 ceph-mon[29691]: pgmap v1053: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 707 B/s rd, 0 B/s wr, 303 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:41.855 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:41 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4287228458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:22:41.855 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:41 smithi080 ceph-mon[40404]: pgmap v1053: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 707 B/s rd, 0 B/s wr, 303 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:41.855 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:41 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4287228458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:22:41.868 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:41.892 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:41.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:41.994 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:42.022 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:42.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:42.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:41 smithi132 ceph-mon[33500]: pgmap v1053: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 707 B/s rd, 0 B/s wr, 303 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:42.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:41 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4287228458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:22:42.125 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:42.151 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:42.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:42.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:42.283 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:42.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:42.384 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:42.411 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:42.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:42.512 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:42.540 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:42.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:42.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:42.642 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:42.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:42.670 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:42.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:42.771 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:42.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:42.799 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:42.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:42.900 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:42.928 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:42.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:43.029 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:43.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:42 smithi132 ceph-mon[33500]: osdmap e491: 8 total, 7 up, 6 in 2024-03-26T05:22:43.058 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:43.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:43.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:43.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:42 smithi080 ceph-mon[40404]: osdmap e491: 8 total, 7 up, 6 in 2024-03-26T05:22:43.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:42 smithi080 ceph-mon[29691]: osdmap e491: 8 total, 7 up, 6 in 2024-03-26T05:22:43.160 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:43.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:43.189 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:43.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:43.290 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:43.318 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:43.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:43.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:43.450 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:43.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:43.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:43.579 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:43.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:43.680 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:43.707 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:43.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:43.809 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:43.839 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:43.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:43.940 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:43.968 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:43.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:43.987 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:43 smithi080 ceph-mon[40404]: pgmap v1055: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 260 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:43.988 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:43 smithi080 ceph-mon[29691]: pgmap v1055: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 260 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:44.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:43 smithi132 ceph-mon[33500]: pgmap v1055: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 260 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:44.069 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:44.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:44.098 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:44.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:44.199 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:44.228 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:44.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:44.312 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:22:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:22:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:22:44.329 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:44.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:44.359 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:44.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:44.460 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:44.490 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:44.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:44.591 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:44.620 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:44.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:22:44.721 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:22:44.723 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:44.752 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:44.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:44.854 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:44.882 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:44.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:44.983 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:45.010 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:45.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:45.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:45.111 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:45.140 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:45.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:45.242 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:45.270 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:45.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:45.370 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:45.395 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:45.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:45.496 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:45.525 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:45.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:45.626 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:45.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:45.653 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:45.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:45.754 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:45.784 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:45.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:45.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:45.886 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:45.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:45.911 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:45.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:46.012 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:46.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:46.038 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:46.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:46.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:45 smithi132 ceph-mon[33500]: pgmap v1056: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 62 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:46.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:45 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:22:46.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:45 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:22:46.079 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:45 smithi080 ceph-mon[40404]: pgmap v1056: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 62 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:46.080 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:45 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:22:46.080 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:45 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:22:46.080 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:45 smithi080 ceph-mon[29691]: pgmap v1056: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 62 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:46.080 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:45 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:22:46.080 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:45 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:22:46.140 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:46.163 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:46.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:46.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:46.265 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:46.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:46.290 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:46.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:46.391 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:46.414 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:46.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:46.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:46.516 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:46.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:46.540 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:46.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:46.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:22:46 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:22:46.641 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:46.669 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:46.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:46.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:46.722 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T05:22:46.771 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:46.800 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:46.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:46.901 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:46.925 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:46.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:46.982 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:46 smithi080 ceph-mon[40404]: osdmap e492: 8 total, 7 up, 6 in 2024-03-26T05:22:46.982 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:46 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:22:46.982 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:46 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:22:46.982 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:46 smithi080 ceph-mon[40404]: osdmap e493: 8 total, 7 up, 6 in 2024-03-26T05:22:46.983 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:46 smithi080 ceph-mon[29691]: osdmap e492: 8 total, 7 up, 6 in 2024-03-26T05:22:46.983 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:46 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:22:46.983 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:46 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:22:46.983 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:46 smithi080 ceph-mon[29691]: osdmap e493: 8 total, 7 up, 6 in 2024-03-26T05:22:47.026 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:46 smithi132 ceph-mon[33500]: osdmap e492: 8 total, 7 up, 6 in 2024-03-26T05:22:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:46 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:22:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:46 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:22:47.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:46 smithi132 ceph-mon[33500]: osdmap e493: 8 total, 7 up, 6 in 2024-03-26T05:22:47.049 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:47.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:47.151 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:47.179 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:47.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:47.282 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:47.306 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:47.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:47.408 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:47.436 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:47.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:47.538 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:47.566 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:47.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:47.667 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:47.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:47.693 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:47.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:47.794 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:47.822 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:47.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:47.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:47 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:47.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:47 smithi080 ceph-mon[29691]: pgmap v1058: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:47.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:47 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:47.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:47 smithi080 ceph-mon[40404]: pgmap v1058: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:47.907 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:22:47.907 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:22:47.923 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:47.950 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:47.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:47.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:47 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:48.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:47 smithi132 ceph-mon[33500]: pgmap v1058: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:48.053 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1 2024-03-26T05:22:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:48.078 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:48.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.1' 2024-03-26T05:22:48.179 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:48.207 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:48.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:48.309 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:48.334 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:48.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:48.435 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:48.462 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:48.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:48.565 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:48.592 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:48.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:48.694 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:48.721 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:48.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:48.792 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'unique_pool_4' removed 2024-03-26T05:22:48.814 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is set 2024-03-26T05:22:48.823 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:48.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:48.854 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:48.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:48.956 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:48.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:48.980 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:48.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:49.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:48 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1054630268' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:22:49.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:48 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4113272984' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:22:49.026 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:48 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:22:49.026 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:48 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1054630268' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:22:49.026 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:48 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4113272984' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:22:49.026 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:48 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:22:49.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:48 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1054630268' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:22:49.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:48 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4113272984' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:22:49.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:48 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:22:49.070 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T05:22:49.082 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:49.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:49.106 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:49.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:49.207 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:49.232 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:49.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:49.334 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:49.358 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:49.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:49.460 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:49.487 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:49.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:49.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:49.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:49.613 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:49.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:49.714 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:49.739 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:49.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:49.841 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6 2024-03-26T05:22:49.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:49 smithi080 ceph-mon[29691]: pgmap v1060: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 786 B/s rd, 0 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:49.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:49 smithi080 ceph-mon[29691]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:22:49.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:49 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1054630268' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:22:49.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:49 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:22:49.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:49 smithi080 ceph-mon[29691]: osdmap e494: 8 total, 7 up, 6 in 2024-03-26T05:22:49.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:49 smithi080 ceph-mon[40404]: pgmap v1060: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 786 B/s rd, 0 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:49.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:49 smithi080 ceph-mon[40404]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:22:49.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:49 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1054630268' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:22:49.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:49 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:22:49.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:49 smithi080 ceph-mon[40404]: osdmap e494: 8 total, 7 up, 6 in 2024-03-26T05:22:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:49.867 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:49.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.6' 2024-03-26T05:22:49.968 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7 2024-03-26T05:22:49.991 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:49.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.7' 2024-03-26T05:22:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:49 smithi132 ceph-mon[33500]: pgmap v1060: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 934 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 786 B/s rd, 0 op/s; 12/708 objects misplaced (1.695%) 2024-03-26T05:22:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:49 smithi132 ceph-mon[33500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:22:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:49 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1054630268' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:22:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:49 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T05:22:50.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:49 smithi132 ceph-mon[33500]: osdmap e494: 8 total, 7 up, 6 in 2024-03-26T05:22:50.093 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3 2024-03-26T05:22:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:50.162 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:50.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.3' 2024-03-26T05:22:50.263 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0 2024-03-26T05:22:50.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:50.288 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:50.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.0' 2024-03-26T05:22:50.389 DEBUG:teuthology.orchestra.run.smithi132:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5 2024-03-26T05:22:50.417 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: docker: command not found 2024-03-26T05:22:50.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:50.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi132 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.5' 2024-03-26T05:22:50.519 DEBUG:teuthology.orchestra.run.smithi080:> sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2 2024-03-26T05:22:50.544 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-26T05:22:50.558 INFO:tasks.thrashosds:joining thrashosds 2024-03-26T05:22:50.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:22:50.559 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: docker: command not found 2024-03-26T05:22:50.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi080 with status 1: 'sudo docker kill -s 1 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd.2' 2024-03-26T05:22:50.794 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:50.830 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:50.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:50 smithi080 ceph-mon[40404]: pgmap v1062: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T05:22:50.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:50 smithi080 ceph-mon[40404]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T05:22:50.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:50 smithi080 ceph-mon[29691]: pgmap v1062: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T05:22:50.924 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:50 smithi080 ceph-mon[29691]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T05:22:50.979 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:22:50.979 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:22:50.979 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 494, 2024-03-26T05:22:50.979 INFO:teuthology.orchestra.run.smithi080.stdout: "fsid": "dd09e9cc-eb2c-11ee-95d0-87774f69a715", 2024-03-26T05:22:50.979 INFO:teuthology.orchestra.run.smithi080.stdout: "created": "2024-03-26T04:54:51.106954+0000", 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "modified": "2024-03-26T05:22:48.790861+0000", 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "last_up_change": "2024-03-26T05:15:47.612973+0000", 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "last_in_change": "2024-03-26T05:20:29.292458+0000", 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_num": 5806080, 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_set": [ 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "nodeep-scrub", 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "noscrub", 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "pglog_hardlimit", 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "purged_snapdirs", 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "recovery_deletes", 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "sortbitwise" 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:22:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_version": 19, 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_snapshot": "", 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_max": 11, 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "max_osd": 8, 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "require_min_compat_client": "luminous", 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "min_compat_client": "luminous", 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "require_osd_release": "squid", 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "allow_crimson": false, 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "pools": [ 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 1, 2024-03-26T05:22:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": ".mgr", 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T04:58:06.876819+0000", 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:22:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 11, 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 11, 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 11, 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 11, 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 11, 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "326", 2024-03-26T05:22:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "324", 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:22:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:22:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: "options": { 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_max": 32, 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_min": 1 2024-03-26T05:22:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": {} 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.190000057220459, 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.190000057220459, 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.75, 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.6399999856948853, 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.6399999856948853, 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 2, 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "datapool", 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:20.705417+0000", 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 8193, 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:22:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 3, 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 3, 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 3, 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 3, 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 3, 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:22:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "71", 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 2, 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 71, 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:22:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:22:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:22:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:22:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:22:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:22:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:22:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:22:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:22:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:22:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:22:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:22:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "rbd": {} 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.6700000762939453, 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.6700000762939453, 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.75, 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 2, 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 2, 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:22:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 3, 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": ".rgw.root", 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:21.150368+0000", 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:22:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:22:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:22:50.996 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:22:50.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:50.996 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "69", 2024-03-26T05:22:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:22:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:22:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:22:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:22:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:22:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:22:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:22:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:22:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:22:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:22:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:22:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:22:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:22:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2, 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2, 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.75, 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.5, 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.5, 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 4, 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.log", 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:23.638065+0000", 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:22:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:22:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "71", 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:22:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:22:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:22:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.5099999904632568, 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.5099999904632568, 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.75, 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 5, 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.control", 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:25.181000+0000", 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:22:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "73", 2024-03-26T05:22:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:22:51.009 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:22:51.009 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:22:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:22:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "options": {}, 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 1.75, 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 1.75, 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.75, 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.309999942779541, 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.309999942779541, 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "pool": 6, 2024-03-26T05:22:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_name": "default.rgw.meta", 2024-03-26T05:22:51.014 INFO:teuthology.orchestra.run.smithi080.stdout: "create_time": "2024-03-26T05:00:27.249998+0000", 2024-03-26T05:22:51.014 INFO:teuthology.orchestra.run.smithi080.stdout: "flags": 1, 2024-03-26T05:22:51.014 INFO:teuthology.orchestra.run.smithi080.stdout: "flags_names": "hashpspool", 2024-03-26T05:22:51.014 INFO:teuthology.orchestra.run.smithi080.stdout: "type": 1, 2024-03-26T05:22:51.014 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 3, 2024-03-26T05:22:51.014 INFO:teuthology.orchestra.run.smithi080.stdout: "min_size": 2, 2024-03-26T05:22:51.014 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_rule": 0, 2024-03-26T05:22:51.014 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_count": 0, 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_target": 0, 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "object_hash": 2, 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_mode": "off", 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num": 32, 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num": 32, 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_placement_num_target": 32, 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_target": 32, 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_num_pending": 32, 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "last_pg_merge_meta": { 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "source_pgid": "0.0", 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "ready_epoch": 0, 2024-03-26T05:22:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_started": 0, 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "last_epoch_clean": 0, 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "source_version": "0'0", 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "target_version": "0'0" 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "last_change": "75", 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend": "0", 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "auid": 0, 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_mode": "selfmanaged", 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_seq": 0, 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "snap_epoch": 0, 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "pool_snaps": [], 2024-03-26T05:22:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps": "[]", 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_bytes": 0, 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "quota_max_objects": 0, 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "tiers": [], 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_of": -1, 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "read_tier": -1, 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "write_tier": -1, 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_mode": "none", 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_bytes": 0, 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "target_max_objects": 0, 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_flush_age": 0, 2024-03-26T05:22:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "cache_min_evict_age": 0, 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profile": "", 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_params": { 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "none" 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_period": 0, 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_count": 0, 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "use_gmt_hitset": true, 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "min_read_recency_for_promote": 0, 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "min_write_recency_for_promote": 0, 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "hit_set_search_last_n": 0, 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "grade_table": [], 2024-03-26T05:22:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "stripe_width": 0, 2024-03-26T05:22:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: "expected_num_objects": 0, 2024-03-26T05:22:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: "fast_read": false, 2024-03-26T05:22:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: "options": { 2024-03-26T05:22:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_autoscale_bias": 4 2024-03-26T05:22:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: "application_metadata": { 2024-03-26T05:22:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": {} 2024-03-26T05:22:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: "read_balance": { 2024-03-26T05:22:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: "score_type": "Fair distribution", 2024-03-26T05:22:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: "score_acting": 2.25, 2024-03-26T05:22:51.020 INFO:teuthology.orchestra.run.smithi080.stdout: "score_stable": 2.25, 2024-03-26T05:22:51.020 INFO:teuthology.orchestra.run.smithi080.stdout: "optimal_score": 0.75, 2024-03-26T05:22:51.020 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_acting": 1.690000057220459, 2024-03-26T05:22:51.020 INFO:teuthology.orchestra.run.smithi080.stdout: "raw_score_stable": 1.690000057220459, 2024-03-26T05:22:51.020 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity_weighted": 1, 2024-03-26T05:22:51.020 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity": 1, 2024-03-26T05:22:51.020 INFO:teuthology.orchestra.run.smithi080.stdout: "average_primary_affinity_weighted": 1 2024-03-26T05:22:51.020 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 0, 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "35a52544-0b0b-47d6-8d6a-571c48ceb366", 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 8, 2024-03-26T05:22:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 477, 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6802", 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6803", 2024-03-26T05:22:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6804", 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6805", 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:22:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6808", 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6809", 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:22:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6806", 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6807", 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1840686622 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6803/1840686622", 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6805/1840686622", 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6809/1840686622", 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6807/1840686622", 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 1, 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "efa5ea33-4f11-49b7-a3f3-f60d5329db23", 2024-03-26T05:22:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 13, 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 481, 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6810", 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6811", 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6812", 2024-03-26T05:22:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6813", 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6816", 2024-03-26T05:22:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6817", 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6814", 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6815", 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1726833996 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6811/1726833996", 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6813/1726833996", 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6817/1726833996", 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6815/1726833996", 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:22:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 2, 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "536bec9b-9636-4e67-b452-ed1a55c22ab6", 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 18, 2024-03-26T05:22:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 473, 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6818", 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6819", 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:22:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6820", 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6821", 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:22:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6824", 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6825", 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:22:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6822", 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6823", 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 128713757 2024-03-26T05:22:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6819/128713757", 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6821/128713757", 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6825/128713757", 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6823/128713757", 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 3, 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "640cfdb6-b9c5-446d-8312-504ae12c7818", 2024-03-26T05:22:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 27, 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 472, 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6826", 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6827", 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6828", 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6829", 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6832", 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6833", 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6830", 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.80:6831", 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 418771441 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.80:6827/418771441", 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.80:6829/418771441", 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.80:6833/418771441", 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.80:6831/418771441", 2024-03-26T05:22:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 4, 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "0cc7822a-ed73-4c79-a30d-1e151ddd6639", 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 0, 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 0, 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 0, 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:22:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 33, 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 336, 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 342, 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6800", 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6801", 2024-03-26T05:22:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6802", 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6803", 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:22:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6806", 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6807", 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:22:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6804", 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6805", 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1441071423 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6801/1441071423", 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6803/1441071423", 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6807/1441071423", 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6805/1441071423", 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "exists" 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 5, 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc", 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 0, 2024-03-26T05:22:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 0, 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 41, 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 442, 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6808", 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.050 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.050 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6809", 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6810", 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6811", 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6814", 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6815", 2024-03-26T05:22:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6812", 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6813", 2024-03-26T05:22:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 794504625 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6809/794504625", 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6811/794504625", 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6815/794504625", 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6813/794504625", 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 6, 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "2970bbab-11b2-4d1b-b57e-037bdac089b5", 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 0, 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 0, 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 51, 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 483, 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 0, 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:22:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6816", 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6817", 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:22:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6818", 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6819", 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6822", 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6823", 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:22:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6820", 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6821", 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 2209078972 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6817/2209078972", 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6819/2209078972", 2024-03-26T05:22:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6823/2209078972", 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6821/2209078972", 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 7, 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "uuid": "12f4b466-28f9-469b-8fd0-65b91c6dc22d", 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "up": 1, 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "in": 1, 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "weight": 1, 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_affinity": 1, 2024-03-26T05:22:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_begin": 61, 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "last_clean_end": 292, 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "up_from": 335, 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "up_thru": 482, 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "down_at": 293, 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "lost_at": 0, 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addrs": { 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6824", 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6825", 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addrs": { 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6826", 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6827", 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addrs": { 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6830", 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6831", 2024-03-26T05:22:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addrs": { 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: "addrvec": [ 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v2", 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6828", 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: "type": "v1", 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: "addr": "172.21.15.132:6829", 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: "nonce": 1165391760 2024-03-26T05:22:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: "public_addr": "172.21.15.132:6825/1165391760", 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: "cluster_addr": "172.21.15.132:6827/1165391760", 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_back_addr": "172.21.15.132:6831/1165391760", 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: "heartbeat_front_addr": "172.21.15.132:6829/1165391760", 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: "state": [ 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: "exists", 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: "up" 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_xinfo": [ 2024-03-26T05:22:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 0, 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:21:45.572907+0000", 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 1, 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:22:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:22:44.978436+0000", 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 2, 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:22:46.322063+0000", 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:22:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 3, 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:22:37.705421+0000", 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 4, 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:22:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 65536, 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:15:24.108106+0000", 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 341 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 5, 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 65536, 2024-03-26T05:22:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:21:47.923174+0000", 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 6, 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:22:21.559380+0000", 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 0 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": 7, 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "down_stamp": "0.000000", 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_probability": 0, 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "laggy_interval": 0, 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "features": 4540701547738038271, 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "old_weight": 0, 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "last_purged_snaps_scrub": "2024-03-26T05:21:44.329029+0000", 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "dead_epoch": 292 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap": [ 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "3.1", 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: 4, 2024-03-26T05:22:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: 5 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.b", 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: "osds": [ 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: 0, 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: 4, 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: 2 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap_items": [ 2024-03-26T05:22:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.2", 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.7", 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:22:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "4.1f", 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 2 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "5.10", 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 6, 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 3 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: "pgid": "6.b", 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: "mappings": [ 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: "from": 1, 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: "to": 3 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_upmap_primaries": [], 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "pg_temp": [], 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "primary_temp": [], 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "blocklist": { 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3992539385": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/2770889532": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/2466284312": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/490056571": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1986135300": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3796611170": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1993138634": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:22:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1832498181": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/2466284312": "2024-03-27T04:55:21.711776+0000", 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1650870847": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/1418683843": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/3147301987": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6801/2770889532": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/571905941": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1451012560": "2024-03-27T04:55:48.542948+0000", 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:0/1235125152": "2024-03-27T05:01:03.689739+0000", 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "172.21.15.80:6800/1418683843": "2024-03-27T05:01:03.689739+0000" 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "range_blocklist": {}, 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "erasure_code_profiles": { 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "default": { 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "crush-failure-domain": "osd", 2024-03-26T05:22:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "k": "2", 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "m": "1", 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "plugin": "jerasure", 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "technique": "reed_sol_van" 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "removed_snaps_queue": [], 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "new_removed_snaps": [], 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "new_purged_snaps": [], 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "crush_node_flags": {}, 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "device_class_flags": {}, 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode": { 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode_enabled": false, 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_bucket_count": 0, 2024-03-26T05:22:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "degraded_stretch_mode": 0, 2024-03-26T05:22:51.079 INFO:teuthology.orchestra.run.smithi080.stdout: "recovering_stretch_mode": 0, 2024-03-26T05:22:51.079 INFO:teuthology.orchestra.run.smithi080.stdout: "stretch_mode_bucket": 0 2024-03-26T05:22:51.079 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:22:51.079 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:22:51.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:50 smithi132 ceph-mon[33500]: pgmap v1062: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T05:22:51.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:50 smithi132 ceph-mon[33500]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T05:22:51.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:22:51 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:22:51.047+0000 7f2fcc605700 -1 mgr get_metadata_python Requested missing service osd.4 2024-03-26T05:22:51.864 INFO:tasks.thrashosds.thrasher:j is {'epoch': 494, 'fsid': 'dd09e9cc-eb2c-11ee-95d0-87774f69a715', 'created': '2024-03-26T04:54:51.106954+0000', 'modified': '2024-03-26T05:22:48.790861+0000', 'last_up_change': '2024-03-26T05:15:47.612973+0000', 'last_in_change': '2024-03-26T05:20:29.292458+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T04:58:06.876819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '326', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '324', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 0.75, 'raw_score_acting': 1.6399999856948853, 'raw_score_stable': 1.6399999856948853, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T05:00:20.705417+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T05:00:21.150368+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T05:00:23.638065+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T05:00:25.181000+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 0.75, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T05:00:27.249998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 0.75, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '35a52544-0b0b-47d6-8d6a-571c48ceb366', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6802', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6803', 'nonce': 1840686622}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6804', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6805', 'nonce': 1840686622}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6808', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6809', 'nonce': 1840686622}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6806', 'nonce': 1840686622}, {'type': 'v1', 'addr': '172.21.15.80:6807', 'nonce': 1840686622}]}, 'public_addr': '172.21.15.80:6803/1840686622', 'cluster_addr': '172.21.15.80:6805/1840686622', 'heartbeat_back_addr': '172.21.15.80:6809/1840686622', 'heartbeat_front_addr': '172.21.15.80:6807/1840686622', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efa5ea33-4f11-49b7-a3f3-f60d5329db23', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 481, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6810', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6811', 'nonce': 1726833996}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6812', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6813', 'nonce': 1726833996}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6816', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6817', 'nonce': 1726833996}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6814', 'nonce': 1726833996}, {'type': 'v1', 'addr': '172.21.15.80:6815', 'nonce': 1726833996}]}, 'public_addr': '172.21.15.80:6811/1726833996', 'cluster_addr': '172.21.15.80:6813/1726833996', 'heartbeat_back_addr': '172.21.15.80:6817/1726833996', 'heartbeat_front_addr': '172.21.15.80:6815/1726833996', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '536bec9b-9636-4e67-b452-ed1a55c22ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 473, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6818', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6819', 'nonce': 128713757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6820', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6821', 'nonce': 128713757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6824', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6825', 'nonce': 128713757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6822', 'nonce': 128713757}, {'type': 'v1', 'addr': '172.21.15.80:6823', 'nonce': 128713757}]}, 'public_addr': '172.21.15.80:6819/128713757', 'cluster_addr': '172.21.15.80:6821/128713757', 'heartbeat_back_addr': '172.21.15.80:6825/128713757', 'heartbeat_front_addr': '172.21.15.80:6823/128713757', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '640cfdb6-b9c5-446d-8312-504ae12c7818', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 472, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6826', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6827', 'nonce': 418771441}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6828', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6829', 'nonce': 418771441}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6832', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6833', 'nonce': 418771441}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6830', 'nonce': 418771441}, {'type': 'v1', 'addr': '172.21.15.80:6831', 'nonce': 418771441}]}, 'public_addr': '172.21.15.80:6827/418771441', 'cluster_addr': '172.21.15.80:6829/418771441', 'heartbeat_back_addr': '172.21.15.80:6833/418771441', 'heartbeat_front_addr': '172.21.15.80:6831/418771441', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0cc7822a-ed73-4c79-a30d-1e151ddd6639', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 336, 'down_at': 342, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6800', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6801', 'nonce': 1441071423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6802', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6803', 'nonce': 1441071423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6806', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6807', 'nonce': 1441071423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6804', 'nonce': 1441071423}, {'type': 'v1', 'addr': '172.21.15.132:6805', 'nonce': 1441071423}]}, 'public_addr': '172.21.15.132:6801/1441071423', 'cluster_addr': '172.21.15.132:6803/1441071423', 'heartbeat_back_addr': '172.21.15.132:6807/1441071423', 'heartbeat_front_addr': '172.21.15.132:6805/1441071423', 'state': ['exists']}, {'osd': 5, 'uuid': 'acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 442, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6808', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6809', 'nonce': 794504625}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6810', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6811', 'nonce': 794504625}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6814', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6815', 'nonce': 794504625}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6812', 'nonce': 794504625}, {'type': 'v1', 'addr': '172.21.15.132:6813', 'nonce': 794504625}]}, 'public_addr': '172.21.15.132:6809/794504625', 'cluster_addr': '172.21.15.132:6811/794504625', 'heartbeat_back_addr': '172.21.15.132:6815/794504625', 'heartbeat_front_addr': '172.21.15.132:6813/794504625', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2970bbab-11b2-4d1b-b57e-037bdac089b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 483, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6816', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6817', 'nonce': 2209078972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6818', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6819', 'nonce': 2209078972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6822', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6823', 'nonce': 2209078972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6820', 'nonce': 2209078972}, {'type': 'v1', 'addr': '172.21.15.132:6821', 'nonce': 2209078972}]}, 'public_addr': '172.21.15.132:6817/2209078972', 'cluster_addr': '172.21.15.132:6819/2209078972', 'heartbeat_back_addr': '172.21.15.132:6823/2209078972', 'heartbeat_front_addr': '172.21.15.132:6821/2209078972', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '12f4b466-28f9-469b-8fd0-65b91c6dc22d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 61, 'last_clean_end': 292, 'up_from': 335, 'up_thru': 482, 'down_at': 293, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6824', 'nonce': 1165391760}, {'type': 'v1', 'addr': '172.21.15.132:6825', 'nonce': 1165391760}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6826', 'nonce': 1165391760}, {'type': 'v1', 'addr': '172.21.15.132:6827', 'nonce': 1165391760}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6830', 'nonce': 1165391760}, {'type': 'v1', 'addr': '172.21.15.132:6831', 'nonce': 1165391760}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6828', 'nonce': 1165391760}, {'type': 'v1', 'addr': '172.21.15.132:6829', 'nonce': 1165391760}]}, 'public_addr': '172.21.15.132:6825/1165391760', 'cluster_addr': '172.21.15.132:6827/1165391760', 'heartbeat_back_addr': '172.21.15.132:6831/1165391760', 'heartbeat_front_addr': '172.21.15.132:6829/1165391760', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:21:45.572907+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:22:44.978436+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:22:46.322063+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:22:37.705421+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T05:15:24.108106+0000', 'dead_epoch': 341}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T05:21:47.923174+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:22:21.559380+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T05:21:44.329029+0000', 'dead_epoch': 292}], 'pg_upmap': [{'pgid': '3.1', 'osds': [4, 0, 5]}, {'pgid': '4.b', 'osds': [0, 4, 2]}], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.10', 'mappings': [{'from': 6, 'to': 3}]}, {'pgid': '6.b', 'mappings': [{'from': 1, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.80:0/3992539385': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/2770889532': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:6801/2466284312': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/490056571': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1986135300': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/3796611170': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1993138634': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1832498181': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/2466284312': '2024-03-27T04:55:21.711776+0000', '172.21.15.80:0/1650870847': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6801/1418683843': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:0/3147301987': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6801/2770889532': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/571905941': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1451012560': '2024-03-27T04:55:48.542948+0000', '172.21.15.80:0/1235125152': '2024-03-27T05:01:03.689739+0000', '172.21.15.80:6800/1418683843': '2024-03-27T05:01:03.689739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T05:22:51.864 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.2 2024-03-26T05:22:51.864 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd rm-pg-upmap-items 4.2 2024-03-26T05:22:52.096 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T05:22:52.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:51 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1437370682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:22:52.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:22:52.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:52.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:51 smithi080 ceph-mon[40404]: osdmap e495: 8 total, 7 up, 6 in 2024-03-26T05:22:52.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:51 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1437370682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:22:52.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:22:52.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:52.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:51 smithi080 ceph-mon[29691]: osdmap e495: 8 total, 7 up, 6 in 2024-03-26T05:22:52.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:51 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1437370682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T05:22:52.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:22:52.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:52.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:51 smithi132 ceph-mon[33500]: osdmap e495: 8 total, 7 up, 6 in 2024-03-26T05:22:53.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:52 smithi080 ceph-mon[29691]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T05:22:53.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:52 smithi080 ceph-mon[40404]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T05:22:53.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:52 smithi132 ceph-mon[33500]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T05:22:53.410 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:53.447 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:54.110 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:22:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:22:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:22:54.336 INFO:teuthology.orchestra.run.smithi080.stdout:ERROR: (22) Invalid argument 2024-03-26T05:22:54.337 INFO:teuthology.orchestra.run.smithi080.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T05:22:55.299 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T05:22:55.553 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:55 smithi080 ceph-mon[40404]: pgmap v1065: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 545 B/s rd, 0 op/s 2024-03-26T05:22:55.553 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:55 smithi080 ceph-mon[29691]: pgmap v1065: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 545 B/s rd, 0 op/s 2024-03-26T05:22:55.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:55 smithi132 ceph-mon[33500]: pgmap v1065: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 545 B/s rd, 0 op/s 2024-03-26T05:22:56.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:22:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:22:56.510 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:56 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:22:56.511 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:56 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:22:56.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:56 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:22:56.902 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:57.961 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0: {} 2024-03-26T05:22:57.980 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1: {} 2024-03-26T05:22:57.999 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2: {} 2024-03-26T05:22:58.018 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:57 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:58.018 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:57 smithi080 ceph-mon[40404]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:22:58.018 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:57 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:22:58.018 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:57 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:22:58.018 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:57 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:22:58.018 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:57 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:58.019 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:57 smithi080 ceph-mon[29691]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:22:58.019 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:57 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:22:58.019 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:57 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:22:58.019 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:57 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:22:58.019 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3: {} 2024-03-26T05:22:58.041 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5: {} 2024-03-26T05:22:58.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:57 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:58.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:57 smithi132 ceph-mon[33500]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:22:58.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:57 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:22:58.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:57 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:22:58.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:57 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:22:58.061 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6: {} 2024-03-26T05:22:58.082 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7: {} 2024-03-26T05:22:58.082 INFO:teuthology.orchestra.run.smithi080.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T05:22:58.082 INFO:teuthology.orchestra.run.smithi080.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T05:22:58.082 INFO:teuthology.orchestra.run.smithi080.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T05:22:58.082 INFO:teuthology.orchestra.run.smithi080.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T05:22:58.082 INFO:teuthology.orchestra.run.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T05:22:58.082 INFO:teuthology.orchestra.run.smithi080.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T05:22:58.082 INFO:teuthology.orchestra.run.smithi080.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T05:22:58.082 INFO:teuthology.orchestra.run.smithi080.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T05:22:58.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/353022171' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:22:58.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/353022171' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:22:59.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/353022171' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T05:22:59.781 INFO:teuthology.orchestra.run.smithi080.stderr:clear 4.2 pg_upmap_items mapping 2024-03-26T05:22:59.782 INFO:teuthology.orchestra.run.smithi080.stderr:noscrub is unset 2024-03-26T05:22:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T05:22:59.797 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T05:23:00.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:59 smithi132 ceph-mon[33500]: pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:23:00.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:59 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1999632451' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]: dispatch 2024-03-26T05:23:00.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:59 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3279891237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:23:00.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:22:59 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:23:00.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:59 smithi080 ceph-mon[29691]: pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:23:00.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:59 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1999632451' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]: dispatch 2024-03-26T05:23:00.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:59 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3279891237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:23:00.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:22:59 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:23:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:59 smithi080 ceph-mon[40404]: pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:23:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:59 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1999632451' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]: dispatch 2024-03-26T05:23:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:59 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3279891237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:23:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:22:59 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T05:23:00.553 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T05:23:00.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:00 smithi080 ceph-mon[29691]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:23:00.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:00 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1999632451' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]': finished 2024-03-26T05:23:00.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:00 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:23:00.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:00 smithi080 ceph-mon[29691]: osdmap e496: 8 total, 7 up, 6 in 2024-03-26T05:23:00.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:00 smithi080 ceph-mon[40404]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:23:00.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:00 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1999632451' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]': finished 2024-03-26T05:23:00.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:00 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:23:00.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:00 smithi080 ceph-mon[40404]: osdmap e496: 8 total, 7 up, 6 in 2024-03-26T05:23:01.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:00 smithi132 ceph-mon[33500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T05:23:01.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:00 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1999632451' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]': finished 2024-03-26T05:23:01.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:00 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T05:23:01.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:00 smithi132 ceph-mon[33500]: osdmap e496: 8 total, 7 up, 6 in 2024-03-26T05:23:01.433 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:01.475 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:23:01.476 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:23:01.476 INFO:teuthology.orchestra.run.smithi080.stdout: "num_ops": 0 2024-03-26T05:23:01.476 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:23:02.015 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:01 smithi080 ceph-mon[40404]: pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 601 B/s rd, 0 op/s 2024-03-26T05:23:02.016 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:01 smithi080 ceph-mon[40404]: 4.1 scrub starts 2024-03-26T05:23:02.016 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:01 smithi080 ceph-mon[40404]: 4.1 scrub ok 2024-03-26T05:23:02.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:01 smithi080 ceph-mon[29691]: pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 601 B/s rd, 0 op/s 2024-03-26T05:23:02.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:01 smithi080 ceph-mon[29691]: 4.1 scrub starts 2024-03-26T05:23:02.016 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:01 smithi080 ceph-mon[29691]: 4.1 scrub ok 2024-03-26T05:23:02.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:01 smithi132 ceph-mon[33500]: pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 601 B/s rd, 0 op/s 2024-03-26T05:23:02.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:01 smithi132 ceph-mon[33500]: 4.1 scrub starts 2024-03-26T05:23:02.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:01 smithi132 ceph-mon[33500]: 4.1 scrub ok 2024-03-26T05:23:02.686 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T05:23:03.164 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:04.022 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:03 smithi080 ceph-mon[29691]: pgmap v1070: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:23:04.022 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:03 smithi080 ceph-mon[29691]: 4.5 scrub starts 2024-03-26T05:23:04.022 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:03 smithi080 ceph-mon[29691]: 4.5 scrub ok 2024-03-26T05:23:04.022 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:03 smithi080 ceph-mon[40404]: pgmap v1070: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:23:04.023 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:03 smithi080 ceph-mon[40404]: 4.5 scrub starts 2024-03-26T05:23:04.023 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:03 smithi080 ceph-mon[40404]: 4.5 scrub ok 2024-03-26T05:23:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:03 smithi132 ceph-mon[33500]: pgmap v1070: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:23:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:03 smithi132 ceph-mon[33500]: 4.5 scrub starts 2024-03-26T05:23:04.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:03 smithi132 ceph-mon[33500]: 4.5 scrub ok 2024-03-26T05:23:04.352 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:23:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:23:04.802 INFO:teuthology.orchestra.run.smithi080.stderr:nodeep-scrub is unset 2024-03-26T05:23:05.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-mon[40404]: 4.1b scrub starts 2024-03-26T05:23:05.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-mon[40404]: 4.1b scrub ok 2024-03-26T05:23:05.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:23:05.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:23:05.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3843296454' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:23:05.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:23:05.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-mon[29691]: 4.1b scrub starts 2024-03-26T05:23:05.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-mon[29691]: 4.1b scrub ok 2024-03-26T05:23:05.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:23:05.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:23:05.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3843296454' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:23:05.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:04 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:23:05.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:04 smithi132 ceph-mon[33500]: 4.1b scrub starts 2024-03-26T05:23:05.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:04 smithi132 ceph-mon[33500]: 4.1b scrub ok 2024-03-26T05:23:05.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:04 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:23:05.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:04 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:23:05.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:04 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3843296454' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:23:05.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:04 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T05:23:05.895 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:23:05.895 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [], 2024-03-26T05:23:05.895 INFO:teuthology.orchestra.run.smithi080.stdout: "complaint_time": 30, 2024-03-26T05:23:05.895 INFO:teuthology.orchestra.run.smithi080.stdout: "num_blocked_ops": 0 2024-03-26T05:23:05.895 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:23:05.950 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-26T05:23:05.950 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-03-26T05:23:05.950 INFO:tasks.cephadm.osd.4:Restarting osd.4 (starting--it wasn't running)... 2024-03-26T05:23:05.950 DEBUG:teuthology.orchestra.run.smithi132:osd.4> sudo journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T05:23:05.953 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl start ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.4 2024-03-26T05:23:06.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:06 smithi132 systemd[1]: Starting Ceph osd.4 for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:23:06.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:05 smithi132 ceph-mon[33500]: pgmap v1071: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:23:06.051 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:05 smithi132 ceph-mon[33500]: 6.12 scrub starts 2024-03-26T05:23:06.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:05 smithi132 ceph-mon[33500]: 6.12 scrub ok 2024-03-26T05:23:06.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:05 smithi132 ceph-mon[33500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:23:06.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:05 smithi132 ceph-mon[33500]: Cluster is now healthy 2024-03-26T05:23:06.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:05 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:23:06.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:05 smithi132 ceph-mon[33500]: osdmap e497: 8 total, 7 up, 6 in 2024-03-26T05:23:06.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:05 smithi132 ceph-mon[33500]: 2.1 scrub starts 2024-03-26T05:23:06.052 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:05 smithi132 ceph-mon[33500]: 2.1 scrub ok 2024-03-26T05:23:06.052 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[40404]: pgmap v1071: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:23:06.052 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[40404]: 6.12 scrub starts 2024-03-26T05:23:06.053 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[40404]: 6.12 scrub ok 2024-03-26T05:23:06.053 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[40404]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:23:06.053 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[40404]: Cluster is now healthy 2024-03-26T05:23:06.053 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:23:06.053 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[40404]: osdmap e497: 8 total, 7 up, 6 in 2024-03-26T05:23:06.053 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[40404]: 2.1 scrub starts 2024-03-26T05:23:06.053 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[40404]: 2.1 scrub ok 2024-03-26T05:23:06.053 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[29691]: pgmap v1071: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:23:06.053 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[29691]: 6.12 scrub starts 2024-03-26T05:23:06.053 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[29691]: 6.12 scrub ok 2024-03-26T05:23:06.053 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[29691]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T05:23:06.054 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[29691]: Cluster is now healthy 2024-03-26T05:23:06.054 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T05:23:06.054 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[29691]: osdmap e497: 8 total, 7 up, 6 in 2024-03-26T05:23:06.054 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[29691]: 2.1 scrub starts 2024-03-26T05:23:06.054 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:05 smithi080 ceph-mon[29691]: 2.1 scrub ok 2024-03-26T05:23:06.307 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:23:06.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:06 smithi132 podman[154161]: 2024-03-26 05:23:06.196176132 +0000 UTC m=+0.018326666 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:23:06.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:06 smithi132 podman[154161]: 2024-03-26 05:23:06.317809834 +0000 UTC m=+0.139960395 container create ecbbc6225c9a10ee41e479de6b55413195b82ec7458baf5550214bd5b1e3a1e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T05:23:06.365 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:23:06.061+0000 7f2fcc605700 -1 mgr get_metadata_python Requested missing service osd.4 2024-03-26T05:23:06.795 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T05:23:06.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[40404]: 5.1a scrub starts 2024-03-26T05:23:06.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[40404]: 5.1a scrub ok 2024-03-26T05:23:06.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[40404]: 4.1d scrub starts 2024-03-26T05:23:06.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[40404]: 4.1d scrub ok 2024-03-26T05:23:06.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [1, 2]}]: dispatch 2024-03-26T05:23:06.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [6, 2]}]: dispatch 2024-03-26T05:23:06.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:23:06.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:06.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:23:06.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:23:06.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[40404]: 4.1e scrub starts 2024-03-26T05:23:06.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[40404]: 4.1e scrub ok 2024-03-26T05:23:06.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[29691]: 5.1a scrub starts 2024-03-26T05:23:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[29691]: 5.1a scrub ok 2024-03-26T05:23:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[29691]: 4.1d scrub starts 2024-03-26T05:23:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[29691]: 4.1d scrub ok 2024-03-26T05:23:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [1, 2]}]: dispatch 2024-03-26T05:23:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [6, 2]}]: dispatch 2024-03-26T05:23:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:23:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:23:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:23:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[29691]: 4.1e scrub starts 2024-03-26T05:23:06.867 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:06 smithi080 ceph-mon[29691]: 4.1e scrub ok 2024-03-26T05:23:06.889 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-mon[33500]: 5.1a scrub starts 2024-03-26T05:23:06.889 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-mon[33500]: 5.1a scrub ok 2024-03-26T05:23:06.889 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-mon[33500]: 4.1d scrub starts 2024-03-26T05:23:06.889 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-mon[33500]: 4.1d scrub ok 2024-03-26T05:23:06.889 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [1, 2]}]: dispatch 2024-03-26T05:23:06.889 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [6, 2]}]: dispatch 2024-03-26T05:23:06.889 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:23:06.889 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:06.889 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:23:06.890 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:23:06.890 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-mon[33500]: 4.1e scrub starts 2024-03-26T05:23:06.890 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:06 smithi132 ceph-mon[33500]: 4.1e scrub ok 2024-03-26T05:23:06.890 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:06 smithi132 podman[154161]: 2024-03-26 05:23:06.595362631 +0000 UTC m=+0.417513147 container init ecbbc6225c9a10ee41e479de6b55413195b82ec7458baf5550214bd5b1e3a1e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-26T05:23:06.890 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:06 smithi132 podman[154161]: 2024-03-26 05:23:06.598782962 +0000 UTC m=+0.420933487 container start ecbbc6225c9a10ee41e479de6b55413195b82ec7458baf5550214bd5b1e3a1e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T05:23:06.890 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:06 smithi132 podman[154161]: 2024-03-26 05:23:06.695458626 +0000 UTC m=+0.517609151 container attach ecbbc6225c9a10ee41e479de6b55413195b82ec7458baf5550214bd5b1e3a1e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-26T05:23:07.043 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:07.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate[154176]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T05:23:07.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate[154176]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T05:23:07.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 bash[154161]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T05:23:07.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 bash[154161]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T05:23:07.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 bash[154161]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T05:23:07.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 bash[154161]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T05:23:07.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 bash[154161]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-26T05:23:07.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 bash[154161]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T05:23:07.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 bash[154161]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-26T05:23:07.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate[154176]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T05:23:07.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate[154176]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T05:23:07.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate[154176]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-26T05:23:07.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate[154176]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T05:23:07.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate[154176]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-26T05:23:07.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 podman[154161]: 2024-03-26 05:23:07.10590586 +0000 UTC m=+0.928056384 container died ecbbc6225c9a10ee41e479de6b55413195b82ec7458baf5550214bd5b1e3a1e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.license=GPLv2) 2024-03-26T05:23:08.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:08.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[29691]: pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:08.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[29691]: 4.18 scrub starts 2024-03-26T05:23:08.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[29691]: 4.18 scrub ok 2024-03-26T05:23:08.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [1, 2]}]': finished 2024-03-26T05:23:08.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [6, 2]}]': finished 2024-03-26T05:23:08.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[29691]: osdmap e498: 8 total, 7 up, 6 in 2024-03-26T05:23:08.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:23:08.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:23:08.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[29691]: 4.7 deep-scrub starts 2024-03-26T05:23:08.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[29691]: 4.7 deep-scrub ok 2024-03-26T05:23:08.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:08.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[40404]: pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:08.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[40404]: 4.18 scrub starts 2024-03-26T05:23:08.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[40404]: 4.18 scrub ok 2024-03-26T05:23:08.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [1, 2]}]': finished 2024-03-26T05:23:08.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [6, 2]}]': finished 2024-03-26T05:23:08.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[40404]: osdmap e498: 8 total, 7 up, 6 in 2024-03-26T05:23:08.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:23:08.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:23:08.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[40404]: 4.7 deep-scrub starts 2024-03-26T05:23:08.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:07 smithi080 ceph-mon[40404]: 4.7 deep-scrub ok 2024-03-26T05:23:08.078 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:08.078 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-mon[33500]: pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:08.078 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-mon[33500]: 4.18 scrub starts 2024-03-26T05:23:08.078 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-mon[33500]: 4.18 scrub ok 2024-03-26T05:23:08.078 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [1, 2]}]': finished 2024-03-26T05:23:08.078 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [6, 2]}]': finished 2024-03-26T05:23:08.078 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-mon[33500]: osdmap e498: 8 total, 7 up, 6 in 2024-03-26T05:23:08.078 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:23:08.078 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:23:08.078 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-mon[33500]: 4.7 deep-scrub starts 2024-03-26T05:23:08.079 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:07 smithi132 ceph-mon[33500]: 4.7 deep-scrub ok 2024-03-26T05:23:08.079 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:07 smithi132 podman[154309]: 2024-03-26 05:23:07.850041382 +0000 UTC m=+0.737993026 container remove ecbbc6225c9a10ee41e479de6b55413195b82ec7458baf5550214bd5b1e3a1e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-26T05:23:08.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:08 smithi132 podman[154380]: 2024-03-26 05:23:07.992930886 +0000 UTC m=+0.042268602 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:23:08.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:08 smithi132 podman[154380]: 2024-03-26 05:23:08.117679395 +0000 UTC m=+0.167017097 container create c3d49c2f4beb095c3be1ee80514079d073f7a68ae16b60dab4ca2bb9079769a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124) 2024-03-26T05:23:08.458 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T05:23:08.580 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-26T05:23:08.580 INFO:teuthology.orchestra.run.smithi080.stdout: "size": 20, 2024-03-26T05:23:08.580 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 600, 2024-03-26T05:23:08.580 INFO:teuthology.orchestra.run.smithi080.stdout: "ops": [ 2024-03-26T05:23:08.580 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.580 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:205421 10.15 10:ac3cc547:::benchmark_data_smithi080_161990_object93698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-03-26T05:23:08.581 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:53.712849+0000", 2024-03-26T05:23:08.581 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 314.87557788100003, 2024-03-26T05:23:08.581 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.99141917899999998, 2024-03-26T05:23:08.581 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.581 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.581 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.581 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:23:08.581 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:08.581 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 205421 2024-03-26T05:23:08.581 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.581 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.581 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.582 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.582 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712849+0000", 2024-03-26T05:23:08.582 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.582 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.582 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.582 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.582 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712849+0000", 2024-03-26T05:23:08.582 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.582 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.582 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.582 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.582 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712851+0000", 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9410000000000001e-06 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712856+0000", 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5310000000000003e-06 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712868+0000", 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1763000000000001e-05 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.583 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712875+0000", 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.6839999999999996e-06 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712938+0000", 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.3108999999999995e-05 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.712955+0000", 2024-03-26T05:23:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6254000000000001e-05 2024-03-26T05:23:08.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.585 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.585 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.585 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.699335+0000", 2024-03-26T05:23:08.586 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.98638049900000002 2024-03-26T05:23:08.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.586 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.586 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.586 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.699435+0000", 2024-03-26T05:23:08.586 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9513999999999996e-05 2024-03-26T05:23:08.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.586 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.586 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.699750+0000", 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00031550600000000001 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704128+0000", 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0043781779999999999 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704137+0000", 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8300000000000002e-06 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.587 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704171+0000", 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3618000000000002e-05 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704236+0000", 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.5011999999999994e-05 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704269+0000", 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2740000000000002e-05 2024-03-26T05:23:08.588 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.589 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.589 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.589 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.589 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.589 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:205443 10.15 10:a8127e64:::benchmark_data_smithi080_161990_object93720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-03-26T05:23:08.589 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:53.730200+0000", 2024-03-26T05:23:08.589 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 314.85822722699999, 2024-03-26T05:23:08.589 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97445138600000003, 2024-03-26T05:23:08.589 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.589 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.589 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.589 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:23:08.590 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:08.590 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 205443 2024-03-26T05:23:08.590 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.590 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.590 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.590 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.590 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730200+0000", 2024-03-26T05:23:08.590 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.590 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.590 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.590 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.590 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730200+0000", 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730202+0000", 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6840000000000001e-06 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730208+0000", 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.2029999999999997e-06 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.591 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730224+0000", 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6398000000000001e-05 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730233+0000", 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5290000000000008e-06 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730292+0000", 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9163000000000003e-05 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.592 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730301+0000", 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.149e-06 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.701474+0000", 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97117258299999998 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.701589+0000", 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115262 2024-03-26T05:23:08.593 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.701769+0000", 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000180373 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704432+0000", 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.002662897 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.594 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704480+0000", 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.7416999999999997e-05 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704558+0000", 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7880999999999995e-05 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704608+0000", 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0430000000000003e-05 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.595 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.598 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.598 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704651+0000", 2024-03-26T05:23:08.598 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3417000000000001e-05 2024-03-26T05:23:08.598 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.598 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.598 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.598 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.598 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.598 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:205446 10.15 10:aad44559:::benchmark_data_smithi080_161990_object93723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:53.730986+0000", 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 314.85744142599998, 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97380755900000004, 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 205446 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.599 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.600 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730986+0000", 2024-03-26T05:23:08.600 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.600 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.600 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.600 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.600 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730986+0000", 2024-03-26T05:23:08.600 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.600 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.600 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.600 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.600 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730987+0000", 2024-03-26T05:23:08.600 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.71e-07 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.730990+0000", 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3179999999999998e-06 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.731001+0000", 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1192999999999999e-05 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.731009+0000", 2024-03-26T05:23:08.601 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7000000000000008e-06 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.731197+0000", 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018755200000000001 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.731206+0000", 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0799999999999995e-06 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.602 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.702711+0000", 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.97150552199999995 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.702790+0000", 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9139000000000002e-05 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.703033+0000", 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00024264999999999999 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.603 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704542+0000", 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.001509355 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704661+0000", 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000118347 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704718+0000", 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7617000000000001e-05 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.604 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704771+0000", 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2320000000000001e-05 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704793+0000", 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.2895e-05 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.605 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15657.0:205463 10.15 10:ab1253e4:::benchmark_data_smithi080_161990_object93740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:17:53.740050+0000", 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 314.84837776299997, 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.974296261, 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15657", 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 205463 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.606 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740050+0000", 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740050+0000", 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740051+0000", 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.482e-06 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.607 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740056+0000", 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.2429999999999996e-06 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740070+0000", 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3883000000000001e-05 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740079+0000", 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.8899999999999996e-06 2024-03-26T05:23:08.608 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.609 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.609 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.609 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740141+0000", 2024-03-26T05:23:08.609 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.1835999999999995e-05 2024-03-26T05:23:08.609 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.609 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.609 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.609 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:53.740151+0000", 2024-03-26T05:23:08.609 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9629999999999993e-06 2024-03-26T05:23:08.609 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.609 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.609 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704843+0000", 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.96469201199999999 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.704956+0000", 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011328 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.705176+0000", 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00021946499999999999 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.610 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.708658+0000", 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0034822059999999998 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.712329+0000", 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0036716689999999998 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.714249+0000", 2024-03-26T05:23:08.611 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0019193299999999999 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.714300+0000", 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.1372999999999998e-05 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:17:54.714346+0000", 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5629e-05 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.612 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:31412 11.a 11:524e5acf:::benchmark_data_smithi080_189724_object31411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:05.721830+0000", 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 242.86659758799999, 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.20363530299999999, 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31412 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.613 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.614 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721830+0000", 2024-03-26T05:23:08.614 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.614 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.614 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.614 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.614 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721830+0000", 2024-03-26T05:23:08.614 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.614 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.614 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.614 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.614 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721830+0000", 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6699999999999999e-07 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721865+0000", 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.4601999999999998e-05 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721872+0000", 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.5340000000000002e-06 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.615 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.721878+0000", 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7219999999999996e-06 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.728760+0000", 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0068817519999999997 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.729030+0000", 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00027042499999999997 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.616 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.921481+0000", 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.19245103599999999 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923500+0000", 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0020189380000000001 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.925083+0000", 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0015828629999999999 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.617 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.925267+0000", 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000183766 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.925415+0000", 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014801899999999999 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.925465+0000", 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0179000000000002e-05 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.618 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:31423 11.a 11:5a11d067:::benchmark_data_smithi080_189724_object31422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:19:05.725877+0000", 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 242.86255022200001, 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.20839273799999999, 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.619 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 31423 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725877+0000", 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725877+0000", 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.620 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725878+0000", 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1689999999999999e-06 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725898+0000", 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9647e-05 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725916+0000", 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8454e-05 2024-03-26T05:23:08.621 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.725924+0000", 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.8760000000000005e-06 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923529+0000", 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.197604479 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.622 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923660+0000", 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013092900000000001 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 5,6", 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.923799+0000", 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000139159 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.925515+0000", 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0017163219999999999 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.623 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.928641+0000", 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0031254880000000001 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.934157+0000", 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055162539999999999 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.934221+0000", 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4067999999999997e-05 2024-03-26T05:23:08.624 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:19:05.934270+0000", 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.8893000000000001e-05 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:129401 11.8 11:1cffb8d6:::benchmark_data_smithi080_189724_object8291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:29.376886+0000", 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 159.21154115900001, 2024-03-26T05:23:08.625 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0428869820000002, 2024-03-26T05:23:08.626 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.626 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.626 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.626 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.626 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.626 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 129401 2024-03-26T05:23:08.626 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.626 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.626 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.626 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.626 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.376886+0000", 2024-03-26T05:23:08.626 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.376886+0000", 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.376888+0000", 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.449e-06 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.627 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.376893+0000", 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4260000000000002e-06 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.376907+0000", 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.4116e-05 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.376916+0000", 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.2620000000000004e-06 2024-03-26T05:23:08.628 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.376986+0000", 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9950000000000006e-05 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.376996+0000", 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.9860000000000006e-06 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.326988+0000", 2024-03-26T05:23:08.629 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.94999215000000004 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.327002+0000", 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.38e-05 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.402719+0000", 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.07571712 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.630 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.402853+0000", 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00013327 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,5,6", 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.403254+0000", 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00040143900000000002 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.411001+0000", 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0077473259999999997 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.631 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.413912+0000", 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0029104629999999999 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.413930+0000", 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8369999999999999e-05 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.419614+0000", 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0056833020000000003 2024-03-26T05:23:08.632 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.419672+0000", 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.8816000000000001e-05 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.419773+0000", 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010073699999999999 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.633 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.634 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:129405 11.8 11:15f21296:::benchmark_data_smithi080_189724_object8295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-26T05:23:08.634 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:29.382959+0000", 2024-03-26T05:23:08.634 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 159.20546879, 2024-03-26T05:23:08.634 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.044621147, 2024-03-26T05:23:08.634 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.634 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.634 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.634 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.634 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.634 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 129405 2024-03-26T05:23:08.634 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.634 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.635 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.635 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.635 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.382959+0000", 2024-03-26T05:23:08.635 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.635 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.635 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.635 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.635 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.382959+0000", 2024-03-26T05:23:08.635 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.635 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.635 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.635 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.382960+0000", 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6700000000000001e-06 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.382966+0000", 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4689999999999996e-06 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.382982+0000", 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6365999999999999e-05 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.636 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.382992+0000", 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.6909999999999992e-06 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.383174+0000", 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018221599999999999 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.383185+0000", 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1172000000000001e-05 2024-03-26T05:23:08.637 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.638 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.638 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.638 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.327032+0000", 2024-03-26T05:23:08.638 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.94384654099999998 2024-03-26T05:23:08.638 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.638 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.638 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.638 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.327041+0000", 2024-03-26T05:23:08.638 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.0389999999999993e-06 2024-03-26T05:23:08.638 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.638 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.638 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.411034+0000", 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0839935599999999 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.411156+0000", 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000121909 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,5,6", 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.411317+0000", 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00016040299999999999 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.413936+0000", 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0026193470000000002 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.413975+0000", 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.9332999999999998e-05 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.426419+0000", 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.012443842 2024-03-26T05:23:08.640 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.427257+0000", 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00083824099999999999 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.427326+0000", 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9035000000000006e-05 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.427580+0000", 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025331300000000002 2024-03-26T05:23:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:129414 11.8 11:1f382fe3:::benchmark_data_smithi080_189724_object8304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:29.392120+0000", 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 159.19630721799999, 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.035912787, 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.642 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 129414 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.392120+0000", 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.392120+0000", 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.643 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.392121+0000", 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1799999999999999e-06 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.392126+0000", 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5299999999999998e-06 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.392137+0000", 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0891e-05 2024-03-26T05:23:08.644 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.392145+0000", 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0490000000000004e-06 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.392217+0000", 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.2716e-05 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.392228+0000", 2024-03-26T05:23:08.645 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0382e-05 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.327067+0000", 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.93483872800000001 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.327075+0000", 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.3100000000000001e-06 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.646 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.414006+0000", 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0869311740000001 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.414115+0000", 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000109385 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,5,6", 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.414234+0000", 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011851 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.647 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.419814+0000", 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0055797859999999998 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.426372+0000", 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0065584689999999999 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.426452+0000", 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9647999999999994e-05 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.648 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.427695+0000", 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0012435040000000001 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.427781+0000", 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.5740000000000002e-05 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.428033+0000", 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00025178499999999998 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.649 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.650 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:129424 11.8 11:18b38af6:::benchmark_data_smithi080_189724_object8314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-26T05:23:08.650 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:29.406463+0000", 2024-03-26T05:23:08.650 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 159.18196386899999, 2024-03-26T05:23:08.650 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0217896610000001, 2024-03-26T05:23:08.650 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.650 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.650 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.650 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.650 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.650 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 129424 2024-03-26T05:23:08.650 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.650 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.406463+0000", 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.406463+0000", 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.406465+0000", 2024-03-26T05:23:08.651 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6160000000000001e-06 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.406471+0000", 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.4509999999999997e-06 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.406490+0000", 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8787000000000002e-05 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.406500+0000", 2024-03-26T05:23:08.652 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.944e-06 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.406570+0000", 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9438999999999997e-05 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.406580+0000", 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0617e-05 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.327099+0000", 2024-03-26T05:23:08.653 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.92051909200000004 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.327108+0000", 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.1610000000000003e-06 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.416627+0000", 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0895190729999999 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.654 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.416810+0000", 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00018375499999999999 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,5,6", 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.416960+0000", 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014969399999999999 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.419861+0000", 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0029011509999999998 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.655 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.426380+0000", 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0065190489999999999 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.426473+0000", 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.3131e-05 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.428088+0000", 2024-03-26T05:23:08.656 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0016146299999999999 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.428143+0000", 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4892999999999998e-05 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.428253+0000", 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000110178 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.657 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:129443 11.8 11:14a57fe3:::benchmark_data_smithi080_189724_object8333:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:20:29.440811+0000", 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 159.14761623000001, 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.9876762050000001, 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 129443 2024-03-26T05:23:08.658 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.440811+0000", 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.440811+0000", 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.440813+0000", 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.486e-06 2024-03-26T05:23:08.659 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.440818+0000", 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.6790000000000002e-06 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.440837+0000", 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8678e-05 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.440848+0000", 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1399999999999999e-05 2024-03-26T05:23:08.660 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.440920+0000", 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.1632999999999997e-05 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:29.440929+0000", 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1300000000000007e-06 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.327134+0000", 2024-03-26T05:23:08.661 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.88620511800000001 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for peered", 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:30.327143+0000", 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.4319999999999999e-06 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.419915+0000", 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0927718829999999 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.420024+0000", 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000109438 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,5,6", 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.420143+0000", 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000119321 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.425487+0000", 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0053440110000000001 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.663 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.426432+0000", 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00094501000000000003 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.428293+0000", 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0018602390000000001 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.428333+0000", 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0114000000000003e-05 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.664 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.428383+0000", 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0643999999999998e-05 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:20:31.428487+0000", 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103989 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.665 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:238570 11.b 11:de6f73e6:::benchmark_data_smithi080_189724_object117460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:22:29.756174+0000", 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 38.832253170000001, 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.13849302999999999, 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 238570 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.666 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.667 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.756174+0000", 2024-03-26T05:23:08.667 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.667 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.667 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.668 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.668 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.756174+0000", 2024-03-26T05:23:08.668 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.668 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.668 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.668 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.668 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.756174+0000", 2024-03-26T05:23:08.668 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.39e-07 2024-03-26T05:23:08.668 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.668 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.669 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.669 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.756177+0000", 2024-03-26T05:23:08.669 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.3640000000000002e-06 2024-03-26T05:23:08.669 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.669 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.669 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.669 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.756182+0000", 2024-03-26T05:23:08.669 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5860000000000004e-06 2024-03-26T05:23:08.669 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.669 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.669 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.669 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.756187+0000", 2024-03-26T05:23:08.670 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9590000000000003e-06 2024-03-26T05:23:08.670 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.670 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.670 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.670 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.758643+0000", 2024-03-26T05:23:08.670 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0024558710000000001 2024-03-26T05:23:08.670 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.670 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.670 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.670 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.758758+0000", 2024-03-26T05:23:08.670 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000115238 2024-03-26T05:23:08.670 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.759755+0000", 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00099611600000000002 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.763916+0000", 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0041611189999999996 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.778167+0000", 2024-03-26T05:23:08.671 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014251674000000001 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894572+0000", 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.116404206 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894626+0000", 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.4762000000000002e-05 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.672 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894667+0000", 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.0796e-05 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:238575 11.b 11:dabd47c7:::benchmark_data_smithi080_189724_object117465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:22:29.765369+0000", 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 38.823058601, 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.12936920700000001, 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.673 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.674 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.674 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 238575 2024-03-26T05:23:08.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.674 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.674 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.674 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.674 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.765369+0000", 2024-03-26T05:23:08.674 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.674 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.674 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.674 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.765369+0000", 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.765371+0000", 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0609999999999998e-06 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.765376+0000", 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5809999999999996e-06 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.675 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.765390+0000", 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3196999999999999e-05 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.765398+0000", 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.2640000000000008e-06 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.765665+0000", 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00026731500000000001 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.676 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.766289+0000", 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00062371900000000001 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.766588+0000", 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000299437 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.778205+0000", 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.011616474999999999 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.677 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.784502+0000", 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0062967930000000002 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894674+0000", 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.11017231199999999 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894708+0000", 2024-03-26T05:23:08.678 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3908000000000001e-05 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894738+0000", 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.0145e-05 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:238584 11.b 11:d21a1754:::benchmark_data_smithi080_189724_object117474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:22:29.775451+0000", 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 38.812976513999999, 2024-03-26T05:23:08.679 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.11935478300000001, 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 238584 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.775451+0000", 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.680 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.775451+0000", 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.775452+0000", 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.466e-06 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.775458+0000", 2024-03-26T05:23:08.681 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.716e-06 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.775470+0000", 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2456000000000001e-05 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.775478+0000", 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 8.0369999999999995e-06 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.682 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.778235+0000", 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0027561560000000001 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.778377+0000", 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014228799999999999 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.778490+0000", 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00011302 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.683 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.792675+0000", 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.014185207999999999 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.795914+0000", 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.003238903 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894744+0000", 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.098829512999999994 2024-03-26T05:23:08.684 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894776+0000", 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2703000000000002e-05 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894806+0000", 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.9317000000000001e-05 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.685 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:238593 11.b 11:d2f31e63:::benchmark_data_smithi080_189724_object117483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:22:29.789456+0000", 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 38.798971275, 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.105427464, 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 238593 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.686 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.789456+0000", 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.789456+0000", 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.789458+0000", 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.0090000000000002e-06 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.687 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.789464+0000", 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.7989999999999999e-06 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.789483+0000", 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.8853999999999999e-05 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.789496+0000", 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3396e-05 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.688 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.789588+0000", 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.1880999999999997e-05 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.789718+0000", 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000130479 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.789844+0000", 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00012543799999999999 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.689 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.796076+0000", 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0062321429999999999 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.796224+0000", 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000148398 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894811+0000", 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.098586437999999998 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.690 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894854+0000", 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.3127000000000002e-05 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894883+0000", 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.9502e-05 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:238607 11.b 11:d7ad6652:::benchmark_data_smithi080_189724_object117497:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:22:29.811301+0000", 2024-03-26T05:23:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 38.777126293999999, 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.083652437999999996, 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 238607 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811301+0000", 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.692 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811301+0000", 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811303+0000", 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.7600000000000001e-06 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811310+0000", 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 6.9949999999999999e-06 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.693 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811325+0000", 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.537e-05 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811336+0000", 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0498e-05 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811409+0000", 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.3008000000000003e-05 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.694 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811553+0000", 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00014460699999999999 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811690+0000", 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000136787 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.816957+0000", 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0052673429999999999 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.695 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.825680+0000", 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0087227409999999991 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894889+0000", 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.069208863999999995 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894923+0000", 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.3844000000000003e-05 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.696 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894953+0000", 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.0620999999999997e-05 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:238608 11.b 11:d29956a8:::benchmark_data_smithi080_189724_object117498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:22:29.811361+0000", 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 38.777066734000002, 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.083664747999999997, 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.697 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 238608 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811361+0000", 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811361+0000", 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811361+0000", 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.0399999999999996e-07 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.811364+0000", 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.2600000000000001e-06 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.816155+0000", 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0047903219999999996 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.699 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.816171+0000", 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.6776e-05 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.816989+0000", 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00081755200000000004 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.817101+0000", 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000111731 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.700 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.817206+0000", 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00010555 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.825716+0000", 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0085092629999999995 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.828094+0000", 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0023783329999999998 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.701 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894959+0000", 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.066864947999999993 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.894996+0000", 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.6850999999999997e-05 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.895025+0000", 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 2.9658e-05 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.702 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:238695 11.b 11:d35f1f67:::benchmark_data_smithi080_189724_object117585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:22:29.904400+0000", 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 38.684027706000002, 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.086444913999999998, 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 238695 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.703 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.904400+0000", 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.904400+0000", 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.904401+0000", 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.128e-06 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.704 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.904405+0000", 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.5410000000000002e-06 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.904416+0000", 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.0363999999999999e-05 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.904423+0000", 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7610000000000007e-06 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.905206+0000", 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00078299399999999996 2024-03-26T05:23:08.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.905326+0000", 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000120037 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.905439+0000", 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000112868 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.913246+0000", 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0078070830000000003 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.706 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.990697+0000", 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.077450964999999997 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.990739+0000", 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.1902e-05 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.990795+0000", 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.5380999999999997e-05 2024-03-26T05:23:08.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.990845+0000", 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9889999999999998e-05 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:238710 11.b 11:dda906dc:::benchmark_data_smithi080_189724_object117600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:22:29.918750+0000", 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 38.669677112999999, 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.092612423999999999, 2024-03-26T05:23:08.708 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 238710 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.918750+0000", 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.918750+0000", 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.918752+0000", 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.3209999999999999e-06 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.918756+0000", 2024-03-26T05:23:08.710 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.9180000000000002e-06 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.918768+0000", 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.1590000000000001e-05 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.918776+0000", 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7039999999999994e-06 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.991037+0000", 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.072260862999999995 2024-03-26T05:23:08.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.991144+0000", 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000107062 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.991247+0000", 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103304 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:30.002398+0000", 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.011150561 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:30.005613+0000", 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0032157850000000001 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:30.011222+0000", 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0056082299999999996 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:30.011316+0000", 2024-03-26T05:23:08.713 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 9.4123000000000007e-05 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:30.011363+0000", 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 4.6962999999999997e-05 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: "description": "osd_op(client.15930.0:238711 11.b 11:de0925e0:::benchmark_data_smithi080_189724_object117601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: "initiated_at": "2024-03-26T05:22:29.923736+0000", 2024-03-26T05:23:08.714 INFO:teuthology.orchestra.run.smithi080.stdout: "age": 38.664691105000003, 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.087831867999999994, 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: "type_data": { 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: "client_info": { 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: "client": "client.15930", 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: "tid": 238711 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: "events": [ 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "initiated", 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.923736+0000", 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.715 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "header_read", 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.923736+0000", 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "throttled", 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.923738+0000", 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.773e-06 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "all_read", 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.923744+0000", 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 5.9739999999999999e-06 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.716 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "dispatched", 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.923757+0000", 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 1.2974e-05 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "queued_for_pg", 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.923765+0000", 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.9060000000000002e-06 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "reached_pg", 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.993253+0000", 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.069487884 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.717 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "started", 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.993355+0000", 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000101989 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "waiting for subops from 0,1", 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:29.993458+0000", 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.000103724 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "op_commit", 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:30.002403+0000", 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0089446300000000003 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.718 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:30.011063+0000", 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.0086594659999999993 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:30.011452+0000", 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 0.00038957400000000001 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "commit_sent", 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:30.011529+0000", 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 7.7025000000000002e-05 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: "event": "done", 2024-03-26T05:23:08.719 INFO:teuthology.orchestra.run.smithi080.stdout: "time": "2024-03-26T05:22:30.011568+0000", 2024-03-26T05:23:08.720 INFO:teuthology.orchestra.run.smithi080.stdout: "duration": 3.8949000000000003e-05 2024-03-26T05:23:08.720 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.720 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.720 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.720 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-26T05:23:08.720 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-26T05:23:08.720 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-26T05:23:08.720 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:08.792 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:08 smithi132 podman[154380]: 2024-03-26 05:23:08.353757242 +0000 UTC m=+0.403094948 container init c3d49c2f4beb095c3be1ee80514079d073f7a68ae16b60dab4ca2bb9079769a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T05:23:08.792 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:08 smithi132 podman[154380]: 2024-03-26 05:23:08.35736848 +0000 UTC m=+0.406706192 container start c3d49c2f4beb095c3be1ee80514079d073f7a68ae16b60dab4ca2bb9079769a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-26T05:23:08.792 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:08 smithi132 ceph-osd[154399]: -- 172.21.15.132:0/1809949073 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d37412d1e0 con 0x55d3740ff000 2024-03-26T05:23:08.792 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:08 smithi132 bash[154380]: c3d49c2f4beb095c3be1ee80514079d073f7a68ae16b60dab4ca2bb9079769a9 2024-03-26T05:23:08.792 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:08 smithi132 systemd[1]: Started Ceph osd.4 for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:23:08.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[29691]: 6.5 deep-scrub starts 2024-03-26T05:23:08.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[29691]: 6.5 deep-scrub ok 2024-03-26T05:23:08.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[29691]: 6.1e scrub starts 2024-03-26T05:23:08.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[29691]: 6.1e scrub ok 2024-03-26T05:23:08.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[29691]: osdmap e499: 8 total, 7 up, 6 in 2024-03-26T05:23:08.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[29691]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:08.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[29691]: 5.6 scrub starts 2024-03-26T05:23:08.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[29691]: 5.6 scrub ok 2024-03-26T05:23:08.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[40404]: 6.5 deep-scrub starts 2024-03-26T05:23:08.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[40404]: 6.5 deep-scrub ok 2024-03-26T05:23:08.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[40404]: 6.1e scrub starts 2024-03-26T05:23:08.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[40404]: 6.1e scrub ok 2024-03-26T05:23:08.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[40404]: osdmap e499: 8 total, 7 up, 6 in 2024-03-26T05:23:08.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[40404]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:08.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[40404]: 5.6 scrub starts 2024-03-26T05:23:08.867 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:08 smithi080 ceph-mon[40404]: 5.6 scrub ok 2024-03-26T05:23:09.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:08 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:08.902+0000 7fd98196a7c0 -1 Falling back to public interface 2024-03-26T05:23:09.105 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:08 smithi132 ceph-mon[33500]: 6.5 deep-scrub starts 2024-03-26T05:23:09.106 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:08 smithi132 ceph-mon[33500]: 6.5 deep-scrub ok 2024-03-26T05:23:09.106 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:08 smithi132 ceph-mon[33500]: 6.1e scrub starts 2024-03-26T05:23:09.106 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:08 smithi132 ceph-mon[33500]: 6.1e scrub ok 2024-03-26T05:23:09.106 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:08 smithi132 ceph-mon[33500]: osdmap e499: 8 total, 7 up, 6 in 2024-03-26T05:23:09.106 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:08 smithi132 ceph-mon[33500]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:09.106 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:08 smithi132 ceph-mon[33500]: 5.6 scrub starts 2024-03-26T05:23:09.106 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:08 smithi132 ceph-mon[33500]: 5.6 scrub ok 2024-03-26T05:23:09.652 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T05:23:10.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:09 smithi132 ceph-mon[33500]: 5.f scrub starts 2024-03-26T05:23:10.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:09 smithi132 ceph-mon[33500]: 5.f scrub ok 2024-03-26T05:23:10.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:09 smithi132 ceph-mon[33500]: 6.7 deep-scrub starts 2024-03-26T05:23:10.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:09 smithi132 ceph-mon[33500]: 6.7 deep-scrub ok 2024-03-26T05:23:10.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:09 smithi080 ceph-mon[40404]: 5.f scrub starts 2024-03-26T05:23:10.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:09 smithi080 ceph-mon[40404]: 5.f scrub ok 2024-03-26T05:23:10.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:09 smithi080 ceph-mon[40404]: 6.7 deep-scrub starts 2024-03-26T05:23:10.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:09 smithi080 ceph-mon[40404]: 6.7 deep-scrub ok 2024-03-26T05:23:10.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:09 smithi080 ceph-mon[29691]: 5.f scrub starts 2024-03-26T05:23:10.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:09 smithi080 ceph-mon[29691]: 5.f scrub ok 2024-03-26T05:23:10.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:09 smithi080 ceph-mon[29691]: 6.7 deep-scrub starts 2024-03-26T05:23:10.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:09 smithi080 ceph-mon[29691]: 6.7 deep-scrub ok 2024-03-26T05:23:10.308 INFO:teuthology.orchestra.run.smithi132.stderr:no valid command found; 10 closest matches: 2024-03-26T05:23:10.308 INFO:teuthology.orchestra.run.smithi132.stderr:0 2024-03-26T05:23:10.309 INFO:teuthology.orchestra.run.smithi132.stderr:1 2024-03-26T05:23:10.309 INFO:teuthology.orchestra.run.smithi132.stderr:2 2024-03-26T05:23:10.309 INFO:teuthology.orchestra.run.smithi132.stderr:abort 2024-03-26T05:23:10.309 INFO:teuthology.orchestra.run.smithi132.stderr:assert 2024-03-26T05:23:10.309 INFO:teuthology.orchestra.run.smithi132.stderr:config diff 2024-03-26T05:23:10.309 INFO:teuthology.orchestra.run.smithi132.stderr:config diff get 2024-03-26T05:23:10.309 INFO:teuthology.orchestra.run.smithi132.stderr:config get 2024-03-26T05:23:10.309 INFO:teuthology.orchestra.run.smithi132.stderr:config help [] 2024-03-26T05:23:10.309 INFO:teuthology.orchestra.run.smithi132.stderr:config set ... 2024-03-26T05:23:10.309 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: invalid command 2024-03-26T05:23:10.944 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:10 smithi132 ceph-mon[33500]: pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T05:23:10.972 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-03-26T05:23:11.038 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:11.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:10 smithi080 ceph-mon[40404]: pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:11.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:10 smithi080 ceph-mon[29691]: pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:11 smithi132 ceph-mon[33500]: 3.e scrub starts 2024-03-26T05:23:12.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:11 smithi132 ceph-mon[33500]: 3.e scrub ok 2024-03-26T05:23:12.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:11 smithi080 ceph-mon[29691]: 3.e scrub starts 2024-03-26T05:23:12.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:11 smithi080 ceph-mon[29691]: 3.e scrub ok 2024-03-26T05:23:12.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:11 smithi080 ceph-mon[40404]: 3.e scrub starts 2024-03-26T05:23:12.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:11 smithi080 ceph-mon[40404]: 3.e scrub ok 2024-03-26T05:23:12.528 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.413+0000 7fd98196a7c0 -1 osd.4 341 log_to_monitors true 2024-03-26T05:23:12.789 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:12.790 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:23:12.790 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2024-03-26T05:23:12.790 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:12.890 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-mon[33500]: 4.a scrub starts 2024-03-26T05:23:12.890 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-mon[33500]: 4.a scrub ok 2024-03-26T05:23:12.890 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-mon[33500]: 5.15 scrub starts 2024-03-26T05:23:12.890 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-mon[33500]: 5.15 scrub ok 2024-03-26T05:23:12.890 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-mon[33500]: pgmap v1078: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:23:12.890 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-mon[33500]: from='osd.4 [v2:172.21.15.132:6800/1991006784,v1:172.21.15.132:6801/1991006784]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T05:23:12.890 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-mon[33500]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T05:23:12.891 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.859+0000 7fd96e755700 -1 osd.4 341 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:12.891 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.891 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.891 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.891 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.892 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.893 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.870+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.894 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.895 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.896 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.896 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.896 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.896 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.896 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.896 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.896 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.898 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.898 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.899 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.899 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.899 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.899 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.899 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.899 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.899 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.899 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.899 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.899 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.899 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.899 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.900 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.901 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.902 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.903 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:12.904 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:12.905 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.871+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:12.906 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:12.907 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.908 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.909 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.910 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:12.911 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:12.912 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.913 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.914 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:12.915 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.916 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.872+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.917 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.918 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:12.919 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:12.920 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:12.920 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:12.920 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:12.920 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:12.920 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:12.920 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:12.920 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:12.920 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:12.920 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:12.928 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:12.928 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:12.928 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:12.928 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:12.928 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:12.928 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:12.928 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:12.928 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:12.928 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:12.929 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:12.930 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:12.931 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:12.931 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:12.931 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:12.931 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:12.931 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:12.931 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.931 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.931 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.931 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.931 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.931 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.931 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.932 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.873+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.933 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:12.934 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:12.935 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:12.936 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:12.937 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.938 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.939 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.940 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.874+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:12.941 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:12.942 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:12.943 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:12.944 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.945 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.946 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.947 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.947 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.947 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.947 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.947 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.947 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.947 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.947 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.947 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:12.948 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:12.948 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:12.948 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:12.948 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:12.948 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:12.948 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:12.948 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:12.948 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:12.948 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:12.948 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:12.948 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:12.948 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:12.949 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:12.950 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.875+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:12.951 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.952 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.952 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.952 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.952 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.952 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.952 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.952 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.952 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.953 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.954 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.955 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.956 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:12.957 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:12.958 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:12.959 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.876+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:12.960 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.961 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.962 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.963 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.964 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.965 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.966 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:12.967 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.877+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:12.968 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:12.969 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:12.970 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.971 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.972 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.973 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:12.974 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:12.975 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:12.975 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:12.975 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:12.975 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:12.975 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:12.975 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:12.975 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:12.975 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:12.975 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:12.975 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:12.975 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:12.976 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:12.977 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.978 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.878+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.979 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:12.980 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:12.981 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:12.982 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:12.983 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:12.984 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.985 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.986 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.987 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.879+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.988 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.989 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.990 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:12.991 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:12.992 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:12.993 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:12.994 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:12.995 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:12.996 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.880+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:12.997 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:12.998 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:12.999 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.000 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.001 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.002 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.003 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.004 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.005 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.881+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.006 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.007 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.008 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.009 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.010 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.011 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.012 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.013 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.014 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.014 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.026 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.026 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.027 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.028 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.882+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.029 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.030 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.031 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.032 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.033 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.034 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.035 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.036 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.037 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.038 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.883+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.039 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.040 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.041 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.884+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.043 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.044 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.045 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.046 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.047 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.048 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.049 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.050 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.051 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.052 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.053 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.054 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.885+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.055 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.057 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.058 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.059 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.060 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.886+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.061 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.062 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.063 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.064 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.065 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.066 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.067 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.068 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.892+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.069 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.070 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.070 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.070 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.070 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.070 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.070 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.070 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.070 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.070 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.070 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.070 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.070 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.071 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.072 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.073 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.074 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.075 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.076 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.077 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.078 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.079 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.079 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.081 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.893+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.082 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.894+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.083 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.084 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.085 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.086 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.087 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.088 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.089 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.090 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.903+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.091 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.092 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.093 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.094 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.095 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.096 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.097 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.100 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.904+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.101 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.102 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.905+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.103 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.104 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.105 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.910+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.106 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.107 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.108 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.109 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.110 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.111 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.112 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.113 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.114 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.911+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.115 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.116 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.117 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.118 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.119 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.120 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.912+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.121 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.122 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.123 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.124 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.125 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.126 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.127 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.129 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.129 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.129 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.920+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.129 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.129 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.129 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.129 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.129 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.129 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.129 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.129 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.129 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.131 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.132 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.133 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.134 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.135 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.921+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.136 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.137 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.922+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.138 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.139 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.140 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.928+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.141 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.142 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95d597700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.143 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.144 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.145 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.146 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.147 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.148 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.149 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.150 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.151 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.929+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.152 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.153 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.154 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.154 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.154 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.154 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.154 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.154 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.154 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.154 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.154 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.154 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.154 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.155 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.156 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd959d90700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.157 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.158 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.930+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.159 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.160 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.936+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.161 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.162 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.163 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.164 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.165 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.166 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.167 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.168 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96159f700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.169 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.170 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.937+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.171 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.172 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.173 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.174 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.175 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.176 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.938+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.177 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.178 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.179 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.180 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.181 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.182 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.183 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.184 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.946+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.185 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.186 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.187 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.188 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.189 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.189 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.189 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.189 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.190 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[40404]: 4.a scrub starts 2024-03-26T05:23:13.190 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[40404]: 4.a scrub ok 2024-03-26T05:23:13.190 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[40404]: 5.15 scrub starts 2024-03-26T05:23:13.190 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[40404]: 5.15 scrub ok 2024-03-26T05:23:13.190 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[40404]: pgmap v1078: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:23:13.190 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[40404]: from='osd.4 [v2:172.21.15.132:6800/1991006784,v1:172.21.15.132:6801/1991006784]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T05:23:13.190 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[40404]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T05:23:13.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[29691]: 4.a scrub starts 2024-03-26T05:23:13.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[29691]: 4.a scrub ok 2024-03-26T05:23:13.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[29691]: 5.15 scrub starts 2024-03-26T05:23:13.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[29691]: 5.15 scrub ok 2024-03-26T05:23:13.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[29691]: pgmap v1078: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:23:13.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[29691]: from='osd.4 [v2:172.21.15.132:6800/1991006784,v1:172.21.15.132:6801/1991006784]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T05:23:13.191 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:12 smithi080 ceph-mon[29691]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T05:23:13.205 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.205 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.205 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.205 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.205 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.205 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.205 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.205 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.205 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.205 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.205 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.205 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.206 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.207 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.208 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.947+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.948+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.948+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.948+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.948+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.948+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.209 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.948+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.948+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.948+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.948+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.210 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.211 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.212 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.214 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.215 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.216 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.953+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.217 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.218 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95dd98700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.219 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.220 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.221 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.222 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.223 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.224 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.954+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.225 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.226 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.955+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.227 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.228 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.229 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.230 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.231 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.232 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.233 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.234 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.235 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.960+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.236 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.237 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.238 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.239 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.240 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.241 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.242 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.243 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.244 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.961+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.962+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.962+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.962+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.245 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.966+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.246 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.247 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.248 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.249 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.250 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.250 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.250 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.250 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.250 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.250 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.250 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.250 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.250 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.251 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.252 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.253 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.254 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.967+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.255 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.256 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.257 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.258 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.259 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.260 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.261 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.968+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.263 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.264 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.969+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.265 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.266 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.267 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.268 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.269 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.976+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.271 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.272 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.273 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.274 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.275 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95bd94700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.276 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.277 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.278 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.279 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.977+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.281 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95fd9c700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.282 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.283 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.283 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.283 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.283 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.283 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.283 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.283 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.283 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.283 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.283 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.286 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.286 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.978+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.286 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.287 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.288 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.289 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.290 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.291 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.293 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.294 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.295 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.985+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.296 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.297 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.298 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.299 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95cd96700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.300 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.301 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.302 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.303 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.986+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.304 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.305 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.306 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.307 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.308 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.993+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.309 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.310 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.311 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.312 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.313 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.314 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.315 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.316 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.317 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.318 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.994+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.319 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.320 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.321 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.322 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.323 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:12.995+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.324 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.325 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.000+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.326 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.327 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.328 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.329 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.330 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.331 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.332 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.333 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.333 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.333 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.333 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.333 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.333 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.333 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.333 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.333 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.333 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.333 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.333 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.334 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.335 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.001+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.336 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.337 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.338 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.339 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.340 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.341 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.002+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.009+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.009+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.009+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.342 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.343 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.344 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.345 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.346 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.347 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.348 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.350 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.010+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95e599700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.351 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.352 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.353 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.354 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.355 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.356 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.357 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.358 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95b593700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.011+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.359 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.360 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.361 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.362 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.363 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.364 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.365 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95f59b700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.366 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.367 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.368 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.017+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.369 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.370 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.371 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.018+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.372 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.373 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.374 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.375 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.376 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.377 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.024+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.378 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.379 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.380 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.381 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.382 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.383 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.384 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.025+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.385 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.386 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.387 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.388 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.389 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.029+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.390 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.391 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.392 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.393 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.394 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd960d9e700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.395 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.396 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.397 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.030+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.398 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.399 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.400 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.031+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.401 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.402 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.403 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.404 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.405 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.406 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.407 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd96059d700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.408 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 342, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.037+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.409 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.410 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 343, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 344, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 345, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 346, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 347, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 348, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 349, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 350, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 351, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 352, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 353, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 354, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 355, got 0 bytes 2024-03-26T05:23:13.411 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 356, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 357, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 358, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 359, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 360, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 361, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 362, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 363, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.412 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.413 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.413 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.413 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.413 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.413 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.413 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.413 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.413 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.413 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.413 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.414 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.415 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95c595700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 364, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 365, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 366, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 367, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 368, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 369, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 370, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 371, got 0 bytes 2024-03-26T05:23:13.416 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.038+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 372, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 373, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 374, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 375, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 376, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 377, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 378, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 379, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 380, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 381, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 382, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 383, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 384, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 385, got 0 bytes 2024-03-26T05:23:13.417 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 386, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 387, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 388, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 389, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 390, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 391, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 392, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 393, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 394, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 395, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 396, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 397, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 398, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 399, got 0 bytes 2024-03-26T05:23:13.418 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 400, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 401, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 402, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 403, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 404, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 405, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 406, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 407, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 408, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 409, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 410, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 411, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 412, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.419 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.045+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.420 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.421 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ad92700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 413, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 414, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 415, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 416, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 417, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 418, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 419, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 420, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.422 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.423 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95ed9a700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 421, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 422, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 423, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 424, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 425, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 426, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 427, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 428, got 0 bytes 2024-03-26T05:23:13.424 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 429, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 430, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 431, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 432, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 433, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 434, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 435, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 436, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 437, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 438, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 439, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 440, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 441, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 442, got 0 bytes 2024-03-26T05:23:13.425 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 443, got 0 bytes 2024-03-26T05:23:13.426 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.046+0000 7fd95a591700 -1 osd.4 445 failed to load OSD map for epoch 444, got 0 bytes 2024-03-26T05:23:13.426 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:23:13.114+0000 7fd972dc2700 -1 osd.4 500 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T05:23:13.954 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T05:23:14.115 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:23:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:23:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[29691]: 3.d scrub starts 2024-03-26T05:23:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[29691]: 3.d scrub ok 2024-03-26T05:23:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[29691]: 1.4 deep-scrub starts 2024-03-26T05:23:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[29691]: 1.4 deep-scrub ok 2024-03-26T05:23:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[29691]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T05:23:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[29691]: from='osd.4 [v2:172.21.15.132:6800/1991006784,v1:172.21.15.132:6801/1991006784]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:23:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[29691]: osdmap e500: 8 total, 7 up, 6 in 2024-03-26T05:23:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[29691]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:23:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:23:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[29691]: 6.d deep-scrub starts 2024-03-26T05:23:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[29691]: 6.d deep-scrub ok 2024-03-26T05:23:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[40404]: 3.d scrub starts 2024-03-26T05:23:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[40404]: 3.d scrub ok 2024-03-26T05:23:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[40404]: 1.4 deep-scrub starts 2024-03-26T05:23:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[40404]: 1.4 deep-scrub ok 2024-03-26T05:23:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[40404]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T05:23:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[40404]: from='osd.4 [v2:172.21.15.132:6800/1991006784,v1:172.21.15.132:6801/1991006784]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:23:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[40404]: osdmap e500: 8 total, 7 up, 6 in 2024-03-26T05:23:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[40404]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:23:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:23:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[40404]: 6.d deep-scrub starts 2024-03-26T05:23:14.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:13 smithi080 ceph-mon[40404]: 6.d deep-scrub ok 2024-03-26T05:23:14.202 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:14.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-mon[33500]: 3.d scrub starts 2024-03-26T05:23:14.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-mon[33500]: 3.d scrub ok 2024-03-26T05:23:14.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-mon[33500]: 1.4 deep-scrub starts 2024-03-26T05:23:14.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-mon[33500]: 1.4 deep-scrub ok 2024-03-26T05:23:14.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-mon[33500]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T05:23:14.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-mon[33500]: from='osd.4 [v2:172.21.15.132:6800/1991006784,v1:172.21.15.132:6801/1991006784]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:23:14.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-mon[33500]: osdmap e500: 8 total, 7 up, 6 in 2024-03-26T05:23:14.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-mon[33500]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-26T05:23:14.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:23:14.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-mon[33500]: 6.d deep-scrub starts 2024-03-26T05:23:14.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:13 smithi132 ceph-mon[33500]: 6.d deep-scrub ok 2024-03-26T05:23:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:14 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:23:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:14 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:23:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:14 smithi132 ceph-mon[33500]: osd.4 [v2:172.21.15.132:6800/1991006784,v1:172.21.15.132:6801/1991006784] boot 2024-03-26T05:23:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:14 smithi132 ceph-mon[33500]: osdmap e501: 8 total, 8 up, 6 in 2024-03-26T05:23:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:14 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:23:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:14 smithi132 ceph-mon[33500]: 2.2 scrub starts 2024-03-26T05:23:15.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:14 smithi132 ceph-mon[33500]: 2.2 scrub ok 2024-03-26T05:23:15.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:14 smithi132 ceph-mon[33500]: pgmap v1081: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 793 B/s rd, 0 op/s 2024-03-26T05:23:15.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:14 smithi132 ceph-mon[33500]: 3.12 scrub starts 2024-03-26T05:23:15.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:14 smithi132 ceph-mon[33500]: 3.12 scrub ok 2024-03-26T05:23:15.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:23:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:23:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[29691]: osd.4 [v2:172.21.15.132:6800/1991006784,v1:172.21.15.132:6801/1991006784] boot 2024-03-26T05:23:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[29691]: osdmap e501: 8 total, 8 up, 6 in 2024-03-26T05:23:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:23:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[29691]: 2.2 scrub starts 2024-03-26T05:23:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[29691]: 2.2 scrub ok 2024-03-26T05:23:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[29691]: pgmap v1081: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 793 B/s rd, 0 op/s 2024-03-26T05:23:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[29691]: 3.12 scrub starts 2024-03-26T05:23:15.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[29691]: 3.12 scrub ok 2024-03-26T05:23:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:23:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:23:15.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[40404]: osd.4 [v2:172.21.15.132:6800/1991006784,v1:172.21.15.132:6801/1991006784] boot 2024-03-26T05:23:15.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[40404]: osdmap e501: 8 total, 8 up, 6 in 2024-03-26T05:23:15.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:23:15.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[40404]: 2.2 scrub starts 2024-03-26T05:23:15.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[40404]: 2.2 scrub ok 2024-03-26T05:23:15.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[40404]: pgmap v1081: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 793 B/s rd, 0 op/s 2024-03-26T05:23:15.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[40404]: 3.12 scrub starts 2024-03-26T05:23:15.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:14 smithi080 ceph-mon[40404]: 3.12 scrub ok 2024-03-26T05:23:15.900 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:15.900 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:23:15.901 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2024-03-26T05:23:15.901 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2024-03-26T05:23:15.901 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:15.974 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T05:23:15.999 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:15 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:23:15.999 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:15 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:23:16.000 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:15 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:23:16.000 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:15 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:23:16.000 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:15 smithi132 ceph-mon[33500]: osdmap e502: 8 total, 8 up, 6 in 2024-03-26T05:23:16.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:15 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:23:16.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:15 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:23:16.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:15 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:23:16.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:15 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:23:16.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:15 smithi080 ceph-mon[29691]: osdmap e502: 8 total, 8 up, 6 in 2024-03-26T05:23:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:15 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:23:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:15 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:23:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:15 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:23:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:15 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:23:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:15 smithi080 ceph-mon[40404]: osdmap e502: 8 total, 8 up, 6 in 2024-03-26T05:23:16.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:23:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:23:16.648 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T05:23:16.748 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:17.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:16 smithi080 ceph-mon[29691]: 1.0 scrub starts 2024-03-26T05:23:17.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:16 smithi080 ceph-mon[29691]: 1.0 scrub ok 2024-03-26T05:23:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:16 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:17.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:16 smithi080 ceph-mon[29691]: pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:23:17.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:16 smithi080 ceph-mon[40404]: 1.0 scrub starts 2024-03-26T05:23:17.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:16 smithi080 ceph-mon[40404]: 1.0 scrub ok 2024-03-26T05:23:17.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:16 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:17.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:16 smithi080 ceph-mon[40404]: pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:23:17.141 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:16 smithi132 ceph-mon[33500]: 1.0 scrub starts 2024-03-26T05:23:17.141 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:16 smithi132 ceph-mon[33500]: 1.0 scrub ok 2024-03-26T05:23:17.141 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:16 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:17.141 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:16 smithi132 ceph-mon[33500]: pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:23:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:18.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:17 smithi080 ceph-mon[29691]: 6.8 scrub starts 2024-03-26T05:23:18.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:17 smithi080 ceph-mon[29691]: 6.8 scrub ok 2024-03-26T05:23:18.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:17 smithi080 ceph-mon[40404]: 6.8 scrub starts 2024-03-26T05:23:18.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:17 smithi080 ceph-mon[40404]: 6.8 scrub ok 2024-03-26T05:23:18.223 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:17 smithi132 ceph-mon[33500]: 6.8 scrub starts 2024-03-26T05:23:18.223 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:17 smithi132 ceph-mon[33500]: 6.8 scrub ok 2024-03-26T05:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2024-03-26T05:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:19.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:18 smithi080 ceph-mon[29691]: 4.16 scrub starts 2024-03-26T05:23:19.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:18 smithi080 ceph-mon[29691]: 4.16 scrub ok 2024-03-26T05:23:19.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:18 smithi080 ceph-mon[29691]: 3.1c scrub starts 2024-03-26T05:23:19.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:18 smithi080 ceph-mon[29691]: 3.1c scrub ok 2024-03-26T05:23:19.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:18 smithi080 ceph-mon[29691]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:19.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:18 smithi080 ceph-mon[40404]: 4.16 scrub starts 2024-03-26T05:23:19.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:18 smithi080 ceph-mon[40404]: 4.16 scrub ok 2024-03-26T05:23:19.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:18 smithi080 ceph-mon[40404]: 3.1c scrub starts 2024-03-26T05:23:19.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:18 smithi080 ceph-mon[40404]: 3.1c scrub ok 2024-03-26T05:23:19.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:18 smithi080 ceph-mon[40404]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:19.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:18 smithi132 ceph-mon[33500]: 4.16 scrub starts 2024-03-26T05:23:19.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:18 smithi132 ceph-mon[33500]: 4.16 scrub ok 2024-03-26T05:23:19.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:18 smithi132 ceph-mon[33500]: 3.1c scrub starts 2024-03-26T05:23:19.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:18 smithi132 ceph-mon[33500]: 3.1c scrub ok 2024-03-26T05:23:19.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:18 smithi132 ceph-mon[33500]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:20.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:19 smithi132 ceph-mon[33500]: 5.17 scrub starts 2024-03-26T05:23:20.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:19 smithi132 ceph-mon[33500]: 5.17 scrub ok 2024-03-26T05:23:20.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:19 smithi132 ceph-mon[33500]: 5.1e deep-scrub starts 2024-03-26T05:23:20.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:19 smithi132 ceph-mon[33500]: 5.1e deep-scrub ok 2024-03-26T05:23:20.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:19 smithi132 ceph-mon[33500]: 6.1 scrub starts 2024-03-26T05:23:20.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:19 smithi132 ceph-mon[33500]: 6.1 scrub ok 2024-03-26T05:23:20.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:19 smithi080 ceph-mon[29691]: 5.17 scrub starts 2024-03-26T05:23:20.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:19 smithi080 ceph-mon[29691]: 5.17 scrub ok 2024-03-26T05:23:20.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:19 smithi080 ceph-mon[29691]: 5.1e deep-scrub starts 2024-03-26T05:23:20.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:19 smithi080 ceph-mon[29691]: 5.1e deep-scrub ok 2024-03-26T05:23:20.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:19 smithi080 ceph-mon[29691]: 6.1 scrub starts 2024-03-26T05:23:20.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:19 smithi080 ceph-mon[29691]: 6.1 scrub ok 2024-03-26T05:23:20.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:19 smithi080 ceph-mon[40404]: 5.17 scrub starts 2024-03-26T05:23:20.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:19 smithi080 ceph-mon[40404]: 5.17 scrub ok 2024-03-26T05:23:20.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:19 smithi080 ceph-mon[40404]: 5.1e deep-scrub starts 2024-03-26T05:23:20.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:19 smithi080 ceph-mon[40404]: 5.1e deep-scrub ok 2024-03-26T05:23:20.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:19 smithi080 ceph-mon[40404]: 6.1 scrub starts 2024-03-26T05:23:20.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:19 smithi080 ceph-mon[40404]: 6.1 scrub ok 2024-03-26T05:23:20.390 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-26T05:23:20.390 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-26T05:23:20.390 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd in 4 2024-03-26T05:23:20.650 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141729 10.9 10:90ac0485:::benchmark_data_smithi080_161990_object30006:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.720442+0000", 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 389.94430907899999, 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.012933866, 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.656 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141729 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720442+0000", 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720442+0000", 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720444+0000", 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.526e-06 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720455+0000", 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0433e-05 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720479+0000", 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.4596999999999998e-05 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720494+0000", 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4133e-05 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.809189+0000", 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.088695464000000002 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.809199+0000", 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5419999999999994e-06 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.733307+0000", 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92410795400000001 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.733376+0000", 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9216999999999998e-05 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141730 10.9 10:92d423bb:::benchmark_data_smithi080_161990_object30007:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:23:20.660 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.720524+0000", 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 389.944226937, 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.013281876, 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141730 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720524+0000", 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720524+0000", 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720524+0000", 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5699999999999998e-07 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720527+0000", 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.3939999999999999e-06 2024-03-26T05:23:20.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720536+0000", 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4809999999999995e-06 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720542+0000", 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1649999999999997e-06 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.809225+0000", 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.088683159999999997 2024-03-26T05:23:20.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.809232+0000", 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.503e-06 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.733739+0000", 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92450684699999996 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.733806+0000", 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6869e-05 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.664 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.665 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.665 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141748 10.16 10:6ac9489a:::benchmark_data_smithi080_161990_object30025:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:23:20.665 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.720548+0000", 2024-03-26T05:23:20.665 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 389.94420312699998, 2024-03-26T05:23:20.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.018819301, 2024-03-26T05:23:20.665 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.665 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:20.665 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.665 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:20.665 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:20.666 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141748 2024-03-26T05:23:20.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.666 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.666 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.666 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720548+0000", 2024-03-26T05:23:20.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.666 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.666 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720548+0000", 2024-03-26T05:23:20.667 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.667 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.667 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720548+0000", 2024-03-26T05:23:20.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6899999999999998e-07 2024-03-26T05:23:20.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.668 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.668 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720551+0000", 2024-03-26T05:23:20.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.5399999999999998e-06 2024-03-26T05:23:20.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.668 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.668 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720560+0000", 2024-03-26T05:23:20.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2979999999999997e-06 2024-03-26T05:23:20.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720565+0000", 2024-03-26T05:23:20.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7489999999999998e-06 2024-03-26T05:23:20.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.809269+0000", 2024-03-26T05:23:20.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.088704029000000004 2024-03-26T05:23:20.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.809276+0000", 2024-03-26T05:23:20.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5989999999999996e-06 2024-03-26T05:23:20.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.730276+0000", 2024-03-26T05:23:20.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92099931300000004 2024-03-26T05:23:20.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.730401+0000", 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000125217 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.730700+0000", 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00029868899999999999 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.732755+0000", 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002055781 2024-03-26T05:23:20.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739245+0000", 2024-03-26T05:23:20.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00648987 2024-03-26T05:23:20.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:20.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739287+0000", 2024-03-26T05:23:20.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1913999999999999e-05 2024-03-26T05:23:20.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739367+0000", 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9932999999999995e-05 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158392 10.11 10:8a795ca1:::benchmark_data_smithi080_161990_object46669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.596536+0000", 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.06821525999999, 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.094042502, 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.674 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158392 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596536+0000", 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596536+0000", 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596538+0000", 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.885e-06 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596542+0000", 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7700000000000001e-06 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596555+0000", 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.292e-05 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596563+0000", 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9459999999999998e-06 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596625+0000", 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1779999999999995e-05 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.596639+0000", 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3797e-05 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.687392+0000", 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0907529970000001 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.687505+0000", 2024-03-26T05:23:20.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000113009 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.687732+0000", 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00022680399999999999 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690443+0000", 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002711364 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:20.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690462+0000", 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.878e-05 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690490+0000", 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.7838e-05 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690536+0000", 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6758000000000002e-05 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690578+0000", 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1854000000000003e-05 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158416 10.11 10:8ed20479:::benchmark_data_smithi080_161990_object46693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.609295+0000", 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.05545595900003, 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.083529985, 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:20.681 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158416 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609295+0000", 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609295+0000", 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609296+0000", 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.395e-06 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609301+0000", 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1089999999999997e-06 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609315+0000", 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3568e-05 2024-03-26T05:23:20.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609323+0000", 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9330000000000001e-06 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609969+0000", 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00064633900000000001 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.609977+0000", 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7929999999999998e-06 2024-03-26T05:23:20.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.689332+0000", 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0793544639999999 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.689422+0000", 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0303999999999995e-05 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:23:20.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.689556+0000", 2024-03-26T05:23:20.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000133931 2024-03-26T05:23:20.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:20.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.691785+0000", 2024-03-26T05:23:20.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0022293230000000001 2024-03-26T05:23:20.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692668+0000", 2024-03-26T05:23:20.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088328400000000002 2024-03-26T05:23:20.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692722+0000", 2024-03-26T05:23:20.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3233999999999999e-05 2024-03-26T05:23:20.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:20.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692781+0000", 2024-03-26T05:23:20.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8940000000000002e-05 2024-03-26T05:23:20.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692825+0000", 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4367999999999999e-05 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158447 10.11 10:8efb7744:::benchmark_data_smithi080_161990_object46724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.633507+0000", 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.03124442699999, 2024-03-26T05:23:20.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0649222920000001, 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158447 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633507+0000", 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633507+0000", 2024-03-26T05:23:20.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633507+0000", 2024-03-26T05:23:20.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.23e-07 2024-03-26T05:23:20.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633511+0000", 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.754e-06 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633521+0000", 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0077e-05 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633527+0000", 2024-03-26T05:23:20.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633578+0000", 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1239999999999997e-05 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.633588+0000", 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6649999999999998e-06 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690639+0000", 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0570507950000001 2024-03-26T05:23:20.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690756+0000", 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000117679 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.690962+0000", 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00020548699999999999 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692888+0000", 2024-03-26T05:23:20.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0019258369999999999 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.694025+0000", 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011376870000000001 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698342+0000", 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043164609999999997 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698390+0000", 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7627999999999999e-05 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698429+0000", 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.926e-05 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158477 10.11 10:8cf6ea54:::benchmark_data_smithi080_161990_object46754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.657292+0000", 2024-03-26T05:23:20.695 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.00745937800002, 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0412966560000001, 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158477 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657292+0000", 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657292+0000", 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657293+0000", 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3599999999999999e-06 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657298+0000", 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2024-03-26T05:23:20.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657312+0000", 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3521e-05 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657321+0000", 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7779999999999996e-06 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657384+0000", 2024-03-26T05:23:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3275999999999995e-05 2024-03-26T05:23:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.657396+0000", 2024-03-26T05:23:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1783999999999999e-05 2024-03-26T05:23:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.691824+0000", 2024-03-26T05:23:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.034427921 2024-03-26T05:23:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:20.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.691892+0000", 2024-03-26T05:23:20.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8178999999999999e-05 2024-03-26T05:23:20.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:23:20.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692026+0000", 2024-03-26T05:23:20.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013440300000000001 2024-03-26T05:23:20.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:20.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.694032+0000", 2024-03-26T05:23:20.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020056230000000002 2024-03-26T05:23:20.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.694070+0000", 2024-03-26T05:23:20.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8408999999999998e-05 2024-03-26T05:23:20.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698509+0000", 2024-03-26T05:23:20.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.004438727 2024-03-26T05:23:20.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:20.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698560+0000", 2024-03-26T05:23:20.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0816000000000003e-05 2024-03-26T05:23:20.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698588+0000", 2024-03-26T05:23:20.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.8438000000000001e-05 2024-03-26T05:23:20.702 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158504 10.11 10:89d6d76b:::benchmark_data_smithi080_161990_object46781:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.677034+0000", 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 370.98771667099999, 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.021703617, 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:20.703 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158504 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677034+0000", 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677034+0000", 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677036+0000", 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4610000000000001e-06 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677041+0000", 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6219999999999997e-06 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677056+0000", 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4630999999999999e-05 2024-03-26T05:23:20.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677064+0000", 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8560000000000007e-06 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677122+0000", 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7734e-05 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.677131+0000", 2024-03-26T05:23:20.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0080000000000008e-06 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.692954+0000", 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.015823565 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.693026+0000", 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1594999999999995e-05 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:23:20.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.693185+0000", 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015974699999999999 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.695963+0000", 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0027773009999999998 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.695973+0000", 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8139999999999995e-06 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698666+0000", 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0026932979999999998 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698703+0000", 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6817999999999999e-05 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698738+0000", 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5166999999999997e-05 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.709 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158528 10.11 10:8c53284c:::benchmark_data_smithi080_161990_object46805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.696752+0000", 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 370.96799869500001, 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.002081698, 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:20.710 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158528 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696752+0000", 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696752+0000", 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696753+0000", 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.159e-06 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696758+0000", 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0189999999999997e-06 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696776+0000", 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.7456999999999999e-05 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696784+0000", 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3739999999999998e-06 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696840+0000", 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6116000000000003e-05 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.696923+0000", 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2599e-05 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.694099+0000", 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.99717579599999995 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.694175+0000", 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6039999999999997e-05 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.694339+0000", 2024-03-26T05:23:20.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000164637 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.695999+0000", 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001659607 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.697054+0000", 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00105466 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698783+0000", 2024-03-26T05:23:20.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001729405 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698811+0000", 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.7582000000000001e-05 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698834+0000", 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.3247e-05 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:158578 10.11 10:89bd8808:::benchmark_data_smithi080_161990_object46855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:09.740795+0000", 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 370.923955686, 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.96429717400000003, 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 158578 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.717 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740795+0000", 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740795+0000", 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740797+0000", 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3030000000000001e-06 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740802+0000", 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1270000000000002e-06 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740816+0000", 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4089e-05 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740823+0000", 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5190000000000003e-06 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740884+0000", 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0424999999999998e-05 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:09.740896+0000", 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2548000000000001e-05 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.695076+0000", 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.95417941799999995 2024-03-26T05:23:20.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.695139+0000", 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3326999999999999e-05 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0,7", 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.695273+0000", 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000133516 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698468+0000", 2024-03-26T05:23:20.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.003195163 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.698747+0000", 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00027941899999999999 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.704990+0000", 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0062431179999999998 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:20.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.705049+0000", 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8473000000000001e-05 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:10.705092+0000", 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3729000000000001e-05 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:129362 11.7 11.e38a9407 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-03-26T05:23:20.723 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:29.327232+0000", 2024-03-26T05:23:20.724 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 171.33751894, 2024-03-26T05:23:20.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.98111769100000001, 2024-03-26T05:23:20.724 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.724 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:23:20.724 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.724 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:23:20.724 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:20.724 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 129362 2024-03-26T05:23:20.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.724 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.724 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.327232+0000", 2024-03-26T05:23:20.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.327232+0000", 2024-03-26T05:23:20.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.327233+0000", 2024-03-26T05:23:20.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.147e-06 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.327237+0000", 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9809999999999997e-06 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.327248+0000", 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1286e-05 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.327257+0000", 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0659999999999993e-06 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.327328+0000", 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0421999999999998e-05 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.327336+0000", 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4910000000000005e-06 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.728 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.308338+0000", 2024-03-26T05:23:20.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.98100134999999999 2024-03-26T05:23:20.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.308350+0000", 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1948e-05 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:129369 11.7 11.a70f1c37 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:29.336054+0000", 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 171.32869708999999, 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.97232951599999995, 2024-03-26T05:23:20.729 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 129369 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.336054+0000", 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.336054+0000", 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.336055+0000", 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.128e-06 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.336059+0000", 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3680000000000004e-06 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.336072+0000", 2024-03-26T05:23:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2561999999999999e-05 2024-03-26T05:23:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.336078+0000", 2024-03-26T05:23:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5590000000000002e-06 2024-03-26T05:23:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.336124+0000", 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5658999999999998e-05 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.336132+0000", 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.763e-06 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.308374+0000", 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.97224249799999996 2024-03-26T05:23:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.308383+0000", 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.9789999999999999e-06 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:129382 11.7 11.3107ce97 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:29.355721+0000", 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 171.309029584, 2024-03-26T05:23:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.95269115800000004, 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 129382 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.355721+0000", 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.355721+0000", 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.355723+0000", 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4699999999999999e-06 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.355728+0000", 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.614e-06 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.355744+0000", 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5145e-05 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.355753+0000", 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8660000000000001e-06 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.355820+0000", 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6790000000000003e-05 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.355832+0000", 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2082999999999999e-05 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.308405+0000", 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.95257229600000004 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.308413+0000", 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8939999999999993e-06 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.738 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:129386 11.7 11.3c371a87 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:29.360890+0000", 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 171.303861152, 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.94755360399999999, 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 129386 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.360890+0000", 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.360890+0000", 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.360891+0000", 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.39e-06 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.360895+0000", 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1810000000000003e-06 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.360911+0000", 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5381000000000001e-05 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.360928+0000", 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.7167e-05 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.360988+0000", 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0544000000000002e-05 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.360999+0000", 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1029999999999999e-05 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.308435+0000", 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.94743542999999997 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.308443+0000", 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4810000000000006e-06 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:129428 11.7 11.46a82bc7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:29.411003+0000", 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 171.25374824400001, 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.89747080800000001, 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 129428 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.411003+0000", 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.411003+0000", 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:20.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.411005+0000", 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.4219999999999999e-06 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.411012+0000", 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1849999999999998e-06 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.411032+0000", 2024-03-26T05:23:20.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.9483e-05 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.411049+0000", 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.736e-05 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.411121+0000", 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1612999999999994e-05 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:20.746 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.411128+0000", 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3599999999999998e-06 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.308465+0000", 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.89733684300000005 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.308474+0000", 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5420000000000005e-06 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:20.747 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:20.748 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:21.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:20 smithi080 ceph-mon[40404]: 6.9 scrub starts 2024-03-26T05:23:21.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:20 smithi080 ceph-mon[40404]: 6.9 scrub ok 2024-03-26T05:23:21.024 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:20 smithi080 ceph-mon[40404]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 688 B/s rd, 0 op/s 2024-03-26T05:23:21.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:20 smithi080 ceph-mon[29691]: 6.9 scrub starts 2024-03-26T05:23:21.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:20 smithi080 ceph-mon[29691]: 6.9 scrub ok 2024-03-26T05:23:21.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:20 smithi080 ceph-mon[29691]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 688 B/s rd, 0 op/s 2024-03-26T05:23:21.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:20 smithi132 ceph-mon[33500]: 6.9 scrub starts 2024-03-26T05:23:21.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:20 smithi132 ceph-mon[33500]: 6.9 scrub ok 2024-03-26T05:23:21.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:20 smithi132 ceph-mon[33500]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 688 B/s rd, 0 op/s 2024-03-26T05:23:21.478 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T05:23:21.720 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:21.978 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:21 smithi080 ceph-mon[29691]: 5.16 scrub starts 2024-03-26T05:23:21.978 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:21 smithi080 ceph-mon[29691]: 5.16 scrub ok 2024-03-26T05:23:21.979 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:21.979 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:21 smithi080 ceph-mon[40404]: 5.16 scrub starts 2024-03-26T05:23:21.979 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:21 smithi080 ceph-mon[40404]: 5.16 scrub ok 2024-03-26T05:23:21.979 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:22.107 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:21 smithi132 ceph-mon[33500]: 5.16 scrub starts 2024-03-26T05:23:22.107 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:21 smithi132 ceph-mon[33500]: 5.16 scrub ok 2024-03-26T05:23:22.107 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:22.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a[29687]: 2024-03-26T05:23:22.435+0000 7f70d2504700 -1 mon.a@0(leader).osd e502 definitely_dead 0 2024-03-26T05:23:22.903 INFO:teuthology.orchestra.run.smithi080.stderr:marked in osd.4. 2024-03-26T05:23:23.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:22 smithi132 ceph-mon[33500]: 6.16 scrub starts 2024-03-26T05:23:23.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:22 smithi132 ceph-mon[33500]: 6.16 scrub ok 2024-03-26T05:23:23.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:22 smithi132 ceph-mon[33500]: pgmap v1086: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:23.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:22 smithi132 ceph-mon[33500]: 4.1c scrub starts 2024-03-26T05:23:23.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:22 smithi132 ceph-mon[33500]: 4.1c scrub ok 2024-03-26T05:23:23.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:22 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/546539659' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-26T05:23:23.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:22 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-26T05:23:23.195 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[29691]: 6.16 scrub starts 2024-03-26T05:23:23.195 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[29691]: 6.16 scrub ok 2024-03-26T05:23:23.195 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[29691]: pgmap v1086: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:23.196 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[29691]: 4.1c scrub starts 2024-03-26T05:23:23.196 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[29691]: 4.1c scrub ok 2024-03-26T05:23:23.196 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/546539659' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-26T05:23:23.196 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-26T05:23:23.196 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[40404]: 6.16 scrub starts 2024-03-26T05:23:23.196 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[40404]: 6.16 scrub ok 2024-03-26T05:23:23.196 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[40404]: pgmap v1086: 142 pgs: 142 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:23.196 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[40404]: 4.1c scrub starts 2024-03-26T05:23:23.196 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[40404]: 4.1c scrub ok 2024-03-26T05:23:23.196 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/546539659' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-26T05:23:23.196 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:22 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-26T05:23:23.346 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:23.346 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:23:23.346 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2024-03-26T05:23:23.346 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:23.586 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-26T05:23:23.586 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-26T05:23:23.586 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-26T05:23:23.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd in 5 2024-03-26T05:23:23.830 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:24.018 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T05:23:24.234 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:23:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:23:24.234 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:23 smithi080 ceph-mon[29691]: 3.10 scrub starts 2024-03-26T05:23:24.235 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:23 smithi080 ceph-mon[29691]: 3.10 scrub ok 2024-03-26T05:23:24.235 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:23 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-26T05:23:24.235 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:23 smithi080 ceph-mon[29691]: osdmap e503: 8 total, 8 up, 7 in 2024-03-26T05:23:24.235 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:23 smithi080 ceph-mon[29691]: 3.0 scrub starts 2024-03-26T05:23:24.235 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:23 smithi080 ceph-mon[29691]: 3.0 scrub ok 2024-03-26T05:23:24.235 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:23 smithi080 ceph-mon[40404]: 3.10 scrub starts 2024-03-26T05:23:24.235 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:23 smithi080 ceph-mon[40404]: 3.10 scrub ok 2024-03-26T05:23:24.235 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:23 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-26T05:23:24.235 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:23 smithi080 ceph-mon[40404]: osdmap e503: 8 total, 8 up, 7 in 2024-03-26T05:23:24.235 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:23 smithi080 ceph-mon[40404]: 3.0 scrub starts 2024-03-26T05:23:24.235 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:23 smithi080 ceph-mon[40404]: 3.0 scrub ok 2024-03-26T05:23:24.267 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:23 smithi132 ceph-mon[33500]: 3.10 scrub starts 2024-03-26T05:23:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:23 smithi132 ceph-mon[33500]: 3.10 scrub ok 2024-03-26T05:23:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:23 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-26T05:23:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:23 smithi132 ceph-mon[33500]: osdmap e503: 8 total, 8 up, 7 in 2024-03-26T05:23:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:23 smithi132 ceph-mon[33500]: 3.0 scrub starts 2024-03-26T05:23:24.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:23 smithi132 ceph-mon[33500]: 3.0 scrub ok 2024-03-26T05:23:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:24 smithi132 ceph-mon[33500]: 3.5 scrub starts 2024-03-26T05:23:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:24 smithi132 ceph-mon[33500]: 3.5 scrub ok 2024-03-26T05:23:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:24 smithi132 ceph-mon[33500]: 6.18 deep-scrub starts 2024-03-26T05:23:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:24 smithi132 ceph-mon[33500]: 6.18 deep-scrub ok 2024-03-26T05:23:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:24 smithi132 ceph-mon[33500]: osdmap e504: 8 total, 8 up, 7 in 2024-03-26T05:23:25.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:24 smithi132 ceph-mon[33500]: pgmap v1089: 142 pgs: 25 peering, 1 unknown, 116 active+clean; 582 KiB data, 7.2 GiB used, 619 GiB / 626 GiB avail 2024-03-26T05:23:25.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-mon[40404]: 3.5 scrub starts 2024-03-26T05:23:25.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-mon[40404]: 3.5 scrub ok 2024-03-26T05:23:25.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-mon[40404]: 6.18 deep-scrub starts 2024-03-26T05:23:25.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-mon[40404]: 6.18 deep-scrub ok 2024-03-26T05:23:25.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-mon[40404]: osdmap e504: 8 total, 8 up, 7 in 2024-03-26T05:23:25.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-mon[40404]: pgmap v1089: 142 pgs: 25 peering, 1 unknown, 116 active+clean; 582 KiB data, 7.2 GiB used, 619 GiB / 626 GiB avail 2024-03-26T05:23:25.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-mon[29691]: 3.5 scrub starts 2024-03-26T05:23:25.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-mon[29691]: 3.5 scrub ok 2024-03-26T05:23:25.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-mon[29691]: 6.18 deep-scrub starts 2024-03-26T05:23:25.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-mon[29691]: 6.18 deep-scrub ok 2024-03-26T05:23:25.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-mon[29691]: osdmap e504: 8 total, 8 up, 7 in 2024-03-26T05:23:25.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:24 smithi080 ceph-mon[29691]: pgmap v1089: 142 pgs: 25 peering, 1 unknown, 116 active+clean; 582 KiB data, 7.2 GiB used, 619 GiB / 626 GiB avail 2024-03-26T05:23:25.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a[29687]: 2024-03-26T05:23:25.720+0000 7f70d2504700 -1 mon.a@0(leader).osd e505 definitely_dead 0 2024-03-26T05:23:25.921 INFO:teuthology.orchestra.run.smithi080.stderr:marked in osd.5. 2024-03-26T05:23:25.927 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:25 smithi132 ceph-mon[33500]: 4.d deep-scrub starts 2024-03-26T05:23:25.927 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:25 smithi132 ceph-mon[33500]: 4.d deep-scrub ok 2024-03-26T05:23:25.927 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:25 smithi132 ceph-mon[33500]: Health check failed: Reduced data availability: 11 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-03-26T05:23:25.927 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:25 smithi132 ceph-mon[33500]: osdmap e505: 8 total, 8 up, 7 in 2024-03-26T05:23:25.927 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:25 smithi132 ceph-mon[33500]: 5.b scrub starts 2024-03-26T05:23:25.927 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:25 smithi132 ceph-mon[33500]: 5.b scrub ok 2024-03-26T05:23:25.928 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:25 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/868703740' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-26T05:23:26.054 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:26.054 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:23:26.054 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2024-03-26T05:23:26.054 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2024-03-26T05:23:26.054 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:26.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[40404]: 4.d deep-scrub starts 2024-03-26T05:23:26.231 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[40404]: 4.d deep-scrub ok 2024-03-26T05:23:26.231 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[40404]: Health check failed: Reduced data availability: 11 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-03-26T05:23:26.231 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[40404]: osdmap e505: 8 total, 8 up, 7 in 2024-03-26T05:23:26.231 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[40404]: 5.b scrub starts 2024-03-26T05:23:26.231 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[40404]: 5.b scrub ok 2024-03-26T05:23:26.231 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/868703740' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-26T05:23:26.231 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[29691]: 4.d deep-scrub starts 2024-03-26T05:23:26.231 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[29691]: 4.d deep-scrub ok 2024-03-26T05:23:26.231 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[29691]: Health check failed: Reduced data availability: 11 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-03-26T05:23:26.231 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[29691]: osdmap e505: 8 total, 8 up, 7 in 2024-03-26T05:23:26.232 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[29691]: 5.b scrub starts 2024-03-26T05:23:26.232 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[29691]: 5.b scrub ok 2024-03-26T05:23:26.232 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:25 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/868703740' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-26T05:23:26.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:23:26 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:23:26.617 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-26T05:23:26.617 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-26T05:23:26.617 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-03-26T05:23:26.727 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T05:23:26.909 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:26.997 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:27.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:26 smithi132 ceph-mon[33500]: 1.7 scrub starts 2024-03-26T05:23:27.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:26 smithi132 ceph-mon[33500]: 1.7 scrub ok 2024-03-26T05:23:27.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:26 smithi132 ceph-mon[33500]: 5.1b scrub starts 2024-03-26T05:23:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:26 smithi132 ceph-mon[33500]: 5.1b scrub ok 2024-03-26T05:23:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:26 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/868703740' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-26T05:23:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:26 smithi132 ceph-mon[33500]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T05:23:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:26 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:26 smithi132 ceph-mon[33500]: pgmap v1092: 142 pgs: 33 peering, 1 unknown, 108 active+clean; 582 KiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 335 B/s, 4 objects/s recovering 2024-03-26T05:23:27.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:26 smithi132 ceph-mon[33500]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T05:23:27.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[40404]: 1.7 scrub starts 2024-03-26T05:23:27.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[40404]: 1.7 scrub ok 2024-03-26T05:23:27.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[40404]: 5.1b scrub starts 2024-03-26T05:23:27.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[40404]: 5.1b scrub ok 2024-03-26T05:23:27.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/868703740' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-26T05:23:27.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[40404]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T05:23:27.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:27.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[40404]: pgmap v1092: 142 pgs: 33 peering, 1 unknown, 108 active+clean; 582 KiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 335 B/s, 4 objects/s recovering 2024-03-26T05:23:27.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[40404]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T05:23:27.180 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[29691]: 1.7 scrub starts 2024-03-26T05:23:27.180 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[29691]: 1.7 scrub ok 2024-03-26T05:23:27.180 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[29691]: 5.1b scrub starts 2024-03-26T05:23:27.181 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[29691]: 5.1b scrub ok 2024-03-26T05:23:27.181 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/868703740' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-26T05:23:27.181 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[29691]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T05:23:27.181 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:27.181 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[29691]: pgmap v1092: 142 pgs: 33 peering, 1 unknown, 108 active+clean; 582 KiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 335 B/s, 4 objects/s recovering 2024-03-26T05:23:27.181 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:26 smithi080 ceph-mon[29691]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T05:23:28.164 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:27 smithi132 ceph-mon[33500]: 3.9 scrub starts 2024-03-26T05:23:28.164 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:27 smithi132 ceph-mon[33500]: 3.9 scrub ok 2024-03-26T05:23:28.164 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:27 smithi132 ceph-mon[33500]: 6.3 deep-scrub starts 2024-03-26T05:23:28.164 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:27 smithi132 ceph-mon[33500]: 6.3 deep-scrub ok 2024-03-26T05:23:28.164 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:27 smithi132 ceph-mon[33500]: 3.7 scrub starts 2024-03-26T05:23:28.164 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:27 smithi132 ceph-mon[33500]: 3.7 scrub ok 2024-03-26T05:23:28.164 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:27 smithi132 ceph-mon[33500]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T05:23:28.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[40404]: 3.9 scrub starts 2024-03-26T05:23:28.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[40404]: 3.9 scrub ok 2024-03-26T05:23:28.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[40404]: 6.3 deep-scrub starts 2024-03-26T05:23:28.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[40404]: 6.3 deep-scrub ok 2024-03-26T05:23:28.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[40404]: 3.7 scrub starts 2024-03-26T05:23:28.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[40404]: 3.7 scrub ok 2024-03-26T05:23:28.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[40404]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T05:23:28.185 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[29691]: 3.9 scrub starts 2024-03-26T05:23:28.185 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[29691]: 3.9 scrub ok 2024-03-26T05:23:28.185 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[29691]: 6.3 deep-scrub starts 2024-03-26T05:23:28.185 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[29691]: 6.3 deep-scrub ok 2024-03-26T05:23:28.185 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[29691]: 3.7 scrub starts 2024-03-26T05:23:28.185 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[29691]: 3.7 scrub ok 2024-03-26T05:23:28.186 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:27 smithi080 ceph-mon[29691]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T05:23:28.597 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:28.597 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:23:28.597 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:23:28.597 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:23:28.597 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.597 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141753 10.10 10:0cd8b062:::benchmark_data_smithi080_161990_object30030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:23:28.597 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.871167+0000", 2024-03-26T05:23:28.598 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 397.73429429399999, 2024-03-26T05:23:28.598 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.85659952900000003, 2024-03-26T05:23:28.598 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.598 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.598 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.598 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.598 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.598 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141753 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871167+0000", 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871167+0000", 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871168+0000", 2024-03-26T05:23:28.599 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.1999999999999998e-07 2024-03-26T05:23:28.600 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.600 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.600 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.600 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871172+0000", 2024-03-26T05:23:28.600 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5999999999999998e-06 2024-03-26T05:23:28.600 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.600 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.600 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.601 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871180+0000", 2024-03-26T05:23:28.601 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2700000000000004e-06 2024-03-26T05:23:28.601 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.601 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.601 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.601 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871186+0000", 2024-03-26T05:23:28.601 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6189999999999999e-06 2024-03-26T05:23:28.601 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.601 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.601 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.601 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871237+0000", 2024-03-26T05:23:28.601 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1038999999999997e-05 2024-03-26T05:23:28.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.602 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.602 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.871247+0000", 2024-03-26T05:23:28.602 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0285999999999999e-05 2024-03-26T05:23:28.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.602 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.602 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.722770+0000", 2024-03-26T05:23:28.602 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.85152329999999998 2024-03-26T05:23:28.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.603 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.722921+0000", 2024-03-26T05:23:28.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000150297 2024-03-26T05:23:28.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.603 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:23:28.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.723205+0000", 2024-03-26T05:23:28.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00028435299999999999 2024-03-26T05:23:28.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.603 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.726976+0000", 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0037705690000000001 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727633+0000", 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065709700000000004 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727682+0000", 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9518000000000003e-05 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.604 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727767+0000", 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4660999999999995e-05 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141755 10.0 10:0360d020:::benchmark_data_smithi080_161990_object30032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.881338+0000", 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 397.72412328600001, 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.84627915200000003, 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.605 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141755 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881338+0000", 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.606 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881338+0000", 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881339+0000", 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2399999999999999e-07 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881340+0000", 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.57e-06 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881345+0000", 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8890000000000001e-06 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881348+0000", 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0960000000000001e-06 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881381+0000", 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2922000000000001e-05 2024-03-26T05:23:28.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.881387+0000", 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.711e-06 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.708793+0000", 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.82740590999999997 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.708904+0000", 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000111308 2024-03-26T05:23:28.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.709245+0000", 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034106199999999999 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.724788+0000", 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.015542861 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.610 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727472+0000", 2024-03-26T05:23:28.611 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0026844220000000001 2024-03-26T05:23:28.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.611 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.611 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.611 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727527+0000", 2024-03-26T05:23:28.611 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4558999999999997e-05 2024-03-26T05:23:28.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.611 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.611 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727617+0000", 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0518000000000003e-05 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141758 10.17 10:e86fd74d:::benchmark_data_smithi080_161990_object30035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.894305+0000", 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 397.71115638100002, 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.833540261, 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.612 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141758 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894305+0000", 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.613 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894305+0000", 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894306+0000", 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4600000000000004e-07 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894309+0000", 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.083e-06 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894319+0000", 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7410000000000004e-06 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894327+0000", 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9840000000000001e-06 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894369+0000", 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1891000000000003e-05 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.615 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.894376+0000", 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3869999999999997e-06 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.724007+0000", 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.82963071399999999 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.724136+0000", 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012902200000000001 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,5", 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.724390+0000", 2024-03-26T05:23:28.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000254813 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727187+0000", 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002796345 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727401+0000", 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021470699999999999 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727758+0000", 2024-03-26T05:23:28.617 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035681899999999997 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727808+0000", 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9663999999999997e-05 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727845+0000", 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7444999999999998e-05 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141763 10.10 10:08b1d820:::benchmark_data_smithi080_161990_object30040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.942376+0000", 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 397.66308529000003, 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.79749960399999997, 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141763 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.619 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942376+0000", 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942376+0000", 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942378+0000", 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4869999999999999e-06 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.620 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942385+0000", 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.469e-06 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942408+0000", 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.2993000000000002e-05 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942423+0000", 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.431e-05 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.621 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.622 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942573+0000", 2024-03-26T05:23:28.622 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000150483 2024-03-26T05:23:28.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.622 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.622 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.622 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.942602+0000", 2024-03-26T05:23:28.622 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.8705999999999999e-05 2024-03-26T05:23:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.725127+0000", 2024-03-26T05:23:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.78252492399999996 2024-03-26T05:23:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.725246+0000", 2024-03-26T05:23:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119441 2024-03-26T05:23:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1", 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.725452+0000", 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00020625900000000001 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.727779+0000", 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0023271519999999999 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739726+0000", 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011946417000000001 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739783+0000", 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6626000000000003e-05 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739876+0000", 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3337000000000004e-05 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141764 10.7 10:e6576bba:::benchmark_data_smithi080_161990_object30041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:23:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.949318+0000", 2024-03-26T05:23:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 397.65614370899999, 2024-03-26T05:23:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.77638933799999998, 2024-03-26T05:23:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141764 2024-03-26T05:23:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949318+0000", 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949318+0000", 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949319+0000", 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.85e-07 2024-03-26T05:23:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949322+0000", 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0400000000000001e-06 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949331+0000", 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.234e-06 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949337+0000", 2024-03-26T05:23:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9000000000000003e-06 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949393+0000", 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6372999999999997e-05 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.949402+0000", 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6770000000000001e-06 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.710744+0000", 2024-03-26T05:23:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.76134257900000002 2024-03-26T05:23:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.710890+0000", 2024-03-26T05:23:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000145816 2024-03-26T05:23:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,5", 2024-03-26T05:23:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.711254+0000", 2024-03-26T05:23:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000364068 2024-03-26T05:23:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.713197+0000", 2024-03-26T05:23:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001942444 2024-03-26T05:23:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.713483+0000", 2024-03-26T05:23:28.633 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00028582 2024-03-26T05:23:28.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.633 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.633 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.725630+0000", 2024-03-26T05:23:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.012147269 2024-03-26T05:23:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.725665+0000", 2024-03-26T05:23:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.4888999999999997e-05 2024-03-26T05:23:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.725707+0000", 2024-03-26T05:23:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2444000000000002e-05 2024-03-26T05:23:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165093 10.16 10:6fcdd48f:::benchmark_data_smithi080_161990_object53370:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:16.654367+0000", 2024-03-26T05:23:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.95109434099999, 2024-03-26T05:23:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0367038669999999, 2024-03-26T05:23:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:23:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165093 2024-03-26T05:23:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.654367+0000", 2024-03-26T05:23:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.654367+0000", 2024-03-26T05:23:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.654367+0000", 2024-03-26T05:23:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.8700000000000002e-07 2024-03-26T05:23:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.654369+0000", 2024-03-26T05:23:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3209999999999999e-06 2024-03-26T05:23:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.654374+0000", 2024-03-26T05:23:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.899e-06 2024-03-26T05:23:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.654377+0000", 2024-03-26T05:23:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2990000000000001e-06 2024-03-26T05:23:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.654412+0000", 2024-03-26T05:23:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5113000000000001e-05 2024-03-26T05:23:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.654420+0000", 2024-03-26T05:23:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4120000000000004e-06 2024-03-26T05:23:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.682231+0000", 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.027811445 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for missing object", 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.682330+0000", 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8948000000000002e-05 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.690994+0000", 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0086638979999999997 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.691071+0000", 2024-03-26T05:23:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7244999999999996e-05 2024-03-26T05:23:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165176 10.16 10:6fe6c8a6:::benchmark_data_smithi080_161990_object53453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:16.731427+0000", 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.87403409699999, 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.95571757400000001, 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165176 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.731427+0000", 2024-03-26T05:23:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.731427+0000", 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.731428+0000", 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3799999999999996e-07 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.731433+0000", 2024-03-26T05:23:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5279999999999997e-06 2024-03-26T05:23:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.731447+0000", 2024-03-26T05:23:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3852000000000001e-05 2024-03-26T05:23:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.731455+0000", 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9109999999999993e-06 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.731503+0000", 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8646000000000001e-05 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.731512+0000", 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6559999999999998e-06 2024-03-26T05:23:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.682582+0000", 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.95107044100000004 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.682675+0000", 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2713999999999994e-05 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.682901+0000", 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00022635600000000001 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.686940+0000", 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.004038402 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.686960+0000", 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.0339e-05 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687026+0000", 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6143999999999996e-05 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687080+0000", 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3501999999999997e-05 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687145+0000", 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5145000000000001e-05 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165216 10.16 10:6d10b735:::benchmark_data_smithi080_161990_object53493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:16.760795+0000", 2024-03-26T05:23:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.84466621000001, 2024-03-26T05:23:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92671128300000005, 2024-03-26T05:23:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165216 2024-03-26T05:23:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.760795+0000", 2024-03-26T05:23:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.760795+0000", 2024-03-26T05:23:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.760796+0000", 2024-03-26T05:23:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5799999999999998e-07 2024-03-26T05:23:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.760800+0000", 2024-03-26T05:23:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7019999999999999e-06 2024-03-26T05:23:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.760810+0000", 2024-03-26T05:23:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0696000000000001e-05 2024-03-26T05:23:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.760818+0000", 2024-03-26T05:23:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3690000000000001e-06 2024-03-26T05:23:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.760871+0000", 2024-03-26T05:23:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3043e-05 2024-03-26T05:23:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.760984+0000", 2024-03-26T05:23:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011288700000000001 2024-03-26T05:23:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.684685+0000", 2024-03-26T05:23:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92370076000000001 2024-03-26T05:23:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.684777+0000", 2024-03-26T05:23:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2942999999999994e-05 2024-03-26T05:23:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.684948+0000", 2024-03-26T05:23:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017036199999999999 2024-03-26T05:23:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687195+0000", 2024-03-26T05:23:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0022471549999999998 2024-03-26T05:23:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687266+0000", 2024-03-26T05:23:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.1094999999999996e-05 2024-03-26T05:23:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687408+0000", 2024-03-26T05:23:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014177300000000001 2024-03-26T05:23:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687448+0000", 2024-03-26T05:23:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9712999999999997e-05 2024-03-26T05:23:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687507+0000", 2024-03-26T05:23:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9027000000000003e-05 2024-03-26T05:23:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165229 10.16 10:6fee7dbe:::benchmark_data_smithi080_161990_object53506:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:16.767572+0000", 2024-03-26T05:23:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.83788956400002, 2024-03-26T05:23:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.92282863599999998, 2024-03-26T05:23:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165229 2024-03-26T05:23:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.767572+0000", 2024-03-26T05:23:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.767572+0000", 2024-03-26T05:23:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.767573+0000", 2024-03-26T05:23:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2200000000000003e-07 2024-03-26T05:23:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.767579+0000", 2024-03-26T05:23:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9859999999999999e-06 2024-03-26T05:23:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.767590+0000", 2024-03-26T05:23:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1039999999999999e-05 2024-03-26T05:23:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.767597+0000", 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8850000000000001e-06 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.767650+0000", 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3483999999999998e-05 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.767659+0000", 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2110000000000004e-06 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.685717+0000", 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.91805746300000002 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.685813+0000", 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6613999999999994e-05 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.685949+0000", 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000135504 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687273+0000", 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001323589 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687356+0000", 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2992999999999996e-05 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.690224+0000", 2024-03-26T05:23:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0028688899999999998 2024-03-26T05:23:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.690282+0000", 2024-03-26T05:23:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7598e-05 2024-03-26T05:23:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.690401+0000", 2024-03-26T05:23:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000118557 2024-03-26T05:23:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165233 10.16 10:6a50c788:::benchmark_data_smithi080_161990_object53510:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:16.774668+0000", 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.83079312299998, 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.91671336199999998, 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165233 2024-03-26T05:23:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.774668+0000", 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.774668+0000", 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.774669+0000", 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7800000000000001e-07 2024-03-26T05:23:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.774673+0000", 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.4149999999999999e-06 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.774681+0000", 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6009999999999995e-06 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.774687+0000", 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5539999999999997e-06 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.774744+0000", 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6971000000000001e-05 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.774754+0000", 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0543999999999999e-05 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687537+0000", 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.91278327400000003 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687617+0000", 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9101e-05 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.687749+0000", 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013256100000000001 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.690468+0000", 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0027189940000000002 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.691157+0000", 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068862199999999995 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.691216+0000", 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9760999999999999e-05 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.691274+0000", 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7085e-05 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.691382+0000", 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108201 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165234 10.16 10:6b14f21a:::benchmark_data_smithi080_161990_object53511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:16.775007+0000", 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.83045410800003, 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.91661356199999999, 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165234 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.775007+0000", 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.775007+0000", 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.775008+0000", 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2500000000000004e-07 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.775013+0000", 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4229999999999998e-06 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.775025+0000", 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2377e-05 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.775033+0000", 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7349999999999996e-06 2024-03-26T05:23:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.775106+0000", 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2867000000000006e-05 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.775115+0000", 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4220000000000006e-06 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.688714+0000", 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.91359921300000002 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.688820+0000", 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000105222 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.689016+0000", 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000196083 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.690622+0000", 2024-03-26T05:23:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001605906 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.691422+0000", 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079994400000000002 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.691468+0000", 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6258999999999999e-05 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.691523+0000", 2024-03-26T05:23:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5325999999999998e-05 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.691621+0000", 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7860000000000002e-05 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165309 10.16 10:6a472b94:::benchmark_data_smithi080_161990_object53586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:16.852922+0000", 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.752539318, 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.84404994499999997, 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165309 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.852922+0000", 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.852922+0000", 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.852923+0000", 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2580000000000001e-06 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.852927+0000", 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.4189999999999998e-06 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.852936+0000", 2024-03-26T05:23:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8200000000000003e-06 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.852943+0000", 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2840000000000001e-06 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.852979+0000", 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.6486000000000003e-05 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.852986+0000", 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.703e-06 2024-03-26T05:23:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.691742+0000", 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.83875548499999997 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.691868+0000", 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012638300000000001 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.692120+0000", 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00025219899999999999 2024-03-26T05:23:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.696709+0000", 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045887619999999997 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.696750+0000", 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0822999999999999e-05 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.696804+0000", 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3664e-05 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.696858+0000", 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4438999999999998e-05 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.696972+0000", 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011422 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165329 10.16 10:68ac504c:::benchmark_data_smithi080_161990_object53606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:16.868275+0000", 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.73718636199999, 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.83216474600000001, 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165329 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.868275+0000", 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.868275+0000", 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.868276+0000", 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.16e-07 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.868279+0000", 2024-03-26T05:23:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2200000000000001e-06 2024-03-26T05:23:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.868287+0000", 2024-03-26T05:23:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7579999999999992e-06 2024-03-26T05:23:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.868291+0000", 2024-03-26T05:23:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4020000000000004e-06 2024-03-26T05:23:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.868344+0000", 2024-03-26T05:23:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2413000000000001e-05 2024-03-26T05:23:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.868356+0000", 2024-03-26T05:23:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2435e-05 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.693081+0000", 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.82472522500000001 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.693200+0000", 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011848799999999999 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.693400+0000", 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000200519 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.696762+0000", 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0033614489999999999 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.697020+0000", 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00025795000000000001 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700273+0000", 2024-03-26T05:23:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.003253546 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700335+0000", 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2005e-05 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700440+0000", 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010452 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165354 10.16 10:6b45ec99:::benchmark_data_smithi080_161990_object53631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:16.888729+0000", 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.71673225500001, 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.81189407499999999, 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165354 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.888729+0000", 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.888729+0000", 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.888730+0000", 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9999999999999998e-07 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.888732+0000", 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.464e-06 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.888740+0000", 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5009999999999998e-06 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.888745+0000", 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4589999999999997e-06 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.888790+0000", 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4316000000000001e-05 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.888798+0000", 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.1669999999999999e-06 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.694193+0000", 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.80539531099999995 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.694270+0000", 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6809000000000003e-05 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.694417+0000", 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014673400000000001 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.696767+0000", 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0023505240000000001 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700205+0000", 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.003438347 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700487+0000", 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00028129300000000001 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700540+0000", 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2843999999999998e-05 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700623+0000", 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3806000000000004e-05 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165358 10.16 10:6aec5801:::benchmark_data_smithi080_161990_object53635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:16.894343+0000", 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.71111849499999, 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.80662611799999995, 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165358 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.894343+0000", 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.894343+0000", 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.894344+0000", 2024-03-26T05:23:28.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7599999999999996e-07 2024-03-26T05:23:28.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.894347+0000", 2024-03-26T05:23:28.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.281e-06 2024-03-26T05:23:28.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.894356+0000", 2024-03-26T05:23:28.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7120000000000006e-06 2024-03-26T05:23:28.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.894361+0000", 2024-03-26T05:23:28.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2870000000000004e-06 2024-03-26T05:23:28.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.894411+0000", 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9401999999999999e-05 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.894421+0000", 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.061e-05 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.695434+0000", 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.80101309600000004 2024-03-26T05:23:28.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.695548+0000", 2024-03-26T05:23:28.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011331 2024-03-26T05:23:28.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.695731+0000", 2024-03-26T05:23:28.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000183487 2024-03-26T05:23:28.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.697033+0000", 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001302175 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700691+0000", 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0036578719999999999 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700778+0000", 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6662000000000005e-05 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700835+0000", 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7528999999999998e-05 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700969+0000", 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000133819 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165414 10.16 10:6e3a0c20:::benchmark_data_smithi080_161990_object53691:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.716 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:16.949440+0000", 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.65602118499999, 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.75178813, 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165414 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.949440+0000", 2024-03-26T05:23:28.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.949440+0000", 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.949442+0000", 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1960000000000001e-06 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.949445+0000", 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.664e-06 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.949455+0000", 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0214000000000001e-05 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.949463+0000", 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4390000000000003e-06 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.949518+0000", 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4935999999999997e-05 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.949528+0000", 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0200999999999999e-05 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.697060+0000", 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.74753225599999995 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.697173+0000", 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000113021 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.697384+0000", 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000210435 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700637+0000", 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0032530240000000002 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.701022+0000", 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038523499999999999 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.701079+0000", 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7153e-05 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.701131+0000", 2024-03-26T05:23:28.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1916000000000002e-05 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.701228+0000", 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7440000000000002e-05 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165445 10.16 10:6bcff478:::benchmark_data_smithi080_161990_object53722:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:16.980588+0000", 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.62487341799999, 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.720895748, 2024-03-26T05:23:28.723 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.724 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.724 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.724 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.724 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.724 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165445 2024-03-26T05:23:28.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.724 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.980588+0000", 2024-03-26T05:23:28.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.980588+0000", 2024-03-26T05:23:28.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.980589+0000", 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5499999999999998e-07 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.980591+0000", 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.5600000000000001e-06 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.980600+0000", 2024-03-26T05:23:28.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6070000000000008e-06 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.980605+0000", 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0810000000000001e-06 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.980643+0000", 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8436999999999998e-05 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:16.980650+0000", 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.939e-06 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.698719+0000", 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.71806917999999997 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.698803+0000", 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3664000000000004e-05 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.698954+0000", 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015122200000000001 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.700643+0000", 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001688822 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.701274+0000", 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00063138099999999998 2024-03-26T05:23:28.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.701332+0000", 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7127999999999999e-05 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.701381+0000", 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9184999999999997e-05 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.701484+0000", 2024-03-26T05:23:28.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103087 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165514 10.16 10:6cbd0425:::benchmark_data_smithi080_161990_object53791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:17.058091+0000", 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.54737011600002, 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.64948547499999998, 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.731 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165514 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.058091+0000", 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.058091+0000", 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.058092+0000", 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.06e-07 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.058099+0000", 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9650000000000002e-06 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.058110+0000", 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0859000000000001e-05 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.058116+0000", 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.224e-06 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.058167+0000", 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0878000000000003e-05 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.058174+0000", 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2320000000000004e-06 2024-03-26T05:23:28.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.701532+0000", 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.64335808100000003 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.701653+0000", 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012053300000000001 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.701854+0000", 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00020123 2024-03-26T05:23:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.707277+0000", 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.005422718 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.707322+0000", 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.5587999999999998e-05 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.707412+0000", 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.9686999999999997e-05 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.707469+0000", 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.719e-05 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.707577+0000", 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010758400000000001 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165579 10.16 10:6e08d3f7:::benchmark_data_smithi080_161990_object53856:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:17.184222+0000", 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.42123941300002, 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.52577287699999997, 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165579 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.184222+0000", 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.184222+0000", 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.184223+0000", 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1930000000000001e-06 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.184229+0000", 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3159999999999996e-06 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.184239+0000", 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0691e-05 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.184248+0000", 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2539999999999992e-06 2024-03-26T05:23:28.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.184300+0000", 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2361999999999998e-05 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.184307+0000", 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2250000000000003e-06 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.703128+0000", 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.51882073399999995 2024-03-26T05:23:28.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.703235+0000", 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000107434 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.703421+0000", 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018524599999999999 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.707398+0000", 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0039777160000000001 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.707589+0000", 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00019117299999999999 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.709740+0000", 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021509379999999998 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.709799+0000", 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9077999999999999e-05 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.709995+0000", 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00019551700000000001 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:165581 10.16 10:6c57421b:::benchmark_data_smithi080_161990_object53858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T05:23:28.744 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:17:17.186654+0000", 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 371.418807319, 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.52353508100000001, 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 165581 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:28.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.186654+0000", 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.186654+0000", 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.186655+0000", 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9599999999999998e-07 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.746 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.186659+0000", 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.704e-06 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.186669+0000", 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0596e-05 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.186678+0000", 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2109999999999998e-06 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.186724+0000", 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.6142e-05 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.186730+0000", 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4559999999999997e-06 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.704555+0000", 2024-03-26T05:23:28.748 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.51782524200000002 2024-03-26T05:23:28.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.749 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:28.749 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.704717+0000", 2024-03-26T05:23:28.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016165299999999999 2024-03-26T05:23:28.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.749 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 1,5", 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.704910+0000", 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00019287300000000001 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.709649+0000", 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0047391309999999997 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.709696+0000", 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7175999999999997e-05 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.751 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:28.751 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.710076+0000", 2024-03-26T05:23:28.751 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037983500000000003 2024-03-26T05:23:28.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.751 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:28.751 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.710120+0000", 2024-03-26T05:23:28.751 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4406999999999997e-05 2024-03-26T05:23:28.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:28.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:28.751 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:28.751 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:17:17.710189+0000", 2024-03-26T05:23:28.752 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8858999999999999e-05 2024-03-26T05:23:28.752 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.752 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.752 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.752 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:28.752 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:28.752 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:28.777 INFO:teuthology.orchestra.run.smithi080.stderr:set pool 1 pgp_num to 11 2024-03-26T05:23:29.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[29691]: 6.0 scrub starts 2024-03-26T05:23:29.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[29691]: 6.0 scrub ok 2024-03-26T05:23:29.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[29691]: pgmap v1095: 142 pgs: 1 remapped+peering, 40 peering, 1 unknown, 100 active+clean; 582 KiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 495 B/s, 12 objects/s recovering 2024-03-26T05:23:29.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[29691]: 6.f scrub starts 2024-03-26T05:23:29.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[29691]: 6.f scrub ok 2024-03-26T05:23:29.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2508300802' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T05:23:29.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T05:23:29.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-26T05:23:29.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[29691]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T05:23:29.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[40404]: 6.0 scrub starts 2024-03-26T05:23:29.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[40404]: 6.0 scrub ok 2024-03-26T05:23:29.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[40404]: pgmap v1095: 142 pgs: 1 remapped+peering, 40 peering, 1 unknown, 100 active+clean; 582 KiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 495 B/s, 12 objects/s recovering 2024-03-26T05:23:29.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[40404]: 6.f scrub starts 2024-03-26T05:23:29.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[40404]: 6.f scrub ok 2024-03-26T05:23:29.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2508300802' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T05:23:29.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T05:23:29.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-26T05:23:29.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:28 smithi080 ceph-mon[40404]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T05:23:29.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:28 smithi132 ceph-mon[33500]: 6.0 scrub starts 2024-03-26T05:23:29.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:28 smithi132 ceph-mon[33500]: 6.0 scrub ok 2024-03-26T05:23:29.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:28 smithi132 ceph-mon[33500]: pgmap v1095: 142 pgs: 1 remapped+peering, 40 peering, 1 unknown, 100 active+clean; 582 KiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 495 B/s, 12 objects/s recovering 2024-03-26T05:23:29.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:28 smithi132 ceph-mon[33500]: 6.f scrub starts 2024-03-26T05:23:29.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:28 smithi132 ceph-mon[33500]: 6.f scrub ok 2024-03-26T05:23:29.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:28 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2508300802' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T05:23:29.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:28 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T05:23:29.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:28 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-26T05:23:29.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:28 smithi132 ceph-mon[33500]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T05:23:29.514 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T05:23:29.586 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-26T05:23:29.754 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:29.830 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:30.165 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:29 smithi132 ceph-mon[33500]: 6.14 scrub starts 2024-03-26T05:23:30.165 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:29 smithi132 ceph-mon[33500]: 6.14 scrub ok 2024-03-26T05:23:30.165 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:29 smithi132 ceph-mon[33500]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T05:23:30.220 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:29 smithi080 ceph-mon[29691]: 6.14 scrub starts 2024-03-26T05:23:30.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:29 smithi080 ceph-mon[29691]: 6.14 scrub ok 2024-03-26T05:23:30.221 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:29 smithi080 ceph-mon[29691]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T05:23:30.221 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:29 smithi080 ceph-mon[40404]: 6.14 scrub starts 2024-03-26T05:23:30.221 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:29 smithi080 ceph-mon[40404]: 6.14 scrub ok 2024-03-26T05:23:30.221 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:29 smithi080 ceph-mon[40404]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T05:23:31.010 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:30 smithi132 ceph-mon[33500]: 1.5 scrub starts 2024-03-26T05:23:31.010 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:30 smithi132 ceph-mon[33500]: 1.5 scrub ok 2024-03-26T05:23:31.010 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:30 smithi132 ceph-mon[33500]: pgmap v1098: 142 pgs: 1 remapped+peering, 18 peering, 123 active+clean; 582 KiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 507 B/s, 24 objects/s recovering 2024-03-26T05:23:31.092 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:30 smithi080 ceph-mon[29691]: 1.5 scrub starts 2024-03-26T05:23:31.092 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:30 smithi080 ceph-mon[29691]: 1.5 scrub ok 2024-03-26T05:23:31.093 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:30 smithi080 ceph-mon[29691]: pgmap v1098: 142 pgs: 1 remapped+peering, 18 peering, 123 active+clean; 582 KiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 507 B/s, 24 objects/s recovering 2024-03-26T05:23:31.093 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:30 smithi080 ceph-mon[40404]: 1.5 scrub starts 2024-03-26T05:23:31.093 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:30 smithi080 ceph-mon[40404]: 1.5 scrub ok 2024-03-26T05:23:31.093 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:30 smithi080 ceph-mon[40404]: pgmap v1098: 142 pgs: 1 remapped+peering, 18 peering, 123 active+clean; 582 KiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 507 B/s, 24 objects/s recovering 2024-03-26T05:23:31.579 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:31.580 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:23:31.580 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2024-03-26T05:23:31.580 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:31.726 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a: {} 2024-03-26T05:23:31.736 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b: {} 2024-03-26T05:23:31.746 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c: {} 2024-03-26T05:23:31.747 INFO:teuthology.orchestra.run.smithi080.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T05:23:31.747 INFO:teuthology.orchestra.run.smithi080.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T05:23:31.747 INFO:teuthology.orchestra.run.smithi080.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T05:23:32.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[29691]: 5.4 deep-scrub starts 2024-03-26T05:23:32.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[29691]: 5.4 deep-scrub ok 2024-03-26T05:23:32.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[29691]: 4.6 scrub starts 2024-03-26T05:23:32.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[29691]: 4.6 scrub ok 2024-03-26T05:23:32.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[29691]: Health check update: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T05:23:32.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[29691]: 5.19 scrub starts 2024-03-26T05:23:32.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[29691]: 5.19 scrub ok 2024-03-26T05:23:32.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[29691]: 5.d scrub starts 2024-03-26T05:23:32.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[29691]: 5.d scrub ok 2024-03-26T05:23:32.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/754922470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:23:32.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[29691]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T05:23:32.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[40404]: 5.4 deep-scrub starts 2024-03-26T05:23:32.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[40404]: 5.4 deep-scrub ok 2024-03-26T05:23:32.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[40404]: 4.6 scrub starts 2024-03-26T05:23:32.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[40404]: 4.6 scrub ok 2024-03-26T05:23:32.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[40404]: Health check update: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T05:23:32.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[40404]: 5.19 scrub starts 2024-03-26T05:23:32.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[40404]: 5.19 scrub ok 2024-03-26T05:23:32.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[40404]: 5.d scrub starts 2024-03-26T05:23:32.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[40404]: 5.d scrub ok 2024-03-26T05:23:32.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/754922470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:23:32.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:31 smithi080 ceph-mon[40404]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T05:23:32.185 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:31 smithi132 ceph-mon[33500]: 5.4 deep-scrub starts 2024-03-26T05:23:32.185 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:31 smithi132 ceph-mon[33500]: 5.4 deep-scrub ok 2024-03-26T05:23:32.185 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:31 smithi132 ceph-mon[33500]: 4.6 scrub starts 2024-03-26T05:23:32.185 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:31 smithi132 ceph-mon[33500]: 4.6 scrub ok 2024-03-26T05:23:32.185 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:31 smithi132 ceph-mon[33500]: Health check update: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T05:23:32.185 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:31 smithi132 ceph-mon[33500]: 5.19 scrub starts 2024-03-26T05:23:32.185 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:31 smithi132 ceph-mon[33500]: 5.19 scrub ok 2024-03-26T05:23:32.185 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:31 smithi132 ceph-mon[33500]: 5.d scrub starts 2024-03-26T05:23:32.185 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:31 smithi132 ceph-mon[33500]: 5.d scrub ok 2024-03-26T05:23:32.185 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:31 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/754922470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:23:32.186 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:31 smithi132 ceph-mon[33500]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T05:23:32.212 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T05:23:32.452 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:32.471 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 0 1 2024-03-26T05:23:32.712 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:33.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[29691]: 6.1c scrub starts 2024-03-26T05:23:33.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[29691]: 6.1c scrub ok 2024-03-26T05:23:33.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[29691]: 3.2 scrub starts 2024-03-26T05:23:33.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[29691]: 3.2 scrub ok 2024-03-26T05:23:33.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[29691]: 3.4 deep-scrub starts 2024-03-26T05:23:33.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[29691]: 3.4 deep-scrub ok 2024-03-26T05:23:33.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[29691]: pgmap v1100: 142 pgs: 142 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 909 B/s rd, 0 op/s; 128 KiB/s, 21 objects/s recovering 2024-03-26T05:23:33.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[40404]: 6.1c scrub starts 2024-03-26T05:23:33.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[40404]: 6.1c scrub ok 2024-03-26T05:23:33.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[40404]: 3.2 scrub starts 2024-03-26T05:23:33.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[40404]: 3.2 scrub ok 2024-03-26T05:23:33.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[40404]: 3.4 deep-scrub starts 2024-03-26T05:23:33.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[40404]: 3.4 deep-scrub ok 2024-03-26T05:23:33.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:32 smithi080 ceph-mon[40404]: pgmap v1100: 142 pgs: 142 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 909 B/s rd, 0 op/s; 128 KiB/s, 21 objects/s recovering 2024-03-26T05:23:33.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:32 smithi132 ceph-mon[33500]: 6.1c scrub starts 2024-03-26T05:23:33.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:32 smithi132 ceph-mon[33500]: 6.1c scrub ok 2024-03-26T05:23:33.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:32 smithi132 ceph-mon[33500]: 3.2 scrub starts 2024-03-26T05:23:33.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:32 smithi132 ceph-mon[33500]: 3.2 scrub ok 2024-03-26T05:23:33.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:32 smithi132 ceph-mon[33500]: 3.4 deep-scrub starts 2024-03-26T05:23:33.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:32 smithi132 ceph-mon[33500]: 3.4 deep-scrub ok 2024-03-26T05:23:33.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:32 smithi132 ceph-mon[33500]: pgmap v1100: 142 pgs: 142 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 909 B/s rd, 0 op/s; 128 KiB/s, 21 objects/s recovering 2024-03-26T05:23:34.146 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:23:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:23:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:23:34.146 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:33 smithi080 ceph-mon[40404]: 2.0 scrub starts 2024-03-26T05:23:34.146 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:33 smithi080 ceph-mon[40404]: 2.0 scrub ok 2024-03-26T05:23:34.146 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:33 smithi080 ceph-mon[40404]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-26T05:23:34.146 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:33 smithi080 ceph-mon[40404]: Cluster is now healthy 2024-03-26T05:23:34.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:33 smithi080 ceph-mon[29691]: 2.0 scrub starts 2024-03-26T05:23:34.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:33 smithi080 ceph-mon[29691]: 2.0 scrub ok 2024-03-26T05:23:34.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:33 smithi080 ceph-mon[29691]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-26T05:23:34.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:33 smithi080 ceph-mon[29691]: Cluster is now healthy 2024-03-26T05:23:34.188 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:33 smithi132 ceph-mon[33500]: 2.0 scrub starts 2024-03-26T05:23:34.188 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:33 smithi132 ceph-mon[33500]: 2.0 scrub ok 2024-03-26T05:23:34.188 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:33 smithi132 ceph-mon[33500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-26T05:23:34.188 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:33 smithi132 ceph-mon[33500]: Cluster is now healthy 2024-03-26T05:23:34.214 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:34.215 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:23:34.215 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2024-03-26T05:23:34.215 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2024-03-26T05:23:34.215 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:34.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T05:23:34.945 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.0 to 1 (10000) 2024-03-26T05:23:35.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:34 smithi132 ceph-mon[33500]: pgmap v1101: 142 pgs: 142 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 96 KiB/s, 16 objects/s recovering 2024-03-26T05:23:35.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:34 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2931936768' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:23:35.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:34 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:23:35.043 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:35.226 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:34 smithi080 ceph-mon[40404]: pgmap v1101: 142 pgs: 142 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 96 KiB/s, 16 objects/s recovering 2024-03-26T05:23:35.226 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:34 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2931936768' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:23:35.227 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:34 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:23:35.227 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:34 smithi080 ceph-mon[29691]: pgmap v1101: 142 pgs: 142 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 96 KiB/s, 16 objects/s recovering 2024-03-26T05:23:35.227 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:34 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2931936768' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:23:35.227 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:34 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:23:35.639 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-26T05:23:35.882 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:36.165 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:23:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:23:36.166 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:35 smithi132 ceph-mon[33500]: 1.6 scrub starts 2024-03-26T05:23:36.166 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:35 smithi132 ceph-mon[33500]: 1.6 scrub ok 2024-03-26T05:23:36.166 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:35 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:23:36.166 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:35 smithi132 ceph-mon[33500]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T05:23:36.166 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:35 smithi132 ceph-mon[33500]: 6.4 scrub starts 2024-03-26T05:23:36.166 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:35 smithi132 ceph-mon[33500]: 6.4 scrub ok 2024-03-26T05:23:36.249 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:35 smithi080 ceph-mon[29691]: 1.6 scrub starts 2024-03-26T05:23:36.249 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:35 smithi080 ceph-mon[29691]: 1.6 scrub ok 2024-03-26T05:23:36.249 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:35 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:23:36.249 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:35 smithi080 ceph-mon[29691]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T05:23:36.249 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:35 smithi080 ceph-mon[29691]: 6.4 scrub starts 2024-03-26T05:23:36.249 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:35 smithi080 ceph-mon[29691]: 6.4 scrub ok 2024-03-26T05:23:36.250 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:35 smithi080 ceph-mon[40404]: 1.6 scrub starts 2024-03-26T05:23:36.250 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:35 smithi080 ceph-mon[40404]: 1.6 scrub ok 2024-03-26T05:23:36.250 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:35 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:23:36.250 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:35 smithi080 ceph-mon[40404]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T05:23:36.250 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:35 smithi080 ceph-mon[40404]: 6.4 scrub starts 2024-03-26T05:23:36.250 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:35 smithi080 ceph-mon[40404]: 6.4 scrub ok 2024-03-26T05:23:36.702 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:36.703 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:23:36.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:23:36.703 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:23:36.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.703 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:88870 10.c 10:3b52b1dc:::benchmark_data_smithi080_161990_object88869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.703 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.499555+0000", 2024-03-26T05:23:36.703 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.21098597399998, 2024-03-26T05:23:36.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.041748498, 2024-03-26T05:23:36.703 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.703 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 88870 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499555+0000", 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499555+0000", 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499556+0000", 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6819999999999999e-06 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499685+0000", 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000128959 2024-03-26T05:23:36.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499697+0000", 2024-03-26T05:23:36.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1698999999999999e-05 2024-03-26T05:23:36.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499707+0000", 2024-03-26T05:23:36.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0196e-05 2024-03-26T05:23:36.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499835+0000", 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012824099999999999 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499846+0000", 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0518e-05 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.524265+0000", 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.024419097 2024-03-26T05:23:36.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.524372+0000", 2024-03-26T05:23:36.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010672500000000001 2024-03-26T05:23:36.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.524474+0000", 2024-03-26T05:23:36.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000102333 2024-03-26T05:23:36.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.531630+0000", 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0071556759999999997 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.541133+0000", 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0095035629999999996 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.541198+0000", 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4717999999999999e-05 2024-03-26T05:23:36.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.541303+0000", 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000105091 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:88884 10.c 10:309480bd:::benchmark_data_smithi080_161990_object88883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.499725+0000", 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.21081607399998, 2024-03-26T05:23:36.710 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0464261029999999, 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 88884 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499725+0000", 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499725+0000", 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499725+0000", 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9999999999999997e-07 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499763+0000", 2024-03-26T05:23:36.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8012999999999997e-05 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499771+0000", 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8860000000000004e-06 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499778+0000", 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9190000000000002e-06 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499875+0000", 2024-03-26T05:23:36.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6834000000000002e-05 2024-03-26T05:23:36.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499881+0000", 2024-03-26T05:23:36.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3289999999999999e-06 2024-03-26T05:23:36.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.526039+0000", 2024-03-26T05:23:36.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.026158071 2024-03-26T05:23:36.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.526121+0000", 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2013000000000002e-05 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.526218+0000", 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6228000000000001e-05 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.531645+0000", 2024-03-26T05:23:36.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0054277359999999998 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.545995+0000", 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014350125 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.546054+0000", 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8368000000000001e-05 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.546151+0000", 2024-03-26T05:23:36.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.6880999999999996e-05 2024-03-26T05:23:36.717 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.717 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.717 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.718 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:88888 10.c 10:38b61580:::benchmark_data_smithi080_161990_object88887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.718 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.499788+0000", 2024-03-26T05:23:36.718 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.21075262699998, 2024-03-26T05:23:36.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0506100009999999, 2024-03-26T05:23:36.718 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.718 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.718 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 88888 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499788+0000", 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499788+0000", 2024-03-26T05:23:36.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499789+0000", 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3000000000000001e-07 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499823+0000", 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.4734999999999999e-05 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499832+0000", 2024-03-26T05:23:36.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3909999999999998e-06 2024-03-26T05:23:36.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499838+0000", 2024-03-26T05:23:36.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4989999999999999e-06 2024-03-26T05:23:36.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499927+0000", 2024-03-26T05:23:36.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8584000000000004e-05 2024-03-26T05:23:36.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499935+0000", 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.1960000000000008e-06 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.527351+0000", 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.027416229 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.527457+0000", 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010566200000000001 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.527601+0000", 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014397999999999999 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.534415+0000", 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0068144 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.724 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550260+0000", 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.015844703000000002 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550315+0000", 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4672000000000001e-05 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550398+0000", 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3419999999999998e-05 2024-03-26T05:23:36.725 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:88968 10.c 10:354a8553:::benchmark_data_smithi080_161990_object88967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.499848+0000", 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.21069316000001, 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.050705132, 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.726 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 88968 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499848+0000", 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499848+0000", 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499848+0000", 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6700000000000003e-07 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499884+0000", 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.5859999999999999e-05 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499891+0000", 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.2620000000000001e-06 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.728 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.499898+0000", 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5139999999999998e-06 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500604+0000", 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070635100000000003 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500610+0000", 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9889999999999997e-06 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.529640+0000", 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.029030406 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.529771+0000", 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013007000000000001 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.529948+0000", 2024-03-26T05:23:36.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017778499999999999 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.537305+0000", 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0073563600000000002 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550435+0000", 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01313073 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550482+0000", 2024-03-26T05:23:36.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7101000000000002e-05 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550553+0000", 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0136999999999996e-05 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:88980 10.c 10:3acd722e:::benchmark_data_smithi080_161990_object88979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.500013+0000", 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.21052758000002, 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0507306460000001, 2024-03-26T05:23:36.732 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 88980 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500013+0000", 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500013+0000", 2024-03-26T05:23:36.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500014+0000", 2024-03-26T05:23:36.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9599999999999999e-07 2024-03-26T05:23:36.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500078+0000", 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4801999999999994e-05 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500089+0000", 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0553999999999999e-05 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500098+0000", 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5250000000000005e-06 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500639+0000", 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054122800000000004 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500644+0000", 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8849999999999998e-06 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.531693+0000", 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0310490880000001 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.531796+0000", 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000102871 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.531890+0000", 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4137999999999999e-05 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.537315+0000", 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0054251600000000001 2024-03-26T05:23:36.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550605+0000", 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013289643 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550660+0000", 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5047999999999999e-05 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550744+0000", 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4108000000000002e-05 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:88983 10.c 10:30d711d6:::benchmark_data_smithi080_161990_object88982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.500110+0000", 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.21043114899999, 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0508523320000001, 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 88983 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500110+0000", 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500110+0000", 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500110+0000", 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6000000000000004e-07 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500174+0000", 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4079999999999996e-05 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500182+0000", 2024-03-26T05:23:36.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.8199999999999997e-06 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500188+0000", 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4590000000000003e-06 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500669+0000", 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000480612 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500672+0000", 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.3799999999999998e-06 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.533163+0000", 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.032490377 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.533250+0000", 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7307999999999998e-05 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.533342+0000", 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.1576000000000005e-05 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.540031+0000", 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0066894659999999998 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550802+0000", 2024-03-26T05:23:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010771012999999999 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550863+0000", 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0392e-05 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550962+0000", 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9289000000000004e-05 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:89004 10.c 10:33732e96:::benchmark_data_smithi080_161990_object89003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.500199+0000", 2024-03-26T05:23:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.21034181800002, 2024-03-26T05:23:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.052890232, 2024-03-26T05:23:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 89004 2024-03-26T05:23:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500199+0000", 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500199+0000", 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500199+0000", 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5799999999999999e-07 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500269+0000", 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9578000000000004e-05 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500278+0000", 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.9230000000000008e-06 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500284+0000", 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4080000000000003e-06 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500697+0000", 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041314500000000002 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500701+0000", 2024-03-26T05:23:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.1559999999999999e-06 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.534458+0000", 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0337575349999999 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.534562+0000", 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104326 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.750 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.534671+0000", 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108312 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.540038+0000", 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053673289999999997 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.552938+0000", 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.012900221999999999 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.553000+0000", 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1902999999999993e-05 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.553089+0000", 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8837000000000003e-05 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.752 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:89035 10.c 10:34a0c1d9:::benchmark_data_smithi080_161990_object89034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.500353+0000", 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.21018757899998, 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0529700019999999, 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 89035 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.753 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500353+0000", 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500353+0000", 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500354+0000", 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5700000000000002e-07 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500424+0000", 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0347999999999998e-05 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500434+0000", 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0019e-05 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500442+0000", 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0879999999999995e-06 2024-03-26T05:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500730+0000", 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00028816300000000003 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500736+0000", 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8629999999999996e-06 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.535948+0000", 2024-03-26T05:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0352115310000001 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.536028+0000", 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9720000000000002e-05 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.536137+0000", 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108959 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.541316+0000", 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0051794720000000001 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.553145+0000", 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011829409000000001 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.553214+0000", 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8507e-05 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.553323+0000", 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000109166 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:89063 10.c 10:377872d5:::benchmark_data_smithi080_161990_object89062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.500456+0000", 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.21008504500003, 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.054700687, 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 89063 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500456+0000", 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.760 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500456+0000", 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500456+0000", 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2300000000000001e-07 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500520+0000", 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3726999999999995e-05 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.761 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500528+0000", 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.1459999999999996e-06 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500534+0000", 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3249999999999996e-06 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500789+0000", 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00025462299999999999 2024-03-26T05:23:36.762 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.763 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.763 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500796+0000", 2024-03-26T05:23:36.763 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4790000000000001e-06 2024-03-26T05:23:36.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.763 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.763 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.537356+0000", 2024-03-26T05:23:36.763 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.036560766 2024-03-26T05:23:36.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.763 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.537448+0000", 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.1632999999999995e-05 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.537547+0000", 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8936000000000003e-05 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.541323+0000", 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0037763800000000002 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.554976+0000", 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013652932 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.555070+0000", 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4142999999999996e-05 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.555156+0000", 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5974000000000001e-05 2024-03-26T05:23:36.765 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:89096 10.c 10:36b74d22:::benchmark_data_smithi080_161990_object89095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.500557+0000", 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.20998342500002, 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.054745955, 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.766 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 89096 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500557+0000", 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500557+0000", 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500558+0000", 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3900000000000005e-07 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500614+0000", 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6459000000000003e-05 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500623+0000", 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.9479999999999997e-06 2024-03-26T05:23:36.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500630+0000", 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6749999999999996e-06 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500824+0000", 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000194172 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.769 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500830+0000", 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3820000000000001e-06 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.538692+0000", 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0378612249999999 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.538822+0000", 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013014499999999999 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.539001+0000", 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017914800000000001 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.544722+0000", 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0057207229999999996 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.555191+0000", 2024-03-26T05:23:36.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010469823 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.555225+0000", 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.3969e-05 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.555303+0000", 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7746999999999999e-05 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:89108 10.c 10:3cc51dcb:::benchmark_data_smithi080_161990_object89107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.500638+0000", 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.20990233499998, 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0566104279999999, 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 89108 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.773 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.774 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500638+0000", 2024-03-26T05:23:36.774 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.774 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.775 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.775 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.775 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500638+0000", 2024-03-26T05:23:36.775 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.775 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500639+0000", 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7399999999999998e-07 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500689+0000", 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0368999999999998e-05 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500696+0000", 2024-03-26T05:23:36.776 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9840000000000004e-06 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500702+0000", 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6999999999999996e-06 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500858+0000", 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015598599999999999 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.777 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500864+0000", 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7119999999999997e-06 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.540078+0000", 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0392140350000001 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.540151+0000", 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3751999999999995e-05 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.540249+0000", 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7177999999999997e-05 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.544729+0000", 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044806990000000003 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.557093+0000", 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.012363779 2024-03-26T05:23:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.557155+0000", 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.232e-05 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.557249+0000", 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.344e-05 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.780 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:89139 10.c 10:39ffa138:::benchmark_data_smithi080_161990_object89138:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.500729+0000", 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.209811828, 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.061777156, 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 89139 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.781 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500729+0000", 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500729+0000", 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500730+0000", 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3099999999999997e-07 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500792+0000", 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2001999999999994e-05 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500800+0000", 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7229999999999993e-06 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500807+0000", 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5610000000000004e-06 2024-03-26T05:23:36.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500937+0000", 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013038700000000001 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500945+0000", 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0909999999999993e-06 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.784 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.541340+0000", 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0403951339999999 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.541410+0000", 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9184999999999996e-05 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.541499+0000", 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.9530000000000005e-05 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.785 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.546163+0000", 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0046641900000000003 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.562330+0000", 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016166848000000001 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.562400+0000", 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0215000000000005e-05 2024-03-26T05:23:36.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.562506+0000", 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000105559 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:89148 10.c 10:30e2ebe2:::benchmark_data_smithi080_161990_object89147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.500818+0000", 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.20972229199998, 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0630231809999999, 2024-03-26T05:23:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 89148 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500818+0000", 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.788 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500818+0000", 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500819+0000", 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4600000000000005e-07 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500872+0000", 2024-03-26T05:23:36.789 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3325999999999997e-05 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500881+0000", 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6940000000000001e-06 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500888+0000", 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9009999999999997e-06 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.790 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501013+0000", 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000124888 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501019+0000", 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0800000000000002e-06 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.542342+0000", 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0413233959999999 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.542416+0000", 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3707000000000004e-05 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.542509+0000", 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2863999999999997e-05 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.546171+0000", 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0036626359999999999 2024-03-26T05:23:36.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.793 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.793 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.793 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.563678+0000", 2024-03-26T05:23:36.793 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.017506891 2024-03-26T05:23:36.793 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.793 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.793 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.793 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.563744+0000", 2024-03-26T05:23:36.793 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5210999999999996e-05 2024-03-26T05:23:36.793 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.793 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.793 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.563842+0000", 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8041000000000005e-05 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:89164 10.c 10:316dca90:::benchmark_data_smithi080_161990_object89163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.500900+0000", 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.209640256, 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.067039863, 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.794 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 89164 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500900+0000", 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.795 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500900+0000", 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500901+0000", 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7899999999999999e-07 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500962+0000", 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0674999999999998e-05 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500970+0000", 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8230000000000001e-06 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500977+0000", 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.9619999999999996e-06 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501087+0000", 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010915499999999999 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.797 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501090+0000", 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7910000000000002e-06 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.543394+0000", 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.042304004 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.543518+0000", 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012350800000000001 2024-03-26T05:23:36.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.543682+0000", 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000164408 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550975+0000", 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0072924449999999998 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.567761+0000", 2024-03-26T05:23:36.799 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016786005 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.567829+0000", 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7929000000000006e-05 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.567940+0000", 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000111679 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.800 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:89167 10.c 10:33b25437:::benchmark_data_smithi080_161990_object89166:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.500989+0000", 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.20955159599998, 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.071831121, 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 89167 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.801 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500989+0000", 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500989+0000", 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.500990+0000", 2024-03-26T05:23:36.802 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8699999999999995e-07 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501042+0000", 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2581999999999999e-05 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501051+0000", 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.5210000000000002e-06 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.803 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501057+0000", 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7410000000000003e-06 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501163+0000", 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000105515 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501171+0000", 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2169999999999994e-06 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.804 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.544766+0000", 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.043594972 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.544844+0000", 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7579999999999999e-05 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.544952+0000", 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108105 2024-03-26T05:23:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550981+0000", 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0060286269999999999 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.572709+0000", 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.021728713 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.572751+0000", 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2181000000000002e-05 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.572820+0000", 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8780000000000002e-05 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:89201 10.c 10:33c904c8:::benchmark_data_smithi080_161990_object89200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e347)", 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:00.501068+0000", 2024-03-26T05:23:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 456.209472608, 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0718850289999999, 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 89201 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501068+0000", 2024-03-26T05:23:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501068+0000", 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501069+0000", 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7100000000000002e-07 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501120+0000", 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1458000000000001e-05 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501129+0000", 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4710000000000007e-06 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501135+0000", 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3860000000000004e-06 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501190+0000", 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4919000000000001e-05 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:00.501193+0000", 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0730000000000001e-06 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.546203+0000", 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.045009879 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.546276+0000", 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3526e-05 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.546412+0000", 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000135623 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.550988+0000", 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045761420000000001 2024-03-26T05:23:36.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.572853+0000", 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.021864844000000001 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.572885+0000", 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2292999999999997e-05 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:01.572953+0000", 2024-03-26T05:23:36.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7843999999999996e-05 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15657.0:141750 10.11 10:89e18f90:::benchmark_data_smithi080_161990_object30027:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:16:50.720953+0000", 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 405.98958794399999, 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0301739569999999, 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.814 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15657", 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3142946873", 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 141750 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720953+0000", 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720953+0000", 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720959+0000", 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.9719999999999997e-06 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720966+0000", 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5909999999999998e-06 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.720992+0000", 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.5887000000000001e-05 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.721005+0000", 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2723e-05 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.822891+0000", 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.101886511 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:50.822925+0000", 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.3618000000000002e-05 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739358+0000", 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.91643326700000005 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739455+0000", 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7088000000000003e-05 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 0", 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.739595+0000", 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013933199999999999 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.818 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.741609+0000", 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020146919999999998 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.751011+0000", 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0094012499999999999 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.751063+0000", 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2747999999999997e-05 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:16:51.751127+0000", 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3278e-05 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:129377 11.5 11:a322a59d:::benchmark_data_smithi080_189724_object8267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:29.350268+0000", 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 187.360272288, 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.07715155, 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.820 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 129377 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.350268+0000", 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.350268+0000", 2024-03-26T05:23:36.821 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.822 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.822 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.350269+0000", 2024-03-26T05:23:36.822 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.0399999999999996e-07 2024-03-26T05:23:36.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.822 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.822 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.350271+0000", 2024-03-26T05:23:36.822 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.4040000000000002e-06 2024-03-26T05:23:36.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.350278+0000", 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2659999999999998e-06 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.350283+0000", 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.1050000000000003e-06 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.350331+0000", 2024-03-26T05:23:36.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8257000000000002e-05 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.350337+0000", 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2219999999999999e-06 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.315209+0000", 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.96487229900000004 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.315221+0000", 2024-03-26T05:23:36.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1958e-05 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.401542+0000", 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0863209460000001 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.401682+0000", 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000140123 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,5,6", 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.402154+0000", 2024-03-26T05:23:36.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00047152899999999999 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.413702+0000", 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011548001 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.421326+0000", 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076244399999999997 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.422030+0000", 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070334300000000005 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.427216+0000", 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0051858090000000004 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.427305+0000", 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.9027e-05 2024-03-26T05:23:36.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.427420+0000", 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011531699999999999 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:129385 11.5 11:a16971fc:::benchmark_data_smithi080_189724_object8275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:29.358743+0000", 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 187.35179781100001, 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.068897363, 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.828 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 129385 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.358743+0000", 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.358743+0000", 2024-03-26T05:23:36.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.358744+0000", 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.5899999999999995e-07 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.358749+0000", 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5029999999999997e-06 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.358759+0000", 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4110000000000002e-06 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.358765+0000", 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.5350000000000001e-06 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.358809+0000", 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4091999999999997e-05 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.358816+0000", 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3309999999999998e-06 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.315251+0000", 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.95643488899999995 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.315260+0000", 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.9190000000000005e-06 2024-03-26T05:23:36.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.413740+0000", 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0984793230000001 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.413881+0000", 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014142399999999999 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,5,6", 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.414014+0000", 2024-03-26T05:23:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013310199999999999 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.421373+0000", 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0073587230000000002 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.421806+0000", 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000432793 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.422036+0000", 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00023011 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.427458+0000", 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0054225710000000002 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.427535+0000", 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6236000000000007e-05 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.427640+0000", 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000105642 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.15930.0:129438 11.5 11:a7edc901:::benchmark_data_smithi080_189724_object8328:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:20:29.426738+0000", 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 187.28380256700001, 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.0011127740000001, 2024-03-26T05:23:36.836 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.15930", 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/3982408208", 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 129438 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.426738+0000", 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.426738+0000", 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.426739+0000", 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3499999999999996e-07 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.426742+0000", 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7479999999999999e-06 2024-03-26T05:23:36.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.426751+0000", 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8640000000000002e-06 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.426757+0000", 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4179999999999996e-06 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.426799+0000", 2024-03-26T05:23:36.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2530000000000001e-05 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:29.426806+0000", 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2129999999999996e-06 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.315285+0000", 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.888479884 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:30.315295+0000", 2024-03-26T05:23:36.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0001e-05 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.416069+0000", 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1007735350000001 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.418490+0000", 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0024205799999999999 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for subops from 3,5,6", 2024-03-26T05:23:36.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.419065+0000", 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057580399999999996 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "op_commit", 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.422041+0000", 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.002975728 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.425364+0000", 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0033227360000000002 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.425437+0000", 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3090000000000007e-05 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "sub_op_commit_rec", 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.427676+0000", 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0022396069999999998 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.427753+0000", 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.6568000000000003e-05 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:36.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:36.844 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:36.844 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:20:31.427851+0000", 2024-03-26T05:23:36.844 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7832999999999997e-05 2024-03-26T05:23:36.844 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.844 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.844 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.844 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:36.844 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:36.844 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:37.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:36 smithi132 ceph-mon[33500]: 5.e scrub starts 2024-03-26T05:23:37.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:36 smithi132 ceph-mon[33500]: 5.e scrub ok 2024-03-26T05:23:37.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:36 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:37.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:36 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:37.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:36 smithi132 ceph-mon[33500]: pgmap v1103: 142 pgs: 142 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 89 KiB/s, 5 objects/s recovering 2024-03-26T05:23:37.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:36 smithi132 ceph-mon[33500]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T05:23:37.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:36 smithi080 ceph-mon[29691]: 5.e scrub starts 2024-03-26T05:23:37.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:36 smithi080 ceph-mon[29691]: 5.e scrub ok 2024-03-26T05:23:37.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:36 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:37.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:36 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:37.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:36 smithi080 ceph-mon[29691]: pgmap v1103: 142 pgs: 142 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 89 KiB/s, 5 objects/s recovering 2024-03-26T05:23:37.289 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:36 smithi080 ceph-mon[29691]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T05:23:37.289 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:36 smithi080 ceph-mon[40404]: 5.e scrub starts 2024-03-26T05:23:37.290 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:36 smithi080 ceph-mon[40404]: 5.e scrub ok 2024-03-26T05:23:37.290 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:36 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:37.290 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:36 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:37.290 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:36 smithi080 ceph-mon[40404]: pgmap v1103: 142 pgs: 142 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 89 KiB/s, 5 objects/s recovering 2024-03-26T05:23:37.290 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:36 smithi080 ceph-mon[40404]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T05:23:37.621 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T05:23:37.871 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:37.953 INFO:teuthology.orchestra.run.smithi080.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-26T05:23:38.228 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:37 smithi132 ceph-mon[33500]: 4.15 scrub starts 2024-03-26T05:23:38.228 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:37 smithi132 ceph-mon[33500]: 4.15 scrub ok 2024-03-26T05:23:38.228 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:37 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2238239232' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:23:38.228 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:37 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:23:38.320 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:37 smithi080 ceph-mon[40404]: 4.15 scrub starts 2024-03-26T05:23:38.321 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:37 smithi080 ceph-mon[40404]: 4.15 scrub ok 2024-03-26T05:23:38.321 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:37 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2238239232' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:23:38.321 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:37 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:23:38.321 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:37 smithi080 ceph-mon[29691]: 4.15 scrub starts 2024-03-26T05:23:38.321 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:37 smithi080 ceph-mon[29691]: 4.15 scrub ok 2024-03-26T05:23:38.321 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:37 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2238239232' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:23:38.321 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:37 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T05:23:38.673 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T05:23:38.923 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:39.207 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[40404]: 1.9 scrub starts 2024-03-26T05:23:39.207 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[40404]: 1.9 scrub ok 2024-03-26T05:23:39.207 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[40404]: 6.11 scrub starts 2024-03-26T05:23:39.207 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[40404]: 6.11 scrub ok 2024-03-26T05:23:39.207 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:23:39.207 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[40404]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T05:23:39.207 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[40404]: pgmap v1106: 142 pgs: 142 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:39.207 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[40404]: 5.a scrub starts 2024-03-26T05:23:39.208 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[40404]: 5.a scrub ok 2024-03-26T05:23:39.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[29691]: 1.9 scrub starts 2024-03-26T05:23:39.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[29691]: 1.9 scrub ok 2024-03-26T05:23:39.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[29691]: 6.11 scrub starts 2024-03-26T05:23:39.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[29691]: 6.11 scrub ok 2024-03-26T05:23:39.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:23:39.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[29691]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T05:23:39.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[29691]: pgmap v1106: 142 pgs: 142 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:39.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[29691]: 5.a scrub starts 2024-03-26T05:23:39.208 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:38 smithi080 ceph-mon[29691]: 5.a scrub ok 2024-03-26T05:23:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:38 smithi132 ceph-mon[33500]: 1.9 scrub starts 2024-03-26T05:23:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:38 smithi132 ceph-mon[33500]: 1.9 scrub ok 2024-03-26T05:23:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:38 smithi132 ceph-mon[33500]: 6.11 scrub starts 2024-03-26T05:23:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:38 smithi132 ceph-mon[33500]: 6.11 scrub ok 2024-03-26T05:23:39.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:38 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-26T05:23:39.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:38 smithi132 ceph-mon[33500]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T05:23:39.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:38 smithi132 ceph-mon[33500]: pgmap v1106: 142 pgs: 142 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:39.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:38 smithi132 ceph-mon[33500]: 5.a scrub starts 2024-03-26T05:23:39.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:38 smithi132 ceph-mon[33500]: 5.a scrub ok 2024-03-26T05:23:39.571 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:39.571 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:23:39.571 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2024-03-26T05:23:39.571 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:40.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:39 smithi080 ceph-mon[29691]: 5.12 scrub starts 2024-03-26T05:23:40.210 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:39 smithi080 ceph-mon[29691]: 5.12 scrub ok 2024-03-26T05:23:40.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:39 smithi080 ceph-mon[29691]: 5.8 scrub starts 2024-03-26T05:23:40.211 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:39 smithi080 ceph-mon[29691]: 5.8 scrub ok 2024-03-26T05:23:40.211 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:39 smithi080 ceph-mon[40404]: 5.12 scrub starts 2024-03-26T05:23:40.211 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:39 smithi080 ceph-mon[40404]: 5.12 scrub ok 2024-03-26T05:23:40.211 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:39 smithi080 ceph-mon[40404]: 5.8 scrub starts 2024-03-26T05:23:40.211 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:39 smithi080 ceph-mon[40404]: 5.8 scrub ok 2024-03-26T05:23:40.231 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:39 smithi132 ceph-mon[33500]: 5.12 scrub starts 2024-03-26T05:23:40.231 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:39 smithi132 ceph-mon[33500]: 5.12 scrub ok 2024-03-26T05:23:40.231 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:39 smithi132 ceph-mon[33500]: 5.8 scrub starts 2024-03-26T05:23:40.231 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:39 smithi132 ceph-mon[33500]: 5.8 scrub ok 2024-03-26T05:23:40.522 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T05:23:40.767 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:40.969 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T05:23:41.210 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:40 smithi132 ceph-mon[33500]: 4.4 scrub starts 2024-03-26T05:23:41.210 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:40 smithi132 ceph-mon[33500]: 4.4 scrub ok 2024-03-26T05:23:41.210 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:40 smithi132 ceph-mon[33500]: pgmap v1107: 142 pgs: 142 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:41.210 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:40 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4059944057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:23:41.210 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:40 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:23:41.338 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:40 smithi080 ceph-mon[29691]: 4.4 scrub starts 2024-03-26T05:23:41.338 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:40 smithi080 ceph-mon[29691]: 4.4 scrub ok 2024-03-26T05:23:41.339 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:40 smithi080 ceph-mon[29691]: pgmap v1107: 142 pgs: 142 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:41.339 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:40 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4059944057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:23:41.339 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:40 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:23:41.339 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:40 smithi080 ceph-mon[40404]: 4.4 scrub starts 2024-03-26T05:23:41.339 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:40 smithi080 ceph-mon[40404]: 4.4 scrub ok 2024-03-26T05:23:41.339 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:40 smithi080 ceph-mon[40404]: pgmap v1107: 142 pgs: 142 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:41.339 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:40 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4059944057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:23:41.339 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:40 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:23:41.780 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-26T05:23:42.025 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:42.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:23:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:23:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[40404]: 5.7 scrub starts 2024-03-26T05:23:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[40404]: 5.7 scrub ok 2024-03-26T05:23:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[40404]: 3.18 scrub starts 2024-03-26T05:23:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[40404]: 3.18 scrub ok 2024-03-26T05:23:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:23:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[40404]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T05:23:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[40404]: 3.1b scrub starts 2024-03-26T05:23:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[40404]: 3.1b scrub ok 2024-03-26T05:23:42.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[40404]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T05:23:42.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:23:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:23:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[29691]: 5.7 scrub starts 2024-03-26T05:23:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[29691]: 5.7 scrub ok 2024-03-26T05:23:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[29691]: 3.18 scrub starts 2024-03-26T05:23:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[29691]: 3.18 scrub ok 2024-03-26T05:23:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:23:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[29691]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T05:23:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[29691]: 3.1b scrub starts 2024-03-26T05:23:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[29691]: 3.1b scrub ok 2024-03-26T05:23:42.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:41 smithi080 ceph-mon[29691]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T05:23:42.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:41 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:23:42.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:41 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:23:42.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:41 smithi132 ceph-mon[33500]: 5.7 scrub starts 2024-03-26T05:23:42.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:41 smithi132 ceph-mon[33500]: 5.7 scrub ok 2024-03-26T05:23:42.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:41 smithi132 ceph-mon[33500]: 3.18 scrub starts 2024-03-26T05:23:42.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:41 smithi132 ceph-mon[33500]: 3.18 scrub ok 2024-03-26T05:23:42.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:41 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:23:42.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:41 smithi132 ceph-mon[33500]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T05:23:42.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:41 smithi132 ceph-mon[33500]: 3.1b scrub starts 2024-03-26T05:23:42.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:41 smithi132 ceph-mon[33500]: 3.1b scrub ok 2024-03-26T05:23:42.282 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:41 smithi132 ceph-mon[33500]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T05:23:42.491 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:42.491 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2024-03-26T05:23:42.492 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2024-03-26T05:23:42.492 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2024-03-26T05:23:42.492 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:43.079 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:42 smithi132 ceph-mon[33500]: pgmap v1110: 142 pgs: 142 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 929 B/s rd, 0 op/s 2024-03-26T05:23:43.108 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T05:23:43.239 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:42 smithi080 ceph-mon[29691]: pgmap v1110: 142 pgs: 142 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 929 B/s rd, 0 op/s 2024-03-26T05:23:43.239 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:42 smithi080 ceph-mon[40404]: pgmap v1110: 142 pgs: 142 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 929 B/s rd, 0 op/s 2024-03-26T05:23:43.348 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:23:44.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:43 smithi132 ceph-mon[33500]: 4.8 scrub starts 2024-03-26T05:23:44.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:43 smithi132 ceph-mon[33500]: 4.8 scrub ok 2024-03-26T05:23:44.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:43 smithi080 ceph-mon[29691]: 4.8 scrub starts 2024-03-26T05:23:44.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:43 smithi080 ceph-mon[29691]: 4.8 scrub ok 2024-03-26T05:23:44.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:23:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:23:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:23:44.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:43 smithi080 ceph-mon[40404]: 4.8 scrub starts 2024-03-26T05:23:44.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:43 smithi080 ceph-mon[40404]: 4.8 scrub ok 2024-03-26T05:23:44.983 INFO:teuthology.orchestra.run.smithi080.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-26T05:23:45.002 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:44 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1452876662' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:23:45.002 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:44 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:23:45.002 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:44 smithi132 ceph-mon[33500]: pgmap v1111: 142 pgs: 142 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 808 B/s rd, 0 op/s 2024-03-26T05:23:45.003 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-26T05:23:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2024-03-26T05:23:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2024-03-26T05:23:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2024-03-26T05:23:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24356.0:4970 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94789128800384 gen 7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-03-26T05:23:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:23.688082+0000", 2024-03-26T05:23:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.322380157000001, 2024-03-26T05:23:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.34734767300000002, 2024-03-26T05:23:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24356", 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 4970 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.688082+0000", 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.688082+0000", 2024-03-26T05:23:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.688084+0000", 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.279e-06 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.688088+0000", 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8739999999999997e-06 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.688096+0000", 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.9470000000000003e-06 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.688104+0000", 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7520000000000002e-06 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.688165+0000", 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1056000000000006e-05 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for peered", 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.688184+0000", 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.8176999999999999e-05 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.945950+0000", 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.25776644599999998 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for missing object", 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.946085+0000", 2024-03-26T05:23:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013527999999999999 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.949722+0000", 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.003636659 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "waiting for rw locks", 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.949800+0000", 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7803e-05 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950460+0000", 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00066057900000000001 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950522+0000", 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.1267999999999997e-05 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.035430+0000", 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.084908552999999998 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(4.8 504/503 [PushOp(4:10010070:lc::lc.29:head, version: 243'28, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:10010070:lc::lc.29:head@243'28, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:23.928252+0000", 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.08221064, 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0013294089999999999, 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.928252+0000", 2024-03-26T05:23:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.928252+0000", 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.928253+0000", 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4000000000000002e-07 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.928266+0000", 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3302000000000001e-05 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.928288+0000", 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.2446000000000002e-05 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.928305+0000", 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6367000000000001e-05 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.929186+0000", 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0008809 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.929197+0000", 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1765e-05 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.929581+0000", 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000384089 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(5.9 504/503 [PushOp(5:93e5b521:::notify.7:head, version: 75'8, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:93e5b521:::notify.7:head@75'8, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:23.929351+0000", 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.081111988, 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0019129730000000001, 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.929351+0000", 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.929351+0000", 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.929351+0000", 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4200000000000001e-07 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.929363+0000", 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2272e-05 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.929377+0000", 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.413e-05 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.929393+0000", 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5116999999999999e-05 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.931069+0000", 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0016760099999999999 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.931080+0000", 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1403000000000001e-05 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.931264+0000", 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000183599 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "pg_update_log_missing(5.9 epoch 504/503 rep_tid 60955 entries 504'13 (0'0) error 5:93e5b521:::notify.7:head by client.24356.0:4974 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-26T05:23:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:23.934664+0000", 2024-03-26T05:23:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.075798161000002, 2024-03-26T05:23:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088263200000000003, 2024-03-26T05:23:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:23:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.934664+0000", 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.934664+0000", 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.934665+0000", 2024-03-26T05:23:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0500000000000003e-07 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.934674+0000", 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6999999999999997e-06 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.935565+0000", 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00089136200000000004 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.935585+0000", 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.0137e-05 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.942268+0000", 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0066828260000000002 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.943491+0000", 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00122269 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(3.f 504/503 [PushOp(3:f5841e89:::zone_info.deb60e34-e9f6-4dc8-8839-55c267d96e78:head, version: 153'6, data_included: [0~875], data_size: 875, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:f5841e89:::zone_info.deb60e34-e9f6-4dc8-8839-55c267d96e78:head@153'6, size: 875, copy_subset: [(0, 875)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 875, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:23.937987+0000", 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.072475948000001, 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042661440000000004, 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.937987+0000", 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.937987+0000", 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.937987+0000", 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.8299999999999997e-07 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.938002+0000", 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5322e-05 2024-03-26T05:23:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.938020+0000", 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.7708999999999999e-05 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.938039+0000", 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.8482e-05 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.941790+0000", 2024-03-26T05:23:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.003751562 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.941803+0000", 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2805e-05 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.942253+0000", 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00044978100000000001 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(5.1f 504/503 [PushOp(5:f95f44c2:::notify.0:head, version: 75'8, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:f95f44c2:::notify.0:head@75'8, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:23.943072+0000", 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.067390238000002, 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001213797, 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.943072+0000", 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.943072+0000", 2024-03-26T05:23:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.943073+0000", 2024-03-26T05:23:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0600000000000001e-07 2024-03-26T05:23:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.943083+0000", 2024-03-26T05:23:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0723000000000001e-05 2024-03-26T05:23:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.943096+0000", 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2969999999999999e-05 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.943110+0000", 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4070000000000001e-05 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.944139+0000", 2024-03-26T05:23:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010280910000000001 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.944150+0000", 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1191e-05 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.944286+0000", 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000136446 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "pg_update_log_missing(5.0 epoch 504/503 rep_tid 224461 entries 504'11 (0'0) error 5:05bf5b68:::notify.1:head by client.24356.0:4971 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:23.948478+0000", 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.061984677000002, 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0014963209999999999, 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.948478+0000", 2024-03-26T05:23:45.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.948478+0000", 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.948478+0000", 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.5100000000000001e-07 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.948484+0000", 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.0399999999999998e-06 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.948495+0000", 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1017e-05 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.948506+0000", 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0417e-05 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.948704+0000", 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00019856400000000001 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.949974+0000", 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0012700319999999999 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "pg_update_log_missing(5.1f epoch 504/503 rep_tid 196887 entries 504'13 (0'0) error 5:f95f44c2:::notify.0:head by client.24356.0:4972 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:23.949055+0000", 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.061407204000002, 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00115011, 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.949055+0000", 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.949055+0000", 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.949056+0000", 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.03e-07 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.949061+0000", 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3990000000000003e-06 2024-03-26T05:23:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.949072+0000", 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0939e-05 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.949082+0000", 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0169e-05 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.949149+0000", 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6426000000000004e-05 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950206+0000", 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001056874 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "pg_update_log_missing(4.5 epoch 504/503 rep_tid 196888 entries 504'53 (0'0) error 4:a60f9269:lc::lua_package_allowlist:head by client.24356.0:4973 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:23.950615+0000", 2024-03-26T05:23:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.059847429000001, 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00097930799999999991, 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950615+0000", 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950615+0000", 2024-03-26T05:23:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950616+0000", 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7000000000000005e-07 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950624+0000", 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4670000000000004e-06 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950641+0000", 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6592e-05 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950655+0000", 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.3721e-05 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.951008+0000", 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035313800000000002 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.951595+0000", 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058682000000000005 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(4.9 504/503 [PushOp(4:921ecdb1:gc::gc.25:head, version: 188'26, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:921ecdb1:gc::gc.25:head@188'26, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:23.950670+0000", 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.059793092, 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0013070989999999999, 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950670+0000", 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950670+0000", 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950670+0000", 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1699999999999999e-07 2024-03-26T05:23:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950682+0000", 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2397e-05 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950697+0000", 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4586999999999999e-05 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950712+0000", 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4921e-05 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950779+0000", 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7274000000000007e-05 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.950790+0000", 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.1202e-05 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.951977+0000", 2024-03-26T05:23:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011863010000000001 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(4.1a 504/503 [PushOp(4:5c7e8a6d:reshard::reshard.0000000015:head, version: 443'29, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:5c7e8a6d:reshard::reshard.0000000015:head@443'29, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:23.969643+0000", 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.040819457000001, 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010753399999999999, 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.969643+0000", 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.969643+0000", 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.969644+0000", 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3799999999999997e-07 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.969654+0000", 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.7999999999999993e-06 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.969672+0000", 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.842e-05 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.969686+0000", 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4219000000000001e-05 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.969775+0000", 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.8540000000000003e-05 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.969787+0000", 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2279000000000001e-05 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.970719+0000", 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00093154399999999997 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(4.13 504/503 [PushOp(4:cd178ae0:reshard::reshard.0000000014:head, version: 443'26, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:cd178ae0:reshard::reshard.0000000014:head@443'26, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:23.971696+0000", 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.038766710000001, 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00095094199999999998, 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.971696+0000", 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.971696+0000", 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.971696+0000", 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.9900000000000001e-07 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.971709+0000", 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2428e-05 2024-03-26T05:23:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.971724+0000", 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.5145e-05 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.971739+0000", 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.502e-05 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.971807+0000", 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7976000000000001e-05 2024-03-26T05:23:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.971819+0000", 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2249000000000001e-05 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:23.972647+0000", 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082762500000000002 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(osd.4.0:1 4.d e504/503 ondisk, result = 0)", 2024-03-26T05:23:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:24.494188+0000", 2024-03-26T05:23:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 20.516275011000001, 2024-03-26T05:23:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010318250000000001, 2024-03-26T05:23:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494188+0000", 2024-03-26T05:23:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494188+0000", 2024-03-26T05:23:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494188+0000", 2024-03-26T05:23:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6400000000000002e-07 2024-03-26T05:23:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494193+0000", 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.7690000000000004e-06 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494203+0000", 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0249e-05 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494211+0000", 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2600000000000005e-06 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495204+0000", 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00099257699999999991 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495209+0000", 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.2449999999999998e-06 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495219+0000", 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0161e-05 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(osd.4.0:1 4.d e504/503 ondisk, result = 0)", 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:24.494243+0000", 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 20.516219405000001, 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001122273, 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494243+0000", 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494243+0000", 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494244+0000", 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.1800000000000001e-07 2024-03-26T05:23:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494246+0000", 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.7070000000000001e-06 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494253+0000", 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.7129999999999999e-06 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494259+0000", 2024-03-26T05:23:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6620000000000002e-06 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495242+0000", 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00098322800000000005 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495248+0000", 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.6570000000000002e-06 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495365+0000", 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011788799999999999 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(osd.4.0:2 4.d e504/503 ondisk, result = 0)", 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:24.494269+0000", 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 20.516194018, 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011994009999999999, 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494269+0000", 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494269+0000", 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494269+0000", 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.2000000000000001e-07 2024-03-26T05:23:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494271+0000", 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.888e-06 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494276+0000", 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3569999999999997e-06 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494282+0000", 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8889999999999999e-06 2024-03-26T05:23:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495396+0000", 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011135470000000001 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495402+0000", 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.4999999999999996e-06 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495468+0000", 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.6000000000000005e-05 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(osd.4.0:2 4.d e504/503 ondisk, result = 0)", 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:24.494348+0000", 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 20.516114162000001, 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00099489200000000004, 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494348+0000", 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494348+0000", 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494349+0000", 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.2199999999999999e-07 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494352+0000", 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.6840000000000001e-06 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494360+0000", 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3799999999999994e-06 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494365+0000", 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.356e-06 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495322+0000", 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00095692599999999998 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495328+0000", 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.2020000000000001e-06 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495343+0000", 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.4922e-05 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(osd.4.0:3 4.d e504/503 ondisk, result = 0)", 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:24.494382+0000", 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 20.516080447, 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011103689999999999, 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494382+0000", 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494382+0000", 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494382+0000", 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.0600000000000001e-07 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494385+0000", 2024-03-26T05:23:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.452e-06 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494392+0000", 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.3679999999999996e-06 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494399+0000", 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.421e-06 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495481+0000", 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001082324 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495485+0000", 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.0210000000000001e-06 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495493+0000", 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4769999999999997e-06 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(osd.4.0:3 4.d e504/503 ondisk, result = 0)", 2024-03-26T05:23:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:24.494423+0000", 2024-03-26T05:23:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 20.516039376999998, 2024-03-26T05:23:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0011398720000000001, 2024-03-26T05:23:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494423+0000", 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494423+0000", 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494424+0000", 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.3799999999999998e-07 2024-03-26T05:23:45.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494426+0000", 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 2.4159999999999998e-06 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494439+0000", 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.2634e-05 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494444+0000", 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8479999999999997e-06 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495515+0000", 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010702859999999999 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495519+0000", 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.7620000000000001e-06 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495563+0000", 2024-03-26T05:23:45.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.4588000000000001e-05 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop_reply(osd.4.0:4 4.d e504/503 ondisk, result = 0)", 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:24.494644+0000", 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 20.515818865, 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010068950000000001, 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494644+0000", 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494644+0000", 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494644+0000", 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.51e-07 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494647+0000", 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2349999999999999e-06 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494655+0000", 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.4549999999999998e-06 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.494662+0000", 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.0280000000000003e-06 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495608+0000", 2024-03-26T05:23:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00094566400000000003 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495611+0000", 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.2830000000000002e-06 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:24.495651+0000", 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.9779e-05 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.24356.0:4988 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94789128800384 gen 8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2024-03-26T05:23:28.688547+0000", 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 16.321916026, 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001081594, 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.24356", 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.80:0/1545910408", 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 4988 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2024-03-26T05:23:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:28.688547+0000", 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:28.688547+0000", 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:28.688547+0000", 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 7.7700000000000004e-07 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:28.688552+0000", 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 4.3610000000000003e-06 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:28.688562+0000", 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.0754000000000001e-05 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:28.688571+0000", 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.1340000000000004e-06 2024-03-26T05:23:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:28.688609+0000", 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 3.8404e-05 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:28.688663+0000", 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.3874e-05 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2024-03-26T05:23:28.689628+0000", 2024-03-26T05:23:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096529000000000005 2024-03-26T05:23:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-26T05:23:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-26T05:23:45.086 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-26T05:23:45.266 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:44 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1452876662' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:23:45.266 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:44 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:23:45.266 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:44 smithi080 ceph-mon[40404]: pgmap v1111: 142 pgs: 142 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 808 B/s rd, 0 op/s 2024-03-26T05:23:45.266 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:44 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1452876662' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:23:45.266 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:44 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T05:23:45.266 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:44 smithi080 ceph-mon[29691]: pgmap v1111: 142 pgs: 142 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 808 B/s rd, 0 op/s 2024-03-26T05:23:45.600 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T05:23:45.861 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:46.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:45 smithi132 ceph-mon[33500]: 3.c deep-scrub starts 2024-03-26T05:23:46.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:45 smithi132 ceph-mon[33500]: 3.c deep-scrub ok 2024-03-26T05:23:46.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:45 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:23:46.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:45 smithi132 ceph-mon[33500]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T05:23:46.320 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:45 smithi080 ceph-mon[40404]: 3.c deep-scrub starts 2024-03-26T05:23:46.321 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:45 smithi080 ceph-mon[40404]: 3.c deep-scrub ok 2024-03-26T05:23:46.321 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:45 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:23:46.321 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:45 smithi080 ceph-mon[40404]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T05:23:46.321 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:45 smithi080 ceph-mon[29691]: 3.c deep-scrub starts 2024-03-26T05:23:46.321 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:45 smithi080 ceph-mon[29691]: 3.c deep-scrub ok 2024-03-26T05:23:46.321 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:45 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-26T05:23:46.321 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:45 smithi080 ceph-mon[29691]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T05:23:46.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:23:46 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:23:47.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[40404]: 4.10 scrub starts 2024-03-26T05:23:47.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[40404]: 4.10 scrub ok 2024-03-26T05:23:47.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:47.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[40404]: pgmap v1113: 142 pgs: 142 active+clean; 582 KiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:23:47.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[40404]: 4.12 scrub starts 2024-03-26T05:23:47.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[40404]: 4.12 scrub ok 2024-03-26T05:23:47.229 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[40404]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T05:23:47.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[29691]: 4.10 scrub starts 2024-03-26T05:23:47.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[29691]: 4.10 scrub ok 2024-03-26T05:23:47.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:47.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[29691]: pgmap v1113: 142 pgs: 142 active+clean; 582 KiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:23:47.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[29691]: 4.12 scrub starts 2024-03-26T05:23:47.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[29691]: 4.12 scrub ok 2024-03-26T05:23:47.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:46 smithi080 ceph-mon[29691]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T05:23:47.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:46 smithi132 ceph-mon[33500]: 4.10 scrub starts 2024-03-26T05:23:47.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:46 smithi132 ceph-mon[33500]: 4.10 scrub ok 2024-03-26T05:23:47.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:46 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:47.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:46 smithi132 ceph-mon[33500]: pgmap v1113: 142 pgs: 142 active+clean; 582 KiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:23:47.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:46 smithi132 ceph-mon[33500]: 4.12 scrub starts 2024-03-26T05:23:47.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:46 smithi132 ceph-mon[33500]: 4.12 scrub ok 2024-03-26T05:23:47.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:46 smithi132 ceph-mon[33500]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T05:23:47.989 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T05:23:48.283 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[40404]: 6.19 scrub starts 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[40404]: 6.19 scrub ok 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[40404]: 6.10 scrub starts 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[40404]: 6.10 scrub ok 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[40404]: 5.c scrub starts 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[40404]: 5.c scrub ok 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1663864787' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[29691]: 6.19 scrub starts 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[29691]: 6.19 scrub ok 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[29691]: 6.10 scrub starts 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[29691]: 6.10 scrub ok 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[29691]: 5.c scrub starts 2024-03-26T05:23:48.284 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[29691]: 5.c scrub ok 2024-03-26T05:23:48.285 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1663864787' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:23:48.285 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:47 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:23:48.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:47 smithi132 ceph-mon[33500]: 6.19 scrub starts 2024-03-26T05:23:48.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:47 smithi132 ceph-mon[33500]: 6.19 scrub ok 2024-03-26T05:23:48.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:47 smithi132 ceph-mon[33500]: 6.10 scrub starts 2024-03-26T05:23:48.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:47 smithi132 ceph-mon[33500]: 6.10 scrub ok 2024-03-26T05:23:48.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:47 smithi132 ceph-mon[33500]: 5.c scrub starts 2024-03-26T05:23:48.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:47 smithi132 ceph-mon[33500]: 5.c scrub ok 2024-03-26T05:23:48.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:47 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1663864787' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:23:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:47 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:23:48.725 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-26T05:23:48.971 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:49.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:48 smithi132 ceph-mon[33500]: 5.1 scrub starts 2024-03-26T05:23:49.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:48 smithi132 ceph-mon[33500]: 5.1 scrub ok 2024-03-26T05:23:49.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:48 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:23:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:48 smithi132 ceph-mon[33500]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T05:23:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:48 smithi132 ceph-mon[33500]: pgmap v1116: 142 pgs: 142 active+clean; 582 KiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:49.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:48 smithi080 ceph-mon[29691]: 5.1 scrub starts 2024-03-26T05:23:49.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:48 smithi080 ceph-mon[29691]: 5.1 scrub ok 2024-03-26T05:23:49.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:48 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:23:49.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:48 smithi080 ceph-mon[29691]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T05:23:49.312 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:48 smithi080 ceph-mon[29691]: pgmap v1116: 142 pgs: 142 active+clean; 582 KiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:49.312 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:48 smithi080 ceph-mon[40404]: 5.1 scrub starts 2024-03-26T05:23:49.312 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:48 smithi080 ceph-mon[40404]: 5.1 scrub ok 2024-03-26T05:23:49.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:48 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:23:49.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:48 smithi080 ceph-mon[40404]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T05:23:49.313 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:48 smithi080 ceph-mon[40404]: pgmap v1116: 142 pgs: 142 active+clean; 582 KiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:50.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:50 smithi080 ceph-mon[29691]: 1.8 scrub starts 2024-03-26T05:23:50.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:50 smithi080 ceph-mon[29691]: 1.8 scrub ok 2024-03-26T05:23:50.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:50 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:23:50.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:50 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:23:50.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:50 smithi080 ceph-mon[29691]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T05:23:50.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:50 smithi080 ceph-mon[29691]: 6.1d scrub starts 2024-03-26T05:23:50.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:50 smithi080 ceph-mon[40404]: 1.8 scrub starts 2024-03-26T05:23:50.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:50 smithi080 ceph-mon[40404]: 1.8 scrub ok 2024-03-26T05:23:50.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:50 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:23:50.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:50 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:23:50.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:50 smithi080 ceph-mon[40404]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T05:23:50.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:50 smithi080 ceph-mon[40404]: 6.1d scrub starts 2024-03-26T05:23:50.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:50 smithi132 ceph-mon[33500]: 1.8 scrub starts 2024-03-26T05:23:50.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:50 smithi132 ceph-mon[33500]: 1.8 scrub ok 2024-03-26T05:23:50.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:50 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:23:50.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:50 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:23:50.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:50 smithi132 ceph-mon[33500]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T05:23:50.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:50 smithi132 ceph-mon[33500]: 6.1d scrub starts 2024-03-26T05:23:51.014 INFO:teuthology.orchestra.run.smithi080.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-26T05:23:51.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:51 smithi132 ceph-mon[33500]: 6.17 scrub starts 2024-03-26T05:23:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:51 smithi132 ceph-mon[33500]: 6.1d scrub ok 2024-03-26T05:23:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:51 smithi132 ceph-mon[33500]: 6.17 scrub ok 2024-03-26T05:23:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:51 smithi132 ceph-mon[33500]: 4.19 deep-scrub starts 2024-03-26T05:23:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:51 smithi132 ceph-mon[33500]: 4.19 deep-scrub ok 2024-03-26T05:23:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:51 smithi132 ceph-mon[33500]: 4.1f scrub starts 2024-03-26T05:23:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:51 smithi132 ceph-mon[33500]: 4.1f scrub ok 2024-03-26T05:23:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:51 smithi132 ceph-mon[33500]: pgmap v1118: 142 pgs: 142 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 965 B/s rd, 0 op/s 2024-03-26T05:23:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:51 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1416631473' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:23:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:51 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:23:51.316 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[40404]: 6.17 scrub starts 2024-03-26T05:23:51.316 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[40404]: 6.1d scrub ok 2024-03-26T05:23:51.316 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[40404]: 6.17 scrub ok 2024-03-26T05:23:51.316 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[40404]: 4.19 deep-scrub starts 2024-03-26T05:23:51.316 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[40404]: 4.19 deep-scrub ok 2024-03-26T05:23:51.316 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[40404]: 4.1f scrub starts 2024-03-26T05:23:51.316 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[40404]: 4.1f scrub ok 2024-03-26T05:23:51.317 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[40404]: pgmap v1118: 142 pgs: 142 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 965 B/s rd, 0 op/s 2024-03-26T05:23:51.317 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1416631473' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:23:51.317 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:23:51.317 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[29691]: 6.17 scrub starts 2024-03-26T05:23:51.317 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[29691]: 6.1d scrub ok 2024-03-26T05:23:51.317 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[29691]: 6.17 scrub ok 2024-03-26T05:23:51.317 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[29691]: 4.19 deep-scrub starts 2024-03-26T05:23:51.317 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[29691]: 4.19 deep-scrub ok 2024-03-26T05:23:51.317 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[29691]: 4.1f scrub starts 2024-03-26T05:23:51.317 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[29691]: 4.1f scrub ok 2024-03-26T05:23:51.317 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[29691]: pgmap v1118: 142 pgs: 142 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 965 B/s rd, 0 op/s 2024-03-26T05:23:51.317 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1416631473' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:23:51.318 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:51 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T05:23:51.619 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 3 1 2024-03-26T05:23:51.864 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:52.278 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[29691]: 3.16 deep-scrub starts 2024-03-26T05:23:52.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[29691]: 3.16 deep-scrub ok 2024-03-26T05:23:52.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[29691]: 3.14 scrub starts 2024-03-26T05:23:52.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[29691]: 3.14 scrub ok 2024-03-26T05:23:52.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[29691]: 4.3 scrub starts 2024-03-26T05:23:52.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[29691]: 4.3 scrub ok 2024-03-26T05:23:52.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:23:52.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[29691]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T05:23:52.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:52.279 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[29691]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T05:23:52.279 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[40404]: 3.16 deep-scrub starts 2024-03-26T05:23:52.279 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[40404]: 3.16 deep-scrub ok 2024-03-26T05:23:52.280 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[40404]: 3.14 scrub starts 2024-03-26T05:23:52.280 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[40404]: 3.14 scrub ok 2024-03-26T05:23:52.280 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[40404]: 4.3 scrub starts 2024-03-26T05:23:52.280 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[40404]: 4.3 scrub ok 2024-03-26T05:23:52.280 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:23:52.280 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[40404]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T05:23:52.281 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:52.281 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:52 smithi080 ceph-mon[40404]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T05:23:52.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:52 smithi132 ceph-mon[33500]: 3.16 deep-scrub starts 2024-03-26T05:23:52.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:52 smithi132 ceph-mon[33500]: 3.16 deep-scrub ok 2024-03-26T05:23:52.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:52 smithi132 ceph-mon[33500]: 3.14 scrub starts 2024-03-26T05:23:52.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:52 smithi132 ceph-mon[33500]: 3.14 scrub ok 2024-03-26T05:23:52.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:52 smithi132 ceph-mon[33500]: 4.3 scrub starts 2024-03-26T05:23:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:52 smithi132 ceph-mon[33500]: 4.3 scrub ok 2024-03-26T05:23:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:52 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-26T05:23:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:52 smithi132 ceph-mon[33500]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T05:23:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:52 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:52 smithi132 ceph-mon[33500]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T05:23:53.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:53 smithi132 ceph-mon[33500]: 3.8 scrub starts 2024-03-26T05:23:53.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:53 smithi132 ceph-mon[33500]: 3.8 scrub ok 2024-03-26T05:23:53.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:53 smithi132 ceph-mon[33500]: 5.1f scrub starts 2024-03-26T05:23:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:53 smithi132 ceph-mon[33500]: 5.1f scrub ok 2024-03-26T05:23:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:53 smithi132 ceph-mon[33500]: 5.5 scrub starts 2024-03-26T05:23:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:53 smithi132 ceph-mon[33500]: 5.5 scrub ok 2024-03-26T05:23:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:53 smithi132 ceph-mon[33500]: pgmap v1121: 142 pgs: 142 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:53.345 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[40404]: 3.8 scrub starts 2024-03-26T05:23:53.345 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[40404]: 3.8 scrub ok 2024-03-26T05:23:53.345 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[40404]: 5.1f scrub starts 2024-03-26T05:23:53.345 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[40404]: 5.1f scrub ok 2024-03-26T05:23:53.345 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[40404]: 5.5 scrub starts 2024-03-26T05:23:53.345 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[40404]: 5.5 scrub ok 2024-03-26T05:23:53.345 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[40404]: pgmap v1121: 142 pgs: 142 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:53.346 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[29691]: 3.8 scrub starts 2024-03-26T05:23:53.346 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[29691]: 3.8 scrub ok 2024-03-26T05:23:53.346 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[29691]: 5.1f scrub starts 2024-03-26T05:23:53.346 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[29691]: 5.1f scrub ok 2024-03-26T05:23:53.346 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[29691]: 5.5 scrub starts 2024-03-26T05:23:53.346 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[29691]: 5.5 scrub ok 2024-03-26T05:23:53.346 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:53 smithi080 ceph-mon[29691]: pgmap v1121: 142 pgs: 142 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:54.024 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.3 to 1 (10000) 2024-03-26T05:23:54.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:54 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/710560918' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:23:54.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:54 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:23:54.340 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:54 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/710560918' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:23:54.340 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:54 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:23:54.340 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:54 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/710560918' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:23:54.341 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:54 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:23:54.341 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:23:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:23:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:23:54.736 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-26T05:23:54.976 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:55.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:55 smithi132 ceph-mon[33500]: 3.1a scrub starts 2024-03-26T05:23:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:55 smithi132 ceph-mon[33500]: 3.1a scrub ok 2024-03-26T05:23:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:55 smithi132 ceph-mon[33500]: 5.3 scrub starts 2024-03-26T05:23:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:55 smithi132 ceph-mon[33500]: 5.3 scrub ok 2024-03-26T05:23:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:55 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:23:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:55 smithi132 ceph-mon[33500]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T05:23:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:55 smithi132 ceph-mon[33500]: pgmap v1123: 142 pgs: 142 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-03-26T05:23:55.354 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[40404]: 3.1a scrub starts 2024-03-26T05:23:55.354 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[40404]: 3.1a scrub ok 2024-03-26T05:23:55.354 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[40404]: 5.3 scrub starts 2024-03-26T05:23:55.354 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[40404]: 5.3 scrub ok 2024-03-26T05:23:55.354 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:23:55.355 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[40404]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T05:23:55.355 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[40404]: pgmap v1123: 142 pgs: 142 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-03-26T05:23:55.355 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[29691]: 3.1a scrub starts 2024-03-26T05:23:55.355 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[29691]: 3.1a scrub ok 2024-03-26T05:23:55.355 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[29691]: 5.3 scrub starts 2024-03-26T05:23:55.355 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[29691]: 5.3 scrub ok 2024-03-26T05:23:55.355 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:23:55.355 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[29691]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T05:23:55.355 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:55 smithi080 ceph-mon[29691]: pgmap v1123: 142 pgs: 142 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-03-26T05:23:56.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:56 smithi080 ceph-mon[40404]: 1.a scrub starts 2024-03-26T05:23:56.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:56 smithi080 ceph-mon[40404]: 1.a scrub ok 2024-03-26T05:23:56.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:56 smithi080 ceph-mon[40404]: 3.6 deep-scrub starts 2024-03-26T05:23:56.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:56 smithi080 ceph-mon[40404]: 3.6 deep-scrub ok 2024-03-26T05:23:56.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:56 smithi080 ceph-mon[40404]: 3.19 scrub starts 2024-03-26T05:23:56.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:56 smithi080 ceph-mon[40404]: 3.19 scrub ok 2024-03-26T05:23:56.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:56 smithi080 ceph-mon[29691]: 1.a scrub starts 2024-03-26T05:23:56.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:56 smithi080 ceph-mon[29691]: 1.a scrub ok 2024-03-26T05:23:56.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:56 smithi080 ceph-mon[29691]: 3.6 deep-scrub starts 2024-03-26T05:23:56.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:56 smithi080 ceph-mon[29691]: 3.6 deep-scrub ok 2024-03-26T05:23:56.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:56 smithi080 ceph-mon[29691]: 3.19 scrub starts 2024-03-26T05:23:56.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:56 smithi080 ceph-mon[29691]: 3.19 scrub ok 2024-03-26T05:23:56.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:23:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:23:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:56 smithi132 ceph-mon[33500]: 1.a scrub starts 2024-03-26T05:23:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:56 smithi132 ceph-mon[33500]: 1.a scrub ok 2024-03-26T05:23:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:56 smithi132 ceph-mon[33500]: 3.6 deep-scrub starts 2024-03-26T05:23:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:56 smithi132 ceph-mon[33500]: 3.6 deep-scrub ok 2024-03-26T05:23:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:56 smithi132 ceph-mon[33500]: 3.19 scrub starts 2024-03-26T05:23:56.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:56 smithi132 ceph-mon[33500]: 3.19 scrub ok 2024-03-26T05:23:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:57 smithi132 ceph-mon[33500]: 3.1 scrub starts 2024-03-26T05:23:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:57 smithi132 ceph-mon[33500]: 3.1 scrub ok 2024-03-26T05:23:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:57 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:57 smithi132 ceph-mon[33500]: pgmap v1124: 142 pgs: 142 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:23:57.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:57 smithi132 ceph-mon[33500]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T05:23:57.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:57 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3783744125' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:23:57.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:57 smithi080 ceph-mon[40404]: 3.1 scrub starts 2024-03-26T05:23:57.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:57 smithi080 ceph-mon[40404]: 3.1 scrub ok 2024-03-26T05:23:57.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:57 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:57.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:57 smithi080 ceph-mon[40404]: pgmap v1124: 142 pgs: 142 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:23:57.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:57 smithi080 ceph-mon[40404]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T05:23:57.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:57 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3783744125' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:23:57.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:57 smithi080 ceph-mon[29691]: 3.1 scrub starts 2024-03-26T05:23:57.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:57 smithi080 ceph-mon[29691]: 3.1 scrub ok 2024-03-26T05:23:57.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:57 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:57.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:57 smithi080 ceph-mon[29691]: pgmap v1124: 142 pgs: 142 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:23:57.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:57 smithi080 ceph-mon[29691]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T05:23:57.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:57 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3783744125' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T05:23:57.788 INFO:teuthology.orchestra.run.smithi080.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-26T05:23:58.365 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[40404]: 4.9 deep-scrub starts 2024-03-26T05:23:58.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[40404]: 4.9 deep-scrub ok 2024-03-26T05:23:58.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[40404]: 1.2 scrub starts 2024-03-26T05:23:58.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[40404]: 1.2 scrub ok 2024-03-26T05:23:58.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:23:58.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[40404]: 4.f scrub starts 2024-03-26T05:23:58.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[40404]: 4.f scrub ok 2024-03-26T05:23:58.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:23:58.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:23:58.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:23:58.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3783744125' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:23:58.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[40404]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T05:23:58.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[29691]: 4.9 deep-scrub starts 2024-03-26T05:23:58.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[29691]: 4.9 deep-scrub ok 2024-03-26T05:23:58.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[29691]: 1.2 scrub starts 2024-03-26T05:23:58.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[29691]: 1.2 scrub ok 2024-03-26T05:23:58.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:23:58.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[29691]: 4.f scrub starts 2024-03-26T05:23:58.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[29691]: 4.f scrub ok 2024-03-26T05:23:58.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:23:58.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:23:58.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:23:58.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3783744125' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:23:58.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:58 smithi080 ceph-mon[29691]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T05:23:58.537 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 5 1 2024-03-26T05:23:58.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:58 smithi132 ceph-mon[33500]: 4.9 deep-scrub starts 2024-03-26T05:23:58.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:58 smithi132 ceph-mon[33500]: 4.9 deep-scrub ok 2024-03-26T05:23:58.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:58 smithi132 ceph-mon[33500]: 1.2 scrub starts 2024-03-26T05:23:58.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:58 smithi132 ceph-mon[33500]: 1.2 scrub ok 2024-03-26T05:23:58.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:58 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:23:58.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:58 smithi132 ceph-mon[33500]: 4.f scrub starts 2024-03-26T05:23:58.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:58 smithi132 ceph-mon[33500]: 4.f scrub ok 2024-03-26T05:23:58.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:58 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:23:58.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:58 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:23:58.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:58 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:23:58.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3783744125' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T05:23:58.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:58 smithi132 ceph-mon[33500]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T05:23:58.779 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:59.213 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[40404]: 1.1 scrub starts 2024-03-26T05:23:59.214 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[40404]: 1.1 scrub ok 2024-03-26T05:23:59.214 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[40404]: 4.c deep-scrub starts 2024-03-26T05:23:59.214 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[40404]: 4.c deep-scrub ok 2024-03-26T05:23:59.214 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[40404]: 3.11 scrub starts 2024-03-26T05:23:59.214 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[40404]: 3.11 scrub ok 2024-03-26T05:23:59.214 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[40404]: 3.17 scrub starts 2024-03-26T05:23:59.214 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[40404]: 3.17 scrub ok 2024-03-26T05:23:59.214 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[40404]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:59.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[29691]: 1.1 scrub starts 2024-03-26T05:23:59.214 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[29691]: 1.1 scrub ok 2024-03-26T05:23:59.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[29691]: 4.c deep-scrub starts 2024-03-26T05:23:59.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[29691]: 4.c deep-scrub ok 2024-03-26T05:23:59.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[29691]: 3.11 scrub starts 2024-03-26T05:23:59.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[29691]: 3.11 scrub ok 2024-03-26T05:23:59.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[29691]: 3.17 scrub starts 2024-03-26T05:23:59.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[29691]: 3.17 scrub ok 2024-03-26T05:23:59.215 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:23:59 smithi080 ceph-mon[29691]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:59.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:59 smithi132 ceph-mon[33500]: 1.1 scrub starts 2024-03-26T05:23:59.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:59 smithi132 ceph-mon[33500]: 1.1 scrub ok 2024-03-26T05:23:59.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:59 smithi132 ceph-mon[33500]: 4.c deep-scrub starts 2024-03-26T05:23:59.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:59 smithi132 ceph-mon[33500]: 4.c deep-scrub ok 2024-03-26T05:23:59.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:59 smithi132 ceph-mon[33500]: 3.11 scrub starts 2024-03-26T05:23:59.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:59 smithi132 ceph-mon[33500]: 3.11 scrub ok 2024-03-26T05:23:59.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:59 smithi132 ceph-mon[33500]: 3.17 scrub starts 2024-03-26T05:23:59.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:59 smithi132 ceph-mon[33500]: 3.17 scrub ok 2024-03-26T05:23:59.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:23:59 smithi132 ceph-mon[33500]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:00.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:00 smithi080 ceph-mon[29691]: 6.15 scrub starts 2024-03-26T05:24:00.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:00 smithi080 ceph-mon[29691]: 6.15 scrub ok 2024-03-26T05:24:00.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:00 smithi080 ceph-mon[29691]: 3.1f scrub starts 2024-03-26T05:24:00.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:00 smithi080 ceph-mon[29691]: 3.1f scrub ok 2024-03-26T05:24:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:00 smithi080 ceph-mon[40404]: 6.15 scrub starts 2024-03-26T05:24:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:00 smithi080 ceph-mon[40404]: 6.15 scrub ok 2024-03-26T05:24:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:00 smithi080 ceph-mon[40404]: 3.1f scrub starts 2024-03-26T05:24:00.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:00 smithi080 ceph-mon[40404]: 3.1f scrub ok 2024-03-26T05:24:00.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:00 smithi132 ceph-mon[33500]: 6.15 scrub starts 2024-03-26T05:24:00.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:00 smithi132 ceph-mon[33500]: 6.15 scrub ok 2024-03-26T05:24:00.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:00 smithi132 ceph-mon[33500]: 3.1f scrub starts 2024-03-26T05:24:00.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:00 smithi132 ceph-mon[33500]: 3.1f scrub ok 2024-03-26T05:24:01.053 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.5 to 1 (10000) 2024-03-26T05:24:01.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[40404]: 6.1a scrub starts 2024-03-26T05:24:01.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[40404]: 6.1a scrub ok 2024-03-26T05:24:01.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[40404]: 5.11 scrub starts 2024-03-26T05:24:01.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[40404]: 5.11 scrub ok 2024-03-26T05:24:01.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[40404]: 4.b scrub starts 2024-03-26T05:24:01.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[40404]: 4.b scrub ok 2024-03-26T05:24:01.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[40404]: pgmap v1128: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 816 B/s rd, 0 op/s 2024-03-26T05:24:01.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1089261499' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:24:01.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[40404]: 4.14 scrub starts 2024-03-26T05:24:01.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[40404]: 4.14 scrub ok 2024-03-26T05:24:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[29691]: 6.1a scrub starts 2024-03-26T05:24:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[29691]: 6.1a scrub ok 2024-03-26T05:24:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[29691]: 5.11 scrub starts 2024-03-26T05:24:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[29691]: 5.11 scrub ok 2024-03-26T05:24:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[29691]: 4.b scrub starts 2024-03-26T05:24:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[29691]: 4.b scrub ok 2024-03-26T05:24:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[29691]: pgmap v1128: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 816 B/s rd, 0 op/s 2024-03-26T05:24:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1089261499' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:24:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[29691]: 4.14 scrub starts 2024-03-26T05:24:01.367 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:01 smithi080 ceph-mon[29691]: 4.14 scrub ok 2024-03-26T05:24:01.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:01 smithi132 ceph-mon[33500]: 6.1a scrub starts 2024-03-26T05:24:01.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:01 smithi132 ceph-mon[33500]: 6.1a scrub ok 2024-03-26T05:24:01.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:01 smithi132 ceph-mon[33500]: 5.11 scrub starts 2024-03-26T05:24:01.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:01 smithi132 ceph-mon[33500]: 5.11 scrub ok 2024-03-26T05:24:01.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:01 smithi132 ceph-mon[33500]: 4.b scrub starts 2024-03-26T05:24:01.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:01 smithi132 ceph-mon[33500]: 4.b scrub ok 2024-03-26T05:24:01.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:01 smithi132 ceph-mon[33500]: pgmap v1128: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 816 B/s rd, 0 op/s 2024-03-26T05:24:01.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:01 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1089261499' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:24:01.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:01 smithi132 ceph-mon[33500]: 4.14 scrub starts 2024-03-26T05:24:01.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:01 smithi132 ceph-mon[33500]: 4.14 scrub ok 2024-03-26T05:24:01.812 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-26T05:24:02.053 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:02.322 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:02 smithi080 ceph-mon[40404]: 3.3 deep-scrub starts 2024-03-26T05:24:02.323 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:02 smithi080 ceph-mon[40404]: 3.3 deep-scrub ok 2024-03-26T05:24:02.323 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:02 smithi080 ceph-mon[40404]: 3.13 scrub starts 2024-03-26T05:24:02.323 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:02 smithi080 ceph-mon[40404]: 3.13 scrub ok 2024-03-26T05:24:02.323 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:02 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1089261499' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T05:24:02.323 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:02 smithi080 ceph-mon[40404]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T05:24:02.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:02 smithi080 ceph-mon[29691]: 3.3 deep-scrub starts 2024-03-26T05:24:02.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:02 smithi080 ceph-mon[29691]: 3.3 deep-scrub ok 2024-03-26T05:24:02.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:02 smithi080 ceph-mon[29691]: 3.13 scrub starts 2024-03-26T05:24:02.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:02 smithi080 ceph-mon[29691]: 3.13 scrub ok 2024-03-26T05:24:02.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:02 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1089261499' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T05:24:02.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:02 smithi080 ceph-mon[29691]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T05:24:02.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:02 smithi132 ceph-mon[33500]: 3.3 deep-scrub starts 2024-03-26T05:24:02.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:02 smithi132 ceph-mon[33500]: 3.3 deep-scrub ok 2024-03-26T05:24:02.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:02 smithi132 ceph-mon[33500]: 3.13 scrub starts 2024-03-26T05:24:02.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:02 smithi132 ceph-mon[33500]: 3.13 scrub ok 2024-03-26T05:24:02.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:02 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1089261499' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T05:24:02.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:02 smithi132 ceph-mon[33500]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T05:24:03.234 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:03 smithi080 ceph-mon[40404]: 5.1d scrub starts 2024-03-26T05:24:03.234 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:03 smithi080 ceph-mon[40404]: 5.1d scrub ok 2024-03-26T05:24:03.234 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:03 smithi080 ceph-mon[40404]: pgmap v1130: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-26T05:24:03.235 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:03 smithi080 ceph-mon[29691]: 5.1d scrub starts 2024-03-26T05:24:03.235 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:03 smithi080 ceph-mon[29691]: 5.1d scrub ok 2024-03-26T05:24:03.235 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:03 smithi080 ceph-mon[29691]: pgmap v1130: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-26T05:24:03.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:03 smithi132 ceph-mon[33500]: 5.1d scrub starts 2024-03-26T05:24:03.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:03 smithi132 ceph-mon[33500]: 5.1d scrub ok 2024-03-26T05:24:03.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:03 smithi132 ceph-mon[33500]: pgmap v1130: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-26T05:24:04.067 INFO:teuthology.orchestra.run.smithi080.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-26T05:24:04.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:04 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/528443977' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:24:04.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:04 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/528443977' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:24:04.367 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:24:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:24:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:24:04.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:04 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/528443977' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T05:24:04.695 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 6 1 2024-03-26T05:24:04.941 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:05.345 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:05 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/528443977' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-26T05:24:05.345 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:05 smithi080 ceph-mon[29691]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T05:24:05.345 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:05 smithi080 ceph-mon[29691]: 4.2 scrub starts 2024-03-26T05:24:05.346 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:05 smithi080 ceph-mon[29691]: 4.2 scrub ok 2024-03-26T05:24:05.346 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:05 smithi080 ceph-mon[29691]: pgmap v1132: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-03-26T05:24:05.346 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:05 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/528443977' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-26T05:24:05.346 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:05 smithi080 ceph-mon[40404]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T05:24:05.346 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:05 smithi080 ceph-mon[40404]: 4.2 scrub starts 2024-03-26T05:24:05.346 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:05 smithi080 ceph-mon[40404]: 4.2 scrub ok 2024-03-26T05:24:05.346 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:05 smithi080 ceph-mon[40404]: pgmap v1132: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-03-26T05:24:05.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:05 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/528443977' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-26T05:24:05.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:05 smithi132 ceph-mon[33500]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T05:24:05.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:05 smithi132 ceph-mon[33500]: 4.2 scrub starts 2024-03-26T05:24:05.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:05 smithi132 ceph-mon[33500]: 4.2 scrub ok 2024-03-26T05:24:05.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:05 smithi132 ceph-mon[33500]: pgmap v1132: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-03-26T05:24:06.354 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:06 smithi080 ceph-mon[40404]: 4.5 scrub starts 2024-03-26T05:24:06.354 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:06 smithi080 ceph-mon[40404]: 4.5 scrub ok 2024-03-26T05:24:06.355 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:06.355 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:06 smithi080 ceph-mon[29691]: 4.5 scrub starts 2024-03-26T05:24:06.355 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:06 smithi080 ceph-mon[29691]: 4.5 scrub ok 2024-03-26T05:24:06.355 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:06.443 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:06 smithi132 ceph-mon[33500]: 4.5 scrub starts 2024-03-26T05:24:06.443 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:06 smithi132 ceph-mon[33500]: 4.5 scrub ok 2024-03-26T05:24:06.443 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:06.443 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:24:06 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:24:07.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:07 smithi132 ceph-mon[33500]: 4.e scrub starts 2024-03-26T05:24:07.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:07 smithi132 ceph-mon[33500]: 4.e scrub ok 2024-03-26T05:24:07.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:07 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:07.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:07 smithi132 ceph-mon[33500]: 1.3 scrub starts 2024-03-26T05:24:07.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:07 smithi132 ceph-mon[33500]: 1.3 scrub ok 2024-03-26T05:24:07.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:07 smithi132 ceph-mon[33500]: pgmap v1133: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:07.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:07 smithi132 ceph-mon[33500]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T05:24:07.458 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:07 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/365857781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:24:07.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[29691]: 4.e scrub starts 2024-03-26T05:24:07.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[29691]: 4.e scrub ok 2024-03-26T05:24:07.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:07.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[29691]: 1.3 scrub starts 2024-03-26T05:24:07.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[29691]: 1.3 scrub ok 2024-03-26T05:24:07.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[29691]: pgmap v1133: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:07.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[29691]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T05:24:07.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/365857781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:24:07.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[40404]: 4.e scrub starts 2024-03-26T05:24:07.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[40404]: 4.e scrub ok 2024-03-26T05:24:07.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:07.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[40404]: 1.3 scrub starts 2024-03-26T05:24:07.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[40404]: 1.3 scrub ok 2024-03-26T05:24:07.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[40404]: pgmap v1133: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:07.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[40404]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T05:24:07.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:07 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/365857781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:24:07.791 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.6 to 1 (10000) 2024-03-26T05:24:08.499 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:08 smithi080 ceph-mon[40404]: 6.a scrub starts 2024-03-26T05:24:08.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:08 smithi080 ceph-mon[40404]: 6.a scrub ok 2024-03-26T05:24:08.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:08 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:24:08.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:08 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:24:08.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:08 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/365857781' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:24:08.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:08 smithi080 ceph-mon[40404]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T05:24:08.500 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:08 smithi080 ceph-mon[29691]: 6.a scrub starts 2024-03-26T05:24:08.500 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:08 smithi080 ceph-mon[29691]: 6.a scrub ok 2024-03-26T05:24:08.500 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:08 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:24:08.500 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:08 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:24:08.500 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:08 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/365857781' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:24:08.500 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:08 smithi080 ceph-mon[29691]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T05:24:08.531 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-26T05:24:08.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:08 smithi132 ceph-mon[33500]: 6.a scrub starts 2024-03-26T05:24:08.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:08 smithi132 ceph-mon[33500]: 6.a scrub ok 2024-03-26T05:24:08.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:08 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:24:08.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:08 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:24:08.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:08 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/365857781' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:24:08.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:08 smithi132 ceph-mon[33500]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T05:24:08.778 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:09.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:09 smithi132 ceph-mon[33500]: 3.f scrub starts 2024-03-26T05:24:09.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:09 smithi132 ceph-mon[33500]: 3.f scrub ok 2024-03-26T05:24:09.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:09 smithi132 ceph-mon[33500]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:09.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:09 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:24:09.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:09 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:24:09.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:09 smithi080 ceph-mon[29691]: 3.f scrub starts 2024-03-26T05:24:09.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:09 smithi080 ceph-mon[29691]: 3.f scrub ok 2024-03-26T05:24:09.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:09 smithi080 ceph-mon[29691]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:09.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:09 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:24:09.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:09 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:24:09.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:09 smithi080 ceph-mon[40404]: 3.f scrub starts 2024-03-26T05:24:09.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:09 smithi080 ceph-mon[40404]: 3.f scrub ok 2024-03-26T05:24:09.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:09 smithi080 ceph-mon[40404]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:09.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:09 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:24:09.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:09 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:24:10.284 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[40404]: 3.1d scrub starts 2024-03-26T05:24:10.285 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[40404]: 3.1d scrub ok 2024-03-26T05:24:10.285 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[40404]: 4.13 scrub starts 2024-03-26T05:24:10.285 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[40404]: 4.13 scrub ok 2024-03-26T05:24:10.285 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[40404]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T05:24:10.285 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[40404]: 5.6 scrub starts 2024-03-26T05:24:10.285 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[40404]: 5.6 scrub ok 2024-03-26T05:24:10.285 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[40404]: 6.1b scrub starts 2024-03-26T05:24:10.285 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[29691]: 3.1d scrub starts 2024-03-26T05:24:10.285 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[29691]: 3.1d scrub ok 2024-03-26T05:24:10.285 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[29691]: 4.13 scrub starts 2024-03-26T05:24:10.286 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[29691]: 4.13 scrub ok 2024-03-26T05:24:10.286 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[29691]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T05:24:10.286 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[29691]: 5.6 scrub starts 2024-03-26T05:24:10.286 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[29691]: 5.6 scrub ok 2024-03-26T05:24:10.286 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:10 smithi080 ceph-mon[29691]: 6.1b scrub starts 2024-03-26T05:24:10.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:10 smithi132 ceph-mon[33500]: 3.1d scrub starts 2024-03-26T05:24:10.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:10 smithi132 ceph-mon[33500]: 3.1d scrub ok 2024-03-26T05:24:10.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:10 smithi132 ceph-mon[33500]: 4.13 scrub starts 2024-03-26T05:24:10.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:10 smithi132 ceph-mon[33500]: 4.13 scrub ok 2024-03-26T05:24:10.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:10 smithi132 ceph-mon[33500]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T05:24:10.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:10 smithi132 ceph-mon[33500]: 5.6 scrub starts 2024-03-26T05:24:10.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:10 smithi132 ceph-mon[33500]: 5.6 scrub ok 2024-03-26T05:24:10.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:10 smithi132 ceph-mon[33500]: 6.1b scrub starts 2024-03-26T05:24:11.166 INFO:teuthology.orchestra.run.smithi080.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-26T05:24:11.465 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:11 smithi080 ceph-mon[40404]: 6.e scrub starts 2024-03-26T05:24:11.465 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:11 smithi080 ceph-mon[40404]: 6.e scrub ok 2024-03-26T05:24:11.465 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:11 smithi080 ceph-mon[40404]: 6.1b scrub ok 2024-03-26T05:24:11.465 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:11 smithi080 ceph-mon[40404]: pgmap v1138: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:11.465 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:11 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2662624067' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:24:11.465 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:11 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:24:11.466 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:11 smithi080 ceph-mon[29691]: 6.e scrub starts 2024-03-26T05:24:11.466 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:11 smithi080 ceph-mon[29691]: 6.e scrub ok 2024-03-26T05:24:11.466 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:11 smithi080 ceph-mon[29691]: 6.1b scrub ok 2024-03-26T05:24:11.466 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:11 smithi080 ceph-mon[29691]: pgmap v1138: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:11.466 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:11 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2662624067' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:24:11.466 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:11 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:24:11.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:11 smithi132 ceph-mon[33500]: 6.e scrub starts 2024-03-26T05:24:11.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:11 smithi132 ceph-mon[33500]: 6.e scrub ok 2024-03-26T05:24:11.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:11 smithi132 ceph-mon[33500]: 6.1b scrub ok 2024-03-26T05:24:11.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:11 smithi132 ceph-mon[33500]: pgmap v1138: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:11.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:11 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2662624067' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:24:11.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:11 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T05:24:11.867 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 7 1 2024-03-26T05:24:12.116 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:12.479 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:12 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:24:12.479 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:12 smithi080 ceph-mon[40404]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T05:24:12.479 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:12 smithi080 ceph-mon[40404]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T05:24:12.479 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:12 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:24:12.479 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:12 smithi080 ceph-mon[29691]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T05:24:12.479 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:12 smithi080 ceph-mon[29691]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T05:24:12.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:12 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-26T05:24:12.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:12 smithi132 ceph-mon[33500]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T05:24:12.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:12 smithi132 ceph-mon[33500]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T05:24:13.390 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:13 smithi080 ceph-mon[29691]: pgmap v1141: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:24:13.390 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:13 smithi080 ceph-mon[40404]: pgmap v1141: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:24:13.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:13 smithi132 ceph-mon[33500]: pgmap v1141: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:24:14.177 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.7 to 1 (10000) 2024-03-26T05:24:14.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:14 smithi080 ceph-mon[29691]: 5.10 scrub starts 2024-03-26T05:24:14.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:14 smithi080 ceph-mon[29691]: 4.1a scrub starts 2024-03-26T05:24:14.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:14 smithi080 ceph-mon[29691]: 5.10 scrub ok 2024-03-26T05:24:14.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:14 smithi080 ceph-mon[29691]: 4.1a scrub ok 2024-03-26T05:24:14.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:14 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3381352991' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:24:14.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:24:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:24:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:24:14.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:14 smithi080 ceph-mon[40404]: 5.10 scrub starts 2024-03-26T05:24:14.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:14 smithi080 ceph-mon[40404]: 4.1a scrub starts 2024-03-26T05:24:14.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:14 smithi080 ceph-mon[40404]: 5.10 scrub ok 2024-03-26T05:24:14.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:14 smithi080 ceph-mon[40404]: 4.1a scrub ok 2024-03-26T05:24:14.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:14 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3381352991' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:24:14.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:14 smithi132 ceph-mon[33500]: 5.10 scrub starts 2024-03-26T05:24:14.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:14 smithi132 ceph-mon[33500]: 4.1a scrub starts 2024-03-26T05:24:14.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:14 smithi132 ceph-mon[33500]: 5.10 scrub ok 2024-03-26T05:24:14.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:14 smithi132 ceph-mon[33500]: 4.1a scrub ok 2024-03-26T05:24:14.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:14 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3381352991' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:24:14.915 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-26T05:24:15.163 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:15.445 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:15 smithi080 ceph-mon[40404]: 3.15 scrub starts 2024-03-26T05:24:15.446 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:15 smithi080 ceph-mon[40404]: 3.15 scrub ok 2024-03-26T05:24:15.446 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:15 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3381352991' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:24:15.446 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:15 smithi080 ceph-mon[40404]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T05:24:15.446 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:15 smithi080 ceph-mon[40404]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 996 B/s rd, 0 op/s 2024-03-26T05:24:15.446 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:15 smithi080 ceph-mon[29691]: 3.15 scrub starts 2024-03-26T05:24:15.446 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:15 smithi080 ceph-mon[29691]: 3.15 scrub ok 2024-03-26T05:24:15.446 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:15 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3381352991' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:24:15.446 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:15 smithi080 ceph-mon[29691]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T05:24:15.446 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:15 smithi080 ceph-mon[29691]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 996 B/s rd, 0 op/s 2024-03-26T05:24:15.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:15 smithi132 ceph-mon[33500]: 3.15 scrub starts 2024-03-26T05:24:15.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:15 smithi132 ceph-mon[33500]: 3.15 scrub ok 2024-03-26T05:24:15.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:15 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3381352991' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:24:15.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:15 smithi132 ceph-mon[33500]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T05:24:15.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:15 smithi132 ceph-mon[33500]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 996 B/s rd, 0 op/s 2024-03-26T05:24:16.449 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:16 smithi080 ceph-mon[29691]: 5.9 scrub starts 2024-03-26T05:24:16.449 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:16 smithi080 ceph-mon[29691]: 5.9 scrub ok 2024-03-26T05:24:16.449 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:16 smithi080 ceph-mon[29691]: 6.2 scrub starts 2024-03-26T05:24:16.449 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:16 smithi080 ceph-mon[29691]: 6.2 scrub ok 2024-03-26T05:24:16.449 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:16 smithi080 ceph-mon[40404]: 5.9 scrub starts 2024-03-26T05:24:16.449 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:16 smithi080 ceph-mon[40404]: 5.9 scrub ok 2024-03-26T05:24:16.449 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:16 smithi080 ceph-mon[40404]: 6.2 scrub starts 2024-03-26T05:24:16.449 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:16 smithi080 ceph-mon[40404]: 6.2 scrub ok 2024-03-26T05:24:16.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:24:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:24:16.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:16 smithi132 ceph-mon[33500]: 5.9 scrub starts 2024-03-26T05:24:16.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:16 smithi132 ceph-mon[33500]: 5.9 scrub ok 2024-03-26T05:24:16.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:16 smithi132 ceph-mon[33500]: 6.2 scrub starts 2024-03-26T05:24:16.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:16 smithi132 ceph-mon[33500]: 6.2 scrub ok 2024-03-26T05:24:17.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:17 smithi132 ceph-mon[33500]: 5.1c scrub starts 2024-03-26T05:24:17.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:17 smithi132 ceph-mon[33500]: 5.1c scrub ok 2024-03-26T05:24:17.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:17 smithi132 ceph-mon[33500]: 6.12 scrub starts 2024-03-26T05:24:17.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:17 smithi132 ceph-mon[33500]: 6.12 scrub ok 2024-03-26T05:24:17.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:17 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:17.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:17 smithi132 ceph-mon[33500]: pgmap v1144: 142 pgs: 142 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:24:17.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:17 smithi132 ceph-mon[33500]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T05:24:17.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:17 smithi132 ceph-mon[33500]: 5.13 scrub starts 2024-03-26T05:24:17.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:17 smithi132 ceph-mon[33500]: 5.13 scrub ok 2024-03-26T05:24:17.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:17 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2590985255' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:24:17.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:17 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:24:17.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[29691]: 5.1c scrub starts 2024-03-26T05:24:17.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[29691]: 5.1c scrub ok 2024-03-26T05:24:17.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[29691]: 6.12 scrub starts 2024-03-26T05:24:17.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[29691]: 6.12 scrub ok 2024-03-26T05:24:17.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:17.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[29691]: pgmap v1144: 142 pgs: 142 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:24:17.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[29691]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T05:24:17.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[29691]: 5.13 scrub starts 2024-03-26T05:24:17.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[29691]: 5.13 scrub ok 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2590985255' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[40404]: 5.1c scrub starts 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[40404]: 5.1c scrub ok 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[40404]: 6.12 scrub starts 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[40404]: 6.12 scrub ok 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[40404]: pgmap v1144: 142 pgs: 142 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[40404]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[40404]: 5.13 scrub starts 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[40404]: 5.13 scrub ok 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2590985255' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:24:17.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:17 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T05:24:17.795 INFO:teuthology.orchestra.run.smithi080.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-26T05:24:18.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:18 smithi080 ceph-mon[40404]: 5.1a scrub starts 2024-03-26T05:24:18.501 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:18 smithi080 ceph-mon[40404]: 5.1a scrub ok 2024-03-26T05:24:18.501 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:18 smithi080 ceph-mon[40404]: 5.18 deep-scrub starts 2024-03-26T05:24:18.501 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:18 smithi080 ceph-mon[40404]: 5.18 deep-scrub ok 2024-03-26T05:24:18.501 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:18 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:24:18.501 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:18 smithi080 ceph-mon[40404]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T05:24:18.501 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:18 smithi080 ceph-mon[29691]: 5.1a scrub starts 2024-03-26T05:24:18.501 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:18 smithi080 ceph-mon[29691]: 5.1a scrub ok 2024-03-26T05:24:18.501 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:18 smithi080 ceph-mon[29691]: 5.18 deep-scrub starts 2024-03-26T05:24:18.501 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:18 smithi080 ceph-mon[29691]: 5.18 deep-scrub ok 2024-03-26T05:24:18.501 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:18 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:24:18.501 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:18 smithi080 ceph-mon[29691]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T05:24:18.532 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T05:24:18.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:18 smithi132 ceph-mon[33500]: 5.1a scrub starts 2024-03-26T05:24:18.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:18 smithi132 ceph-mon[33500]: 5.1a scrub ok 2024-03-26T05:24:18.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:18 smithi132 ceph-mon[33500]: 5.18 deep-scrub starts 2024-03-26T05:24:18.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:18 smithi132 ceph-mon[33500]: 5.18 deep-scrub ok 2024-03-26T05:24:18.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:18 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-26T05:24:18.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:18 smithi132 ceph-mon[33500]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T05:24:18.774 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:19.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:19 smithi132 ceph-mon[33500]: 5.15 scrub starts 2024-03-26T05:24:19.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:19 smithi132 ceph-mon[33500]: 5.15 scrub ok 2024-03-26T05:24:19.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:19 smithi132 ceph-mon[33500]: pgmap v1147: 142 pgs: 142 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:19.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:19 smithi132 ceph-mon[33500]: 3.1e scrub starts 2024-03-26T05:24:19.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:19 smithi132 ceph-mon[33500]: 3.1e scrub ok 2024-03-26T05:24:19.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:19 smithi080 ceph-mon[29691]: 5.15 scrub starts 2024-03-26T05:24:19.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:19 smithi080 ceph-mon[29691]: 5.15 scrub ok 2024-03-26T05:24:19.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:19 smithi080 ceph-mon[29691]: pgmap v1147: 142 pgs: 142 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:19.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:19 smithi080 ceph-mon[29691]: 3.1e scrub starts 2024-03-26T05:24:19.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:19 smithi080 ceph-mon[29691]: 3.1e scrub ok 2024-03-26T05:24:19.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:19 smithi080 ceph-mon[40404]: 5.15 scrub starts 2024-03-26T05:24:19.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:19 smithi080 ceph-mon[40404]: 5.15 scrub ok 2024-03-26T05:24:19.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:19 smithi080 ceph-mon[40404]: pgmap v1147: 142 pgs: 142 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:19.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:19 smithi080 ceph-mon[40404]: 3.1e scrub starts 2024-03-26T05:24:19.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:19 smithi080 ceph-mon[40404]: 3.1e scrub ok 2024-03-26T05:24:20.298 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[29691]: 2.2 deep-scrub starts 2024-03-26T05:24:20.298 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[29691]: 2.2 deep-scrub ok 2024-03-26T05:24:20.298 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:24:20.298 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:24:20.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[29691]: 4.1 scrub starts 2024-03-26T05:24:20.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[29691]: 4.1 scrub ok 2024-03-26T05:24:20.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[29691]: 4.16 deep-scrub starts 2024-03-26T05:24:20.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[29691]: 4.16 deep-scrub ok 2024-03-26T05:24:20.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[29691]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T05:24:20.299 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[29691]: 4.7 scrub starts 2024-03-26T05:24:20.300 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[29691]: 4.7 scrub ok 2024-03-26T05:24:20.300 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[40404]: 2.2 deep-scrub starts 2024-03-26T05:24:20.300 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[40404]: 2.2 deep-scrub ok 2024-03-26T05:24:20.300 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:24:20.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:24:20.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[40404]: 4.1 scrub starts 2024-03-26T05:24:20.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[40404]: 4.1 scrub ok 2024-03-26T05:24:20.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[40404]: 4.16 deep-scrub starts 2024-03-26T05:24:20.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[40404]: 4.16 deep-scrub ok 2024-03-26T05:24:20.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[40404]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T05:24:20.302 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[40404]: 4.7 scrub starts 2024-03-26T05:24:20.302 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:20 smithi080 ceph-mon[40404]: 4.7 scrub ok 2024-03-26T05:24:20.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:20 smithi132 ceph-mon[33500]: 2.2 deep-scrub starts 2024-03-26T05:24:20.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:20 smithi132 ceph-mon[33500]: 2.2 deep-scrub ok 2024-03-26T05:24:20.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:20 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:24:20.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:20 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:24:20.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:20 smithi132 ceph-mon[33500]: 4.1 scrub starts 2024-03-26T05:24:20.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:20 smithi132 ceph-mon[33500]: 4.1 scrub ok 2024-03-26T05:24:20.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:20 smithi132 ceph-mon[33500]: 4.16 deep-scrub starts 2024-03-26T05:24:20.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:20 smithi132 ceph-mon[33500]: 4.16 deep-scrub ok 2024-03-26T05:24:20.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:20 smithi132 ceph-mon[33500]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T05:24:20.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:20 smithi132 ceph-mon[33500]: 4.7 scrub starts 2024-03-26T05:24:20.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:20 smithi132 ceph-mon[33500]: 4.7 scrub ok 2024-03-26T05:24:21.214 INFO:teuthology.orchestra.run.smithi080.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T05:24:21.540 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[40404]: 6.5 scrub starts 2024-03-26T05:24:21.540 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[40404]: 6.5 scrub ok 2024-03-26T05:24:21.540 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[40404]: 6.1e scrub starts 2024-03-26T05:24:21.541 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[40404]: 6.1e scrub ok 2024-03-26T05:24:21.541 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[40404]: pgmap v1149: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:21.541 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/236893813' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:24:21.541 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:24:21.541 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[40404]: 4.11 scrub starts 2024-03-26T05:24:21.541 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[40404]: 4.11 scrub ok 2024-03-26T05:24:21.541 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:21.541 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[29691]: 6.5 scrub starts 2024-03-26T05:24:21.541 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[29691]: 6.5 scrub ok 2024-03-26T05:24:21.541 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[29691]: 6.1e scrub starts 2024-03-26T05:24:21.541 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[29691]: 6.1e scrub ok 2024-03-26T05:24:21.541 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[29691]: pgmap v1149: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:21.542 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/236893813' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:24:21.542 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:24:21.542 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[29691]: 4.11 scrub starts 2024-03-26T05:24:21.542 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[29691]: 4.11 scrub ok 2024-03-26T05:24:21.542 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:21 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:21.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:21 smithi132 ceph-mon[33500]: 6.5 scrub starts 2024-03-26T05:24:21.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:21 smithi132 ceph-mon[33500]: 6.5 scrub ok 2024-03-26T05:24:21.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:21 smithi132 ceph-mon[33500]: 6.1e scrub starts 2024-03-26T05:24:21.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:21 smithi132 ceph-mon[33500]: 6.1e scrub ok 2024-03-26T05:24:21.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:21 smithi132 ceph-mon[33500]: pgmap v1149: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:21.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:21 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/236893813' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:24:21.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:21 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:24:21.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:21 smithi132 ceph-mon[33500]: 4.11 scrub starts 2024-03-26T05:24:21.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:21 smithi132 ceph-mon[33500]: 4.11 scrub ok 2024-03-26T05:24:21.543 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:21 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:21.980 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-26T05:24:22.224 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:22.461 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:22 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:24:22.461 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:22 smithi080 ceph-mon[40404]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T05:24:22.462 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:22 smithi080 ceph-mon[40404]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T05:24:22.462 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:22 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:24:22.462 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:22 smithi080 ceph-mon[29691]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T05:24:22.462 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:22 smithi080 ceph-mon[29691]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T05:24:22.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:22 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:24:22.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:22 smithi132 ceph-mon[33500]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T05:24:22.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:22 smithi132 ceph-mon[33500]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T05:24:23.459 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:23 smithi080 ceph-mon[40404]: 4.18 scrub starts 2024-03-26T05:24:23.460 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:23 smithi080 ceph-mon[40404]: 4.18 scrub ok 2024-03-26T05:24:23.460 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:23 smithi080 ceph-mon[40404]: pgmap v1152: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:24:23.460 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:23 smithi080 ceph-mon[40404]: 6.c scrub starts 2024-03-26T05:24:23.460 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:23 smithi080 ceph-mon[40404]: 6.c scrub ok 2024-03-26T05:24:23.460 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:23 smithi080 ceph-mon[29691]: 4.18 scrub starts 2024-03-26T05:24:23.460 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:23 smithi080 ceph-mon[29691]: 4.18 scrub ok 2024-03-26T05:24:23.460 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:23 smithi080 ceph-mon[29691]: pgmap v1152: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:24:23.460 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:23 smithi080 ceph-mon[29691]: 6.c scrub starts 2024-03-26T05:24:23.461 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:23 smithi080 ceph-mon[29691]: 6.c scrub ok 2024-03-26T05:24:23.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:23 smithi132 ceph-mon[33500]: 4.18 scrub starts 2024-03-26T05:24:23.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:23 smithi132 ceph-mon[33500]: 4.18 scrub ok 2024-03-26T05:24:23.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:23 smithi132 ceph-mon[33500]: pgmap v1152: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:24:23.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:23 smithi132 ceph-mon[33500]: 6.c scrub starts 2024-03-26T05:24:23.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:23 smithi132 ceph-mon[33500]: 6.c scrub ok 2024-03-26T05:24:24.223 INFO:teuthology.orchestra.run.smithi080.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-26T05:24:24.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-mon[29691]: 6.9 deep-scrub starts 2024-03-26T05:24:24.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-mon[29691]: 6.9 deep-scrub ok 2024-03-26T05:24:24.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-mon[29691]: 5.14 scrub starts 2024-03-26T05:24:24.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-mon[29691]: 5.14 scrub ok 2024-03-26T05:24:24.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3460777970' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:24:24.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:24:24.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:24:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:24:24.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-mon[40404]: 6.9 deep-scrub starts 2024-03-26T05:24:24.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-mon[40404]: 6.9 deep-scrub ok 2024-03-26T05:24:24.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-mon[40404]: 5.14 scrub starts 2024-03-26T05:24:24.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-mon[40404]: 5.14 scrub ok 2024-03-26T05:24:24.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3460777970' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:24:24.367 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:24 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:24:24.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:24 smithi132 ceph-mon[33500]: 6.9 deep-scrub starts 2024-03-26T05:24:24.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:24 smithi132 ceph-mon[33500]: 6.9 deep-scrub ok 2024-03-26T05:24:24.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:24 smithi132 ceph-mon[33500]: 5.14 scrub starts 2024-03-26T05:24:24.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:24 smithi132 ceph-mon[33500]: 5.14 scrub ok 2024-03-26T05:24:24.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:24 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3460777970' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:24:24.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:24 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T05:24:24.996 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-26T05:24:24.996 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-26T05:24:24.996 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-26T05:24:24.996 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-26T05:24:24.997 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T05:24:24.997 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T05:24:25.240 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:25.486 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:25 smithi080 ceph-mon[40404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:24:25.487 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:25 smithi080 ceph-mon[40404]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T05:24:25.487 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:25 smithi080 ceph-mon[40404]: pgmap v1154: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-26T05:24:25.487 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:25 smithi080 ceph-mon[29691]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:24:25.487 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:25 smithi080 ceph-mon[29691]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T05:24:25.487 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:25 smithi080 ceph-mon[29691]: pgmap v1154: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-26T05:24:25.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:25 smithi132 ceph-mon[33500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-26T05:24:25.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:25 smithi132 ceph-mon[33500]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T05:24:25.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:25 smithi132 ceph-mon[33500]: pgmap v1154: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-26T05:24:26.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:24:26 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:24:27.029 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:24:27.029 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":540,"fsid":"dd09e9cc-eb2c-11ee-95d0-87774f69a715","created":"2024-03-26T04:54:51.106954+0000","modified":"2024-03-26T05:24:26.792757+0000","last_up_change":"2024-03-26T05:23:13.847196+0000","last_in_change":"2024-03-26T05:23:25.910605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:58:06.876819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"509","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"324","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T05:00:20.705417+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T05:00:21.150368+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T05:00:23.638065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T05:00:25.181000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T05:00:27.249998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"35a52544-0b0b-47d6-8d6a-571c48ceb366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":508,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6803","nonce":1840686622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6805","nonce":1840686622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6809","nonce":1840686622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6807","nonce":1840686622}]},"public_addr":"172.21.15.80:6803/1840686622","cluster_addr":"172.21.15.80:6805/1840686622","heartbeat_back_addr":"172.21.15.80:6809/1840686622","heartbeat_front_addr":"172.21.15.80:6807/1840686622","state":["exists","up"]},{"osd":1,"uuid":"efa5ea33-4f11-49b7-a3f3-f60d5329db23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":506,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6811","nonce":1726833996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6813","nonce":1726833996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6817","nonce":1726833996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6815","nonce":1726833996}]},"public_addr":"172.21.15.80:6811/1726833996","cluster_addr":"172.21.15.80:6813/1726833996","heartbeat_back_addr":"172.21.15.80:6817/1726833996","heartbeat_front_addr":"172.21.15.80:6815/1726833996","state":["exists","up"]},{"osd":2,"uuid":"536bec9b-9636-4e67-b452-ed1a55c22ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":508,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6819","nonce":128713757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6821","nonce":128713757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6825","nonce":128713757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6823","nonce":128713757}]},"public_addr":"172.21.15.80:6819/128713757","cluster_addr":"172.21.15.80:6821/128713757","heartbeat_back_addr":"172.21.15.80:6825/128713757","heartbeat_front_addr":"172.21.15.80:6823/128713757","state":["exists","up"]},{"osd":3,"uuid":"640cfdb6-b9c5-446d-8312-504ae12c7818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6827","nonce":418771441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6829","nonce":418771441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6833","nonce":418771441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6831","nonce":418771441}]},"public_addr":"172.21.15.80:6827/418771441","cluster_addr":"172.21.15.80:6829/418771441","heartbeat_back_addr":"172.21.15.80:6833/418771441","heartbeat_front_addr":"172.21.15.80:6831/418771441","state":["exists","up"]},{"osd":4,"uuid":"0cc7822a-ed73-4c79-a30d-1e151ddd6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":341,"up_from":501,"up_thru":506,"down_at":342,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1991006784},{"type":"v1","addr":"172.21.15.132:6801","nonce":1991006784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1991006784},{"type":"v1","addr":"172.21.15.132:6803","nonce":1991006784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1991006784},{"type":"v1","addr":"172.21.15.132:6807","nonce":1991006784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1991006784},{"type":"v1","addr":"172.21.15.132:6805","nonce":1991006784}]},"public_addr":"172.21.15.132:6801/1991006784","cluster_addr":"172.21.15.132:6803/1991006784","heartbeat_back_addr":"172.21.15.132:6807/1991006784","heartbeat_front_addr":"172.21.15.132:6805/1991006784","state":["exists","up"]},{"osd":5,"uuid":"acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6809","nonce":794504625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6811","nonce":794504625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6815","nonce":794504625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6813","nonce":794504625}]},"public_addr":"172.21.15.132:6809/794504625","cluster_addr":"172.21.15.132:6811/794504625","heartbeat_back_addr":"172.21.15.132:6815/794504625","heartbeat_front_addr":"172.21.15.132:6813/794504625","state":["exists","up"]},{"osd":6,"uuid":"2970bbab-11b2-4d1b-b57e-037bdac089b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":506,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6817","nonce":2209078972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6819","nonce":2209078972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6823","nonce":2209078972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6821","nonce":2209078972}]},"public_addr":"172.21.15.132:6817/2209078972","cluster_addr":"172.21.15.132:6819/2209078972","heartbeat_back_addr":"172.21.15.132:6823/2209078972","heartbeat_front_addr":"172.21.15.132:6821/2209078972","state":["exists","up"]},{"osd":7,"uuid":"12f4b466-28f9-469b-8fd0-65b91c6dc22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":61,"last_clean_end":292,"up_from":335,"up_thru":506,"down_at":293,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":1165391760},{"type":"v1","addr":"172.21.15.132:6825","nonce":1165391760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1165391760},{"type":"v1","addr":"172.21.15.132:6827","nonce":1165391760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1165391760},{"type":"v1","addr":"172.21.15.132:6831","nonce":1165391760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1165391760},{"type":"v1","addr":"172.21.15.132:6829","nonce":1165391760}]},"public_addr":"172.21.15.132:6825/1165391760","cluster_addr":"172.21.15.132:6827/1165391760","heartbeat_back_addr":"172.21.15.132:6831/1165391760","heartbeat_front_addr":"172.21.15.132:6829/1165391760","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:23:03.816390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:24:07.106368+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:24:08.554071+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:23:48.668907+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:24:18.539639+0000","dead_epoch":341},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:23:06.084714+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:23:40.227279+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:23:07.176040+0000","dead_epoch":292}],"pg_upmap":[{"pgid":"3.1","osds":[4,0,5]},{"pgid":"4.b","osds":[0,4,2]}],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":1,"to":2}]},{"pgid":"3.1d","mappings":[{"from":6,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"5.10","mappings":[{"from":6,"to":3}]},{"pgid":"6.b","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.80:0/3992539385":"2024-03-27T05:01:03.689739+0000","172.21.15.80:6800/2770889532":"2024-03-27T04:55:48.542948+0000","172.21.15.80:6801/2466284312":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/490056571":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/1986135300":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/3796611170":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/1993138634":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/1832498181":"2024-03-27T05:01:03.689739+0000","172.21.15.80:6800/2466284312":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/1650870847":"2024-03-27T05:01:03.689739+0000","172.21.15.80:6801/1418683843":"2024-03-27T05:01:03.689739+0000","172.21.15.80:0/3147301987":"2024-03-27T05:01:03.689739+0000","172.21.15.80:6801/2770889532":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/571905941":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/1451012560":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/1235125152":"2024-03-27T05:01:03.689739+0000","172.21.15.80:6800/1418683843":"2024-03-27T05:01:03.689739+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T05:24:27.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[29691]: 4.1b scrub starts 2024-03-26T05:24:27.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[29691]: 4.1b scrub ok 2024-03-26T05:24:27.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:27.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[29691]: pgmap v1155: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:24:27.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[29691]: 1.0 scrub starts 2024-03-26T05:24:27.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[29691]: 1.0 scrub ok 2024-03-26T05:24:27.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[29691]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T05:24:27.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2854472050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:24:27.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[40404]: 4.1b scrub starts 2024-03-26T05:24:27.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[40404]: 4.1b scrub ok 2024-03-26T05:24:27.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:27.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[40404]: pgmap v1155: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:24:27.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[40404]: 1.0 scrub starts 2024-03-26T05:24:27.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[40404]: 1.0 scrub ok 2024-03-26T05:24:27.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[40404]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T05:24:27.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:27 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2854472050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:24:27.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:27 smithi132 ceph-mon[33500]: 4.1b scrub starts 2024-03-26T05:24:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:27 smithi132 ceph-mon[33500]: 4.1b scrub ok 2024-03-26T05:24:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:27 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:27 smithi132 ceph-mon[33500]: pgmap v1155: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:24:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:27 smithi132 ceph-mon[33500]: 1.0 scrub starts 2024-03-26T05:24:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:27 smithi132 ceph-mon[33500]: 1.0 scrub ok 2024-03-26T05:24:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:27 smithi132 ceph-mon[33500]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T05:24:27.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:27 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2854472050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:24:27.796 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T05:24:27.796 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T05:24:28.041 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:28.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:28 smithi080 ceph-mon[40404]: 3.e scrub starts 2024-03-26T05:24:28.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:28 smithi080 ceph-mon[40404]: 3.e scrub ok 2024-03-26T05:24:28.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:28 smithi080 ceph-mon[40404]: 3.b scrub starts 2024-03-26T05:24:28.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:28 smithi080 ceph-mon[40404]: 3.b scrub ok 2024-03-26T05:24:28.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:28 smithi080 ceph-mon[29691]: 3.e scrub starts 2024-03-26T05:24:28.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:28 smithi080 ceph-mon[29691]: 3.e scrub ok 2024-03-26T05:24:28.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:28 smithi080 ceph-mon[29691]: 3.b scrub starts 2024-03-26T05:24:28.471 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:28 smithi080 ceph-mon[29691]: 3.b scrub ok 2024-03-26T05:24:28.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:28 smithi132 ceph-mon[33500]: 3.e scrub starts 2024-03-26T05:24:28.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:28 smithi132 ceph-mon[33500]: 3.e scrub ok 2024-03-26T05:24:28.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:28 smithi132 ceph-mon[33500]: 3.b scrub starts 2024-03-26T05:24:28.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:28 smithi132 ceph-mon[33500]: 3.b scrub ok 2024-03-26T05:24:29.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:29 smithi080 ceph-mon[29691]: pgmap v1157: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 785 B/s rd, 0 op/s 2024-03-26T05:24:29.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:29 smithi080 ceph-mon[29691]: 6.7 scrub starts 2024-03-26T05:24:29.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:29 smithi080 ceph-mon[29691]: 6.7 scrub ok 2024-03-26T05:24:29.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:29 smithi080 ceph-mon[40404]: pgmap v1157: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 785 B/s rd, 0 op/s 2024-03-26T05:24:29.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:29 smithi080 ceph-mon[40404]: 6.7 scrub starts 2024-03-26T05:24:29.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:29 smithi080 ceph-mon[40404]: 6.7 scrub ok 2024-03-26T05:24:29.768 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:24:29.768 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":540,"fsid":"dd09e9cc-eb2c-11ee-95d0-87774f69a715","created":"2024-03-26T04:54:51.106954+0000","modified":"2024-03-26T05:24:26.792757+0000","last_up_change":"2024-03-26T05:23:13.847196+0000","last_in_change":"2024-03-26T05:23:25.910605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:58:06.876819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"509","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"324","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T05:00:20.705417+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T05:00:21.150368+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T05:00:23.638065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T05:00:25.181000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T05:00:27.249998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"35a52544-0b0b-47d6-8d6a-571c48ceb366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":508,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6803","nonce":1840686622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6805","nonce":1840686622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6809","nonce":1840686622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":1840686622},{"type":"v1","addr":"172.21.15.80:6807","nonce":1840686622}]},"public_addr":"172.21.15.80:6803/1840686622","cluster_addr":"172.21.15.80:6805/1840686622","heartbeat_back_addr":"172.21.15.80:6809/1840686622","heartbeat_front_addr":"172.21.15.80:6807/1840686622","state":["exists","up"]},{"osd":1,"uuid":"efa5ea33-4f11-49b7-a3f3-f60d5329db23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":506,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6811","nonce":1726833996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6813","nonce":1726833996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6817","nonce":1726833996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":1726833996},{"type":"v1","addr":"172.21.15.80:6815","nonce":1726833996}]},"public_addr":"172.21.15.80:6811/1726833996","cluster_addr":"172.21.15.80:6813/1726833996","heartbeat_back_addr":"172.21.15.80:6817/1726833996","heartbeat_front_addr":"172.21.15.80:6815/1726833996","state":["exists","up"]},{"osd":2,"uuid":"536bec9b-9636-4e67-b452-ed1a55c22ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":508,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6819","nonce":128713757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6821","nonce":128713757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6825","nonce":128713757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":128713757},{"type":"v1","addr":"172.21.15.80:6823","nonce":128713757}]},"public_addr":"172.21.15.80:6819/128713757","cluster_addr":"172.21.15.80:6821/128713757","heartbeat_back_addr":"172.21.15.80:6825/128713757","heartbeat_front_addr":"172.21.15.80:6823/128713757","state":["exists","up"]},{"osd":3,"uuid":"640cfdb6-b9c5-446d-8312-504ae12c7818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6827","nonce":418771441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6829","nonce":418771441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6833","nonce":418771441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":418771441},{"type":"v1","addr":"172.21.15.80:6831","nonce":418771441}]},"public_addr":"172.21.15.80:6827/418771441","cluster_addr":"172.21.15.80:6829/418771441","heartbeat_back_addr":"172.21.15.80:6833/418771441","heartbeat_front_addr":"172.21.15.80:6831/418771441","state":["exists","up"]},{"osd":4,"uuid":"0cc7822a-ed73-4c79-a30d-1e151ddd6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":341,"up_from":501,"up_thru":506,"down_at":342,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1991006784},{"type":"v1","addr":"172.21.15.132:6801","nonce":1991006784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1991006784},{"type":"v1","addr":"172.21.15.132:6803","nonce":1991006784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1991006784},{"type":"v1","addr":"172.21.15.132:6807","nonce":1991006784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1991006784},{"type":"v1","addr":"172.21.15.132:6805","nonce":1991006784}]},"public_addr":"172.21.15.132:6801/1991006784","cluster_addr":"172.21.15.132:6803/1991006784","heartbeat_back_addr":"172.21.15.132:6807/1991006784","heartbeat_front_addr":"172.21.15.132:6805/1991006784","state":["exists","up"]},{"osd":5,"uuid":"acee0ff1-e5e5-45c4-aed0-8f8b32a2e6dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6809","nonce":794504625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6811","nonce":794504625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6815","nonce":794504625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":794504625},{"type":"v1","addr":"172.21.15.132:6813","nonce":794504625}]},"public_addr":"172.21.15.132:6809/794504625","cluster_addr":"172.21.15.132:6811/794504625","heartbeat_back_addr":"172.21.15.132:6815/794504625","heartbeat_front_addr":"172.21.15.132:6813/794504625","state":["exists","up"]},{"osd":6,"uuid":"2970bbab-11b2-4d1b-b57e-037bdac089b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":506,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6817","nonce":2209078972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6819","nonce":2209078972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6823","nonce":2209078972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2209078972},{"type":"v1","addr":"172.21.15.132:6821","nonce":2209078972}]},"public_addr":"172.21.15.132:6817/2209078972","cluster_addr":"172.21.15.132:6819/2209078972","heartbeat_back_addr":"172.21.15.132:6823/2209078972","heartbeat_front_addr":"172.21.15.132:6821/2209078972","state":["exists","up"]},{"osd":7,"uuid":"12f4b466-28f9-469b-8fd0-65b91c6dc22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":61,"last_clean_end":292,"up_from":335,"up_thru":506,"down_at":293,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":1165391760},{"type":"v1","addr":"172.21.15.132:6825","nonce":1165391760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1165391760},{"type":"v1","addr":"172.21.15.132:6827","nonce":1165391760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1165391760},{"type":"v1","addr":"172.21.15.132:6831","nonce":1165391760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1165391760},{"type":"v1","addr":"172.21.15.132:6829","nonce":1165391760}]},"public_addr":"172.21.15.132:6825/1165391760","cluster_addr":"172.21.15.132:6827/1165391760","heartbeat_back_addr":"172.21.15.132:6831/1165391760","heartbeat_front_addr":"172.21.15.132:6829/1165391760","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:23:03.816390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:24:07.106368+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:24:08.554071+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:23:48.668907+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:24:18.539639+0000","dead_epoch":341},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:23:06.084714+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:23:40.227279+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:23:07.176040+0000","dead_epoch":292}],"pg_upmap":[{"pgid":"3.1","osds":[4,0,5]},{"pgid":"4.b","osds":[0,4,2]}],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":1,"to":2}]},{"pgid":"3.1d","mappings":[{"from":6,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"5.10","mappings":[{"from":6,"to":3}]},{"pgid":"6.b","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.80:0/3992539385":"2024-03-27T05:01:03.689739+0000","172.21.15.80:6800/2770889532":"2024-03-27T04:55:48.542948+0000","172.21.15.80:6801/2466284312":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/490056571":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/1986135300":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/3796611170":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/1993138634":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/1832498181":"2024-03-27T05:01:03.689739+0000","172.21.15.80:6800/2466284312":"2024-03-27T04:55:21.711776+0000","172.21.15.80:0/1650870847":"2024-03-27T05:01:03.689739+0000","172.21.15.80:6801/1418683843":"2024-03-27T05:01:03.689739+0000","172.21.15.80:0/3147301987":"2024-03-27T05:01:03.689739+0000","172.21.15.80:6801/2770889532":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/571905941":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/1451012560":"2024-03-27T04:55:48.542948+0000","172.21.15.80:0/1235125152":"2024-03-27T05:01:03.689739+0000","172.21.15.80:6800/1418683843":"2024-03-27T05:01:03.689739+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T05:24:29.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:29 smithi132 ceph-mon[33500]: pgmap v1157: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 785 B/s rd, 0 op/s 2024-03-26T05:24:29.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:29 smithi132 ceph-mon[33500]: 6.7 scrub starts 2024-03-26T05:24:29.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:29 smithi132 ceph-mon[33500]: 6.7 scrub ok 2024-03-26T05:24:30.395 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T05:24:30.395 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T05:24:30.396 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T05:24:30.396 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T05:24:30.396 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T05:24:30.397 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T05:24:30.398 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T05:24:30.398 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T05:24:30.433 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[40404]: 3.12 scrub starts 2024-03-26T05:24:30.433 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[40404]: 3.12 scrub ok 2024-03-26T05:24:30.434 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:24:30.434 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:24:30.434 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[40404]: 6.8 deep-scrub starts 2024-03-26T05:24:30.434 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[40404]: 6.8 deep-scrub ok 2024-03-26T05:24:30.434 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:24:30.434 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:24:30.434 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2119019500' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:24:30.434 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:24:30.434 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:24:30.434 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[29691]: 3.12 scrub starts 2024-03-26T05:24:30.435 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[29691]: 3.12 scrub ok 2024-03-26T05:24:30.435 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:24:30.435 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:24:30.435 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[29691]: 6.8 deep-scrub starts 2024-03-26T05:24:30.435 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[29691]: 6.8 deep-scrub ok 2024-03-26T05:24:30.435 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:24:30.435 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:24:30.435 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2119019500' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:24:30.435 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:24:30.435 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:30 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:24:30.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:30 smithi132 ceph-mon[33500]: 3.12 scrub starts 2024-03-26T05:24:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:30 smithi132 ceph-mon[33500]: 3.12 scrub ok 2024-03-26T05:24:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:30 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:24:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:30 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:24:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:30 smithi132 ceph-mon[33500]: 6.8 deep-scrub starts 2024-03-26T05:24:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:30 smithi132 ceph-mon[33500]: 6.8 deep-scrub ok 2024-03-26T05:24:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:30 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:24:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:30 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:24:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:30 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2119019500' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:24:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:30 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:24:30.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:30 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:24:30.935 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:30.936 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:30.937 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:30.939 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:30.939 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:30.940 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:30.942 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:30.944 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:31.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:31 smithi132 ceph-mon[33500]: pgmap v1158: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:24:31.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:31 smithi132 ceph-mon[33500]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T05:24:31.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:31 smithi080 ceph-mon[29691]: pgmap v1158: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:24:31.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:31 smithi080 ceph-mon[29691]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T05:24:31.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:31 smithi080 ceph-mon[40404]: pgmap v1158: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:24:31.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:31 smithi080 ceph-mon[40404]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T05:24:32.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:32 smithi080 ceph-mon[29691]: 3.1c scrub starts 2024-03-26T05:24:32.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:32 smithi080 ceph-mon[29691]: 3.1c scrub ok 2024-03-26T05:24:32.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:32 smithi080 ceph-mon[29691]: 2.1 deep-scrub starts 2024-03-26T05:24:32.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:32 smithi080 ceph-mon[29691]: 2.1 deep-scrub ok 2024-03-26T05:24:32.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:32 smithi080 ceph-mon[29691]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T05:24:32.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:32 smithi080 ceph-mon[40404]: 3.1c scrub starts 2024-03-26T05:24:32.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:32 smithi080 ceph-mon[40404]: 3.1c scrub ok 2024-03-26T05:24:32.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:32 smithi080 ceph-mon[40404]: 2.1 deep-scrub starts 2024-03-26T05:24:32.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:32 smithi080 ceph-mon[40404]: 2.1 deep-scrub ok 2024-03-26T05:24:32.617 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:32 smithi080 ceph-mon[40404]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T05:24:32.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:32 smithi132 ceph-mon[33500]: 3.1c scrub starts 2024-03-26T05:24:32.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:32 smithi132 ceph-mon[33500]: 3.1c scrub ok 2024-03-26T05:24:32.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:32 smithi132 ceph-mon[33500]: 2.1 deep-scrub starts 2024-03-26T05:24:32.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:32 smithi132 ceph-mon[33500]: 2.1 deep-scrub ok 2024-03-26T05:24:32.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:32 smithi132 ceph-mon[33500]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T05:24:33.588 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[29691]: 4.1e scrub starts 2024-03-26T05:24:33.588 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[29691]: 4.1e scrub ok 2024-03-26T05:24:33.588 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[29691]: pgmap v1161: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:33.588 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[29691]: 4.1c scrub starts 2024-03-26T05:24:33.588 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[29691]: 4.1c scrub ok 2024-03-26T05:24:33.588 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[29691]: 4.0 scrub starts 2024-03-26T05:24:33.588 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[29691]: 4.0 scrub ok 2024-03-26T05:24:33.588 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[40404]: 4.1e scrub starts 2024-03-26T05:24:33.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[40404]: 4.1e scrub ok 2024-03-26T05:24:33.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[40404]: pgmap v1161: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:33.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[40404]: 4.1c scrub starts 2024-03-26T05:24:33.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[40404]: 4.1c scrub ok 2024-03-26T05:24:33.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[40404]: 4.0 scrub starts 2024-03-26T05:24:33.589 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:33 smithi080 ceph-mon[40404]: 4.0 scrub ok 2024-03-26T05:24:33.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:33 smithi132 ceph-mon[33500]: 4.1e scrub starts 2024-03-26T05:24:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:33 smithi132 ceph-mon[33500]: 4.1e scrub ok 2024-03-26T05:24:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:33 smithi132 ceph-mon[33500]: pgmap v1161: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:33 smithi132 ceph-mon[33500]: 4.1c scrub starts 2024-03-26T05:24:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:33 smithi132 ceph-mon[33500]: 4.1c scrub ok 2024-03-26T05:24:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:33 smithi132 ceph-mon[33500]: 4.0 scrub starts 2024-03-26T05:24:33.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:33 smithi132 ceph-mon[33500]: 4.0 scrub ok 2024-03-26T05:24:34.288 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:24:34 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:24:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:24:34.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:34 smithi080 ceph-mon[29691]: 3.0 deep-scrub starts 2024-03-26T05:24:34.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:34 smithi080 ceph-mon[29691]: 3.0 deep-scrub ok 2024-03-26T05:24:34.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:34 smithi080 ceph-mon[40404]: 3.0 deep-scrub starts 2024-03-26T05:24:34.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:34 smithi080 ceph-mon[40404]: 3.0 deep-scrub ok 2024-03-26T05:24:34.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:34 smithi132 ceph-mon[33500]: 3.0 deep-scrub starts 2024-03-26T05:24:34.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:34 smithi132 ceph-mon[33500]: 3.0 deep-scrub ok 2024-03-26T05:24:35.533 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:35 smithi080 ceph-mon[40404]: 6.1 scrub starts 2024-03-26T05:24:35.534 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:35 smithi080 ceph-mon[40404]: 6.1 scrub ok 2024-03-26T05:24:35.534 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:35 smithi080 ceph-mon[40404]: pgmap v1162: 142 pgs: 142 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-26T05:24:35.534 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:35 smithi080 ceph-mon[40404]: 6.d scrub starts 2024-03-26T05:24:35.534 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:35 smithi080 ceph-mon[40404]: 6.d scrub ok 2024-03-26T05:24:35.534 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:35 smithi080 ceph-mon[40404]: 6.1f scrub starts 2024-03-26T05:24:35.534 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:35 smithi080 ceph-mon[29691]: 6.1 scrub starts 2024-03-26T05:24:35.534 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:35 smithi080 ceph-mon[29691]: 6.1 scrub ok 2024-03-26T05:24:35.534 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:35 smithi080 ceph-mon[29691]: pgmap v1162: 142 pgs: 142 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-26T05:24:35.534 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:35 smithi080 ceph-mon[29691]: 6.d scrub starts 2024-03-26T05:24:35.534 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:35 smithi080 ceph-mon[29691]: 6.d scrub ok 2024-03-26T05:24:35.535 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:35 smithi080 ceph-mon[29691]: 6.1f scrub starts 2024-03-26T05:24:35.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:35 smithi132 ceph-mon[33500]: 6.1 scrub starts 2024-03-26T05:24:35.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:35 smithi132 ceph-mon[33500]: 6.1 scrub ok 2024-03-26T05:24:35.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:35 smithi132 ceph-mon[33500]: pgmap v1162: 142 pgs: 142 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-26T05:24:35.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:35 smithi132 ceph-mon[33500]: 6.d scrub starts 2024-03-26T05:24:35.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:35 smithi132 ceph-mon[33500]: 6.d scrub ok 2024-03-26T05:24:35.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:35 smithi132 ceph-mon[33500]: 6.1f scrub starts 2024-03-26T05:24:35.795 INFO:teuthology.orchestra.run.smithi080.stdout:55834575174 2024-03-26T05:24:35.796 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T05:24:36.386 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:36 smithi080 ceph-mon[40404]: 6.18 deep-scrub starts 2024-03-26T05:24:36.387 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:36 smithi080 ceph-mon[40404]: 6.18 deep-scrub ok 2024-03-26T05:24:36.387 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:36 smithi080 ceph-mon[40404]: 6.1f scrub ok 2024-03-26T05:24:36.387 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:36 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:36.387 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:36 smithi080 ceph-mon[29691]: 6.18 deep-scrub starts 2024-03-26T05:24:36.387 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:36 smithi080 ceph-mon[29691]: 6.18 deep-scrub ok 2024-03-26T05:24:36.387 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:36 smithi080 ceph-mon[29691]: 6.1f scrub ok 2024-03-26T05:24:36.387 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:36 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:36.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:36 smithi132 ceph-mon[33500]: 6.18 deep-scrub starts 2024-03-26T05:24:36.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:36 smithi132 ceph-mon[33500]: 6.18 deep-scrub ok 2024-03-26T05:24:36.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:36 smithi132 ceph-mon[33500]: 6.1f scrub ok 2024-03-26T05:24:36.542 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:36 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:36.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:24:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:24:37.994 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:37 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:37.994 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:37 smithi080 ceph-mon[40404]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:37.995 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:37 smithi080 ceph-mon[40404]: 1.7 scrub starts 2024-03-26T05:24:37.995 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:37 smithi080 ceph-mon[40404]: 1.7 scrub ok 2024-03-26T05:24:37.995 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:37 smithi080 ceph-mon[40404]: 4.17 deep-scrub starts 2024-03-26T05:24:37.995 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:37 smithi080 ceph-mon[40404]: 4.17 deep-scrub ok 2024-03-26T05:24:37.995 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:37 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:37.995 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:37 smithi080 ceph-mon[29691]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:37.995 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:37 smithi080 ceph-mon[29691]: 1.7 scrub starts 2024-03-26T05:24:37.995 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:37 smithi080 ceph-mon[29691]: 1.7 scrub ok 2024-03-26T05:24:37.995 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:37 smithi080 ceph-mon[29691]: 4.17 deep-scrub starts 2024-03-26T05:24:37.995 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:37 smithi080 ceph-mon[29691]: 4.17 deep-scrub ok 2024-03-26T05:24:38.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:37 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:38.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:37 smithi132 ceph-mon[33500]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:38.041 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:37 smithi132 ceph-mon[33500]: 1.7 scrub starts 2024-03-26T05:24:38.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:37 smithi132 ceph-mon[33500]: 1.7 scrub ok 2024-03-26T05:24:38.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:37 smithi132 ceph-mon[33500]: 4.17 deep-scrub starts 2024-03-26T05:24:38.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:37 smithi132 ceph-mon[33500]: 4.17 deep-scrub ok 2024-03-26T05:24:39.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:38 smithi080 ceph-mon[29691]: 6.13 scrub starts 2024-03-26T05:24:39.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:38 smithi080 ceph-mon[29691]: 6.13 scrub ok 2024-03-26T05:24:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:38 smithi080 ceph-mon[40404]: 6.13 scrub starts 2024-03-26T05:24:39.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:38 smithi080 ceph-mon[40404]: 6.13 scrub ok 2024-03-26T05:24:39.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:38 smithi132 ceph-mon[33500]: 6.13 scrub starts 2024-03-26T05:24:39.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:38 smithi132 ceph-mon[33500]: 6.13 scrub ok 2024-03-26T05:24:39.890 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[29691]: pgmap v1164: 142 pgs: 142 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:39.890 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[29691]: 5.f scrub starts 2024-03-26T05:24:39.890 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[29691]: 5.f scrub ok 2024-03-26T05:24:39.891 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[29691]: 6.3 scrub starts 2024-03-26T05:24:39.891 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[29691]: 5.2 scrub starts 2024-03-26T05:24:39.891 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[29691]: 5.2 scrub ok 2024-03-26T05:24:39.891 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[29691]: 6.3 scrub ok 2024-03-26T05:24:39.891 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[29691]: 6.6 scrub starts 2024-03-26T05:24:39.891 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[29691]: 6.6 scrub ok 2024-03-26T05:24:39.891 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[40404]: pgmap v1164: 142 pgs: 142 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:39.891 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[40404]: 5.f scrub starts 2024-03-26T05:24:39.891 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[40404]: 5.f scrub ok 2024-03-26T05:24:39.891 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[40404]: 6.3 scrub starts 2024-03-26T05:24:39.891 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[40404]: 5.2 scrub starts 2024-03-26T05:24:39.891 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[40404]: 5.2 scrub ok 2024-03-26T05:24:39.892 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[40404]: 6.3 scrub ok 2024-03-26T05:24:39.892 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[40404]: 6.6 scrub starts 2024-03-26T05:24:39.892 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:39 smithi080 ceph-mon[40404]: 6.6 scrub ok 2024-03-26T05:24:40.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:39 smithi132 ceph-mon[33500]: pgmap v1164: 142 pgs: 142 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:39 smithi132 ceph-mon[33500]: 5.f scrub starts 2024-03-26T05:24:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:39 smithi132 ceph-mon[33500]: 5.f scrub ok 2024-03-26T05:24:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:39 smithi132 ceph-mon[33500]: 6.3 scrub starts 2024-03-26T05:24:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:39 smithi132 ceph-mon[33500]: 5.2 scrub starts 2024-03-26T05:24:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:39 smithi132 ceph-mon[33500]: 5.2 scrub ok 2024-03-26T05:24:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:39 smithi132 ceph-mon[33500]: 6.3 scrub ok 2024-03-26T05:24:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:39 smithi132 ceph-mon[33500]: 6.6 scrub starts 2024-03-26T05:24:40.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:39 smithi132 ceph-mon[33500]: 6.6 scrub ok 2024-03-26T05:24:41.047 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:40 smithi080 ceph-mon[40404]: 5.e deep-scrub starts 2024-03-26T05:24:41.047 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:40 smithi080 ceph-mon[40404]: 5.e deep-scrub ok 2024-03-26T05:24:41.047 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:40 smithi080 ceph-mon[40404]: 5.d scrub starts 2024-03-26T05:24:41.047 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:40 smithi080 ceph-mon[40404]: 5.d scrub ok 2024-03-26T05:24:41.047 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:40 smithi080 ceph-mon[29691]: 5.e deep-scrub starts 2024-03-26T05:24:41.047 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:40 smithi080 ceph-mon[29691]: 5.e deep-scrub ok 2024-03-26T05:24:41.047 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:40 smithi080 ceph-mon[29691]: 5.d scrub starts 2024-03-26T05:24:41.047 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:40 smithi080 ceph-mon[29691]: 5.d scrub ok 2024-03-26T05:24:41.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:40 smithi132 ceph-mon[33500]: 5.e deep-scrub starts 2024-03-26T05:24:41.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:40 smithi132 ceph-mon[33500]: 5.e deep-scrub ok 2024-03-26T05:24:41.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:40 smithi132 ceph-mon[33500]: 5.d scrub starts 2024-03-26T05:24:41.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:40 smithi132 ceph-mon[33500]: 5.d scrub ok 2024-03-26T05:24:41.850 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:41 smithi080 ceph-mon[40404]: pgmap v1165: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T05:24:41.850 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:41 smithi080 ceph-mon[29691]: pgmap v1165: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T05:24:42.179 INFO:teuthology.orchestra.run.smithi080.stdout:219043332398 2024-03-26T05:24:42.180 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T05:24:42.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:41 smithi132 ceph-mon[33500]: pgmap v1165: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T05:24:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:42 smithi080 ceph-mon[29691]: 3.d scrub starts 2024-03-26T05:24:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:42 smithi080 ceph-mon[29691]: 3.d scrub ok 2024-03-26T05:24:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:42 smithi080 ceph-mon[29691]: 3.a scrub starts 2024-03-26T05:24:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:42 smithi080 ceph-mon[29691]: 3.a scrub ok 2024-03-26T05:24:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:42 smithi080 ceph-mon[29691]: 5.b scrub starts 2024-03-26T05:24:43.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:42 smithi080 ceph-mon[29691]: 5.b scrub ok 2024-03-26T05:24:43.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:42 smithi080 ceph-mon[40404]: 3.d scrub starts 2024-03-26T05:24:43.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:42 smithi080 ceph-mon[40404]: 3.d scrub ok 2024-03-26T05:24:43.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:42 smithi080 ceph-mon[40404]: 3.a scrub starts 2024-03-26T05:24:43.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:42 smithi080 ceph-mon[40404]: 3.a scrub ok 2024-03-26T05:24:43.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:42 smithi080 ceph-mon[40404]: 5.b scrub starts 2024-03-26T05:24:43.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:42 smithi080 ceph-mon[40404]: 5.b scrub ok 2024-03-26T05:24:43.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:42 smithi132 ceph-mon[33500]: 3.d scrub starts 2024-03-26T05:24:43.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:42 smithi132 ceph-mon[33500]: 3.d scrub ok 2024-03-26T05:24:43.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:42 smithi132 ceph-mon[33500]: 3.a scrub starts 2024-03-26T05:24:43.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:42 smithi132 ceph-mon[33500]: 3.a scrub ok 2024-03-26T05:24:43.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:42 smithi132 ceph-mon[33500]: 5.b scrub starts 2024-03-26T05:24:43.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:42 smithi132 ceph-mon[33500]: 5.b scrub ok 2024-03-26T05:24:43.301 INFO:teuthology.orchestra.run.smithi080.stdout:176093659443 2024-03-26T05:24:43.301 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T05:24:44.105 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:24:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:24:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:24:44.106 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[29691]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-03-26T05:24:44.106 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[29691]: 5.1b scrub starts 2024-03-26T05:24:44.106 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[29691]: 5.1b scrub ok 2024-03-26T05:24:44.106 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[29691]: 1.6 scrub starts 2024-03-26T05:24:44.106 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[29691]: 1.6 scrub ok 2024-03-26T05:24:44.106 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[29691]: 5.a deep-scrub starts 2024-03-26T05:24:44.106 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[29691]: 5.a deep-scrub ok 2024-03-26T05:24:44.106 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[40404]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-03-26T05:24:44.106 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[40404]: 5.1b scrub starts 2024-03-26T05:24:44.107 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[40404]: 5.1b scrub ok 2024-03-26T05:24:44.107 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[40404]: 1.6 scrub starts 2024-03-26T05:24:44.107 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[40404]: 1.6 scrub ok 2024-03-26T05:24:44.107 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[40404]: 5.a deep-scrub starts 2024-03-26T05:24:44.107 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:43 smithi080 ceph-mon[40404]: 5.a deep-scrub ok 2024-03-26T05:24:44.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:43 smithi132 ceph-mon[33500]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-03-26T05:24:44.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:43 smithi132 ceph-mon[33500]: 5.1b scrub starts 2024-03-26T05:24:44.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:43 smithi132 ceph-mon[33500]: 5.1b scrub ok 2024-03-26T05:24:44.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:43 smithi132 ceph-mon[33500]: 1.6 scrub starts 2024-03-26T05:24:44.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:43 smithi132 ceph-mon[33500]: 1.6 scrub ok 2024-03-26T05:24:44.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:43 smithi132 ceph-mon[33500]: 5.a deep-scrub starts 2024-03-26T05:24:44.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:43 smithi132 ceph-mon[33500]: 5.a deep-scrub ok 2024-03-26T05:24:45.046 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:44 smithi080 ceph-mon[40404]: 5.4 scrub starts 2024-03-26T05:24:45.046 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:44 smithi080 ceph-mon[40404]: 5.4 scrub ok 2024-03-26T05:24:45.047 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:44 smithi080 ceph-mon[29691]: 5.4 scrub starts 2024-03-26T05:24:45.047 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:44 smithi080 ceph-mon[29691]: 5.4 scrub ok 2024-03-26T05:24:45.074 INFO:teuthology.orchestra.run.smithi080.stdout:77309411651 2024-03-26T05:24:45.075 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T05:24:45.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:44 smithi132 ceph-mon[33500]: 5.4 scrub starts 2024-03-26T05:24:45.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:44 smithi132 ceph-mon[33500]: 5.4 scrub ok 2024-03-26T05:24:45.891 INFO:teuthology.orchestra.run.smithi080.stdout:115964117309 2024-03-26T05:24:45.891 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T05:24:46.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:45 smithi080 ceph-mon[40404]: pgmap v1167: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:46.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:45 smithi080 ceph-mon[40404]: 4.6 scrub starts 2024-03-26T05:24:46.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:45 smithi080 ceph-mon[40404]: 4.6 scrub ok 2024-03-26T05:24:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:45 smithi080 ceph-mon[29691]: pgmap v1167: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:45 smithi080 ceph-mon[29691]: 4.6 scrub starts 2024-03-26T05:24:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:45 smithi080 ceph-mon[29691]: 4.6 scrub ok 2024-03-26T05:24:46.160 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:45 smithi132 ceph-mon[33500]: pgmap v1167: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:46.161 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:45 smithi132 ceph-mon[33500]: 4.6 scrub starts 2024-03-26T05:24:46.161 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:45 smithi132 ceph-mon[33500]: 4.6 scrub ok 2024-03-26T05:24:46.370 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:46.412 INFO:teuthology.orchestra.run.smithi080.stdout:34359738701 2024-03-26T05:24:46.412 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T05:24:46.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:24:46 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:24:46.782 INFO:teuthology.orchestra.run.smithi080.stdout:1438814044272 2024-03-26T05:24:46.782 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T05:24:46.850 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:47.031 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:46 smithi080 ceph-mon[40404]: 6.16 scrub starts 2024-03-26T05:24:47.031 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:46 smithi080 ceph-mon[40404]: 6.16 scrub ok 2024-03-26T05:24:47.032 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:46 smithi080 ceph-mon[29691]: 6.16 scrub starts 2024-03-26T05:24:47.032 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:46 smithi080 ceph-mon[29691]: 6.16 scrub ok 2024-03-26T05:24:47.260 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:46 smithi132 ceph-mon[33500]: 6.16 scrub starts 2024-03-26T05:24:47.261 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:46 smithi132 ceph-mon[33500]: 6.16 scrub ok 2024-03-26T05:24:48.078 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:47 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:48.078 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:47 smithi080 ceph-mon[40404]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:48.078 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:47 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:48.078 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:47 smithi080 ceph-mon[29691]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:48.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:47 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:48.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:47 smithi132 ceph-mon[33500]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:49.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[29691]: 5.1e scrub starts 2024-03-26T05:24:49.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[29691]: 5.1e scrub ok 2024-03-26T05:24:49.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[29691]: 4.4 deep-scrub starts 2024-03-26T05:24:49.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[29691]: 4.4 deep-scrub ok 2024-03-26T05:24:49.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[29691]: pgmap v1169: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:49.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[29691]: 6.14 scrub starts 2024-03-26T05:24:49.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[29691]: 6.14 scrub ok 2024-03-26T05:24:49.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[40404]: 5.1e scrub starts 2024-03-26T05:24:49.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[40404]: 5.1e scrub ok 2024-03-26T05:24:49.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[40404]: 4.4 deep-scrub starts 2024-03-26T05:24:49.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[40404]: 4.4 deep-scrub ok 2024-03-26T05:24:49.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[40404]: pgmap v1169: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:49.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[40404]: 6.14 scrub starts 2024-03-26T05:24:49.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:48 smithi080 ceph-mon[40404]: 6.14 scrub ok 2024-03-26T05:24:49.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:48 smithi132 ceph-mon[33500]: 5.1e scrub starts 2024-03-26T05:24:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:48 smithi132 ceph-mon[33500]: 5.1e scrub ok 2024-03-26T05:24:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:48 smithi132 ceph-mon[33500]: 4.4 deep-scrub starts 2024-03-26T05:24:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:48 smithi132 ceph-mon[33500]: 4.4 deep-scrub ok 2024-03-26T05:24:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:48 smithi132 ceph-mon[33500]: pgmap v1169: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:48 smithi132 ceph-mon[33500]: 6.14 scrub starts 2024-03-26T05:24:49.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:48 smithi132 ceph-mon[33500]: 6.14 scrub ok 2024-03-26T05:24:49.800 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:49.803 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:49.803 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:49.803 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:49.832 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:49 smithi080 ceph-mon[29691]: 5.8 scrub starts 2024-03-26T05:24:49.832 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:49 smithi080 ceph-mon[29691]: 5.8 scrub ok 2024-03-26T05:24:50.087 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:49 smithi080 ceph-mon[40404]: 5.8 scrub starts 2024-03-26T05:24:50.087 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:49 smithi080 ceph-mon[40404]: 5.8 scrub ok 2024-03-26T05:24:50.209 INFO:teuthology.orchestra.run.smithi080.stdout:2151778615316 2024-03-26T05:24:50.210 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T05:24:50.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:49 smithi132 ceph-mon[33500]: 5.8 scrub starts 2024-03-26T05:24:50.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:49 smithi132 ceph-mon[33500]: 5.8 scrub ok 2024-03-26T05:24:51.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:50 smithi080 ceph-mon[40404]: 1.5 scrub starts 2024-03-26T05:24:51.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:50 smithi080 ceph-mon[40404]: 1.5 scrub ok 2024-03-26T05:24:51.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:50 smithi080 ceph-mon[40404]: 5.0 scrub starts 2024-03-26T05:24:51.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:50 smithi080 ceph-mon[40404]: 5.0 scrub ok 2024-03-26T05:24:51.009 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:50 smithi080 ceph-mon[40404]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:51.009 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:50 smithi080 ceph-mon[29691]: 1.5 scrub starts 2024-03-26T05:24:51.009 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:50 smithi080 ceph-mon[29691]: 1.5 scrub ok 2024-03-26T05:24:51.009 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:50 smithi080 ceph-mon[29691]: 5.0 scrub starts 2024-03-26T05:24:51.009 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:50 smithi080 ceph-mon[29691]: 5.0 scrub ok 2024-03-26T05:24:51.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:50 smithi080 ceph-mon[29691]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:51.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:50 smithi132 ceph-mon[33500]: 1.5 scrub starts 2024-03-26T05:24:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:50 smithi132 ceph-mon[33500]: 1.5 scrub ok 2024-03-26T05:24:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:50 smithi132 ceph-mon[33500]: 5.0 scrub starts 2024-03-26T05:24:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:50 smithi132 ceph-mon[33500]: 5.0 scrub ok 2024-03-26T05:24:51.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:50 smithi132 ceph-mon[33500]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:51.941 INFO:teuthology.orchestra.run.smithi080.stdout:219043332400 2024-03-26T05:24:52.107 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[40404]: 4.8 scrub starts 2024-03-26T05:24:52.108 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[40404]: 4.8 scrub ok 2024-03-26T05:24:52.108 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[40404]: 3.9 deep-scrub starts 2024-03-26T05:24:52.108 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[40404]: 3.9 deep-scrub ok 2024-03-26T05:24:52.108 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[40404]: 4.a deep-scrub starts 2024-03-26T05:24:52.108 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[40404]: 4.a deep-scrub ok 2024-03-26T05:24:52.108 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:52.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[29691]: 4.8 scrub starts 2024-03-26T05:24:52.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[29691]: 4.8 scrub ok 2024-03-26T05:24:52.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[29691]: 3.9 deep-scrub starts 2024-03-26T05:24:52.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[29691]: 3.9 deep-scrub ok 2024-03-26T05:24:52.109 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[29691]: 4.a deep-scrub starts 2024-03-26T05:24:52.109 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[29691]: 4.a deep-scrub ok 2024-03-26T05:24:52.109 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:51 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:52.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:51 smithi132 ceph-mon[33500]: 4.8 scrub starts 2024-03-26T05:24:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:51 smithi132 ceph-mon[33500]: 4.8 scrub ok 2024-03-26T05:24:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:51 smithi132 ceph-mon[33500]: 3.9 deep-scrub starts 2024-03-26T05:24:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:51 smithi132 ceph-mon[33500]: 3.9 deep-scrub ok 2024-03-26T05:24:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:51 smithi132 ceph-mon[33500]: 4.a deep-scrub starts 2024-03-26T05:24:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:51 smithi132 ceph-mon[33500]: 4.a deep-scrub ok 2024-03-26T05:24:52.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:51 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:52.515 INFO:teuthology.orchestra.run.smithi080.stdout:55834575177 2024-03-26T05:24:53.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:24:53.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:24:53.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[29691]: 5.16 scrub starts 2024-03-26T05:24:53.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[29691]: 5.16 scrub ok 2024-03-26T05:24:53.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[29691]: 4.d deep-scrub starts 2024-03-26T05:24:53.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[29691]: 4.d deep-scrub ok 2024-03-26T05:24:53.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[29691]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T05:24:53.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[29691]: 3.10 scrub starts 2024-03-26T05:24:53.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[29691]: 3.10 scrub ok 2024-03-26T05:24:53.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/3072778855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T05:24:53.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[29691]: pgmap v1172: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:24:53.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1969143675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T05:24:53.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:24:53.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:24:53.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[40404]: 5.16 scrub starts 2024-03-26T05:24:53.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[40404]: 5.16 scrub ok 2024-03-26T05:24:53.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[40404]: 4.d deep-scrub starts 2024-03-26T05:24:53.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[40404]: 4.d deep-scrub ok 2024-03-26T05:24:53.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[40404]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T05:24:53.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[40404]: 3.10 scrub starts 2024-03-26T05:24:53.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[40404]: 3.10 scrub ok 2024-03-26T05:24:53.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/3072778855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T05:24:53.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[40404]: pgmap v1172: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:24:53.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:52 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1969143675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T05:24:53.252 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575174 got 55834575177 for osd.1 2024-03-26T05:24:53.252 DEBUG:teuthology.parallel:result is None 2024-03-26T05:24:53.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:52 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:24:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:52 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:24:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:52 smithi132 ceph-mon[33500]: 5.16 scrub starts 2024-03-26T05:24:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:52 smithi132 ceph-mon[33500]: 5.16 scrub ok 2024-03-26T05:24:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:52 smithi132 ceph-mon[33500]: 4.d deep-scrub starts 2024-03-26T05:24:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:52 smithi132 ceph-mon[33500]: 4.d deep-scrub ok 2024-03-26T05:24:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:52 smithi132 ceph-mon[33500]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T05:24:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:52 smithi132 ceph-mon[33500]: 3.10 scrub starts 2024-03-26T05:24:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:52 smithi132 ceph-mon[33500]: 3.10 scrub ok 2024-03-26T05:24:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:52 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/3072778855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T05:24:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:52 smithi132 ceph-mon[33500]: pgmap v1172: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:24:53.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:52 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1969143675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T05:24:54.185 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:24:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:24:54.821 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:54.852 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332398 got 219043332400 for osd.6 2024-03-26T05:24:54.852 DEBUG:teuthology.parallel:result is None 2024-03-26T05:24:55.088 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-mon[40404]: 3.5 scrub starts 2024-03-26T05:24:55.089 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-mon[40404]: 3.5 scrub ok 2024-03-26T05:24:55.089 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-mon[40404]: purged_snaps scrub starts 2024-03-26T05:24:55.089 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-mon[40404]: purged_snaps scrub ok 2024-03-26T05:24:55.089 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-mon[40404]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T05:24:55.089 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-mon[40404]: pgmap v1174: 142 pgs: 142 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:24:55.089 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-mon[29691]: 3.5 scrub starts 2024-03-26T05:24:55.089 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-mon[29691]: 3.5 scrub ok 2024-03-26T05:24:55.089 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-mon[29691]: purged_snaps scrub starts 2024-03-26T05:24:55.089 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-mon[29691]: purged_snaps scrub ok 2024-03-26T05:24:55.090 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-mon[29691]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T05:24:55.090 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:54 smithi080 ceph-mon[29691]: pgmap v1174: 142 pgs: 142 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:24:55.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:54 smithi132 ceph-mon[33500]: 3.5 scrub starts 2024-03-26T05:24:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:54 smithi132 ceph-mon[33500]: 3.5 scrub ok 2024-03-26T05:24:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:54 smithi132 ceph-mon[33500]: purged_snaps scrub starts 2024-03-26T05:24:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:54 smithi132 ceph-mon[33500]: purged_snaps scrub ok 2024-03-26T05:24:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:54 smithi132 ceph-mon[33500]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T05:24:55.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:54 smithi132 ceph-mon[33500]: pgmap v1174: 142 pgs: 142 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:24:56.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:55 smithi080 ceph-mon[29691]: 1.9 scrub starts 2024-03-26T05:24:56.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:55 smithi080 ceph-mon[29691]: 1.9 scrub ok 2024-03-26T05:24:56.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:55 smithi080 ceph-mon[29691]: 6.b scrub starts 2024-03-26T05:24:56.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:55 smithi080 ceph-mon[40404]: 1.9 scrub starts 2024-03-26T05:24:56.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:55 smithi080 ceph-mon[40404]: 1.9 scrub ok 2024-03-26T05:24:56.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:55 smithi080 ceph-mon[40404]: 6.b scrub starts 2024-03-26T05:24:56.165 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:55 smithi132 ceph-mon[33500]: 1.9 scrub starts 2024-03-26T05:24:56.165 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:55 smithi132 ceph-mon[33500]: 1.9 scrub ok 2024-03-26T05:24:56.165 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:55 smithi132 ceph-mon[33500]: 6.b scrub starts 2024-03-26T05:24:56.365 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:24:56.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:24:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:24:57.020 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:56 smithi080 ceph-mon[40404]: 6.b scrub ok 2024-03-26T05:24:57.021 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:56 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:57.021 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:56 smithi080 ceph-mon[40404]: pgmap v1175: 142 pgs: 142 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:57.021 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:56 smithi080 ceph-mon[29691]: 6.b scrub ok 2024-03-26T05:24:57.021 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:56 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:57.021 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:56 smithi080 ceph-mon[29691]: pgmap v1175: 142 pgs: 142 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:57.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:56 smithi132 ceph-mon[33500]: 6.b scrub ok 2024-03-26T05:24:57.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:56 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:57.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:56 smithi132 ceph-mon[33500]: pgmap v1175: 142 pgs: 142 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:58.086 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:57 smithi080 ceph-mon[40404]: 1.8 scrub starts 2024-03-26T05:24:58.086 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:57 smithi080 ceph-mon[40404]: 1.8 scrub ok 2024-03-26T05:24:58.086 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:57 smithi080 ceph-mon[40404]: 6.0 scrub starts 2024-03-26T05:24:58.086 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:57 smithi080 ceph-mon[40404]: 6.0 scrub ok 2024-03-26T05:24:58.086 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:57 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:24:58.087 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:57 smithi080 ceph-mon[29691]: 1.8 scrub starts 2024-03-26T05:24:58.087 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:57 smithi080 ceph-mon[29691]: 1.8 scrub ok 2024-03-26T05:24:58.087 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:57 smithi080 ceph-mon[29691]: 6.0 scrub starts 2024-03-26T05:24:58.087 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:57 smithi080 ceph-mon[29691]: 6.0 scrub ok 2024-03-26T05:24:58.087 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:57 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:24:58.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:57 smithi132 ceph-mon[33500]: 1.8 scrub starts 2024-03-26T05:24:58.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:57 smithi132 ceph-mon[33500]: 1.8 scrub ok 2024-03-26T05:24:58.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:57 smithi132 ceph-mon[33500]: 6.0 scrub starts 2024-03-26T05:24:58.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:57 smithi132 ceph-mon[33500]: 6.0 scrub ok 2024-03-26T05:24:58.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:57 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:24:58.333 INFO:teuthology.orchestra.run.smithi080.stdout:34359738703 2024-03-26T05:24:58.549 INFO:teuthology.orchestra.run.smithi080.stdout:176093659446 2024-03-26T05:24:59.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: 6.19 scrub starts 2024-03-26T05:24:59.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: 6.19 scrub ok 2024-03-26T05:24:59.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: 2.0 scrub starts 2024-03-26T05:24:59.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: 2.0 scrub ok 2024-03-26T05:24:59.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: 6.f deep-scrub starts 2024-03-26T05:24:59.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: pgmap v1176: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:59.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2067070928' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T05:24:59.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:24:59.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:24:59.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:24:59.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: 6.f deep-scrub ok 2024-03-26T05:24:59.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/2190058781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T05:24:59.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/197021885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T05:24:59.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: 6.19 scrub starts 2024-03-26T05:24:59.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: 6.19 scrub ok 2024-03-26T05:24:59.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: 2.0 scrub starts 2024-03-26T05:24:59.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: 2.0 scrub ok 2024-03-26T05:24:59.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: 6.f deep-scrub starts 2024-03-26T05:24:59.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: pgmap v1176: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:59.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2067070928' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T05:24:59.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:24:59.148 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:24:59.148 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:24:59.148 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: 6.f deep-scrub ok 2024-03-26T05:24:59.148 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/2190058781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T05:24:59.148 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:58 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/197021885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T05:24:59.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: 6.19 scrub starts 2024-03-26T05:24:59.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: 6.19 scrub ok 2024-03-26T05:24:59.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: 2.0 scrub starts 2024-03-26T05:24:59.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: 2.0 scrub ok 2024-03-26T05:24:59.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: 6.f deep-scrub starts 2024-03-26T05:24:59.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: pgmap v1176: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:59.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2067070928' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T05:24:59.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:24:59.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:24:59.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:24:59.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: 6.f deep-scrub ok 2024-03-26T05:24:59.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/2190058781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T05:24:59.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:58 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/197021885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T05:24:59.585 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738701 got 34359738703 for osd.0 2024-03-26T05:24:59.586 DEBUG:teuthology.parallel:result is None 2024-03-26T05:25:00.060 INFO:teuthology.orchestra.run.smithi080.stdout:77309411654 2024-03-26T05:25:00.163 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:59 smithi080 ceph-mon[29691]: 5.1f scrub starts 2024-03-26T05:25:00.163 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:59 smithi080 ceph-mon[29691]: 5.1f scrub ok 2024-03-26T05:25:00.163 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:59 smithi080 ceph-mon[29691]: 3.4 scrub starts 2024-03-26T05:25:00.163 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:59 smithi080 ceph-mon[29691]: 3.4 scrub ok 2024-03-26T05:25:00.163 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:24:59 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1103652697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T05:25:00.163 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:59 smithi080 ceph-mon[40404]: 5.1f scrub starts 2024-03-26T05:25:00.163 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:59 smithi080 ceph-mon[40404]: 5.1f scrub ok 2024-03-26T05:25:00.163 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:59 smithi080 ceph-mon[40404]: 3.4 scrub starts 2024-03-26T05:25:00.164 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:59 smithi080 ceph-mon[40404]: 3.4 scrub ok 2024-03-26T05:25:00.164 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:24:59 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1103652697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T05:25:00.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:59 smithi132 ceph-mon[33500]: 5.1f scrub starts 2024-03-26T05:25:00.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:59 smithi132 ceph-mon[33500]: 5.1f scrub ok 2024-03-26T05:25:00.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:59 smithi132 ceph-mon[33500]: 3.4 scrub starts 2024-03-26T05:25:00.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:59 smithi132 ceph-mon[33500]: 3.4 scrub ok 2024-03-26T05:25:00.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:24:59 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1103652697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T05:25:00.328 INFO:teuthology.orchestra.run.smithi080.stdout:115964117312 2024-03-26T05:25:00.861 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117309 got 115964117312 for osd.3 2024-03-26T05:25:00.861 DEBUG:teuthology.parallel:result is None 2024-03-26T05:25:01.108 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:00 smithi080 ceph-mon[40404]: 6.17 scrub starts 2024-03-26T05:25:01.108 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:00 smithi080 ceph-mon[40404]: 6.17 scrub ok 2024-03-26T05:25:01.108 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:00 smithi080 ceph-mon[40404]: pgmap v1177: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-03-26T05:25:01.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:00 smithi080 ceph-mon[29691]: 6.17 scrub starts 2024-03-26T05:25:01.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:00 smithi080 ceph-mon[29691]: 6.17 scrub ok 2024-03-26T05:25:01.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:00 smithi080 ceph-mon[29691]: pgmap v1177: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-03-26T05:25:01.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:00 smithi132 ceph-mon[33500]: 6.17 scrub starts 2024-03-26T05:25:01.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:00 smithi132 ceph-mon[33500]: 6.17 scrub ok 2024-03-26T05:25:01.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:00 smithi132 ceph-mon[33500]: pgmap v1177: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-03-26T05:25:01.793 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411651 got 77309411654 for osd.2 2024-03-26T05:25:01.794 DEBUG:teuthology.parallel:result is None 2024-03-26T05:25:02.070 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:01 smithi080 ceph-mon[40404]: 4.1d scrub starts 2024-03-26T05:25:02.070 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:01 smithi080 ceph-mon[40404]: 4.1d scrub ok 2024-03-26T05:25:02.071 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:01 smithi080 ceph-mon[29691]: 4.1d scrub starts 2024-03-26T05:25:02.071 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:01 smithi080 ceph-mon[29691]: 4.1d scrub ok 2024-03-26T05:25:02.177 INFO:teuthology.orchestra.run.smithi080.stdout:1438814044275 2024-03-26T05:25:02.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:01 smithi132 ceph-mon[33500]: 4.1d scrub starts 2024-03-26T05:25:02.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:01 smithi132 ceph-mon[33500]: 4.1d scrub ok 2024-03-26T05:25:02.372 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659443 got 176093659446 for osd.5 2024-03-26T05:25:02.372 DEBUG:teuthology.parallel:result is None 2024-03-26T05:25:02.869 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1438814044272 got 1438814044275 for osd.7 2024-03-26T05:25:02.869 DEBUG:teuthology.parallel:result is None 2024-03-26T05:25:03.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:02 smithi132 ceph-mon[33500]: 3.14 deep-scrub starts 2024-03-26T05:25:03.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:02 smithi132 ceph-mon[33500]: 3.14 deep-scrub ok 2024-03-26T05:25:03.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:02 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/180239657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T05:25:03.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:02 smithi132 ceph-mon[33500]: 5.c scrub starts 2024-03-26T05:25:03.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:02 smithi132 ceph-mon[33500]: 5.c scrub ok 2024-03-26T05:25:03.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:02 smithi132 ceph-mon[33500]: pgmap v1178: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:25:03.337 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:02 smithi080 ceph-mon[29691]: 3.14 deep-scrub starts 2024-03-26T05:25:03.337 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:02 smithi080 ceph-mon[29691]: 3.14 deep-scrub ok 2024-03-26T05:25:03.337 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:02 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/180239657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T05:25:03.337 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:02 smithi080 ceph-mon[29691]: 5.c scrub starts 2024-03-26T05:25:03.337 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:02 smithi080 ceph-mon[29691]: 5.c scrub ok 2024-03-26T05:25:03.337 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:02 smithi080 ceph-mon[29691]: pgmap v1178: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:25:03.337 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:02 smithi080 ceph-mon[40404]: 3.14 deep-scrub starts 2024-03-26T05:25:03.337 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:02 smithi080 ceph-mon[40404]: 3.14 deep-scrub ok 2024-03-26T05:25:03.337 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:02 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/180239657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T05:25:03.337 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:02 smithi080 ceph-mon[40404]: 5.c scrub starts 2024-03-26T05:25:03.338 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:02 smithi080 ceph-mon[40404]: 5.c scrub ok 2024-03-26T05:25:03.338 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:02 smithi080 ceph-mon[40404]: pgmap v1178: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:25:03.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:03 smithi080 ceph-mon[40404]: 1.1 deep-scrub starts 2024-03-26T05:25:03.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:03 smithi080 ceph-mon[40404]: 1.1 deep-scrub ok 2024-03-26T05:25:03.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:03 smithi080 ceph-mon[40404]: 3.1 deep-scrub starts 2024-03-26T05:25:03.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:03 smithi080 ceph-mon[40404]: 3.1 deep-scrub ok 2024-03-26T05:25:03.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:03 smithi080 ceph-mon[40404]: 4.3 scrub starts 2024-03-26T05:25:03.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:03 smithi080 ceph-mon[40404]: 4.3 scrub ok 2024-03-26T05:25:03.924 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:03 smithi080 ceph-mon[29691]: 1.1 deep-scrub starts 2024-03-26T05:25:03.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:03 smithi080 ceph-mon[29691]: 1.1 deep-scrub ok 2024-03-26T05:25:03.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:03 smithi080 ceph-mon[29691]: 3.1 deep-scrub starts 2024-03-26T05:25:03.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:03 smithi080 ceph-mon[29691]: 3.1 deep-scrub ok 2024-03-26T05:25:03.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:03 smithi080 ceph-mon[29691]: 4.3 scrub starts 2024-03-26T05:25:03.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:03 smithi080 ceph-mon[29691]: 4.3 scrub ok 2024-03-26T05:25:04.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:03 smithi132 ceph-mon[33500]: 1.1 deep-scrub starts 2024-03-26T05:25:04.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:03 smithi132 ceph-mon[33500]: 1.1 deep-scrub ok 2024-03-26T05:25:04.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:03 smithi132 ceph-mon[33500]: 3.1 deep-scrub starts 2024-03-26T05:25:04.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:03 smithi132 ceph-mon[33500]: 3.1 deep-scrub ok 2024-03-26T05:25:04.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:03 smithi132 ceph-mon[33500]: 4.3 scrub starts 2024-03-26T05:25:04.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:03 smithi132 ceph-mon[33500]: 4.3 scrub ok 2024-03-26T05:25:04.366 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:04 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:25:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:25:04.381 INFO:teuthology.orchestra.run.smithi080.stdout:2151778615319 2024-03-26T05:25:05.012 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:04 smithi080 ceph-mon[29691]: 6.1d scrub starts 2024-03-26T05:25:05.012 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:04 smithi080 ceph-mon[29691]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-03-26T05:25:05.012 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:04 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/4189079172' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T05:25:05.012 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:04 smithi080 ceph-mon[29691]: 6.1d scrub ok 2024-03-26T05:25:05.013 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:04 smithi080 ceph-mon[40404]: 6.1d scrub starts 2024-03-26T05:25:05.013 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:04 smithi080 ceph-mon[40404]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-03-26T05:25:05.013 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:04 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/4189079172' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T05:25:05.013 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:04 smithi080 ceph-mon[40404]: 6.1d scrub ok 2024-03-26T05:25:05.039 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2151778615316 got 2151778615319 for osd.4 2024-03-26T05:25:05.039 DEBUG:teuthology.parallel:result is None 2024-03-26T05:25:05.039 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T05:25:05.039 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:25:05.282 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:25:05.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:04 smithi132 ceph-mon[33500]: 6.1d scrub starts 2024-03-26T05:25:05.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:04 smithi132 ceph-mon[33500]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-03-26T05:25:05.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:04 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/4189079172' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T05:25:05.294 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:04 smithi132 ceph-mon[33500]: 6.1d scrub ok 2024-03-26T05:25:06.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:05 smithi080 ceph-mon[40404]: 4.15 scrub starts 2024-03-26T05:25:06.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:05 smithi080 ceph-mon[40404]: 4.15 scrub ok 2024-03-26T05:25:06.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:05 smithi080 ceph-mon[29691]: 4.15 scrub starts 2024-03-26T05:25:06.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:05 smithi080 ceph-mon[29691]: 4.15 scrub ok 2024-03-26T05:25:06.172 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:05 smithi132 ceph-mon[33500]: 4.15 scrub starts 2024-03-26T05:25:06.173 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:05 smithi132 ceph-mon[33500]: 4.15 scrub ok 2024-03-26T05:25:06.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:25:06 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:25:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[29691]: 3.16 scrub starts 2024-03-26T05:25:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[29691]: 3.16 scrub ok 2024-03-26T05:25:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[29691]: 4.c scrub starts 2024-03-26T05:25:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[29691]: 4.c scrub ok 2024-03-26T05:25:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[29691]: 5.5 scrub starts 2024-03-26T05:25:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[29691]: 5.5 scrub ok 2024-03-26T05:25:07.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:07.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[29691]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:07.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[29691]: pgmap v1180: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:25:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[40404]: 3.16 scrub starts 2024-03-26T05:25:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[40404]: 3.16 scrub ok 2024-03-26T05:25:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[40404]: 4.c scrub starts 2024-03-26T05:25:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[40404]: 4.c scrub ok 2024-03-26T05:25:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[40404]: 5.5 scrub starts 2024-03-26T05:25:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[40404]: 5.5 scrub ok 2024-03-26T05:25:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[40404]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:07.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:06 smithi080 ceph-mon[40404]: pgmap v1180: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:25:07.209 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:25:07.212 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:25:07.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:06 smithi132 ceph-mon[33500]: 3.16 scrub starts 2024-03-26T05:25:07.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:06 smithi132 ceph-mon[33500]: 3.16 scrub ok 2024-03-26T05:25:07.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:06 smithi132 ceph-mon[33500]: 4.c scrub starts 2024-03-26T05:25:07.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:06 smithi132 ceph-mon[33500]: 4.c scrub ok 2024-03-26T05:25:07.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:06 smithi132 ceph-mon[33500]: 5.5 scrub starts 2024-03-26T05:25:07.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:06 smithi132 ceph-mon[33500]: 5.5 scrub ok 2024-03-26T05:25:07.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:06 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:07.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:06 smithi132 ceph-mon[33500]: from='client.24382 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:07.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:06 smithi132 ceph-mon[33500]: pgmap v1180: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:25:07.815 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":1180,"stamp":"2024-03-26T05:25:06.318279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":2485,"num_read_kb":2205,"num_write":1200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":929567,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2858,"ondisk_log_size":2858,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":434,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6887864,"kb_used_data":1152312,"kb_used_omap":21,"kb_used_meta":5735466,"kb_avail":743073352,"statfs":{"total":767960285184,"available":760907112448,"internally_reserved":0,"allocated":1179967488,"data_stored":1177385173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22119,"internal_metadata":5873117593},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002920"},"pg_stats":[{"pgid":"4.c","version":"524'35","reported_seq":114,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866935+0000","last_change":"2024-03-26T05:23:57.476800+0000","last_active":"2024-03-26T05:24:53.866935+0000","last_peered":"2024-03-26T05:24:53.866935+0000","last_clean":"2024-03-26T05:24:53.866935+0000","last_became_active":"2024-03-26T05:23:23.934466+0000","last_became_peered":"2024-03-26T05:23:23.934466+0000","last_unstale":"2024-03-26T05:24:53.866935+0000","last_undegraded":"2024-03-26T05:24:53.866935+0000","last_fullsized":"2024-03-26T05:24:53.866935+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"524'35","last_scrub_stamp":"2024-03-26T05:23:57.476717+0000","last_deep_scrub":"524'35","last_deep_scrub_stamp":"2024-03-26T05:23:57.476717+0000","last_clean_scrub_stamp":"2024-03-26T05:23:57.476717+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:02.375662+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1060,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628245+0000","last_change":"2024-03-26T05:24:27.927986+0000","last_active":"2024-03-26T05:24:54.628245+0000","last_peered":"2024-03-26T05:24:54.628245+0000","last_clean":"2024-03-26T05:24:54.628245+0000","last_became_active":"2024-03-26T05:23:23.934367+0000","last_became_peered":"2024-03-26T05:23:23.934367+0000","last_unstale":"2024-03-26T05:24:54.628245+0000","last_undegraded":"2024-03-26T05:24:54.628245+0000","last_fullsized":"2024-03-26T05:24:54.628245+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:27.927903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:24:27.927903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:37.235808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"508'15","reported_seq":1410,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704427+0000","last_change":"2024-03-26T05:24:40.508225+0000","last_active":"2024-03-26T05:24:58.704427+0000","last_peered":"2024-03-26T05:24:58.704427+0000","last_clean":"2024-03-26T05:24:58.704427+0000","last_became_active":"2024-03-26T05:23:26.805008+0000","last_became_peered":"2024-03-26T05:23:26.805008+0000","last_unstale":"2024-03-26T05:24:58.704427+0000","last_undegraded":"2024-03-26T05:24:58.704427+0000","last_fullsized":"2024-03-26T05:24:58.704427+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'15","last_scrub_stamp":"2024-03-26T05:24:40.508138+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:24:40.508138+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:09.226754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.867118+0000","last_change":"2024-03-26T05:24:09.665071+0000","last_active":"2024-03-26T05:24:53.867118+0000","last_peered":"2024-03-26T05:24:53.867118+0000","last_clean":"2024-03-26T05:24:53.867118+0000","last_became_active":"2024-03-26T05:23:23.943566+0000","last_became_peered":"2024-03-26T05:23:23.943566+0000","last_unstale":"2024-03-26T05:24:53.867118+0000","last_undegraded":"2024-03-26T05:24:53.867118+0000","last_fullsized":"2024-03-26T05:24:53.867118+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:09.664959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:09.664959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:21.887749+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"88'128","reported_seq":1233,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628098+0000","last_change":"2024-03-26T05:24:54.628147+0000","last_active":"2024-03-26T05:24:54.628098+0000","last_peered":"2024-03-26T05:24:54.628098+0000","last_clean":"2024-03-26T05:24:54.628098+0000","last_became_active":"2024-03-26T05:15:33.616505+0000","last_became_peered":"2024-03-26T05:15:33.616505+0000","last_unstale":"2024-03-26T05:24:54.628098+0000","last_undegraded":"2024-03-26T05:24:54.628098+0000","last_fullsized":"2024-03-26T05:24:54.628098+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:54.627969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:54.627969+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:03.373012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"504'53","reported_seq":131,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.867097+0000","last_change":"2024-03-26T05:24:51.724793+0000","last_active":"2024-03-26T05:24:53.867097+0000","last_peered":"2024-03-26T05:24:53.867097+0000","last_clean":"2024-03-26T05:24:53.867097+0000","last_became_active":"2024-03-26T05:23:23.936434+0000","last_became_peered":"2024-03-26T05:23:23.936434+0000","last_unstale":"2024-03-26T05:24:53.867097+0000","last_undegraded":"2024-03-26T05:24:53.867097+0000","last_fullsized":"2024-03-26T05:24:53.867097+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'53","last_scrub_stamp":"2024-03-26T05:24:51.724667+0000","last_deep_scrub":"504'53","last_deep_scrub_stamp":"2024-03-26T05:24:51.724667+0000","last_clean_scrub_stamp":"2024-03-26T05:24:51.724667+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:19.890634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1108,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.945645+0000","last_change":"2024-03-26T05:24:41.749017+0000","last_active":"2024-03-26T05:24:54.945645+0000","last_peered":"2024-03-26T05:24:54.945645+0000","last_clean":"2024-03-26T05:24:54.945645+0000","last_became_active":"2024-03-26T05:23:23.935162+0000","last_became_peered":"2024-03-26T05:23:23.935162+0000","last_unstale":"2024-03-26T05:24:54.945645+0000","last_undegraded":"2024-03-26T05:24:54.945645+0000","last_fullsized":"2024-03-26T05:24:54.945645+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:41.748912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:24:41.748912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:00.092226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1047,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613898+0000","last_change":"2024-03-26T05:23:47.382629+0000","last_active":"2024-03-26T05:24:54.613898+0000","last_peered":"2024-03-26T05:24:54.613898+0000","last_clean":"2024-03-26T05:24:54.613898+0000","last_became_active":"2024-03-26T05:23:23.924351+0000","last_became_peered":"2024-03-26T05:23:23.924351+0000","last_unstale":"2024-03-26T05:24:54.613898+0000","last_undegraded":"2024-03-26T05:24:54.613898+0000","last_fullsized":"2024-03-26T05:24:54.613898+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:47.382551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:23:47.382551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1054,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.516785+0000","last_change":"2024-03-26T05:24:58.516825+0000","last_active":"2024-03-26T05:24:58.516785+0000","last_peered":"2024-03-26T05:24:58.516785+0000","last_clean":"2024-03-26T05:24:58.516785+0000","last_became_active":"2024-03-26T05:23:23.943972+0000","last_became_peered":"2024-03-26T05:23:23.943972+0000","last_unstale":"2024-03-26T05:24:58.516785+0000","last_undegraded":"2024-03-26T05:24:58.516785+0000","last_fullsized":"2024-03-26T05:24:58.516785+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:58.516670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:58.516670+0000","last_clean_scrub_stamp":"2024-03-26T05:24:58.516670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.259476+0000","scrub_duration":230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"88'128","reported_seq":629,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:56.447301+0000","last_change":"2024-03-26T05:24:56.447341+0000","last_active":"2024-03-26T05:24:56.447301+0000","last_peered":"2024-03-26T05:24:56.447301+0000","last_clean":"2024-03-26T05:24:56.447301+0000","last_became_active":"2024-03-26T05:23:29.790949+0000","last_became_peered":"2024-03-26T05:23:29.790949+0000","last_unstale":"2024-03-26T05:24:56.447301+0000","last_undegraded":"2024-03-26T05:24:56.447301+0000","last_fullsized":"2024-03-26T05:24:56.447301+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:56.447202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:56.447202+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.745143+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"443'33","reported_seq":111,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866969+0000","last_change":"2024-03-26T05:24:05.544239+0000","last_active":"2024-03-26T05:24:53.866969+0000","last_peered":"2024-03-26T05:24:53.866969+0000","last_clean":"2024-03-26T05:24:53.866969+0000","last_became_active":"2024-03-26T05:23:23.942559+0000","last_became_peered":"2024-03-26T05:23:23.942559+0000","last_unstale":"2024-03-26T05:24:53.866969+0000","last_undegraded":"2024-03-26T05:24:53.866969+0000","last_fullsized":"2024-03-26T05:24:53.866969+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'33","last_scrub_stamp":"2024-03-26T05:24:05.544072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:05.544072+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:08.325712+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":105,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866283+0000","last_change":"2024-03-26T05:24:50.679081+0000","last_active":"2024-03-26T05:24:53.866283+0000","last_peered":"2024-03-26T05:24:53.866283+0000","last_clean":"2024-03-26T05:24:53.866283+0000","last_became_active":"2024-03-26T05:23:23.940539+0000","last_became_peered":"2024-03-26T05:23:23.940539+0000","last_unstale":"2024-03-26T05:24:53.866283+0000","last_undegraded":"2024-03-26T05:24:53.866283+0000","last_fullsized":"2024-03-26T05:24:53.866283+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:50.678973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:50.678973+0000","last_clean_scrub_stamp":"2024-03-26T05:24:50.678973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.106668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114413+0000","last_change":"2024-03-26T05:24:38.362574+0000","last_active":"2024-03-26T05:24:55.114413+0000","last_peered":"2024-03-26T05:24:55.114413+0000","last_clean":"2024-03-26T05:24:55.114413+0000","last_became_active":"2024-03-26T05:23:26.811352+0000","last_became_peered":"2024-03-26T05:23:26.811352+0000","last_unstale":"2024-03-26T05:24:55.114413+0000","last_undegraded":"2024-03-26T05:24:55.114413+0000","last_fullsized":"2024-03-26T05:24:55.114413+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:38.362489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:13:07.883082+0000","last_clean_scrub_stamp":"2024-03-26T05:24:38.362489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:06.510293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1052,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628824+0000","last_change":"2024-03-26T05:24:22.885221+0000","last_active":"2024-03-26T05:24:54.628824+0000","last_peered":"2024-03-26T05:24:54.628824+0000","last_clean":"2024-03-26T05:24:54.628824+0000","last_became_active":"2024-03-26T05:23:26.798774+0000","last_became_peered":"2024-03-26T05:23:26.798774+0000","last_unstale":"2024-03-26T05:24:54.628824+0000","last_undegraded":"2024-03-26T05:24:54.628824+0000","last_fullsized":"2024-03-26T05:24:54.628824+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:22.885144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:22.885144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:44.127918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"443'46","reported_seq":1159,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613969+0000","last_change":"2024-03-26T05:23:57.258003+0000","last_active":"2024-03-26T05:24:54.613969+0000","last_peered":"2024-03-26T05:24:54.613969+0000","last_clean":"2024-03-26T05:24:54.613969+0000","last_became_active":"2024-03-26T05:23:23.923212+0000","last_became_peered":"2024-03-26T05:23:23.923212+0000","last_unstale":"2024-03-26T05:24:54.613969+0000","last_undegraded":"2024-03-26T05:24:54.613969+0000","last_fullsized":"2024-03-26T05:24:54.613969+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'46","last_scrub_stamp":"2024-03-26T05:23:57.257836+0000","last_deep_scrub":"292'34","last_deep_scrub_stamp":"2024-03-26T05:12:23.778630+0000","last_clean_scrub_stamp":"2024-03-26T05:23:57.257836+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:12.134655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1042,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628118+0000","last_change":"2024-03-26T05:23:51.643958+0000","last_active":"2024-03-26T05:24:54.628118+0000","last_peered":"2024-03-26T05:24:54.628118+0000","last_clean":"2024-03-26T05:24:54.628118+0000","last_became_active":"2024-03-26T05:15:32.626265+0000","last_became_peered":"2024-03-26T05:15:32.626265+0000","last_unstale":"2024-03-26T05:24:54.628118+0000","last_undegraded":"2024-03-26T05:24:54.628118+0000","last_fullsized":"2024-03-26T05:24:54.628118+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:51.643877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:23:51.643877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:12.355149+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"542'16","reported_seq":141,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704607+0000","last_change":"2024-03-26T05:24:39.595828+0000","last_active":"2024-03-26T05:24:58.704607+0000","last_peered":"2024-03-26T05:24:58.704607+0000","last_clean":"2024-03-26T05:24:58.704607+0000","last_became_active":"2024-03-26T05:23:26.815826+0000","last_became_peered":"2024-03-26T05:23:26.815826+0000","last_unstale":"2024-03-26T05:24:58.704607+0000","last_undegraded":"2024-03-26T05:24:58.704607+0000","last_fullsized":"2024-03-26T05:24:58.704607+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"542'16","last_scrub_stamp":"2024-03-26T05:24:39.595738+0000","last_deep_scrub":"542'16","last_deep_scrub_stamp":"2024-03-26T05:24:39.595738+0000","last_clean_scrub_stamp":"2024-03-26T05:24:39.595738+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.026484+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.115178+0000","last_change":"2024-03-26T05:24:34.502280+0000","last_active":"2024-03-26T05:24:55.115178+0000","last_peered":"2024-03-26T05:24:55.115178+0000","last_clean":"2024-03-26T05:24:55.115178+0000","last_became_active":"2024-03-26T05:23:26.806027+0000","last_became_peered":"2024-03-26T05:23:26.806027+0000","last_unstale":"2024-03-26T05:24:55.115178+0000","last_undegraded":"2024-03-26T05:24:55.115178+0000","last_fullsized":"2024-03-26T05:24:55.115178+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:34.502151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:13.155026+0000","last_clean_scrub_stamp":"2024-03-26T05:24:34.502151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:37.567669+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"88'128","reported_seq":620,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.115401+0000","last_change":"2024-03-26T05:23:54.384405+0000","last_active":"2024-03-26T05:24:55.115401+0000","last_peered":"2024-03-26T05:24:55.115401+0000","last_clean":"2024-03-26T05:24:55.115401+0000","last_became_active":"2024-03-26T05:23:29.788529+0000","last_became_peered":"2024-03-26T05:23:29.788529+0000","last_unstale":"2024-03-26T05:24:55.115401+0000","last_undegraded":"2024-03-26T05:24:55.115401+0000","last_fullsized":"2024-03-26T05:24:55.115401+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:23:54.384271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:23:54.384271+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:08.342345+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866211+0000","last_change":"2024-03-26T05:23:47.499758+0000","last_active":"2024-03-26T05:24:53.866211+0000","last_peered":"2024-03-26T05:24:53.866211+0000","last_clean":"2024-03-26T05:24:53.866211+0000","last_became_active":"2024-03-26T05:23:23.927317+0000","last_became_peered":"2024-03-26T05:23:23.927317+0000","last_unstale":"2024-03-26T05:24:53.866211+0000","last_undegraded":"2024-03-26T05:24:53.866211+0000","last_fullsized":"2024-03-26T05:24:53.866211+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:47.499614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:23:47.499614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:11.795722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"443'64","reported_seq":1149,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628254+0000","last_change":"2024-03-26T05:24:32.900767+0000","last_active":"2024-03-26T05:24:54.628254+0000","last_peered":"2024-03-26T05:24:54.628254+0000","last_clean":"2024-03-26T05:24:54.628254+0000","last_became_active":"2024-03-26T05:15:32.636466+0000","last_became_peered":"2024-03-26T05:15:32.636466+0000","last_unstale":"2024-03-26T05:24:54.628254+0000","last_undegraded":"2024-03-26T05:24:54.628254+0000","last_fullsized":"2024-03-26T05:24:54.628254+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'64","last_scrub_stamp":"2024-03-26T05:24:32.900670+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:24:32.900670+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:32.919847+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":74,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1034,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628776+0000","last_change":"2024-03-26T05:23:26.606642+0000","last_active":"2024-03-26T05:24:54.628776+0000","last_peered":"2024-03-26T05:24:54.628776+0000","last_clean":"2024-03-26T05:24:54.628776+0000","last_became_active":"2024-03-26T05:15:32.636983+0000","last_became_peered":"2024-03-26T05:15:32.636983+0000","last_unstale":"2024-03-26T05:24:54.628776+0000","last_undegraded":"2024-03-26T05:24:54.628776+0000","last_fullsized":"2024-03-26T05:24:54.628776+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:26.606563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:23:26.606563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866427+0000","last_change":"2024-03-26T05:24:14.649708+0000","last_active":"2024-03-26T05:24:53.866427+0000","last_peered":"2024-03-26T05:24:53.866427+0000","last_clean":"2024-03-26T05:24:53.866427+0000","last_became_active":"2024-03-26T05:23:23.937610+0000","last_became_peered":"2024-03-26T05:23:23.937610+0000","last_unstale":"2024-03-26T05:24:53.866427+0000","last_undegraded":"2024-03-26T05:24:53.866427+0000","last_fullsized":"2024-03-26T05:24:53.866427+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:14.649618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:14.649618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:38.751980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"88'128","reported_seq":709,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.867109+0000","last_change":"2024-03-26T05:24:49.650076+0000","last_active":"2024-03-26T05:24:53.867109+0000","last_peered":"2024-03-26T05:24:53.867109+0000","last_clean":"2024-03-26T05:24:53.867109+0000","last_became_active":"2024-03-26T05:23:26.794528+0000","last_became_peered":"2024-03-26T05:23:26.794528+0000","last_unstale":"2024-03-26T05:24:53.867109+0000","last_undegraded":"2024-03-26T05:24:53.867109+0000","last_fullsized":"2024-03-26T05:24:53.867109+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:49.649969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:49.649969+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:54.547383+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"504'11","reported_seq":1347,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704729+0000","last_change":"2024-03-26T05:24:49.894562+0000","last_active":"2024-03-26T05:24:58.704729+0000","last_peered":"2024-03-26T05:24:58.704729+0000","last_clean":"2024-03-26T05:24:58.704729+0000","last_became_active":"2024-03-26T05:23:23.935379+0000","last_became_peered":"2024-03-26T05:23:23.935379+0000","last_unstale":"2024-03-26T05:24:58.704729+0000","last_undegraded":"2024-03-26T05:24:58.704729+0000","last_fullsized":"2024-03-26T05:24:58.704729+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'11","last_scrub_stamp":"2024-03-26T05:24:49.894423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:49.894423+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:03.719125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"443'48","reported_seq":139,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866510+0000","last_change":"2024-03-26T05:24:18.678081+0000","last_active":"2024-03-26T05:24:53.866510+0000","last_peered":"2024-03-26T05:24:53.866510+0000","last_clean":"2024-03-26T05:24:53.866510+0000","last_became_active":"2024-03-26T05:23:26.821811+0000","last_became_peered":"2024-03-26T05:23:26.821811+0000","last_unstale":"2024-03-26T05:24:53.866510+0000","last_undegraded":"2024-03-26T05:24:53.866510+0000","last_fullsized":"2024-03-26T05:24:53.866510+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'48","last_scrub_stamp":"2024-03-26T05:24:18.677949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:18.677949+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:47.140363+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":62,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"523'2","reported_seq":1077,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863767+0000","last_change":"2024-03-26T05:23:54.815597+0000","last_active":"2024-03-26T05:24:53.863767+0000","last_peered":"2024-03-26T05:24:53.863767+0000","last_clean":"2024-03-26T05:24:53.863767+0000","last_became_active":"2024-03-26T05:23:23.943577+0000","last_became_peered":"2024-03-26T05:23:23.943577+0000","last_unstale":"2024-03-26T05:24:53.863767+0000","last_undegraded":"2024-03-26T05:24:53.863767+0000","last_fullsized":"2024-03-26T05:24:53.863767+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'2","last_scrub_stamp":"2024-03-26T05:23:54.815515+0000","last_deep_scrub":"523'2","last_deep_scrub_stamp":"2024-03-26T05:23:54.815515+0000","last_clean_scrub_stamp":"2024-03-26T05:23:54.815515+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:13.249966+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":958,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627724+0000","last_change":"2024-03-26T05:24:38.724006+0000","last_active":"2024-03-26T05:24:54.627724+0000","last_peered":"2024-03-26T05:24:54.627724+0000","last_clean":"2024-03-26T05:24:54.627724+0000","last_became_active":"2024-03-26T05:15:32.630135+0000","last_became_peered":"2024-03-26T05:15:32.630135+0000","last_unstale":"2024-03-26T05:24:54.627724+0000","last_undegraded":"2024-03-26T05:24:54.627724+0000","last_fullsized":"2024-03-26T05:24:54.627724+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:38.723902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:26.563380+0000","last_clean_scrub_stamp":"2024-03-26T05:24:38.723902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:00.966186+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"396'129","reported_seq":878,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628367+0000","last_change":"2024-03-26T05:23:29.791291+0000","last_active":"2024-03-26T05:24:54.628367+0000","last_peered":"2024-03-26T05:24:54.628367+0000","last_clean":"2024-03-26T05:24:54.628367+0000","last_became_active":"2024-03-26T05:23:29.790874+0000","last_became_peered":"2024-03-26T05:23:29.790874+0000","last_unstale":"2024-03-26T05:24:54.628367+0000","last_undegraded":"2024-03-26T05:24:54.628367+0000","last_fullsized":"2024-03-26T05:24:54.628367+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"396'129","last_scrub_stamp":"2024-03-26T05:23:12.709531+0000","last_deep_scrub":"396'129","last_deep_scrub_stamp":"2024-03-26T05:23:12.709531+0000","last_clean_scrub_stamp":"2024-03-26T05:23:12.709531+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"508'13","reported_seq":1399,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.864011+0000","last_change":"2024-03-26T05:23:53.830513+0000","last_active":"2024-03-26T05:24:53.864011+0000","last_peered":"2024-03-26T05:24:53.864011+0000","last_clean":"2024-03-26T05:24:53.864011+0000","last_became_active":"2024-03-26T05:23:26.811064+0000","last_became_peered":"2024-03-26T05:23:26.811064+0000","last_unstale":"2024-03-26T05:24:53.864011+0000","last_undegraded":"2024-03-26T05:24:53.864011+0000","last_fullsized":"2024-03-26T05:24:53.864011+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'13","last_scrub_stamp":"2024-03-26T05:23:53.830404+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:23:53.830404+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:16.286405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"443'36","reported_seq":297,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613554+0000","last_change":"2024-03-26T05:24:04.154903+0000","last_active":"2024-03-26T05:24:54.613554+0000","last_peered":"2024-03-26T05:24:54.613554+0000","last_clean":"2024-03-26T05:24:54.613554+0000","last_became_active":"2024-03-26T05:23:26.814397+0000","last_became_peered":"2024-03-26T05:23:26.814397+0000","last_unstale":"2024-03-26T05:24:54.613554+0000","last_undegraded":"2024-03-26T05:24:54.613554+0000","last_fullsized":"2024-03-26T05:24:54.613554+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'36","last_scrub_stamp":"2024-03-26T05:24:04.154794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:04.154794+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:32.051611+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114760+0000","last_change":"2024-03-26T05:24:53.460669+0000","last_active":"2024-03-26T05:24:55.114760+0000","last_peered":"2024-03-26T05:24:55.114760+0000","last_clean":"2024-03-26T05:24:55.114760+0000","last_became_active":"2024-03-26T05:23:26.803216+0000","last_became_peered":"2024-03-26T05:23:26.803216+0000","last_unstale":"2024-03-26T05:24:55.114760+0000","last_undegraded":"2024-03-26T05:24:55.114760+0000","last_fullsized":"2024-03-26T05:24:55.114760+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:53.460562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:24:53.460562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:20.820850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1050,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:57.108239+0000","last_change":"2024-03-26T05:24:57.108261+0000","last_active":"2024-03-26T05:24:57.108239+0000","last_peered":"2024-03-26T05:24:57.108239+0000","last_clean":"2024-03-26T05:24:57.108239+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:24:57.108239+0000","last_undegraded":"2024-03-26T05:24:57.108239+0000","last_fullsized":"2024-03-26T05:24:57.108239+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:57.108166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:20:04.436614+0000","last_clean_scrub_stamp":"2024-03-26T05:24:57.108166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:12.832195+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"88'128","reported_seq":810,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.946296+0000","last_change":"2024-03-26T05:24:36.738001+0000","last_active":"2024-03-26T05:24:54.946296+0000","last_peered":"2024-03-26T05:24:54.946296+0000","last_clean":"2024-03-26T05:24:54.946296+0000","last_became_active":"2024-03-26T05:15:33.617538+0000","last_became_peered":"2024-03-26T05:15:33.617538+0000","last_unstale":"2024-03-26T05:24:54.946296+0000","last_undegraded":"2024-03-26T05:24:54.946296+0000","last_fullsized":"2024-03-26T05:24:54.946296+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:36.737913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:36.737913+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:06.450085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1106,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.945052+0000","last_change":"2024-03-26T05:24:38.705636+0000","last_active":"2024-03-26T05:24:54.945052+0000","last_peered":"2024-03-26T05:24:54.945052+0000","last_clean":"2024-03-26T05:24:54.945052+0000","last_became_active":"2024-03-26T05:23:26.815745+0000","last_became_peered":"2024-03-26T05:23:26.815745+0000","last_unstale":"2024-03-26T05:24:54.945052+0000","last_undegraded":"2024-03-26T05:24:54.945052+0000","last_fullsized":"2024-03-26T05:24:54.945052+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:38.705558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:24:38.705558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:50.201177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"443'54","reported_seq":1248,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863703+0000","last_change":"2024-03-26T05:23:50.867092+0000","last_active":"2024-03-26T05:24:53.863703+0000","last_peered":"2024-03-26T05:24:53.863703+0000","last_clean":"2024-03-26T05:24:53.863703+0000","last_became_active":"2024-03-26T05:23:26.802524+0000","last_became_peered":"2024-03-26T05:23:26.802524+0000","last_unstale":"2024-03-26T05:24:53.863703+0000","last_undegraded":"2024-03-26T05:24:53.863703+0000","last_fullsized":"2024-03-26T05:24:53.863703+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'54","last_scrub_stamp":"2024-03-26T05:23:50.866997+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:23:50.866997+0000","objects_scrubbed":9,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:56.597923+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1089,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:59.241821+0000","last_change":"2024-03-26T05:24:59.241863+0000","last_active":"2024-03-26T05:24:59.241821+0000","last_peered":"2024-03-26T05:24:59.241821+0000","last_clean":"2024-03-26T05:24:59.241821+0000","last_became_active":"2024-03-26T05:23:26.814433+0000","last_became_peered":"2024-03-26T05:23:26.814433+0000","last_unstale":"2024-03-26T05:24:59.241821+0000","last_undegraded":"2024-03-26T05:24:59.241821+0000","last_fullsized":"2024-03-26T05:24:59.241821+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:59.241718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:32.076613+0000","last_clean_scrub_stamp":"2024-03-26T05:24:59.241718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:08.528257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1045,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613382+0000","last_change":"2024-03-26T05:24:34.192183+0000","last_active":"2024-03-26T05:24:54.613382+0000","last_peered":"2024-03-26T05:24:54.613382+0000","last_clean":"2024-03-26T05:24:54.613382+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:24:54.613382+0000","last_undegraded":"2024-03-26T05:24:54.613382+0000","last_fullsized":"2024-03-26T05:24:54.613382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:34.192057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:24:34.192057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:43.301832+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"88'128","reported_seq":812,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.946224+0000","last_change":"2024-03-26T05:24:42.704622+0000","last_active":"2024-03-26T05:24:54.946224+0000","last_peered":"2024-03-26T05:24:54.946224+0000","last_clean":"2024-03-26T05:24:54.946224+0000","last_became_active":"2024-03-26T05:15:22.624234+0000","last_became_peered":"2024-03-26T05:15:22.624234+0000","last_unstale":"2024-03-26T05:24:54.946224+0000","last_undegraded":"2024-03-26T05:24:54.946224+0000","last_fullsized":"2024-03-26T05:24:54.946224+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:42.704521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:42.704521+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:12.560237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114666+0000","last_change":"2024-03-26T05:23:40.433667+0000","last_active":"2024-03-26T05:24:55.114666+0000","last_peered":"2024-03-26T05:24:55.114666+0000","last_clean":"2024-03-26T05:24:55.114666+0000","last_became_active":"2024-03-26T05:23:26.803617+0000","last_became_peered":"2024-03-26T05:23:26.803617+0000","last_unstale":"2024-03-26T05:24:55.114666+0000","last_undegraded":"2024-03-26T05:24:55.114666+0000","last_fullsized":"2024-03-26T05:24:55.114666+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:40.433567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:23:40.433567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:05.903491+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"443'42","reported_seq":1156,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863696+0000","last_change":"2024-03-26T05:24:45.067962+0000","last_active":"2024-03-26T05:24:53.863696+0000","last_peered":"2024-03-26T05:24:53.863696+0000","last_clean":"2024-03-26T05:24:53.863696+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:24:53.863696+0000","last_undegraded":"2024-03-26T05:24:53.863696+0000","last_fullsized":"2024-03-26T05:24:53.863696+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'42","last_scrub_stamp":"2024-03-26T05:24:45.067806+0000","last_deep_scrub":"254'30","last_deep_scrub_stamp":"2024-03-26T05:20:05.430706+0000","last_clean_scrub_stamp":"2024-03-26T05:24:45.067806+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:13.288527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":2242,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:04.679404+0000","last_change":"2024-03-26T05:24:57.647759+0000","last_active":"2024-03-26T05:25:04.679404+0000","last_peered":"2024-03-26T05:25:04.679404+0000","last_clean":"2024-03-26T05:25:04.679404+0000","last_became_active":"2024-03-26T05:15:32.633061+0000","last_became_peered":"2024-03-26T05:15:32.633061+0000","last_unstale":"2024-03-26T05:25:04.679404+0000","last_undegraded":"2024-03-26T05:25:04.679404+0000","last_fullsized":"2024-03-26T05:25:04.679404+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:24:57.647662+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:24:57.647662+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:01.073763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":638,"num_read_kb":633,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":783,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:02.684312+0000","last_change":"2024-03-26T05:25:02.684337+0000","last_active":"2024-03-26T05:25:02.684312+0000","last_peered":"2024-03-26T05:25:02.684312+0000","last_clean":"2024-03-26T05:25:02.684312+0000","last_became_active":"2024-03-26T05:23:26.814167+0000","last_became_peered":"2024-03-26T05:23:26.814167+0000","last_unstale":"2024-03-26T05:25:02.684312+0000","last_undegraded":"2024-03-26T05:25:02.684312+0000","last_fullsized":"2024-03-26T05:25:02.684312+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:25:02.684226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:25:02.684226+0000","last_clean_scrub_stamp":"2024-03-26T05:25:02.684226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:19.002393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1069,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613646+0000","last_change":"2024-03-26T05:23:35.035033+0000","last_active":"2024-03-26T05:24:54.613646+0000","last_peered":"2024-03-26T05:24:54.613646+0000","last_clean":"2024-03-26T05:24:54.613646+0000","last_became_active":"2024-03-26T05:23:26.813893+0000","last_became_peered":"2024-03-26T05:23:26.813893+0000","last_unstale":"2024-03-26T05:24:54.613646+0000","last_undegraded":"2024-03-26T05:24:54.613646+0000","last_fullsized":"2024-03-26T05:24:54.613646+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:35.034957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:23:35.034957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:59.347136+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"88'128","reported_seq":877,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613557+0000","last_change":"2024-03-26T05:24:06.151857+0000","last_active":"2024-03-26T05:24:54.613557+0000","last_peered":"2024-03-26T05:24:54.613557+0000","last_clean":"2024-03-26T05:24:54.613557+0000","last_became_active":"2024-03-26T05:23:26.793568+0000","last_became_peered":"2024-03-26T05:23:26.793568+0000","last_unstale":"2024-03-26T05:24:54.613557+0000","last_undegraded":"2024-03-26T05:24:54.613557+0000","last_fullsized":"2024-03-26T05:24:54.613557+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:06.151711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:06.151711+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:06.690101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":991,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627941+0000","last_change":"2024-03-26T05:24:43.640437+0000","last_active":"2024-03-26T05:24:54.627941+0000","last_peered":"2024-03-26T05:24:54.627941+0000","last_clean":"2024-03-26T05:24:54.627941+0000","last_became_active":"2024-03-26T05:23:26.796563+0000","last_became_peered":"2024-03-26T05:23:26.796563+0000","last_unstale":"2024-03-26T05:24:54.627941+0000","last_undegraded":"2024-03-26T05:24:54.627941+0000","last_fullsized":"2024-03-26T05:24:54.627941+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:43.640310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:30.459226+0000","last_clean_scrub_stamp":"2024-03-26T05:24:43.640310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:12.108706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"504'53","reported_seq":1480,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704655+0000","last_change":"2024-03-26T05:24:05.768474+0000","last_active":"2024-03-26T05:24:58.704655+0000","last_peered":"2024-03-26T05:24:58.704655+0000","last_clean":"2024-03-26T05:24:58.704655+0000","last_became_active":"2024-03-26T05:23:23.943584+0000","last_became_peered":"2024-03-26T05:23:23.943584+0000","last_unstale":"2024-03-26T05:24:58.704655+0000","last_undegraded":"2024-03-26T05:24:58.704655+0000","last_fullsized":"2024-03-26T05:24:58.704655+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'53","last_scrub_stamp":"2024-03-26T05:24:05.768347+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:24:05.768347+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:21.004994+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1071,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628122+0000","last_change":"2024-03-26T05:23:31.577895+0000","last_active":"2024-03-26T05:24:54.628122+0000","last_peered":"2024-03-26T05:24:54.628122+0000","last_clean":"2024-03-26T05:24:54.628122+0000","last_became_active":"2024-03-26T05:23:26.803298+0000","last_became_peered":"2024-03-26T05:23:26.803298+0000","last_unstale":"2024-03-26T05:24:54.628122+0000","last_undegraded":"2024-03-26T05:24:54.628122+0000","last_fullsized":"2024-03-26T05:24:54.628122+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:31.577805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:23:31.577805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:58.663662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":1227,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627452+0000","last_change":"2024-03-26T05:24:26.606821+0000","last_active":"2024-03-26T05:24:54.627452+0000","last_peered":"2024-03-26T05:24:54.627452+0000","last_clean":"2024-03-26T05:24:54.627452+0000","last_became_active":"2024-03-26T05:15:32.632929+0000","last_became_peered":"2024-03-26T05:15:32.632929+0000","last_unstale":"2024-03-26T05:24:54.627452+0000","last_undegraded":"2024-03-26T05:24:54.627452+0000","last_fullsized":"2024-03-26T05:24:54.627452+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:26.606708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:26.606708+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:53.269117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114532+0000","last_change":"2024-03-26T05:24:28.556704+0000","last_active":"2024-03-26T05:24:55.114532+0000","last_peered":"2024-03-26T05:24:55.114532+0000","last_clean":"2024-03-26T05:24:55.114532+0000","last_became_active":"2024-03-26T05:23:26.811489+0000","last_became_peered":"2024-03-26T05:23:26.811489+0000","last_unstale":"2024-03-26T05:24:55.114532+0000","last_undegraded":"2024-03-26T05:24:55.114532+0000","last_fullsized":"2024-03-26T05:24:55.114532+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:28.556602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:08.881287+0000","last_clean_scrub_stamp":"2024-03-26T05:24:28.556602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:32.860328+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1091,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306879+0000","last_change":"2024-03-26T05:24:09.569441+0000","last_active":"2024-03-26T05:24:54.306879+0000","last_peered":"2024-03-26T05:24:54.306879+0000","last_clean":"2024-03-26T05:24:54.306879+0000","last_became_active":"2024-03-26T05:23:26.807099+0000","last_became_peered":"2024-03-26T05:23:26.807099+0000","last_unstale":"2024-03-26T05:24:54.306879+0000","last_undegraded":"2024-03-26T05:24:54.306879+0000","last_fullsized":"2024-03-26T05:24:54.306879+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:09.569314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:09.569314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:20.702045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"498'33","reported_seq":926,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306977+0000","last_change":"2024-03-26T05:24:19.574330+0000","last_active":"2024-03-26T05:24:54.306977+0000","last_peered":"2024-03-26T05:24:54.306977+0000","last_clean":"2024-03-26T05:24:54.306977+0000","last_became_active":"2024-03-26T05:23:26.799540+0000","last_became_peered":"2024-03-26T05:23:26.799540+0000","last_unstale":"2024-03-26T05:24:54.306977+0000","last_undegraded":"2024-03-26T05:24:54.306977+0000","last_fullsized":"2024-03-26T05:24:54.306977+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'33","last_scrub_stamp":"2024-03-26T05:24:19.574229+0000","last_deep_scrub":"498'33","last_deep_scrub_stamp":"2024-03-26T05:23:07.541360+0000","last_clean_scrub_stamp":"2024-03-26T05:24:19.574229+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:21.097430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1077,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306274+0000","last_change":"2024-03-26T05:24:31.549896+0000","last_active":"2024-03-26T05:24:54.306274+0000","last_peered":"2024-03-26T05:24:54.306274+0000","last_clean":"2024-03-26T05:24:54.306274+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:24:54.306274+0000","last_undegraded":"2024-03-26T05:24:54.306274+0000","last_fullsized":"2024-03-26T05:24:54.306274+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:31.549805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:31.549805+0000","last_clean_scrub_stamp":"2024-03-26T05:24:31.549805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:42.611302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1058,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613357+0000","last_change":"2024-03-26T05:24:33.081578+0000","last_active":"2024-03-26T05:24:54.613357+0000","last_peered":"2024-03-26T05:24:54.613357+0000","last_clean":"2024-03-26T05:24:54.613357+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:24:54.613357+0000","last_undegraded":"2024-03-26T05:24:54.613357+0000","last_fullsized":"2024-03-26T05:24:54.613357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:33.081489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:33.081489+0000","last_clean_scrub_stamp":"2024-03-26T05:24:33.081489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:51.071265+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":919,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628108+0000","last_change":"2024-03-26T05:24:19.621770+0000","last_active":"2024-03-26T05:24:54.628108+0000","last_peered":"2024-03-26T05:24:54.628108+0000","last_clean":"2024-03-26T05:24:54.628108+0000","last_became_active":"2024-03-26T05:15:32.633633+0000","last_became_peered":"2024-03-26T05:15:32.633633+0000","last_unstale":"2024-03-26T05:24:54.628108+0000","last_undegraded":"2024-03-26T05:24:54.628108+0000","last_fullsized":"2024-03-26T05:24:54.628108+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:19.621638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:07.434449+0000","last_clean_scrub_stamp":"2024-03-26T05:24:19.621638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:28.971628+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"88'128","reported_seq":1230,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627337+0000","last_change":"2024-03-26T05:23:56.568904+0000","last_active":"2024-03-26T05:24:54.627337+0000","last_peered":"2024-03-26T05:24:54.627337+0000","last_clean":"2024-03-26T05:24:54.627337+0000","last_became_active":"2024-03-26T05:23:26.800521+0000","last_became_peered":"2024-03-26T05:23:26.800521+0000","last_unstale":"2024-03-26T05:24:54.627337+0000","last_undegraded":"2024-03-26T05:24:54.627337+0000","last_fullsized":"2024-03-26T05:24:54.627337+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:23:56.568770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:23:56.568770+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:01.457115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1051,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863611+0000","last_change":"2024-03-26T05:23:51.858662+0000","last_active":"2024-03-26T05:24:53.863611+0000","last_peered":"2024-03-26T05:24:53.863611+0000","last_clean":"2024-03-26T05:24:53.863611+0000","last_became_active":"2024-03-26T05:23:23.942931+0000","last_became_peered":"2024-03-26T05:23:23.942931+0000","last_unstale":"2024-03-26T05:24:53.863611+0000","last_undegraded":"2024-03-26T05:24:53.863611+0000","last_fullsized":"2024-03-26T05:24:53.863611+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:51.858583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:23:51.858583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:57.686902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"542'69","reported_seq":1467,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613828+0000","last_change":"2024-03-26T05:24:48.226927+0000","last_active":"2024-03-26T05:24:54.613828+0000","last_peered":"2024-03-26T05:24:54.613828+0000","last_clean":"2024-03-26T05:24:54.613828+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:24:54.613828+0000","last_undegraded":"2024-03-26T05:24:54.613828+0000","last_fullsized":"2024-03-26T05:24:54.613828+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"542'69","last_scrub_stamp":"2024-03-26T05:24:48.226785+0000","last_deep_scrub":"542'69","last_deep_scrub_stamp":"2024-03-26T05:24:48.226785+0000","last_clean_scrub_stamp":"2024-03-26T05:24:48.226785+0000","objects_scrubbed":11,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:51.157070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866570+0000","last_change":"2024-03-26T05:24:00.547165+0000","last_active":"2024-03-26T05:24:53.866570+0000","last_peered":"2024-03-26T05:24:53.866570+0000","last_clean":"2024-03-26T05:24:53.866570+0000","last_became_active":"2024-03-26T05:23:23.934085+0000","last_became_peered":"2024-03-26T05:23:23.934085+0000","last_unstale":"2024-03-26T05:24:53.866570+0000","last_undegraded":"2024-03-26T05:24:53.866570+0000","last_fullsized":"2024-03-26T05:24:53.866570+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:00.547062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:00.547062+0000","last_clean_scrub_stamp":"2024-03-26T05:24:00.547062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:28.867272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"535'3","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114296+0000","last_change":"2024-03-26T05:24:18.432690+0000","last_active":"2024-03-26T05:24:55.114296+0000","last_peered":"2024-03-26T05:24:55.114296+0000","last_clean":"2024-03-26T05:24:55.114296+0000","last_became_active":"2024-03-26T05:23:26.810556+0000","last_became_peered":"2024-03-26T05:23:26.810556+0000","last_unstale":"2024-03-26T05:24:55.114296+0000","last_undegraded":"2024-03-26T05:24:55.114296+0000","last_fullsized":"2024-03-26T05:24:55.114296+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'3","last_scrub_stamp":"2024-03-26T05:24:18.432581+0000","last_deep_scrub":"535'3","last_deep_scrub_stamp":"2024-03-26T05:24:18.432581+0000","last_clean_scrub_stamp":"2024-03-26T05:24:18.432581+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:22.693785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1029,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628902+0000","last_change":"2024-03-26T05:24:38.852984+0000","last_active":"2024-03-26T05:24:54.628902+0000","last_peered":"2024-03-26T05:24:54.628902+0000","last_clean":"2024-03-26T05:24:54.628902+0000","last_became_active":"2024-03-26T05:23:23.935102+0000","last_became_peered":"2024-03-26T05:23:23.935102+0000","last_unstale":"2024-03-26T05:24:54.628902+0000","last_undegraded":"2024-03-26T05:24:54.628902+0000","last_fullsized":"2024-03-26T05:24:54.628902+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:38.852902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:38.852902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:39.795769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"88'128","reported_seq":634,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:02.596706+0000","last_change":"2024-03-26T05:25:02.596736+0000","last_active":"2024-03-26T05:25:02.596706+0000","last_peered":"2024-03-26T05:25:02.596706+0000","last_clean":"2024-03-26T05:25:02.596706+0000","last_became_active":"2024-03-26T05:23:29.788305+0000","last_became_peered":"2024-03-26T05:23:29.788305+0000","last_unstale":"2024-03-26T05:25:02.596706+0000","last_undegraded":"2024-03-26T05:25:02.596706+0000","last_fullsized":"2024-03-26T05:25:02.596706+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:25:02.596620+0000","last_deep_scrub":"88'128","last_deep_scrub_stamp":"2024-03-26T05:25:02.596620+0000","last_clean_scrub_stamp":"2024-03-26T05:25:02.596620+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:16.110610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"443'27","reported_seq":1153,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863984+0000","last_change":"2024-03-26T05:23:59.846869+0000","last_active":"2024-03-26T05:24:53.863984+0000","last_peered":"2024-03-26T05:24:53.863984+0000","last_clean":"2024-03-26T05:24:53.863984+0000","last_became_active":"2024-03-26T05:23:23.943394+0000","last_became_peered":"2024-03-26T05:23:23.943394+0000","last_unstale":"2024-03-26T05:24:53.863984+0000","last_undegraded":"2024-03-26T05:24:53.863984+0000","last_fullsized":"2024-03-26T05:24:53.863984+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'27","last_scrub_stamp":"2024-03-26T05:23:59.846748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:23:59.846748+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:15.920009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114678+0000","last_change":"2024-03-26T05:23:44.371115+0000","last_active":"2024-03-26T05:24:55.114678+0000","last_peered":"2024-03-26T05:24:55.114678+0000","last_clean":"2024-03-26T05:24:55.114678+0000","last_became_active":"2024-03-26T05:23:26.805885+0000","last_became_peered":"2024-03-26T05:23:26.805885+0000","last_unstale":"2024-03-26T05:24:55.114678+0000","last_undegraded":"2024-03-26T05:24:55.114678+0000","last_fullsized":"2024-03-26T05:24:55.114678+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:44.371020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:44.371020+0000","last_clean_scrub_stamp":"2024-03-26T05:23:44.371020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:11.566186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1067,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306528+0000","last_change":"2024-03-26T05:24:43.546754+0000","last_active":"2024-03-26T05:24:54.306528+0000","last_peered":"2024-03-26T05:24:54.306528+0000","last_clean":"2024-03-26T05:24:54.306528+0000","last_became_active":"2024-03-26T05:23:23.947964+0000","last_became_peered":"2024-03-26T05:23:23.947964+0000","last_unstale":"2024-03-26T05:24:54.306528+0000","last_undegraded":"2024-03-26T05:24:54.306528+0000","last_fullsized":"2024-03-26T05:24:54.306528+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:43.546668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:43.546668+0000","last_clean_scrub_stamp":"2024-03-26T05:24:43.546668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:12.718060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1041,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863401+0000","last_change":"2024-03-26T05:24:23.073367+0000","last_active":"2024-03-26T05:24:53.863401+0000","last_peered":"2024-03-26T05:24:53.863401+0000","last_clean":"2024-03-26T05:24:53.863401+0000","last_became_active":"2024-03-26T05:15:32.639943+0000","last_became_peered":"2024-03-26T05:15:32.639943+0000","last_unstale":"2024-03-26T05:24:53.863401+0000","last_undegraded":"2024-03-26T05:24:53.863401+0000","last_fullsized":"2024-03-26T05:24:53.863401+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:23.073227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:23.073227+0000","last_clean_scrub_stamp":"2024-03-26T05:24:23.073227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:46.652413+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"443'67","reported_seq":1172,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.945771+0000","last_change":"2024-03-26T05:24:50.883481+0000","last_active":"2024-03-26T05:24:54.945771+0000","last_peered":"2024-03-26T05:24:54.945771+0000","last_clean":"2024-03-26T05:24:54.945771+0000","last_became_active":"2024-03-26T05:15:32.626640+0000","last_became_peered":"2024-03-26T05:15:32.626640+0000","last_unstale":"2024-03-26T05:24:54.945771+0000","last_undegraded":"2024-03-26T05:24:54.945771+0000","last_fullsized":"2024-03-26T05:24:54.945771+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'67","last_scrub_stamp":"2024-03-26T05:24:50.883350+0000","last_deep_scrub":"443'67","last_deep_scrub_stamp":"2024-03-26T05:24:50.883350+0000","last_clean_scrub_stamp":"2024-03-26T05:24:50.883350+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:56.268769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":952,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628455+0000","last_change":"2024-03-26T05:24:41.666156+0000","last_active":"2024-03-26T05:24:54.628455+0000","last_peered":"2024-03-26T05:24:54.628455+0000","last_clean":"2024-03-26T05:24:54.628455+0000","last_became_active":"2024-03-26T05:23:26.798089+0000","last_became_peered":"2024-03-26T05:23:26.798089+0000","last_unstale":"2024-03-26T05:24:54.628455+0000","last_undegraded":"2024-03-26T05:24:54.628455+0000","last_fullsized":"2024-03-26T05:24:54.628455+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:41.666052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:46.539478+0000","last_clean_scrub_stamp":"2024-03-26T05:24:41.666052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:11.363597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1083,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306654+0000","last_change":"2024-03-26T05:24:42.548874+0000","last_active":"2024-03-26T05:24:54.306654+0000","last_peered":"2024-03-26T05:24:54.306654+0000","last_clean":"2024-03-26T05:24:54.306654+0000","last_became_active":"2024-03-26T05:23:26.799302+0000","last_became_peered":"2024-03-26T05:23:26.799302+0000","last_unstale":"2024-03-26T05:24:54.306654+0000","last_undegraded":"2024-03-26T05:24:54.306654+0000","last_fullsized":"2024-03-26T05:24:54.306654+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:42.548771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:42.548771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:01.372124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":951,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627078+0000","last_change":"2024-03-26T05:24:29.649735+0000","last_active":"2024-03-26T05:24:54.627078+0000","last_peered":"2024-03-26T05:24:54.627078+0000","last_clean":"2024-03-26T05:24:54.627078+0000","last_became_active":"2024-03-26T05:15:32.633481+0000","last_became_peered":"2024-03-26T05:15:32.633481+0000","last_unstale":"2024-03-26T05:24:54.627078+0000","last_undegraded":"2024-03-26T05:24:54.627078+0000","last_fullsized":"2024-03-26T05:24:54.627078+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:29.649637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:29.649637+0000","last_clean_scrub_stamp":"2024-03-26T05:24:29.649637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:30.844423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1060,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306297+0000","last_change":"2024-03-26T05:24:49.385957+0000","last_active":"2024-03-26T05:24:54.306297+0000","last_peered":"2024-03-26T05:24:54.306297+0000","last_clean":"2024-03-26T05:24:54.306297+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:24:54.306297+0000","last_undegraded":"2024-03-26T05:24:54.306297+0000","last_fullsized":"2024-03-26T05:24:54.306297+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:49.385874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:52.271117+0000","last_clean_scrub_stamp":"2024-03-26T05:24:49.385874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:05.350862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"443'43","reported_seq":113,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866147+0000","last_change":"2024-03-26T05:23:56.461342+0000","last_active":"2024-03-26T05:24:53.866147+0000","last_peered":"2024-03-26T05:24:53.866147+0000","last_clean":"2024-03-26T05:24:53.866147+0000","last_became_active":"2024-03-26T05:23:23.921513+0000","last_became_peered":"2024-03-26T05:23:23.921513+0000","last_unstale":"2024-03-26T05:24:53.866147+0000","last_undegraded":"2024-03-26T05:24:53.866147+0000","last_fullsized":"2024-03-26T05:24:53.866147+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'43","last_scrub_stamp":"2024-03-26T05:23:56.461189+0000","last_deep_scrub":"443'43","last_deep_scrub_stamp":"2024-03-26T05:23:56.461189+0000","last_clean_scrub_stamp":"2024-03-26T05:23:56.461189+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:14.572658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":975,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628378+0000","last_change":"2024-03-26T05:24:27.651958+0000","last_active":"2024-03-26T05:24:54.628378+0000","last_peered":"2024-03-26T05:24:54.628378+0000","last_clean":"2024-03-26T05:24:54.628378+0000","last_became_active":"2024-03-26T05:23:23.923551+0000","last_became_peered":"2024-03-26T05:23:23.923551+0000","last_unstale":"2024-03-26T05:24:54.628378+0000","last_undegraded":"2024-03-26T05:24:54.628378+0000","last_fullsized":"2024-03-26T05:24:54.628378+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:27.651844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:24:27.651844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:38.146689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1035,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.072770+0000","last_change":"2024-03-26T05:24:55.072793+0000","last_active":"2024-03-26T05:24:55.072770+0000","last_peered":"2024-03-26T05:24:55.072770+0000","last_clean":"2024-03-26T05:24:55.072770+0000","last_became_active":"2024-03-26T05:20:15.376729+0000","last_became_peered":"2024-03-26T05:20:15.376729+0000","last_unstale":"2024-03-26T05:24:55.072770+0000","last_undegraded":"2024-03-26T05:24:55.072770+0000","last_fullsized":"2024-03-26T05:24:55.072770+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:55.072657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:55.072657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:58.783140+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"504'13","reported_seq":1227,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704603+0000","last_change":"2024-03-26T05:24:14.563653+0000","last_active":"2024-03-26T05:24:58.704603+0000","last_peered":"2024-03-26T05:24:58.704603+0000","last_clean":"2024-03-26T05:24:58.704603+0000","last_became_active":"2024-03-26T05:23:23.927485+0000","last_became_peered":"2024-03-26T05:23:23.927485+0000","last_unstale":"2024-03-26T05:24:58.704603+0000","last_undegraded":"2024-03-26T05:24:58.704603+0000","last_fullsized":"2024-03-26T05:24:58.704603+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'13","last_scrub_stamp":"2024-03-26T05:24:14.563317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:14.563317+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:30.281483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"443'60","reported_seq":120,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114001+0000","last_change":"2024-03-26T05:24:50.472093+0000","last_active":"2024-03-26T05:24:55.114001+0000","last_peered":"2024-03-26T05:24:55.114001+0000","last_clean":"2024-03-26T05:24:55.114001+0000","last_became_active":"2024-03-26T05:23:26.807667+0000","last_became_peered":"2024-03-26T05:23:26.807667+0000","last_unstale":"2024-03-26T05:24:55.114001+0000","last_undegraded":"2024-03-26T05:24:55.114001+0000","last_fullsized":"2024-03-26T05:24:55.114001+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'60","last_scrub_stamp":"2024-03-26T05:24:50.471950+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:24:50.471950+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:19.488139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":974,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628451+0000","last_change":"2024-03-26T05:24:07.574811+0000","last_active":"2024-03-26T05:24:54.628451+0000","last_peered":"2024-03-26T05:24:54.628451+0000","last_clean":"2024-03-26T05:24:54.628451+0000","last_became_active":"2024-03-26T05:23:23.920284+0000","last_became_peered":"2024-03-26T05:23:23.920284+0000","last_unstale":"2024-03-26T05:24:54.628451+0000","last_undegraded":"2024-03-26T05:24:54.628451+0000","last_fullsized":"2024-03-26T05:24:54.628451+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:24:07.574701+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_clean_scrub_stamp":"2024-03-26T05:24:07.574701+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:31.102020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":96,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114802+0000","last_change":"2024-03-26T05:24:06.352202+0000","last_active":"2024-03-26T05:24:55.114802+0000","last_peered":"2024-03-26T05:24:55.114802+0000","last_clean":"2024-03-26T05:24:55.114802+0000","last_became_active":"2024-03-26T05:23:26.809224+0000","last_became_peered":"2024-03-26T05:23:26.809224+0000","last_unstale":"2024-03-26T05:24:55.114802+0000","last_undegraded":"2024-03-26T05:24:55.114802+0000","last_fullsized":"2024-03-26T05:24:55.114802+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:24:06.352098+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:24:06.352098+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:23.245143+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"542'28","reported_seq":1080,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.629130+0000","last_change":"2024-03-26T05:24:36.788404+0000","last_active":"2024-03-26T05:24:54.629130+0000","last_peered":"2024-03-26T05:24:54.629130+0000","last_clean":"2024-03-26T05:24:54.629130+0000","last_became_active":"2024-03-26T05:15:32.640871+0000","last_became_peered":"2024-03-26T05:15:32.640871+0000","last_unstale":"2024-03-26T05:24:54.629130+0000","last_undegraded":"2024-03-26T05:24:54.629130+0000","last_fullsized":"2024-03-26T05:24:54.629130+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"542'28","last_scrub_stamp":"2024-03-26T05:24:36.788275+0000","last_deep_scrub":"542'28","last_deep_scrub_stamp":"2024-03-26T05:24:36.788275+0000","last_clean_scrub_stamp":"2024-03-26T05:24:36.788275+0000","objects_scrubbed":4,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:44.656721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":32,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":1128,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.945050+0000","last_change":"2024-03-26T05:24:51.876501+0000","last_active":"2024-03-26T05:24:54.945050+0000","last_peered":"2024-03-26T05:24:54.945050+0000","last_clean":"2024-03-26T05:24:54.945050+0000","last_became_active":"2024-03-26T05:23:26.814453+0000","last_became_peered":"2024-03-26T05:23:26.814453+0000","last_unstale":"2024-03-26T05:24:54.945050+0000","last_undegraded":"2024-03-26T05:24:54.945050+0000","last_fullsized":"2024-03-26T05:24:54.945050+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:24:51.876276+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:17:50.147911+0000","last_clean_scrub_stamp":"2024-03-26T05:24:51.876276+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:11.333124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114426+0000","last_change":"2024-03-26T05:24:51.503660+0000","last_active":"2024-03-26T05:24:55.114426+0000","last_peered":"2024-03-26T05:24:55.114426+0000","last_clean":"2024-03-26T05:24:55.114426+0000","last_became_active":"2024-03-26T05:23:26.800666+0000","last_became_peered":"2024-03-26T05:23:26.800666+0000","last_unstale":"2024-03-26T05:24:55.114426+0000","last_undegraded":"2024-03-26T05:24:55.114426+0000","last_fullsized":"2024-03-26T05:24:55.114426+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:51.503539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:51.503539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.057042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":956,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628204+0000","last_change":"2024-03-26T05:23:58.627233+0000","last_active":"2024-03-26T05:24:54.628204+0000","last_peered":"2024-03-26T05:24:54.628204+0000","last_clean":"2024-03-26T05:24:54.628204+0000","last_became_active":"2024-03-26T05:23:23.925613+0000","last_became_peered":"2024-03-26T05:23:23.925613+0000","last_unstale":"2024-03-26T05:24:54.628204+0000","last_undegraded":"2024-03-26T05:24:54.628204+0000","last_fullsized":"2024-03-26T05:24:54.628204+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:58.627125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:23:58.627125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:10.873968+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"443'39","reported_seq":1120,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863888+0000","last_change":"2024-03-26T05:24:18.886779+0000","last_active":"2024-03-26T05:24:53.863888+0000","last_peered":"2024-03-26T05:24:53.863888+0000","last_clean":"2024-03-26T05:24:53.863888+0000","last_became_active":"2024-03-26T05:15:32.634434+0000","last_became_peered":"2024-03-26T05:15:32.634434+0000","last_unstale":"2024-03-26T05:24:53.863888+0000","last_undegraded":"2024-03-26T05:24:53.863888+0000","last_fullsized":"2024-03-26T05:24:53.863888+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'39","last_scrub_stamp":"2024-03-26T05:24:18.886672+0000","last_deep_scrub":"443'39","last_deep_scrub_stamp":"2024-03-26T05:24:18.886672+0000","last_clean_scrub_stamp":"2024-03-26T05:24:18.886672+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:41.880187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":925,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627523+0000","last_change":"2024-03-26T05:23:57.572827+0000","last_active":"2024-03-26T05:24:54.627523+0000","last_peered":"2024-03-26T05:24:54.627523+0000","last_clean":"2024-03-26T05:24:54.627523+0000","last_became_active":"2024-03-26T05:23:23.925756+0000","last_became_peered":"2024-03-26T05:23:23.925756+0000","last_unstale":"2024-03-26T05:24:54.627523+0000","last_undegraded":"2024-03-26T05:24:54.627523+0000","last_fullsized":"2024-03-26T05:24:54.627523+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:57.572747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:23:57.572747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:01.874256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1025,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.629034+0000","last_change":"2024-03-26T05:23:18.608175+0000","last_active":"2024-03-26T05:24:54.629034+0000","last_peered":"2024-03-26T05:24:54.629034+0000","last_clean":"2024-03-26T05:24:54.629034+0000","last_became_active":"2024-03-26T05:15:32.641031+0000","last_became_peered":"2024-03-26T05:15:32.641031+0000","last_unstale":"2024-03-26T05:24:54.629034+0000","last_undegraded":"2024-03-26T05:24:54.629034+0000","last_fullsized":"2024-03-26T05:24:54.629034+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:18.608076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:23:18.608076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1063,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306900+0000","last_change":"2024-03-26T05:24:48.425579+0000","last_active":"2024-03-26T05:24:54.306900+0000","last_peered":"2024-03-26T05:24:54.306900+0000","last_clean":"2024-03-26T05:24:54.306900+0000","last_became_active":"2024-03-26T05:23:23.945259+0000","last_became_peered":"2024-03-26T05:23:23.945259+0000","last_unstale":"2024-03-26T05:24:54.306900+0000","last_undegraded":"2024-03-26T05:24:54.306900+0000","last_fullsized":"2024-03-26T05:24:54.306900+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:48.425489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:48.425489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:01.965172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"443'32","reported_seq":112,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:04.536324+0000","last_change":"2024-03-26T05:25:04.536358+0000","last_active":"2024-03-26T05:25:04.536324+0000","last_peered":"2024-03-26T05:25:04.536324+0000","last_clean":"2024-03-26T05:25:04.536324+0000","last_became_active":"2024-03-26T05:23:26.798031+0000","last_became_peered":"2024-03-26T05:23:26.798031+0000","last_unstale":"2024-03-26T05:25:04.536324+0000","last_undegraded":"2024-03-26T05:25:04.536324+0000","last_fullsized":"2024-03-26T05:25:04.536324+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'32","last_scrub_stamp":"2024-03-26T05:25:04.536248+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:25:04.536248+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:22.353963+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1060,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863470+0000","last_change":"2024-03-26T05:24:28.910622+0000","last_active":"2024-03-26T05:24:53.863470+0000","last_peered":"2024-03-26T05:24:53.863470+0000","last_clean":"2024-03-26T05:24:53.863470+0000","last_became_active":"2024-03-26T05:15:32.632387+0000","last_became_peered":"2024-03-26T05:15:32.632387+0000","last_unstale":"2024-03-26T05:24:53.863470+0000","last_undegraded":"2024-03-26T05:24:53.863470+0000","last_fullsized":"2024-03-26T05:24:53.863470+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:28.910532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:45.352678+0000","last_clean_scrub_stamp":"2024-03-26T05:24:28.910532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:57.527714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"336'11","reported_seq":1344,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704446+0000","last_change":"2024-03-26T05:24:23.840064+0000","last_active":"2024-03-26T05:24:58.704446+0000","last_peered":"2024-03-26T05:24:58.704446+0000","last_clean":"2024-03-26T05:24:58.704446+0000","last_became_active":"2024-03-26T05:15:32.636392+0000","last_became_peered":"2024-03-26T05:15:32.636392+0000","last_unstale":"2024-03-26T05:24:58.704446+0000","last_undegraded":"2024-03-26T05:24:58.704446+0000","last_fullsized":"2024-03-26T05:24:58.704446+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-03-26T05:24:23.839981+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:24:23.839981+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:43.739324+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":111,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:59.794453+0000","last_change":"2024-03-26T05:24:59.794477+0000","last_active":"2024-03-26T05:24:59.794453+0000","last_peered":"2024-03-26T05:24:59.794453+0000","last_clean":"2024-03-26T05:24:59.794453+0000","last_became_active":"2024-03-26T05:23:26.814104+0000","last_became_peered":"2024-03-26T05:23:26.814104+0000","last_unstale":"2024-03-26T05:24:59.794453+0000","last_undegraded":"2024-03-26T05:24:59.794453+0000","last_fullsized":"2024-03-26T05:24:59.794453+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:59.794371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:59.794371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:25.337751+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"443'49","reported_seq":1121,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628941+0000","last_change":"2024-03-26T05:24:00.812655+0000","last_active":"2024-03-26T05:24:54.628941+0000","last_peered":"2024-03-26T05:24:54.628941+0000","last_clean":"2024-03-26T05:24:54.628941+0000","last_became_active":"2024-03-26T05:15:32.626448+0000","last_became_peered":"2024-03-26T05:15:32.626448+0000","last_unstale":"2024-03-26T05:24:54.628941+0000","last_undegraded":"2024-03-26T05:24:54.628941+0000","last_fullsized":"2024-03-26T05:24:54.628941+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'49","last_scrub_stamp":"2024-03-26T05:24:00.812557+0000","last_deep_scrub":"345'39","last_deep_scrub_stamp":"2024-03-26T05:15:56.674423+0000","last_clean_scrub_stamp":"2024-03-26T05:24:00.812557+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:06.645746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":959,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627933+0000","last_change":"2024-03-26T05:24:00.584116+0000","last_active":"2024-03-26T05:24:54.627933+0000","last_peered":"2024-03-26T05:24:54.627933+0000","last_clean":"2024-03-26T05:24:54.627933+0000","last_became_active":"2024-03-26T05:23:23.924133+0000","last_became_peered":"2024-03-26T05:23:23.924133+0000","last_unstale":"2024-03-26T05:24:54.627933+0000","last_undegraded":"2024-03-26T05:24:54.627933+0000","last_fullsized":"2024-03-26T05:24:54.627933+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:00.584025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:24:00.584025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:28.716607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"508'12","reported_seq":116,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704700+0000","last_change":"2024-03-26T05:24:17.405498+0000","last_active":"2024-03-26T05:24:58.704700+0000","last_peered":"2024-03-26T05:24:58.704700+0000","last_clean":"2024-03-26T05:24:58.704700+0000","last_became_active":"2024-03-26T05:23:26.806234+0000","last_became_peered":"2024-03-26T05:23:26.806234+0000","last_unstale":"2024-03-26T05:24:58.704700+0000","last_undegraded":"2024-03-26T05:24:58.704700+0000","last_fullsized":"2024-03-26T05:24:58.704700+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'12","last_scrub_stamp":"2024-03-26T05:24:17.405360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:17.405360+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:29.961077+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1048,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863661+0000","last_change":"2024-03-26T05:24:46.144452+0000","last_active":"2024-03-26T05:24:53.863661+0000","last_peered":"2024-03-26T05:24:53.863661+0000","last_clean":"2024-03-26T05:24:53.863661+0000","last_became_active":"2024-03-26T05:15:32.631444+0000","last_became_peered":"2024-03-26T05:15:32.631444+0000","last_unstale":"2024-03-26T05:24:53.863661+0000","last_undegraded":"2024-03-26T05:24:53.863661+0000","last_fullsized":"2024-03-26T05:24:53.863661+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:46.144355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:59.508745+0000","last_clean_scrub_stamp":"2024-03-26T05:24:46.144355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:46.492028+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"443'32","reported_seq":109,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866578+0000","last_change":"2024-03-26T05:24:08.668366+0000","last_active":"2024-03-26T05:24:53.866578+0000","last_peered":"2024-03-26T05:24:53.866578+0000","last_clean":"2024-03-26T05:24:53.866578+0000","last_became_active":"2024-03-26T05:23:23.931656+0000","last_became_peered":"2024-03-26T05:23:23.931656+0000","last_unstale":"2024-03-26T05:24:53.866578+0000","last_undegraded":"2024-03-26T05:24:53.866578+0000","last_fullsized":"2024-03-26T05:24:53.866578+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'32","last_scrub_stamp":"2024-03-26T05:24:08.668274+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:08.668274+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:22.481085+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":105,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:01.681192+0000","last_change":"2024-03-26T05:25:01.681239+0000","last_active":"2024-03-26T05:25:01.681192+0000","last_peered":"2024-03-26T05:25:01.681192+0000","last_clean":"2024-03-26T05:25:01.681192+0000","last_became_active":"2024-03-26T05:23:23.931645+0000","last_became_peered":"2024-03-26T05:23:23.931645+0000","last_unstale":"2024-03-26T05:25:01.681192+0000","last_undegraded":"2024-03-26T05:25:01.681192+0000","last_fullsized":"2024-03-26T05:25:01.681192+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:25:01.681095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:25:01.681095+0000","last_clean_scrub_stamp":"2024-03-26T05:25:01.681095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:23.546578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1080,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613567+0000","last_change":"2024-03-26T05:23:39.196051+0000","last_active":"2024-03-26T05:24:54.613567+0000","last_peered":"2024-03-26T05:24:54.613567+0000","last_clean":"2024-03-26T05:24:54.613567+0000","last_became_active":"2024-03-26T05:23:26.813536+0000","last_became_peered":"2024-03-26T05:23:26.813536+0000","last_unstale":"2024-03-26T05:24:54.613567+0000","last_undegraded":"2024-03-26T05:24:54.613567+0000","last_fullsized":"2024-03-26T05:24:54.613567+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:39.195947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:55.496800+0000","last_clean_scrub_stamp":"2024-03-26T05:23:39.195947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:02.584927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1044,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628103+0000","last_change":"2024-03-26T05:23:37.641257+0000","last_active":"2024-03-26T05:24:54.628103+0000","last_peered":"2024-03-26T05:24:54.628103+0000","last_clean":"2024-03-26T05:24:54.628103+0000","last_became_active":"2024-03-26T05:23:26.802281+0000","last_became_peered":"2024-03-26T05:23:26.802281+0000","last_unstale":"2024-03-26T05:24:54.628103+0000","last_undegraded":"2024-03-26T05:24:54.628103+0000","last_fullsized":"2024-03-26T05:24:54.628103+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:37.641159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:23:37.641159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"443'21","reported_seq":1086,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613655+0000","last_change":"2024-03-26T05:23:46.347900+0000","last_active":"2024-03-26T05:24:54.613655+0000","last_peered":"2024-03-26T05:24:54.613655+0000","last_clean":"2024-03-26T05:24:54.613655+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:24:54.613655+0000","last_undegraded":"2024-03-26T05:24:54.613655+0000","last_fullsized":"2024-03-26T05:24:54.613655+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'21","last_scrub_stamp":"2024-03-26T05:23:46.347776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:23:46.347776+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:03.079113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":930,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628241+0000","last_change":"2024-03-26T05:24:13.545357+0000","last_active":"2024-03-26T05:24:54.628241+0000","last_peered":"2024-03-26T05:24:54.628241+0000","last_clean":"2024-03-26T05:24:54.628241+0000","last_became_active":"2024-03-26T05:23:23.930076+0000","last_became_peered":"2024-03-26T05:23:23.930076+0000","last_unstale":"2024-03-26T05:24:54.628241+0000","last_undegraded":"2024-03-26T05:24:54.628241+0000","last_fullsized":"2024-03-26T05:24:54.628241+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:13.545257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:24:13.545257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:42.876092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1032,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628355+0000","last_change":"2024-03-26T05:24:16.861274+0000","last_active":"2024-03-26T05:24:54.628355+0000","last_peered":"2024-03-26T05:24:54.628355+0000","last_clean":"2024-03-26T05:24:54.628355+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:24:54.628355+0000","last_undegraded":"2024-03-26T05:24:54.628355+0000","last_fullsized":"2024-03-26T05:24:54.628355+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:16.861192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:24:16.861192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:35.838026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1071,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863121+0000","last_change":"2024-03-26T05:23:46.864136+0000","last_active":"2024-03-26T05:24:53.863121+0000","last_peered":"2024-03-26T05:24:53.863121+0000","last_clean":"2024-03-26T05:24:53.863121+0000","last_became_active":"2024-03-26T05:23:26.809570+0000","last_became_peered":"2024-03-26T05:23:26.809570+0000","last_unstale":"2024-03-26T05:24:53.863121+0000","last_undegraded":"2024-03-26T05:24:53.863121+0000","last_fullsized":"2024-03-26T05:24:53.863121+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:46.864038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:23:46.864038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:01.818874+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"443'33","reported_seq":1092,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628368+0000","last_change":"2024-03-26T05:24:20.888480+0000","last_active":"2024-03-26T05:24:54.628368+0000","last_peered":"2024-03-26T05:24:54.628368+0000","last_clean":"2024-03-26T05:24:54.628368+0000","last_became_active":"2024-03-26T05:15:32.636359+0000","last_became_peered":"2024-03-26T05:15:32.636359+0000","last_unstale":"2024-03-26T05:24:54.628368+0000","last_undegraded":"2024-03-26T05:24:54.628368+0000","last_fullsized":"2024-03-26T05:24:54.628368+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'33","last_scrub_stamp":"2024-03-26T05:24:20.888373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:20.888373+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:48.499006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":100,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:05.509193+0000","last_change":"2024-03-26T05:25:05.509240+0000","last_active":"2024-03-26T05:25:05.509193+0000","last_peered":"2024-03-26T05:25:05.509193+0000","last_clean":"2024-03-26T05:25:05.509193+0000","last_became_active":"2024-03-26T05:23:26.801339+0000","last_became_peered":"2024-03-26T05:23:26.801339+0000","last_unstale":"2024-03-26T05:25:05.509193+0000","last_undegraded":"2024-03-26T05:25:05.509193+0000","last_fullsized":"2024-03-26T05:25:05.509193+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:25:05.509098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:50.406745+0000","last_clean_scrub_stamp":"2024-03-26T05:25:05.509098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:33.584124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":968,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627411+0000","last_change":"2024-03-26T05:24:12.573345+0000","last_active":"2024-03-26T05:24:54.627411+0000","last_peered":"2024-03-26T05:24:54.627411+0000","last_clean":"2024-03-26T05:24:54.627411+0000","last_became_active":"2024-03-26T05:23:23.923115+0000","last_became_peered":"2024-03-26T05:23:23.923115+0000","last_unstale":"2024-03-26T05:24:54.627411+0000","last_undegraded":"2024-03-26T05:24:54.627411+0000","last_fullsized":"2024-03-26T05:24:54.627411+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:12.573245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:24:12.573245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:19.220471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1036,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628606+0000","last_change":"2024-03-26T05:24:37.960766+0000","last_active":"2024-03-26T05:24:54.628606+0000","last_peered":"2024-03-26T05:24:54.628606+0000","last_clean":"2024-03-26T05:24:54.628606+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:24:54.628606+0000","last_undegraded":"2024-03-26T05:24:54.628606+0000","last_fullsized":"2024-03-26T05:24:54.628606+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:37.960659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:53.017168+0000","last_clean_scrub_stamp":"2024-03-26T05:24:37.960659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:02.519656+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"443'14","reported_seq":1056,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628081+0000","last_change":"2024-03-26T05:23:45.659583+0000","last_active":"2024-03-26T05:24:54.628081+0000","last_peered":"2024-03-26T05:24:54.628081+0000","last_clean":"2024-03-26T05:24:54.628081+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:24:54.628081+0000","last_undegraded":"2024-03-26T05:24:54.628081+0000","last_fullsized":"2024-03-26T05:24:54.628081+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'14","last_scrub_stamp":"2024-03-26T05:23:45.659465+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:23:45.659465+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:58.734800+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1083,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863851+0000","last_change":"2024-03-26T05:23:57.771237+0000","last_active":"2024-03-26T05:24:53.863851+0000","last_peered":"2024-03-26T05:24:53.863851+0000","last_clean":"2024-03-26T05:24:53.863851+0000","last_became_active":"2024-03-26T05:23:26.809979+0000","last_became_peered":"2024-03-26T05:23:26.809979+0000","last_unstale":"2024-03-26T05:24:53.863851+0000","last_undegraded":"2024-03-26T05:24:53.863851+0000","last_fullsized":"2024-03-26T05:24:53.863851+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:57.771156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:11.070341+0000","last_clean_scrub_stamp":"2024-03-26T05:23:57.771156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:04.197280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1057,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.945906+0000","last_change":"2024-03-26T05:23:59.761514+0000","last_active":"2024-03-26T05:24:54.945906+0000","last_peered":"2024-03-26T05:24:54.945906+0000","last_clean":"2024-03-26T05:24:54.945906+0000","last_became_active":"2024-03-26T05:23:23.936128+0000","last_became_peered":"2024-03-26T05:23:23.936128+0000","last_unstale":"2024-03-26T05:24:54.945906+0000","last_undegraded":"2024-03-26T05:24:54.945906+0000","last_fullsized":"2024-03-26T05:24:54.945906+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:59.761427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:23:59.761427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:10.149772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":955,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627283+0000","last_change":"2024-03-26T05:24:15.611940+0000","last_active":"2024-03-26T05:24:54.627283+0000","last_peered":"2024-03-26T05:24:54.627283+0000","last_clean":"2024-03-26T05:24:54.627283+0000","last_became_active":"2024-03-26T05:23:23.923501+0000","last_became_peered":"2024-03-26T05:23:23.923501+0000","last_unstale":"2024-03-26T05:24:54.627283+0000","last_undegraded":"2024-03-26T05:24:54.627283+0000","last_fullsized":"2024-03-26T05:24:54.627283+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:15.611825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:15.611825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:42.657185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"443'39","reported_seq":1239,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.945493+0000","last_change":"2024-03-26T05:23:49.645029+0000","last_active":"2024-03-26T05:24:54.945493+0000","last_peered":"2024-03-26T05:24:54.945493+0000","last_clean":"2024-03-26T05:24:54.945493+0000","last_became_active":"2024-03-26T05:23:26.814859+0000","last_became_peered":"2024-03-26T05:23:26.814859+0000","last_unstale":"2024-03-26T05:24:54.945493+0000","last_undegraded":"2024-03-26T05:24:54.945493+0000","last_fullsized":"2024-03-26T05:24:54.945493+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'39","last_scrub_stamp":"2024-03-26T05:23:49.644913+0000","last_deep_scrub":"413'31","last_deep_scrub_stamp":"2024-03-26T05:18:46.881054+0000","last_clean_scrub_stamp":"2024-03-26T05:23:49.644913+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:09.049245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1040,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628012+0000","last_change":"2024-03-26T05:23:40.634385+0000","last_active":"2024-03-26T05:24:54.628012+0000","last_peered":"2024-03-26T05:24:54.628012+0000","last_clean":"2024-03-26T05:24:54.628012+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:24:54.628012+0000","last_undegraded":"2024-03-26T05:24:54.628012+0000","last_fullsized":"2024-03-26T05:24:54.628012+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:40.634265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:47.776444+0000","last_clean_scrub_stamp":"2024-03-26T05:23:40.634265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1046,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863953+0000","last_change":"2024-03-26T05:24:48.133964+0000","last_active":"2024-03-26T05:24:53.863953+0000","last_peered":"2024-03-26T05:24:53.863953+0000","last_clean":"2024-03-26T05:24:53.863953+0000","last_became_active":"2024-03-26T05:15:32.634716+0000","last_became_peered":"2024-03-26T05:15:32.634716+0000","last_unstale":"2024-03-26T05:24:53.863953+0000","last_undegraded":"2024-03-26T05:24:53.863953+0000","last_fullsized":"2024-03-26T05:24:53.863953+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:48.133880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:18.758720+0000","last_clean_scrub_stamp":"2024-03-26T05:24:48.133880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.554815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1066,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.614016+0000","last_change":"2024-03-26T05:23:49.553078+0000","last_active":"2024-03-26T05:24:54.614016+0000","last_peered":"2024-03-26T05:24:54.614016+0000","last_clean":"2024-03-26T05:24:54.614016+0000","last_became_active":"2024-03-26T05:23:26.814722+0000","last_became_peered":"2024-03-26T05:23:26.814722+0000","last_unstale":"2024-03-26T05:24:54.614016+0000","last_undegraded":"2024-03-26T05:24:54.614016+0000","last_fullsized":"2024-03-26T05:24:54.614016+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:49.552975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:23:49.552975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"443'42","reported_seq":1121,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863816+0000","last_change":"2024-03-26T05:24:31.996949+0000","last_active":"2024-03-26T05:24:53.863816+0000","last_peered":"2024-03-26T05:24:53.863816+0000","last_clean":"2024-03-26T05:24:53.863816+0000","last_became_active":"2024-03-26T05:15:32.634216+0000","last_became_peered":"2024-03-26T05:15:32.634216+0000","last_unstale":"2024-03-26T05:24:53.863816+0000","last_undegraded":"2024-03-26T05:24:53.863816+0000","last_fullsized":"2024-03-26T05:24:53.863816+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'42","last_scrub_stamp":"2024-03-26T05:24:31.996840+0000","last_deep_scrub":"290'32","last_deep_scrub_stamp":"2024-03-26T05:12:15.970313+0000","last_clean_scrub_stamp":"2024-03-26T05:24:31.996840+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:44.828643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":1064,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613257+0000","last_change":"2024-03-26T05:23:55.228827+0000","last_active":"2024-03-26T05:24:54.613257+0000","last_peered":"2024-03-26T05:24:54.613257+0000","last_clean":"2024-03-26T05:24:54.613257+0000","last_became_active":"2024-03-26T05:23:23.930527+0000","last_became_peered":"2024-03-26T05:23:23.930527+0000","last_unstale":"2024-03-26T05:24:54.613257+0000","last_undegraded":"2024-03-26T05:24:54.613257+0000","last_fullsized":"2024-03-26T05:24:54.613257+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:23:55.228704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:23:55.228704+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:24.587746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"504'13","reported_seq":1401,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.929348+0000","last_change":"2024-03-26T05:24:58.929348+0000","last_active":"2024-03-26T05:24:58.929348+0000","last_peered":"2024-03-26T05:24:58.929348+0000","last_clean":"2024-03-26T05:24:58.929348+0000","last_became_active":"2024-03-26T05:23:23.935698+0000","last_became_peered":"2024-03-26T05:23:23.935698+0000","last_unstale":"2024-03-26T05:24:58.929348+0000","last_undegraded":"2024-03-26T05:24:58.929348+0000","last_fullsized":"2024-03-26T05:24:58.929348+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'13","last_scrub_stamp":"2024-03-26T05:24:58.929226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:58.929226+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.689329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"429'2","reported_seq":987,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627530+0000","last_change":"2024-03-26T05:23:31.438151+0000","last_active":"2024-03-26T05:24:54.627530+0000","last_peered":"2024-03-26T05:24:54.627530+0000","last_clean":"2024-03-26T05:24:54.627530+0000","last_became_active":"2024-03-26T05:23:26.797077+0000","last_became_peered":"2024-03-26T05:23:26.797077+0000","last_unstale":"2024-03-26T05:24:54.627530+0000","last_undegraded":"2024-03-26T05:24:54.627530+0000","last_fullsized":"2024-03-26T05:24:54.627530+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'2","last_scrub_stamp":"2024-03-26T05:23:31.438043+0000","last_deep_scrub":"429'2","last_deep_scrub_stamp":"2024-03-26T05:19:54.275264+0000","last_clean_scrub_stamp":"2024-03-26T05:23:31.438043+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:56.650917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"443'38","reported_seq":1131,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:00.935067+0000","last_change":"2024-03-26T05:25:00.935129+0000","last_active":"2024-03-26T05:25:00.935067+0000","last_peered":"2024-03-26T05:25:00.935067+0000","last_clean":"2024-03-26T05:25:00.935067+0000","last_became_active":"2024-03-26T05:23:23.935314+0000","last_became_peered":"2024-03-26T05:23:23.935314+0000","last_unstale":"2024-03-26T05:25:00.935067+0000","last_undegraded":"2024-03-26T05:25:00.935067+0000","last_fullsized":"2024-03-26T05:25:00.935067+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'38","last_scrub_stamp":"2024-03-26T05:25:00.934972+0000","last_deep_scrub":"322'32","last_deep_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_clean_scrub_stamp":"2024-03-26T05:25:00.934972+0000","objects_scrubbed":6,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:13.778155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866780+0000","last_change":"2024-03-26T05:23:53.488591+0000","last_active":"2024-03-26T05:24:53.866780+0000","last_peered":"2024-03-26T05:24:53.866780+0000","last_clean":"2024-03-26T05:24:53.866780+0000","last_became_active":"2024-03-26T05:23:23.940669+0000","last_became_peered":"2024-03-26T05:23:23.940669+0000","last_unstale":"2024-03-26T05:24:53.866780+0000","last_undegraded":"2024-03-26T05:24:53.866780+0000","last_fullsized":"2024-03-26T05:24:53.866780+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:53.488486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:23:53.488486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:04.275443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.867090+0000","last_change":"2024-03-26T05:24:15.607395+0000","last_active":"2024-03-26T05:24:53.867090+0000","last_peered":"2024-03-26T05:24:53.867090+0000","last_clean":"2024-03-26T05:24:53.867090+0000","last_became_active":"2024-03-26T05:23:23.937195+0000","last_became_peered":"2024-03-26T05:23:23.937195+0000","last_unstale":"2024-03-26T05:24:53.867090+0000","last_undegraded":"2024-03-26T05:24:53.867090+0000","last_fullsized":"2024-03-26T05:24:53.867090+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:15.607260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:15.607260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:19.431547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1062,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.629087+0000","last_change":"2024-03-26T05:24:35.078487+0000","last_active":"2024-03-26T05:24:54.629087+0000","last_peered":"2024-03-26T05:24:54.629087+0000","last_clean":"2024-03-26T05:24:54.629087+0000","last_became_active":"2024-03-26T05:23:26.803457+0000","last_became_peered":"2024-03-26T05:23:26.803457+0000","last_unstale":"2024-03-26T05:24:54.629087+0000","last_undegraded":"2024-03-26T05:24:54.629087+0000","last_fullsized":"2024-03-26T05:24:54.629087+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:35.078380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:20:01.576615+0000","last_clean_scrub_stamp":"2024-03-26T05:24:35.078380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:40.685466+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"443'39","reported_seq":1123,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306655+0000","last_change":"2024-03-26T05:24:32.574978+0000","last_active":"2024-03-26T05:24:54.306655+0000","last_peered":"2024-03-26T05:24:54.306655+0000","last_clean":"2024-03-26T05:24:54.306655+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:24:54.306655+0000","last_undegraded":"2024-03-26T05:24:54.306655+0000","last_fullsized":"2024-03-26T05:24:54.306655+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'39","last_scrub_stamp":"2024-03-26T05:24:32.574853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:32.574853+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:54.392821+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1058,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863831+0000","last_change":"2024-03-26T05:23:41.727083+0000","last_active":"2024-03-26T05:24:53.863831+0000","last_peered":"2024-03-26T05:24:53.863831+0000","last_clean":"2024-03-26T05:24:53.863831+0000","last_became_active":"2024-03-26T05:23:23.929621+0000","last_became_peered":"2024-03-26T05:23:23.929621+0000","last_unstale":"2024-03-26T05:24:53.863831+0000","last_undegraded":"2024-03-26T05:24:53.863831+0000","last_fullsized":"2024-03-26T05:24:53.863831+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:41.727001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:23:41.727001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:07.634291+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1052,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613902+0000","last_change":"2024-03-26T05:24:02.227643+0000","last_active":"2024-03-26T05:24:54.613902+0000","last_peered":"2024-03-26T05:24:54.613902+0000","last_clean":"2024-03-26T05:24:54.613902+0000","last_became_active":"2024-03-26T05:23:23.930052+0000","last_became_peered":"2024-03-26T05:23:23.930052+0000","last_unstale":"2024-03-26T05:24:54.613902+0000","last_undegraded":"2024-03-26T05:24:54.613902+0000","last_fullsized":"2024-03-26T05:24:54.613902+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:02.227560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:02.227560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:23.706995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":103,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.867096+0000","last_change":"2024-03-26T05:24:19.669790+0000","last_active":"2024-03-26T05:24:53.867096+0000","last_peered":"2024-03-26T05:24:53.867096+0000","last_clean":"2024-03-26T05:24:53.867096+0000","last_became_active":"2024-03-26T05:23:26.816310+0000","last_became_peered":"2024-03-26T05:23:26.816310+0000","last_unstale":"2024-03-26T05:24:53.867096+0000","last_undegraded":"2024-03-26T05:24:53.867096+0000","last_fullsized":"2024-03-26T05:24:53.867096+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:19.669658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:24:19.669658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:36.243542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"443'18","reported_seq":105,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866419+0000","last_change":"2024-03-26T05:24:25.668061+0000","last_active":"2024-03-26T05:24:53.866419+0000","last_peered":"2024-03-26T05:24:53.866419+0000","last_clean":"2024-03-26T05:24:53.866419+0000","last_became_active":"2024-03-26T05:23:23.927909+0000","last_became_peered":"2024-03-26T05:23:23.927909+0000","last_unstale":"2024-03-26T05:24:53.866419+0000","last_undegraded":"2024-03-26T05:24:53.866419+0000","last_fullsized":"2024-03-26T05:24:53.866419+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-03-26T05:24:25.667897+0000","last_deep_scrub":"321'16","last_deep_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_clean_scrub_stamp":"2024-03-26T05:24:25.667897+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:26.347631+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114170+0000","last_change":"2024-03-26T05:24:31.419665+0000","last_active":"2024-03-26T05:24:55.114170+0000","last_peered":"2024-03-26T05:24:55.114170+0000","last_clean":"2024-03-26T05:24:55.114170+0000","last_became_active":"2024-03-26T05:23:26.811878+0000","last_became_peered":"2024-03-26T05:23:26.811878+0000","last_unstale":"2024-03-26T05:24:55.114170+0000","last_undegraded":"2024-03-26T05:24:55.114170+0000","last_fullsized":"2024-03-26T05:24:55.114170+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:31.419548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:00.635308+0000","last_clean_scrub_stamp":"2024-03-26T05:24:31.419548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:36.294443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":966,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627532+0000","last_change":"2024-03-26T05:24:16.565833+0000","last_active":"2024-03-26T05:24:54.627532+0000","last_peered":"2024-03-26T05:24:54.627532+0000","last_clean":"2024-03-26T05:24:54.627532+0000","last_became_active":"2024-03-26T05:23:23.924963+0000","last_became_peered":"2024-03-26T05:23:23.924963+0000","last_unstale":"2024-03-26T05:24:54.627532+0000","last_undegraded":"2024-03-26T05:24:54.627532+0000","last_fullsized":"2024-03-26T05:24:54.627532+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:16.565704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_clean_scrub_stamp":"2024-03-26T05:24:16.565704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:17.798547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":100,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:57.570572+0000","last_change":"2024-03-26T05:24:57.570614+0000","last_active":"2024-03-26T05:24:57.570572+0000","last_peered":"2024-03-26T05:24:57.570572+0000","last_clean":"2024-03-26T05:24:57.570572+0000","last_became_active":"2024-03-26T05:23:26.802657+0000","last_became_peered":"2024-03-26T05:23:26.802657+0000","last_unstale":"2024-03-26T05:24:57.570572+0000","last_undegraded":"2024-03-26T05:24:57.570572+0000","last_fullsized":"2024-03-26T05:24:57.570572+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:57.570470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:57.570470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:18.424129+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"443'33","reported_seq":111,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866361+0000","last_change":"2024-03-26T05:24:12.574819+0000","last_active":"2024-03-26T05:24:53.866361+0000","last_peered":"2024-03-26T05:24:53.866361+0000","last_clean":"2024-03-26T05:24:53.866361+0000","last_became_active":"2024-03-26T05:23:23.930895+0000","last_became_peered":"2024-03-26T05:23:23.930895+0000","last_unstale":"2024-03-26T05:24:53.866361+0000","last_undegraded":"2024-03-26T05:24:53.866361+0000","last_fullsized":"2024-03-26T05:24:53.866361+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'33","last_scrub_stamp":"2024-03-26T05:24:12.574705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:12.574705+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:28.337465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.115042+0000","last_change":"2024-03-26T05:24:08.338763+0000","last_active":"2024-03-26T05:24:55.115042+0000","last_peered":"2024-03-26T05:24:55.115042+0000","last_clean":"2024-03-26T05:24:55.115042+0000","last_became_active":"2024-03-26T05:23:26.811430+0000","last_became_peered":"2024-03-26T05:23:26.811430+0000","last_unstale":"2024-03-26T05:24:55.115042+0000","last_undegraded":"2024-03-26T05:24:55.115042+0000","last_fullsized":"2024-03-26T05:24:55.115042+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:08.338680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:24:08.338680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:31.121401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114102+0000","last_change":"2024-03-26T05:24:42.348161+0000","last_active":"2024-03-26T05:24:55.114102+0000","last_peered":"2024-03-26T05:24:55.114102+0000","last_clean":"2024-03-26T05:24:55.114102+0000","last_became_active":"2024-03-26T05:23:26.793735+0000","last_became_peered":"2024-03-26T05:23:26.793735+0000","last_unstale":"2024-03-26T05:24:55.114102+0000","last_undegraded":"2024-03-26T05:24:55.114102+0000","last_fullsized":"2024-03-26T05:24:55.114102+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:42.348053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:42.348053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:57.786404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1048,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863362+0000","last_change":"2024-03-26T05:24:35.038720+0000","last_active":"2024-03-26T05:24:53.863362+0000","last_peered":"2024-03-26T05:24:53.863362+0000","last_clean":"2024-03-26T05:24:53.863362+0000","last_became_active":"2024-03-26T05:15:32.640052+0000","last_became_peered":"2024-03-26T05:15:32.640052+0000","last_unstale":"2024-03-26T05:24:53.863362+0000","last_undegraded":"2024-03-26T05:24:53.863362+0000","last_fullsized":"2024-03-26T05:24:53.863362+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:35.038610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:35.038610+0000","last_clean_scrub_stamp":"2024-03-26T05:24:35.038610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:45.104003+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"520'74","reported_seq":1156,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628355+0000","last_change":"2024-03-26T05:23:49.635195+0000","last_active":"2024-03-26T05:24:54.628355+0000","last_peered":"2024-03-26T05:24:54.628355+0000","last_clean":"2024-03-26T05:24:54.628355+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:24:54.628355+0000","last_undegraded":"2024-03-26T05:24:54.628355+0000","last_fullsized":"2024-03-26T05:24:54.628355+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'74","last_scrub_stamp":"2024-03-26T05:23:49.635086+0000","last_deep_scrub":"520'74","last_deep_scrub_stamp":"2024-03-26T05:23:49.635086+0000","last_clean_scrub_stamp":"2024-03-26T05:23:49.635086+0000","objects_scrubbed":11,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:14.138112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":98,"num_read_kb":85,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1042,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628605+0000","last_change":"2024-03-26T05:24:18.885706+0000","last_active":"2024-03-26T05:24:54.628605+0000","last_peered":"2024-03-26T05:24:54.628605+0000","last_clean":"2024-03-26T05:24:54.628605+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:24:54.628605+0000","last_undegraded":"2024-03-26T05:24:54.628605+0000","last_fullsized":"2024-03-26T05:24:54.628605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:18.885629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:24:18.885629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:43.265977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.865836+0000","last_change":"2024-03-26T05:24:16.642162+0000","last_active":"2024-03-26T05:24:53.865836+0000","last_peered":"2024-03-26T05:24:53.865836+0000","last_clean":"2024-03-26T05:24:53.865836+0000","last_became_active":"2024-03-26T05:23:23.927943+0000","last_became_peered":"2024-03-26T05:23:23.927943+0000","last_unstale":"2024-03-26T05:24:53.865836+0000","last_undegraded":"2024-03-26T05:24:53.865836+0000","last_fullsized":"2024-03-26T05:24:53.865836+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:16.642049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:16.642049+0000","last_clean_scrub_stamp":"2024-03-26T05:24:16.642049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:44.028895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1020,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628379+0000","last_change":"2024-03-26T05:24:10.134983+0000","last_active":"2024-03-26T05:24:54.628379+0000","last_peered":"2024-03-26T05:24:54.628379+0000","last_clean":"2024-03-26T05:24:54.628379+0000","last_became_active":"2024-03-26T05:15:32.637717+0000","last_became_peered":"2024-03-26T05:15:32.637717+0000","last_unstale":"2024-03-26T05:24:54.628379+0000","last_undegraded":"2024-03-26T05:24:54.628379+0000","last_fullsized":"2024-03-26T05:24:54.628379+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:10.134860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:10.134860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:16.518730+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"443'27","reported_seq":109,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866010+0000","last_change":"2024-03-26T05:24:21.705542+0000","last_active":"2024-03-26T05:24:53.866010+0000","last_peered":"2024-03-26T05:24:53.866010+0000","last_clean":"2024-03-26T05:24:53.866010+0000","last_became_active":"2024-03-26T05:23:23.928598+0000","last_became_peered":"2024-03-26T05:23:23.928598+0000","last_unstale":"2024-03-26T05:24:53.866010+0000","last_undegraded":"2024-03-26T05:24:53.866010+0000","last_fullsized":"2024-03-26T05:24:53.866010+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'27","last_scrub_stamp":"2024-03-26T05:24:21.705431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:21.705431+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:42.216471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1076,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863734+0000","last_change":"2024-03-26T05:23:58.803459+0000","last_active":"2024-03-26T05:24:53.863734+0000","last_peered":"2024-03-26T05:24:53.863734+0000","last_clean":"2024-03-26T05:24:53.863734+0000","last_became_active":"2024-03-26T05:23:26.810639+0000","last_became_peered":"2024-03-26T05:23:26.810639+0000","last_unstale":"2024-03-26T05:24:53.863734+0000","last_undegraded":"2024-03-26T05:24:53.863734+0000","last_fullsized":"2024-03-26T05:24:53.863734+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:58.803375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:23:58.803375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:00.549208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1065,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613316+0000","last_change":"2024-03-26T05:23:31.098901+0000","last_active":"2024-03-26T05:24:54.613316+0000","last_peered":"2024-03-26T05:24:54.613316+0000","last_clean":"2024-03-26T05:24:54.613316+0000","last_became_active":"2024-03-26T05:23:26.818750+0000","last_became_peered":"2024-03-26T05:23:26.818750+0000","last_unstale":"2024-03-26T05:24:54.613316+0000","last_undegraded":"2024-03-26T05:24:54.613316+0000","last_fullsized":"2024-03-26T05:24:54.613316+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:31.098786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:23:31.098786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:59.884114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":103,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866510+0000","last_change":"2024-03-26T05:23:59.562452+0000","last_active":"2024-03-26T05:24:53.866510+0000","last_peered":"2024-03-26T05:24:53.866510+0000","last_clean":"2024-03-26T05:24:53.866510+0000","last_became_active":"2024-03-26T05:23:26.820867+0000","last_became_peered":"2024-03-26T05:23:26.820867+0000","last_unstale":"2024-03-26T05:24:53.866510+0000","last_undegraded":"2024-03-26T05:24:53.866510+0000","last_fullsized":"2024-03-26T05:24:53.866510+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:59.562345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:23:59.562345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:08.378788+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":638,"num_read_kb":633,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1823,"num_read_kb":1548,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":7018,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3666,"internal_metadata":0},"log_size":1321,"ondisk_log_size":1321,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":335,"seq":1438814044277,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":523564,"kb_used_data":1060,"kb_used_omap":4,"kb_used_meta":522491,"kb_avail":93221588,"statfs":{"total":95995035648,"available":95458906112,"internally_reserved":0,"allocated":1085440,"data_stored":766308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":535031464},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332403,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":731612,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93013540,"statfs":{"total":95995035648,"available":95245864960,"internally_reserved":0,"allocated":741376,"data_stored":437112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":748419526},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575180,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938496,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92806656,"statfs":{"total":95995035648,"available":95034015744,"internally_reserved":0,"allocated":778240,"data_stored":437936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738705,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952376,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92792776,"statfs":{"total":95995035648,"available":95019802624,"internally_reserved":0,"allocated":770048,"data_stored":438781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411656,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":731880,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93013272,"statfs":{"total":95995035648,"available":95245590528,"internally_reserved":0,"allocated":1015808,"data_stored":700054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117314,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907132,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92838020,"statfs":{"total":95995035648,"available":95066132480,"internally_reserved":0,"allocated":1363968,"data_stored":1027960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":927529415},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":501,"seq":2151778615320,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1208176,"kb_used_data":1145384,"kb_used_omap":7,"kb_used_meta":62776,"kb_avail":92536976,"statfs":{"total":95995035648,"available":94757863424,"internally_reserved":0,"allocated":1172873216,"data_stored":1172549216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8112,"internal_metadata":64282704},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659448,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894628,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":92850524,"statfs":{"total":95995035648,"available":95078936576,"internally_reserved":0,"allocated":1339392,"data_stored":1027806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":914749893},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3666,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:25:07.821 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:25:08.067 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:25:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[29691]: 5.7 scrub starts 2024-03-26T05:25:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[29691]: 5.7 scrub ok 2024-03-26T05:25:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[29691]: 6.1c scrub starts 2024-03-26T05:25:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[29691]: 6.1c scrub ok 2024-03-26T05:25:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[29691]: 3.1a scrub starts 2024-03-26T05:25:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[29691]: 3.1a scrub ok 2024-03-26T05:25:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[29691]: 6.4 deep-scrub starts 2024-03-26T05:25:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[29691]: 6.4 deep-scrub ok 2024-03-26T05:25:08.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[29691]: from='client.16407 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:25:08.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[40404]: 5.7 scrub starts 2024-03-26T05:25:08.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[40404]: 5.7 scrub ok 2024-03-26T05:25:08.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[40404]: 6.1c scrub starts 2024-03-26T05:25:08.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[40404]: 6.1c scrub ok 2024-03-26T05:25:08.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[40404]: 3.1a scrub starts 2024-03-26T05:25:08.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[40404]: 3.1a scrub ok 2024-03-26T05:25:08.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[40404]: 6.4 deep-scrub starts 2024-03-26T05:25:08.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[40404]: 6.4 deep-scrub ok 2024-03-26T05:25:08.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:07 smithi080 ceph-mon[40404]: from='client.16407 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:25:08.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:07 smithi132 ceph-mon[33500]: 5.7 scrub starts 2024-03-26T05:25:08.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:07 smithi132 ceph-mon[33500]: 5.7 scrub ok 2024-03-26T05:25:08.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:07 smithi132 ceph-mon[33500]: 6.1c scrub starts 2024-03-26T05:25:08.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:07 smithi132 ceph-mon[33500]: 6.1c scrub ok 2024-03-26T05:25:08.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:07 smithi132 ceph-mon[33500]: 3.1a scrub starts 2024-03-26T05:25:08.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:07 smithi132 ceph-mon[33500]: 3.1a scrub ok 2024-03-26T05:25:08.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:07 smithi132 ceph-mon[33500]: 6.4 deep-scrub starts 2024-03-26T05:25:08.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:07 smithi132 ceph-mon[33500]: 6.4 deep-scrub ok 2024-03-26T05:25:08.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:07 smithi132 ceph-mon[33500]: from='client.16407 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:25:09.162 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:08 smithi080 ceph-mon[40404]: 3.1f scrub starts 2024-03-26T05:25:09.163 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:08 smithi080 ceph-mon[40404]: 3.1f scrub ok 2024-03-26T05:25:09.163 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:08 smithi080 ceph-mon[40404]: pgmap v1181: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:25:09.163 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:08 smithi080 ceph-mon[29691]: 3.1f scrub starts 2024-03-26T05:25:09.163 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:08 smithi080 ceph-mon[29691]: 3.1f scrub ok 2024-03-26T05:25:09.163 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:08 smithi080 ceph-mon[29691]: pgmap v1181: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:25:09.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:08 smithi132 ceph-mon[33500]: 3.1f scrub starts 2024-03-26T05:25:09.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:08 smithi132 ceph-mon[33500]: 3.1f scrub ok 2024-03-26T05:25:09.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:08 smithi132 ceph-mon[33500]: pgmap v1181: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:25:09.872 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-26T05:25:09.874 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-26T05:25:10.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:09 smithi080 ceph-mon[40404]: 1.2 scrub starts 2024-03-26T05:25:10.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:09 smithi080 ceph-mon[40404]: 1.2 scrub ok 2024-03-26T05:25:10.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:09 smithi080 ceph-mon[29691]: 1.2 scrub starts 2024-03-26T05:25:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:09 smithi080 ceph-mon[29691]: 1.2 scrub ok 2024-03-26T05:25:10.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:09 smithi132 ceph-mon[33500]: 1.2 scrub starts 2024-03-26T05:25:10.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:09 smithi132 ceph-mon[33500]: 1.2 scrub ok 2024-03-26T05:25:10.540 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":1181,"stamp":"2024-03-26T05:25:08.318777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":2485,"num_read_kb":2205,"num_write":1200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":929567,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2858,"ondisk_log_size":2858,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":434,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6887864,"kb_used_data":1152312,"kb_used_omap":21,"kb_used_meta":5735466,"kb_avail":743073352,"statfs":{"total":767960285184,"available":760907112448,"internally_reserved":0,"allocated":1179967488,"data_stored":1177385173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22119,"internal_metadata":5873117593},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002876"},"pg_stats":[{"pgid":"4.c","version":"524'35","reported_seq":114,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866935+0000","last_change":"2024-03-26T05:23:57.476800+0000","last_active":"2024-03-26T05:24:53.866935+0000","last_peered":"2024-03-26T05:24:53.866935+0000","last_clean":"2024-03-26T05:24:53.866935+0000","last_became_active":"2024-03-26T05:23:23.934466+0000","last_became_peered":"2024-03-26T05:23:23.934466+0000","last_unstale":"2024-03-26T05:24:53.866935+0000","last_undegraded":"2024-03-26T05:24:53.866935+0000","last_fullsized":"2024-03-26T05:24:53.866935+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"524'35","last_scrub_stamp":"2024-03-26T05:23:57.476717+0000","last_deep_scrub":"524'35","last_deep_scrub_stamp":"2024-03-26T05:23:57.476717+0000","last_clean_scrub_stamp":"2024-03-26T05:23:57.476717+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:02.375662+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1060,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628245+0000","last_change":"2024-03-26T05:24:27.927986+0000","last_active":"2024-03-26T05:24:54.628245+0000","last_peered":"2024-03-26T05:24:54.628245+0000","last_clean":"2024-03-26T05:24:54.628245+0000","last_became_active":"2024-03-26T05:23:23.934367+0000","last_became_peered":"2024-03-26T05:23:23.934367+0000","last_unstale":"2024-03-26T05:24:54.628245+0000","last_undegraded":"2024-03-26T05:24:54.628245+0000","last_fullsized":"2024-03-26T05:24:54.628245+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:27.927903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:24:27.927903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:37.235808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"508'15","reported_seq":1410,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704427+0000","last_change":"2024-03-26T05:24:40.508225+0000","last_active":"2024-03-26T05:24:58.704427+0000","last_peered":"2024-03-26T05:24:58.704427+0000","last_clean":"2024-03-26T05:24:58.704427+0000","last_became_active":"2024-03-26T05:23:26.805008+0000","last_became_peered":"2024-03-26T05:23:26.805008+0000","last_unstale":"2024-03-26T05:24:58.704427+0000","last_undegraded":"2024-03-26T05:24:58.704427+0000","last_fullsized":"2024-03-26T05:24:58.704427+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'15","last_scrub_stamp":"2024-03-26T05:24:40.508138+0000","last_deep_scrub":"117'9","last_deep_scrub_stamp":"2024-03-26T05:03:09.035843+0000","last_clean_scrub_stamp":"2024-03-26T05:24:40.508138+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:09.226754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.867118+0000","last_change":"2024-03-26T05:24:09.665071+0000","last_active":"2024-03-26T05:24:53.867118+0000","last_peered":"2024-03-26T05:24:53.867118+0000","last_clean":"2024-03-26T05:24:53.867118+0000","last_became_active":"2024-03-26T05:23:23.943566+0000","last_became_peered":"2024-03-26T05:23:23.943566+0000","last_unstale":"2024-03-26T05:24:53.867118+0000","last_undegraded":"2024-03-26T05:24:53.867118+0000","last_fullsized":"2024-03-26T05:24:53.867118+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:09.664959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:09.664959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:21.887749+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"88'128","reported_seq":1233,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628098+0000","last_change":"2024-03-26T05:24:54.628147+0000","last_active":"2024-03-26T05:24:54.628098+0000","last_peered":"2024-03-26T05:24:54.628098+0000","last_clean":"2024-03-26T05:24:54.628098+0000","last_became_active":"2024-03-26T05:15:33.616505+0000","last_became_peered":"2024-03-26T05:15:33.616505+0000","last_unstale":"2024-03-26T05:24:54.628098+0000","last_undegraded":"2024-03-26T05:24:54.628098+0000","last_fullsized":"2024-03-26T05:24:54.628098+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:54.627969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:54.627969+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:03.373012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"504'53","reported_seq":131,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.867097+0000","last_change":"2024-03-26T05:24:51.724793+0000","last_active":"2024-03-26T05:24:53.867097+0000","last_peered":"2024-03-26T05:24:53.867097+0000","last_clean":"2024-03-26T05:24:53.867097+0000","last_became_active":"2024-03-26T05:23:23.936434+0000","last_became_peered":"2024-03-26T05:23:23.936434+0000","last_unstale":"2024-03-26T05:24:53.867097+0000","last_undegraded":"2024-03-26T05:24:53.867097+0000","last_fullsized":"2024-03-26T05:24:53.867097+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'53","last_scrub_stamp":"2024-03-26T05:24:51.724667+0000","last_deep_scrub":"504'53","last_deep_scrub_stamp":"2024-03-26T05:24:51.724667+0000","last_clean_scrub_stamp":"2024-03-26T05:24:51.724667+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:19.890634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1108,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.945645+0000","last_change":"2024-03-26T05:24:41.749017+0000","last_active":"2024-03-26T05:24:54.945645+0000","last_peered":"2024-03-26T05:24:54.945645+0000","last_clean":"2024-03-26T05:24:54.945645+0000","last_became_active":"2024-03-26T05:23:23.935162+0000","last_became_peered":"2024-03-26T05:23:23.935162+0000","last_unstale":"2024-03-26T05:24:54.945645+0000","last_undegraded":"2024-03-26T05:24:54.945645+0000","last_fullsized":"2024-03-26T05:24:54.945645+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:41.748912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:45.549936+0000","last_clean_scrub_stamp":"2024-03-26T05:24:41.748912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:00.092226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1055,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:02.204587+0000","last_change":"2024-03-26T05:25:02.204608+0000","last_active":"2024-03-26T05:25:02.204587+0000","last_peered":"2024-03-26T05:25:02.204587+0000","last_clean":"2024-03-26T05:25:02.204587+0000","last_became_active":"2024-03-26T05:23:23.924351+0000","last_became_peered":"2024-03-26T05:23:23.924351+0000","last_unstale":"2024-03-26T05:25:02.204587+0000","last_undegraded":"2024-03-26T05:25:02.204587+0000","last_fullsized":"2024-03-26T05:25:02.204587+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:25:02.204526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:09:56.509231+0000","last_clean_scrub_stamp":"2024-03-26T05:25:02.204526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:09.387881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1054,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.516785+0000","last_change":"2024-03-26T05:24:58.516825+0000","last_active":"2024-03-26T05:24:58.516785+0000","last_peered":"2024-03-26T05:24:58.516785+0000","last_clean":"2024-03-26T05:24:58.516785+0000","last_became_active":"2024-03-26T05:23:23.943972+0000","last_became_peered":"2024-03-26T05:23:23.943972+0000","last_unstale":"2024-03-26T05:24:58.516785+0000","last_undegraded":"2024-03-26T05:24:58.516785+0000","last_fullsized":"2024-03-26T05:24:58.516785+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:58.516670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:58.516670+0000","last_clean_scrub_stamp":"2024-03-26T05:24:58.516670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.259476+0000","scrub_duration":230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"88'128","reported_seq":629,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:56.447301+0000","last_change":"2024-03-26T05:24:56.447341+0000","last_active":"2024-03-26T05:24:56.447301+0000","last_peered":"2024-03-26T05:24:56.447301+0000","last_clean":"2024-03-26T05:24:56.447301+0000","last_became_active":"2024-03-26T05:23:29.790949+0000","last_became_peered":"2024-03-26T05:23:29.790949+0000","last_unstale":"2024-03-26T05:24:56.447301+0000","last_undegraded":"2024-03-26T05:24:56.447301+0000","last_fullsized":"2024-03-26T05:24:56.447301+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:56.447202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:56.447202+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.745143+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"443'33","reported_seq":111,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866969+0000","last_change":"2024-03-26T05:24:05.544239+0000","last_active":"2024-03-26T05:24:53.866969+0000","last_peered":"2024-03-26T05:24:53.866969+0000","last_clean":"2024-03-26T05:24:53.866969+0000","last_became_active":"2024-03-26T05:23:23.942559+0000","last_became_peered":"2024-03-26T05:23:23.942559+0000","last_unstale":"2024-03-26T05:24:53.866969+0000","last_undegraded":"2024-03-26T05:24:53.866969+0000","last_fullsized":"2024-03-26T05:24:53.866969+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'33","last_scrub_stamp":"2024-03-26T05:24:05.544072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:05.544072+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:08.325712+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":105,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866283+0000","last_change":"2024-03-26T05:24:50.679081+0000","last_active":"2024-03-26T05:24:53.866283+0000","last_peered":"2024-03-26T05:24:53.866283+0000","last_clean":"2024-03-26T05:24:53.866283+0000","last_became_active":"2024-03-26T05:23:23.940539+0000","last_became_peered":"2024-03-26T05:23:23.940539+0000","last_unstale":"2024-03-26T05:24:53.866283+0000","last_undegraded":"2024-03-26T05:24:53.866283+0000","last_fullsized":"2024-03-26T05:24:53.866283+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:50.678973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:50.678973+0000","last_clean_scrub_stamp":"2024-03-26T05:24:50.678973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.106668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114413+0000","last_change":"2024-03-26T05:24:38.362574+0000","last_active":"2024-03-26T05:24:55.114413+0000","last_peered":"2024-03-26T05:24:55.114413+0000","last_clean":"2024-03-26T05:24:55.114413+0000","last_became_active":"2024-03-26T05:23:26.811352+0000","last_became_peered":"2024-03-26T05:23:26.811352+0000","last_unstale":"2024-03-26T05:24:55.114413+0000","last_undegraded":"2024-03-26T05:24:55.114413+0000","last_fullsized":"2024-03-26T05:24:55.114413+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:38.362489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:13:07.883082+0000","last_clean_scrub_stamp":"2024-03-26T05:24:38.362489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:06.510293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1052,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628824+0000","last_change":"2024-03-26T05:24:22.885221+0000","last_active":"2024-03-26T05:24:54.628824+0000","last_peered":"2024-03-26T05:24:54.628824+0000","last_clean":"2024-03-26T05:24:54.628824+0000","last_became_active":"2024-03-26T05:23:26.798774+0000","last_became_peered":"2024-03-26T05:23:26.798774+0000","last_unstale":"2024-03-26T05:24:54.628824+0000","last_undegraded":"2024-03-26T05:24:54.628824+0000","last_fullsized":"2024-03-26T05:24:54.628824+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:22.885144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:22.885144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:44.127918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"443'46","reported_seq":1159,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613969+0000","last_change":"2024-03-26T05:23:57.258003+0000","last_active":"2024-03-26T05:24:54.613969+0000","last_peered":"2024-03-26T05:24:54.613969+0000","last_clean":"2024-03-26T05:24:54.613969+0000","last_became_active":"2024-03-26T05:23:23.923212+0000","last_became_peered":"2024-03-26T05:23:23.923212+0000","last_unstale":"2024-03-26T05:24:54.613969+0000","last_undegraded":"2024-03-26T05:24:54.613969+0000","last_fullsized":"2024-03-26T05:24:54.613969+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'46","last_scrub_stamp":"2024-03-26T05:23:57.257836+0000","last_deep_scrub":"292'34","last_deep_scrub_stamp":"2024-03-26T05:12:23.778630+0000","last_clean_scrub_stamp":"2024-03-26T05:23:57.257836+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:12.134655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1042,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628118+0000","last_change":"2024-03-26T05:23:51.643958+0000","last_active":"2024-03-26T05:24:54.628118+0000","last_peered":"2024-03-26T05:24:54.628118+0000","last_clean":"2024-03-26T05:24:54.628118+0000","last_became_active":"2024-03-26T05:15:32.626265+0000","last_became_peered":"2024-03-26T05:15:32.626265+0000","last_unstale":"2024-03-26T05:24:54.628118+0000","last_undegraded":"2024-03-26T05:24:54.628118+0000","last_fullsized":"2024-03-26T05:24:54.628118+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:51.643877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:23:51.643877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:12.355149+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"542'16","reported_seq":141,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704607+0000","last_change":"2024-03-26T05:24:39.595828+0000","last_active":"2024-03-26T05:24:58.704607+0000","last_peered":"2024-03-26T05:24:58.704607+0000","last_clean":"2024-03-26T05:24:58.704607+0000","last_became_active":"2024-03-26T05:23:26.815826+0000","last_became_peered":"2024-03-26T05:23:26.815826+0000","last_unstale":"2024-03-26T05:24:58.704607+0000","last_undegraded":"2024-03-26T05:24:58.704607+0000","last_fullsized":"2024-03-26T05:24:58.704607+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"542'16","last_scrub_stamp":"2024-03-26T05:24:39.595738+0000","last_deep_scrub":"542'16","last_deep_scrub_stamp":"2024-03-26T05:24:39.595738+0000","last_clean_scrub_stamp":"2024-03-26T05:24:39.595738+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.026484+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.115178+0000","last_change":"2024-03-26T05:24:34.502280+0000","last_active":"2024-03-26T05:24:55.115178+0000","last_peered":"2024-03-26T05:24:55.115178+0000","last_clean":"2024-03-26T05:24:55.115178+0000","last_became_active":"2024-03-26T05:23:26.806027+0000","last_became_peered":"2024-03-26T05:23:26.806027+0000","last_unstale":"2024-03-26T05:24:55.115178+0000","last_undegraded":"2024-03-26T05:24:55.115178+0000","last_fullsized":"2024-03-26T05:24:55.115178+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:34.502151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:13.155026+0000","last_clean_scrub_stamp":"2024-03-26T05:24:34.502151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:37.567669+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"88'128","reported_seq":620,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.115401+0000","last_change":"2024-03-26T05:23:54.384405+0000","last_active":"2024-03-26T05:24:55.115401+0000","last_peered":"2024-03-26T05:24:55.115401+0000","last_clean":"2024-03-26T05:24:55.115401+0000","last_became_active":"2024-03-26T05:23:29.788529+0000","last_became_peered":"2024-03-26T05:23:29.788529+0000","last_unstale":"2024-03-26T05:24:55.115401+0000","last_undegraded":"2024-03-26T05:24:55.115401+0000","last_fullsized":"2024-03-26T05:24:55.115401+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:23:54.384271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:23:54.384271+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:08.342345+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866211+0000","last_change":"2024-03-26T05:23:47.499758+0000","last_active":"2024-03-26T05:24:53.866211+0000","last_peered":"2024-03-26T05:24:53.866211+0000","last_clean":"2024-03-26T05:24:53.866211+0000","last_became_active":"2024-03-26T05:23:23.927317+0000","last_became_peered":"2024-03-26T05:23:23.927317+0000","last_unstale":"2024-03-26T05:24:53.866211+0000","last_undegraded":"2024-03-26T05:24:53.866211+0000","last_fullsized":"2024-03-26T05:24:53.866211+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:47.499614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:23:47.499614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:11.795722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"443'64","reported_seq":1149,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628254+0000","last_change":"2024-03-26T05:24:32.900767+0000","last_active":"2024-03-26T05:24:54.628254+0000","last_peered":"2024-03-26T05:24:54.628254+0000","last_clean":"2024-03-26T05:24:54.628254+0000","last_became_active":"2024-03-26T05:15:32.636466+0000","last_became_peered":"2024-03-26T05:15:32.636466+0000","last_unstale":"2024-03-26T05:24:54.628254+0000","last_undegraded":"2024-03-26T05:24:54.628254+0000","last_fullsized":"2024-03-26T05:24:54.628254+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'64","last_scrub_stamp":"2024-03-26T05:24:32.900670+0000","last_deep_scrub":"96'40","last_deep_scrub_stamp":"2024-03-26T05:03:14.123706+0000","last_clean_scrub_stamp":"2024-03-26T05:24:32.900670+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:32.919847+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":74,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1034,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628776+0000","last_change":"2024-03-26T05:23:26.606642+0000","last_active":"2024-03-26T05:24:54.628776+0000","last_peered":"2024-03-26T05:24:54.628776+0000","last_clean":"2024-03-26T05:24:54.628776+0000","last_became_active":"2024-03-26T05:15:32.636983+0000","last_became_peered":"2024-03-26T05:15:32.636983+0000","last_unstale":"2024-03-26T05:24:54.628776+0000","last_undegraded":"2024-03-26T05:24:54.628776+0000","last_fullsized":"2024-03-26T05:24:54.628776+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:26.606563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:23:26.606563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866427+0000","last_change":"2024-03-26T05:24:14.649708+0000","last_active":"2024-03-26T05:24:53.866427+0000","last_peered":"2024-03-26T05:24:53.866427+0000","last_clean":"2024-03-26T05:24:53.866427+0000","last_became_active":"2024-03-26T05:23:23.937610+0000","last_became_peered":"2024-03-26T05:23:23.937610+0000","last_unstale":"2024-03-26T05:24:53.866427+0000","last_undegraded":"2024-03-26T05:24:53.866427+0000","last_fullsized":"2024-03-26T05:24:53.866427+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:14.649618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:14.649618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:38.751980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"88'128","reported_seq":709,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.867109+0000","last_change":"2024-03-26T05:24:49.650076+0000","last_active":"2024-03-26T05:24:53.867109+0000","last_peered":"2024-03-26T05:24:53.867109+0000","last_clean":"2024-03-26T05:24:53.867109+0000","last_became_active":"2024-03-26T05:23:26.794528+0000","last_became_peered":"2024-03-26T05:23:26.794528+0000","last_unstale":"2024-03-26T05:24:53.867109+0000","last_undegraded":"2024-03-26T05:24:53.867109+0000","last_fullsized":"2024-03-26T05:24:53.867109+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:49.649969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:49.649969+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:54.547383+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"504'11","reported_seq":1347,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704729+0000","last_change":"2024-03-26T05:24:49.894562+0000","last_active":"2024-03-26T05:24:58.704729+0000","last_peered":"2024-03-26T05:24:58.704729+0000","last_clean":"2024-03-26T05:24:58.704729+0000","last_became_active":"2024-03-26T05:23:23.935379+0000","last_became_peered":"2024-03-26T05:23:23.935379+0000","last_unstale":"2024-03-26T05:24:58.704729+0000","last_undegraded":"2024-03-26T05:24:58.704729+0000","last_fullsized":"2024-03-26T05:24:58.704729+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'11","last_scrub_stamp":"2024-03-26T05:24:49.894423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:49.894423+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:03.719125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"443'48","reported_seq":139,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866510+0000","last_change":"2024-03-26T05:24:18.678081+0000","last_active":"2024-03-26T05:24:53.866510+0000","last_peered":"2024-03-26T05:24:53.866510+0000","last_clean":"2024-03-26T05:24:53.866510+0000","last_became_active":"2024-03-26T05:23:26.821811+0000","last_became_peered":"2024-03-26T05:23:26.821811+0000","last_unstale":"2024-03-26T05:24:53.866510+0000","last_undegraded":"2024-03-26T05:24:53.866510+0000","last_fullsized":"2024-03-26T05:24:53.866510+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'48","last_scrub_stamp":"2024-03-26T05:24:18.677949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:18.677949+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:47.140363+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":62,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"523'2","reported_seq":1077,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863767+0000","last_change":"2024-03-26T05:23:54.815597+0000","last_active":"2024-03-26T05:24:53.863767+0000","last_peered":"2024-03-26T05:24:53.863767+0000","last_clean":"2024-03-26T05:24:53.863767+0000","last_became_active":"2024-03-26T05:23:23.943577+0000","last_became_peered":"2024-03-26T05:23:23.943577+0000","last_unstale":"2024-03-26T05:24:53.863767+0000","last_undegraded":"2024-03-26T05:24:53.863767+0000","last_fullsized":"2024-03-26T05:24:53.863767+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'2","last_scrub_stamp":"2024-03-26T05:23:54.815515+0000","last_deep_scrub":"523'2","last_deep_scrub_stamp":"2024-03-26T05:23:54.815515+0000","last_clean_scrub_stamp":"2024-03-26T05:23:54.815515+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:13.249966+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":958,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627724+0000","last_change":"2024-03-26T05:24:38.724006+0000","last_active":"2024-03-26T05:24:54.627724+0000","last_peered":"2024-03-26T05:24:54.627724+0000","last_clean":"2024-03-26T05:24:54.627724+0000","last_became_active":"2024-03-26T05:15:32.630135+0000","last_became_peered":"2024-03-26T05:15:32.630135+0000","last_unstale":"2024-03-26T05:24:54.627724+0000","last_undegraded":"2024-03-26T05:24:54.627724+0000","last_fullsized":"2024-03-26T05:24:54.627724+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:38.723902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:26.563380+0000","last_clean_scrub_stamp":"2024-03-26T05:24:38.723902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:00.966186+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"396'129","reported_seq":878,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628367+0000","last_change":"2024-03-26T05:23:29.791291+0000","last_active":"2024-03-26T05:24:54.628367+0000","last_peered":"2024-03-26T05:24:54.628367+0000","last_clean":"2024-03-26T05:24:54.628367+0000","last_became_active":"2024-03-26T05:23:29.790874+0000","last_became_peered":"2024-03-26T05:23:29.790874+0000","last_unstale":"2024-03-26T05:24:54.628367+0000","last_undegraded":"2024-03-26T05:24:54.628367+0000","last_fullsized":"2024-03-26T05:24:54.628367+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"396'129","last_scrub_stamp":"2024-03-26T05:23:12.709531+0000","last_deep_scrub":"396'129","last_deep_scrub_stamp":"2024-03-26T05:23:12.709531+0000","last_clean_scrub_stamp":"2024-03-26T05:23:12.709531+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"508'13","reported_seq":1400,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704428+0000","last_change":"2024-03-26T05:23:53.830513+0000","last_active":"2024-03-26T05:24:58.704428+0000","last_peered":"2024-03-26T05:24:58.704428+0000","last_clean":"2024-03-26T05:24:58.704428+0000","last_became_active":"2024-03-26T05:23:26.811064+0000","last_became_peered":"2024-03-26T05:23:26.811064+0000","last_unstale":"2024-03-26T05:24:58.704428+0000","last_undegraded":"2024-03-26T05:24:58.704428+0000","last_fullsized":"2024-03-26T05:24:58.704428+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'13","last_scrub_stamp":"2024-03-26T05:23:53.830404+0000","last_deep_scrub":"100'9","last_deep_scrub_stamp":"2024-03-26T05:05:11.014574+0000","last_clean_scrub_stamp":"2024-03-26T05:23:53.830404+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:16.286405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"443'36","reported_seq":297,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613554+0000","last_change":"2024-03-26T05:24:04.154903+0000","last_active":"2024-03-26T05:24:54.613554+0000","last_peered":"2024-03-26T05:24:54.613554+0000","last_clean":"2024-03-26T05:24:54.613554+0000","last_became_active":"2024-03-26T05:23:26.814397+0000","last_became_peered":"2024-03-26T05:23:26.814397+0000","last_unstale":"2024-03-26T05:24:54.613554+0000","last_undegraded":"2024-03-26T05:24:54.613554+0000","last_fullsized":"2024-03-26T05:24:54.613554+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'36","last_scrub_stamp":"2024-03-26T05:24:04.154794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:04.154794+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:32.051611+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114760+0000","last_change":"2024-03-26T05:24:53.460669+0000","last_active":"2024-03-26T05:24:55.114760+0000","last_peered":"2024-03-26T05:24:55.114760+0000","last_clean":"2024-03-26T05:24:55.114760+0000","last_became_active":"2024-03-26T05:23:26.803216+0000","last_became_peered":"2024-03-26T05:23:26.803216+0000","last_unstale":"2024-03-26T05:24:55.114760+0000","last_undegraded":"2024-03-26T05:24:55.114760+0000","last_fullsized":"2024-03-26T05:24:55.114760+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:53.460562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:24:53.460562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:20.820850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1050,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:57.108239+0000","last_change":"2024-03-26T05:24:57.108261+0000","last_active":"2024-03-26T05:24:57.108239+0000","last_peered":"2024-03-26T05:24:57.108239+0000","last_clean":"2024-03-26T05:24:57.108239+0000","last_became_active":"2024-03-26T05:00:28.986920+0000","last_became_peered":"2024-03-26T05:00:28.986920+0000","last_unstale":"2024-03-26T05:24:57.108239+0000","last_undegraded":"2024-03-26T05:24:57.108239+0000","last_fullsized":"2024-03-26T05:24:57.108239+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:57.108166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:20:04.436614+0000","last_clean_scrub_stamp":"2024-03-26T05:24:57.108166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:12.832195+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"88'128","reported_seq":810,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.946296+0000","last_change":"2024-03-26T05:24:36.738001+0000","last_active":"2024-03-26T05:24:54.946296+0000","last_peered":"2024-03-26T05:24:54.946296+0000","last_clean":"2024-03-26T05:24:54.946296+0000","last_became_active":"2024-03-26T05:15:33.617538+0000","last_became_peered":"2024-03-26T05:15:33.617538+0000","last_unstale":"2024-03-26T05:24:54.946296+0000","last_undegraded":"2024-03-26T05:24:54.946296+0000","last_fullsized":"2024-03-26T05:24:54.946296+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:36.737913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:36.737913+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:06.450085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1106,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.945052+0000","last_change":"2024-03-26T05:24:38.705636+0000","last_active":"2024-03-26T05:24:54.945052+0000","last_peered":"2024-03-26T05:24:54.945052+0000","last_clean":"2024-03-26T05:24:54.945052+0000","last_became_active":"2024-03-26T05:23:26.815745+0000","last_became_peered":"2024-03-26T05:23:26.815745+0000","last_unstale":"2024-03-26T05:24:54.945052+0000","last_undegraded":"2024-03-26T05:24:54.945052+0000","last_fullsized":"2024-03-26T05:24:54.945052+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:38.705558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:59.261180+0000","last_clean_scrub_stamp":"2024-03-26T05:24:38.705558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:50.201177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"443'54","reported_seq":1256,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:03.055936+0000","last_change":"2024-03-26T05:25:03.055974+0000","last_active":"2024-03-26T05:25:03.055936+0000","last_peered":"2024-03-26T05:25:03.055936+0000","last_clean":"2024-03-26T05:25:03.055936+0000","last_became_active":"2024-03-26T05:23:26.802524+0000","last_became_peered":"2024-03-26T05:23:26.802524+0000","last_unstale":"2024-03-26T05:25:03.055936+0000","last_undegraded":"2024-03-26T05:25:03.055936+0000","last_fullsized":"2024-03-26T05:25:03.055936+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'54","last_scrub_stamp":"2024-03-26T05:25:03.055874+0000","last_deep_scrub":"243'34","last_deep_scrub_stamp":"2024-03-26T05:09:53.978203+0000","last_clean_scrub_stamp":"2024-03-26T05:25:03.055874+0000","objects_scrubbed":9,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:12.091770+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1089,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:59.241821+0000","last_change":"2024-03-26T05:24:59.241863+0000","last_active":"2024-03-26T05:24:59.241821+0000","last_peered":"2024-03-26T05:24:59.241821+0000","last_clean":"2024-03-26T05:24:59.241821+0000","last_became_active":"2024-03-26T05:23:26.814433+0000","last_became_peered":"2024-03-26T05:23:26.814433+0000","last_unstale":"2024-03-26T05:24:59.241821+0000","last_undegraded":"2024-03-26T05:24:59.241821+0000","last_fullsized":"2024-03-26T05:24:59.241821+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:59.241718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:32.076613+0000","last_clean_scrub_stamp":"2024-03-26T05:24:59.241718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:08.528257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1045,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613382+0000","last_change":"2024-03-26T05:24:34.192183+0000","last_active":"2024-03-26T05:24:54.613382+0000","last_peered":"2024-03-26T05:24:54.613382+0000","last_clean":"2024-03-26T05:24:54.613382+0000","last_became_active":"2024-03-26T05:00:29.461191+0000","last_became_peered":"2024-03-26T05:00:29.461191+0000","last_unstale":"2024-03-26T05:24:54.613382+0000","last_undegraded":"2024-03-26T05:24:54.613382+0000","last_fullsized":"2024-03-26T05:24:54.613382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:34.192057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:20.739242+0000","last_clean_scrub_stamp":"2024-03-26T05:24:34.192057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:43.301832+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"88'128","reported_seq":812,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.946224+0000","last_change":"2024-03-26T05:24:42.704622+0000","last_active":"2024-03-26T05:24:54.946224+0000","last_peered":"2024-03-26T05:24:54.946224+0000","last_clean":"2024-03-26T05:24:54.946224+0000","last_became_active":"2024-03-26T05:15:22.624234+0000","last_became_peered":"2024-03-26T05:15:22.624234+0000","last_unstale":"2024-03-26T05:24:54.946224+0000","last_undegraded":"2024-03-26T05:24:54.946224+0000","last_fullsized":"2024-03-26T05:24:54.946224+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:42.704521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:42.704521+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:12.560237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114666+0000","last_change":"2024-03-26T05:23:40.433667+0000","last_active":"2024-03-26T05:24:55.114666+0000","last_peered":"2024-03-26T05:24:55.114666+0000","last_clean":"2024-03-26T05:24:55.114666+0000","last_became_active":"2024-03-26T05:23:26.803617+0000","last_became_peered":"2024-03-26T05:23:26.803617+0000","last_unstale":"2024-03-26T05:24:55.114666+0000","last_undegraded":"2024-03-26T05:24:55.114666+0000","last_fullsized":"2024-03-26T05:24:55.114666+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:40.433567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:23:40.433567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:05.903491+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"443'42","reported_seq":1156,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863696+0000","last_change":"2024-03-26T05:24:45.067962+0000","last_active":"2024-03-26T05:24:53.863696+0000","last_peered":"2024-03-26T05:24:53.863696+0000","last_clean":"2024-03-26T05:24:53.863696+0000","last_became_active":"2024-03-26T05:05:36.414709+0000","last_became_peered":"2024-03-26T05:05:36.414709+0000","last_unstale":"2024-03-26T05:24:53.863696+0000","last_undegraded":"2024-03-26T05:24:53.863696+0000","last_fullsized":"2024-03-26T05:24:53.863696+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'42","last_scrub_stamp":"2024-03-26T05:24:45.067806+0000","last_deep_scrub":"254'30","last_deep_scrub_stamp":"2024-03-26T05:20:05.430706+0000","last_clean_scrub_stamp":"2024-03-26T05:24:45.067806+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:13.288527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'6","reported_seq":2242,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:04.679404+0000","last_change":"2024-03-26T05:24:57.647759+0000","last_active":"2024-03-26T05:25:04.679404+0000","last_peered":"2024-03-26T05:25:04.679404+0000","last_clean":"2024-03-26T05:25:04.679404+0000","last_became_active":"2024-03-26T05:15:32.633061+0000","last_became_peered":"2024-03-26T05:15:32.633061+0000","last_unstale":"2024-03-26T05:25:04.679404+0000","last_undegraded":"2024-03-26T05:25:04.679404+0000","last_fullsized":"2024-03-26T05:25:04.679404+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-26T05:24:57.647662+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-26T05:01:36.780518+0000","last_clean_scrub_stamp":"2024-03-26T05:24:57.647662+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:01.073763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00086382199999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":638,"num_read_kb":633,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":783,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:02.684312+0000","last_change":"2024-03-26T05:25:02.684337+0000","last_active":"2024-03-26T05:25:02.684312+0000","last_peered":"2024-03-26T05:25:02.684312+0000","last_clean":"2024-03-26T05:25:02.684312+0000","last_became_active":"2024-03-26T05:23:26.814167+0000","last_became_peered":"2024-03-26T05:23:26.814167+0000","last_unstale":"2024-03-26T05:25:02.684312+0000","last_undegraded":"2024-03-26T05:25:02.684312+0000","last_fullsized":"2024-03-26T05:25:02.684312+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:25:02.684226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:25:02.684226+0000","last_clean_scrub_stamp":"2024-03-26T05:25:02.684226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:19.002393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1069,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613646+0000","last_change":"2024-03-26T05:23:35.035033+0000","last_active":"2024-03-26T05:24:54.613646+0000","last_peered":"2024-03-26T05:24:54.613646+0000","last_clean":"2024-03-26T05:24:54.613646+0000","last_became_active":"2024-03-26T05:23:26.813893+0000","last_became_peered":"2024-03-26T05:23:26.813893+0000","last_unstale":"2024-03-26T05:24:54.613646+0000","last_undegraded":"2024-03-26T05:24:54.613646+0000","last_fullsized":"2024-03-26T05:24:54.613646+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:35.034957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:23:35.034957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:59.347136+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"88'128","reported_seq":877,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613557+0000","last_change":"2024-03-26T05:24:06.151857+0000","last_active":"2024-03-26T05:24:54.613557+0000","last_peered":"2024-03-26T05:24:54.613557+0000","last_clean":"2024-03-26T05:24:54.613557+0000","last_became_active":"2024-03-26T05:23:26.793568+0000","last_became_peered":"2024-03-26T05:23:26.793568+0000","last_unstale":"2024-03-26T05:24:54.613557+0000","last_undegraded":"2024-03-26T05:24:54.613557+0000","last_fullsized":"2024-03-26T05:24:54.613557+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":3,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:06.151711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:06.151711+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:06.690101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":991,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627941+0000","last_change":"2024-03-26T05:24:43.640437+0000","last_active":"2024-03-26T05:24:54.627941+0000","last_peered":"2024-03-26T05:24:54.627941+0000","last_clean":"2024-03-26T05:24:54.627941+0000","last_became_active":"2024-03-26T05:23:26.796563+0000","last_became_peered":"2024-03-26T05:23:26.796563+0000","last_unstale":"2024-03-26T05:24:54.627941+0000","last_undegraded":"2024-03-26T05:24:54.627941+0000","last_fullsized":"2024-03-26T05:24:54.627941+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:43.640310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:30.459226+0000","last_clean_scrub_stamp":"2024-03-26T05:24:43.640310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:12.108706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"504'53","reported_seq":1480,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704655+0000","last_change":"2024-03-26T05:24:05.768474+0000","last_active":"2024-03-26T05:24:58.704655+0000","last_peered":"2024-03-26T05:24:58.704655+0000","last_clean":"2024-03-26T05:24:58.704655+0000","last_became_active":"2024-03-26T05:23:23.943584+0000","last_became_peered":"2024-03-26T05:23:23.943584+0000","last_unstale":"2024-03-26T05:24:58.704655+0000","last_undegraded":"2024-03-26T05:24:58.704655+0000","last_fullsized":"2024-03-26T05:24:58.704655+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'53","last_scrub_stamp":"2024-03-26T05:24:05.768347+0000","last_deep_scrub":"100'24","last_deep_scrub_stamp":"2024-03-26T05:02:00.251579+0000","last_clean_scrub_stamp":"2024-03-26T05:24:05.768347+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:21.004994+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1071,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628122+0000","last_change":"2024-03-26T05:23:31.577895+0000","last_active":"2024-03-26T05:24:54.628122+0000","last_peered":"2024-03-26T05:24:54.628122+0000","last_clean":"2024-03-26T05:24:54.628122+0000","last_became_active":"2024-03-26T05:23:26.803298+0000","last_became_peered":"2024-03-26T05:23:26.803298+0000","last_unstale":"2024-03-26T05:24:54.628122+0000","last_undegraded":"2024-03-26T05:24:54.628122+0000","last_fullsized":"2024-03-26T05:24:54.628122+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:31.577805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:23:31.577805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:58.663662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'128","reported_seq":1227,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627452+0000","last_change":"2024-03-26T05:24:26.606821+0000","last_active":"2024-03-26T05:24:54.627452+0000","last_peered":"2024-03-26T05:24:54.627452+0000","last_clean":"2024-03-26T05:24:54.627452+0000","last_became_active":"2024-03-26T05:15:32.632929+0000","last_became_peered":"2024-03-26T05:15:32.632929+0000","last_unstale":"2024-03-26T05:24:54.627452+0000","last_undegraded":"2024-03-26T05:24:54.627452+0000","last_fullsized":"2024-03-26T05:24:54.627452+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:24:26.606708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:26.606708+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:53.269117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114532+0000","last_change":"2024-03-26T05:24:28.556704+0000","last_active":"2024-03-26T05:24:55.114532+0000","last_peered":"2024-03-26T05:24:55.114532+0000","last_clean":"2024-03-26T05:24:55.114532+0000","last_became_active":"2024-03-26T05:23:26.811489+0000","last_became_peered":"2024-03-26T05:23:26.811489+0000","last_unstale":"2024-03-26T05:24:55.114532+0000","last_undegraded":"2024-03-26T05:24:55.114532+0000","last_fullsized":"2024-03-26T05:24:55.114532+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:28.556602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:08.881287+0000","last_clean_scrub_stamp":"2024-03-26T05:24:28.556602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:32.860328+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1091,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306879+0000","last_change":"2024-03-26T05:24:09.569441+0000","last_active":"2024-03-26T05:24:54.306879+0000","last_peered":"2024-03-26T05:24:54.306879+0000","last_clean":"2024-03-26T05:24:54.306879+0000","last_became_active":"2024-03-26T05:23:26.807099+0000","last_became_peered":"2024-03-26T05:23:26.807099+0000","last_unstale":"2024-03-26T05:24:54.306879+0000","last_undegraded":"2024-03-26T05:24:54.306879+0000","last_fullsized":"2024-03-26T05:24:54.306879+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:09.569314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:09.569314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:20.702045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"498'33","reported_seq":926,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306977+0000","last_change":"2024-03-26T05:24:19.574330+0000","last_active":"2024-03-26T05:24:54.306977+0000","last_peered":"2024-03-26T05:24:54.306977+0000","last_clean":"2024-03-26T05:24:54.306977+0000","last_became_active":"2024-03-26T05:23:26.799540+0000","last_became_peered":"2024-03-26T05:23:26.799540+0000","last_unstale":"2024-03-26T05:24:54.306977+0000","last_undegraded":"2024-03-26T05:24:54.306977+0000","last_fullsized":"2024-03-26T05:24:54.306977+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'33","last_scrub_stamp":"2024-03-26T05:24:19.574229+0000","last_deep_scrub":"498'33","last_deep_scrub_stamp":"2024-03-26T05:23:07.541360+0000","last_clean_scrub_stamp":"2024-03-26T05:24:19.574229+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:21.097430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1077,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306274+0000","last_change":"2024-03-26T05:24:31.549896+0000","last_active":"2024-03-26T05:24:54.306274+0000","last_peered":"2024-03-26T05:24:54.306274+0000","last_clean":"2024-03-26T05:24:54.306274+0000","last_became_active":"2024-03-26T05:00:21.763221+0000","last_became_peered":"2024-03-26T05:00:21.763221+0000","last_unstale":"2024-03-26T05:24:54.306274+0000","last_undegraded":"2024-03-26T05:24:54.306274+0000","last_fullsized":"2024-03-26T05:24:54.306274+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:31.549805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:31.549805+0000","last_clean_scrub_stamp":"2024-03-26T05:24:31.549805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:42.611302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092137600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1058,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613357+0000","last_change":"2024-03-26T05:24:33.081578+0000","last_active":"2024-03-26T05:24:54.613357+0000","last_peered":"2024-03-26T05:24:54.613357+0000","last_clean":"2024-03-26T05:24:54.613357+0000","last_became_active":"2024-03-26T05:00:23.346493+0000","last_became_peered":"2024-03-26T05:00:23.346493+0000","last_unstale":"2024-03-26T05:24:54.613357+0000","last_undegraded":"2024-03-26T05:24:54.613357+0000","last_fullsized":"2024-03-26T05:24:54.613357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:33.081489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:33.081489+0000","last_clean_scrub_stamp":"2024-03-26T05:24:33.081489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:51.071265+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":919,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628108+0000","last_change":"2024-03-26T05:24:19.621770+0000","last_active":"2024-03-26T05:24:54.628108+0000","last_peered":"2024-03-26T05:24:54.628108+0000","last_clean":"2024-03-26T05:24:54.628108+0000","last_became_active":"2024-03-26T05:15:32.633633+0000","last_became_peered":"2024-03-26T05:15:32.633633+0000","last_unstale":"2024-03-26T05:24:54.628108+0000","last_undegraded":"2024-03-26T05:24:54.628108+0000","last_fullsized":"2024-03-26T05:24:54.628108+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:19.621638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:07.434449+0000","last_clean_scrub_stamp":"2024-03-26T05:24:19.621638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:28.971628+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"88'128","reported_seq":1230,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627337+0000","last_change":"2024-03-26T05:23:56.568904+0000","last_active":"2024-03-26T05:24:54.627337+0000","last_peered":"2024-03-26T05:24:54.627337+0000","last_clean":"2024-03-26T05:24:54.627337+0000","last_became_active":"2024-03-26T05:23:26.800521+0000","last_became_peered":"2024-03-26T05:23:26.800521+0000","last_unstale":"2024-03-26T05:24:54.627337+0000","last_undegraded":"2024-03-26T05:24:54.627337+0000","last_fullsized":"2024-03-26T05:24:54.627337+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:23:56.568770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:58:07.465271+0000","last_clean_scrub_stamp":"2024-03-26T05:23:56.568770+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:01.457115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1059,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:06.069108+0000","last_change":"2024-03-26T05:25:06.069127+0000","last_active":"2024-03-26T05:25:06.069108+0000","last_peered":"2024-03-26T05:25:06.069108+0000","last_clean":"2024-03-26T05:25:06.069108+0000","last_became_active":"2024-03-26T05:23:23.942931+0000","last_became_peered":"2024-03-26T05:23:23.942931+0000","last_unstale":"2024-03-26T05:25:06.069108+0000","last_undegraded":"2024-03-26T05:25:06.069108+0000","last_fullsized":"2024-03-26T05:25:06.069108+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:25:06.069049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:25:06.069049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:12.072428+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"542'69","reported_seq":1468,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704439+0000","last_change":"2024-03-26T05:24:48.226927+0000","last_active":"2024-03-26T05:24:58.704439+0000","last_peered":"2024-03-26T05:24:58.704439+0000","last_clean":"2024-03-26T05:24:58.704439+0000","last_became_active":"2024-03-26T05:00:24.938513+0000","last_became_peered":"2024-03-26T05:00:24.938513+0000","last_unstale":"2024-03-26T05:24:58.704439+0000","last_undegraded":"2024-03-26T05:24:58.704439+0000","last_fullsized":"2024-03-26T05:24:58.704439+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"542'69","last_scrub_stamp":"2024-03-26T05:24:48.226785+0000","last_deep_scrub":"542'69","last_deep_scrub_stamp":"2024-03-26T05:24:48.226785+0000","last_clean_scrub_stamp":"2024-03-26T05:24:48.226785+0000","objects_scrubbed":11,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:51.157070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866570+0000","last_change":"2024-03-26T05:24:00.547165+0000","last_active":"2024-03-26T05:24:53.866570+0000","last_peered":"2024-03-26T05:24:53.866570+0000","last_clean":"2024-03-26T05:24:53.866570+0000","last_became_active":"2024-03-26T05:23:23.934085+0000","last_became_peered":"2024-03-26T05:23:23.934085+0000","last_unstale":"2024-03-26T05:24:53.866570+0000","last_undegraded":"2024-03-26T05:24:53.866570+0000","last_fullsized":"2024-03-26T05:24:53.866570+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:00.547062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:00.547062+0000","last_clean_scrub_stamp":"2024-03-26T05:24:00.547062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:28.867272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"535'3","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114296+0000","last_change":"2024-03-26T05:24:18.432690+0000","last_active":"2024-03-26T05:24:55.114296+0000","last_peered":"2024-03-26T05:24:55.114296+0000","last_clean":"2024-03-26T05:24:55.114296+0000","last_became_active":"2024-03-26T05:23:26.810556+0000","last_became_peered":"2024-03-26T05:23:26.810556+0000","last_unstale":"2024-03-26T05:24:55.114296+0000","last_undegraded":"2024-03-26T05:24:55.114296+0000","last_fullsized":"2024-03-26T05:24:55.114296+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'3","last_scrub_stamp":"2024-03-26T05:24:18.432581+0000","last_deep_scrub":"535'3","last_deep_scrub_stamp":"2024-03-26T05:24:18.432581+0000","last_clean_scrub_stamp":"2024-03-26T05:24:18.432581+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:22.693785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080961800000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1029,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628902+0000","last_change":"2024-03-26T05:24:38.852984+0000","last_active":"2024-03-26T05:24:54.628902+0000","last_peered":"2024-03-26T05:24:54.628902+0000","last_clean":"2024-03-26T05:24:54.628902+0000","last_became_active":"2024-03-26T05:23:23.935102+0000","last_became_peered":"2024-03-26T05:23:23.935102+0000","last_unstale":"2024-03-26T05:24:54.628902+0000","last_undegraded":"2024-03-26T05:24:54.628902+0000","last_fullsized":"2024-03-26T05:24:54.628902+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:38.852902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:38.852902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:39.795769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"88'128","reported_seq":634,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:02.596706+0000","last_change":"2024-03-26T05:25:02.596736+0000","last_active":"2024-03-26T05:25:02.596706+0000","last_peered":"2024-03-26T05:25:02.596706+0000","last_clean":"2024-03-26T05:25:02.596706+0000","last_became_active":"2024-03-26T05:23:29.788305+0000","last_became_peered":"2024-03-26T05:23:29.788305+0000","last_unstale":"2024-03-26T05:25:02.596706+0000","last_undegraded":"2024-03-26T05:25:02.596706+0000","last_fullsized":"2024-03-26T05:25:02.596706+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":324,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"88'128","last_scrub_stamp":"2024-03-26T05:25:02.596620+0000","last_deep_scrub":"88'128","last_deep_scrub_stamp":"2024-03-26T05:25:02.596620+0000","last_clean_scrub_stamp":"2024-03-26T05:25:02.596620+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:16.110610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"443'27","reported_seq":1153,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863984+0000","last_change":"2024-03-26T05:23:59.846869+0000","last_active":"2024-03-26T05:24:53.863984+0000","last_peered":"2024-03-26T05:24:53.863984+0000","last_clean":"2024-03-26T05:24:53.863984+0000","last_became_active":"2024-03-26T05:23:23.943394+0000","last_became_peered":"2024-03-26T05:23:23.943394+0000","last_unstale":"2024-03-26T05:24:53.863984+0000","last_undegraded":"2024-03-26T05:24:53.863984+0000","last_fullsized":"2024-03-26T05:24:53.863984+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'27","last_scrub_stamp":"2024-03-26T05:23:59.846748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:23:59.846748+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:15.920009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114678+0000","last_change":"2024-03-26T05:23:44.371115+0000","last_active":"2024-03-26T05:24:55.114678+0000","last_peered":"2024-03-26T05:24:55.114678+0000","last_clean":"2024-03-26T05:24:55.114678+0000","last_became_active":"2024-03-26T05:23:26.805885+0000","last_became_peered":"2024-03-26T05:23:26.805885+0000","last_unstale":"2024-03-26T05:24:55.114678+0000","last_undegraded":"2024-03-26T05:24:55.114678+0000","last_fullsized":"2024-03-26T05:24:55.114678+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:44.371020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:44.371020+0000","last_clean_scrub_stamp":"2024-03-26T05:23:44.371020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:11.566186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1067,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306528+0000","last_change":"2024-03-26T05:24:43.546754+0000","last_active":"2024-03-26T05:24:54.306528+0000","last_peered":"2024-03-26T05:24:54.306528+0000","last_clean":"2024-03-26T05:24:54.306528+0000","last_became_active":"2024-03-26T05:23:23.947964+0000","last_became_peered":"2024-03-26T05:23:23.947964+0000","last_unstale":"2024-03-26T05:24:54.306528+0000","last_undegraded":"2024-03-26T05:24:54.306528+0000","last_fullsized":"2024-03-26T05:24:54.306528+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:43.546668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:43.546668+0000","last_clean_scrub_stamp":"2024-03-26T05:24:43.546668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:12.718060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1041,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863401+0000","last_change":"2024-03-26T05:24:23.073367+0000","last_active":"2024-03-26T05:24:53.863401+0000","last_peered":"2024-03-26T05:24:53.863401+0000","last_clean":"2024-03-26T05:24:53.863401+0000","last_became_active":"2024-03-26T05:15:32.639943+0000","last_became_peered":"2024-03-26T05:15:32.639943+0000","last_unstale":"2024-03-26T05:24:53.863401+0000","last_undegraded":"2024-03-26T05:24:53.863401+0000","last_fullsized":"2024-03-26T05:24:53.863401+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:23.073227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:23.073227+0000","last_clean_scrub_stamp":"2024-03-26T05:24:23.073227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:46.652413+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"443'67","reported_seq":1172,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.945771+0000","last_change":"2024-03-26T05:24:50.883481+0000","last_active":"2024-03-26T05:24:54.945771+0000","last_peered":"2024-03-26T05:24:54.945771+0000","last_clean":"2024-03-26T05:24:54.945771+0000","last_became_active":"2024-03-26T05:15:32.626640+0000","last_became_peered":"2024-03-26T05:15:32.626640+0000","last_unstale":"2024-03-26T05:24:54.945771+0000","last_undegraded":"2024-03-26T05:24:54.945771+0000","last_fullsized":"2024-03-26T05:24:54.945771+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'67","last_scrub_stamp":"2024-03-26T05:24:50.883350+0000","last_deep_scrub":"443'67","last_deep_scrub_stamp":"2024-03-26T05:24:50.883350+0000","last_clean_scrub_stamp":"2024-03-26T05:24:50.883350+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:56.268769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":952,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628455+0000","last_change":"2024-03-26T05:24:41.666156+0000","last_active":"2024-03-26T05:24:54.628455+0000","last_peered":"2024-03-26T05:24:54.628455+0000","last_clean":"2024-03-26T05:24:54.628455+0000","last_became_active":"2024-03-26T05:23:26.798089+0000","last_became_peered":"2024-03-26T05:23:26.798089+0000","last_unstale":"2024-03-26T05:24:54.628455+0000","last_undegraded":"2024-03-26T05:24:54.628455+0000","last_fullsized":"2024-03-26T05:24:54.628455+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:41.666052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:46.539478+0000","last_clean_scrub_stamp":"2024-03-26T05:24:41.666052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:11.363597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1083,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306654+0000","last_change":"2024-03-26T05:24:42.548874+0000","last_active":"2024-03-26T05:24:54.306654+0000","last_peered":"2024-03-26T05:24:54.306654+0000","last_clean":"2024-03-26T05:24:54.306654+0000","last_became_active":"2024-03-26T05:23:26.799302+0000","last_became_peered":"2024-03-26T05:23:26.799302+0000","last_unstale":"2024-03-26T05:24:54.306654+0000","last_undegraded":"2024-03-26T05:24:54.306654+0000","last_fullsized":"2024-03-26T05:24:54.306654+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:42.548771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:42.548771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:01.372124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":951,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627078+0000","last_change":"2024-03-26T05:24:29.649735+0000","last_active":"2024-03-26T05:24:54.627078+0000","last_peered":"2024-03-26T05:24:54.627078+0000","last_clean":"2024-03-26T05:24:54.627078+0000","last_became_active":"2024-03-26T05:15:32.633481+0000","last_became_peered":"2024-03-26T05:15:32.633481+0000","last_unstale":"2024-03-26T05:24:54.627078+0000","last_undegraded":"2024-03-26T05:24:54.627078+0000","last_fullsized":"2024-03-26T05:24:54.627078+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:29.649637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:29.649637+0000","last_clean_scrub_stamp":"2024-03-26T05:24:29.649637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:30.844423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1060,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306297+0000","last_change":"2024-03-26T05:24:49.385957+0000","last_active":"2024-03-26T05:24:54.306297+0000","last_peered":"2024-03-26T05:24:54.306297+0000","last_clean":"2024-03-26T05:24:54.306297+0000","last_became_active":"2024-03-26T05:00:26.960656+0000","last_became_peered":"2024-03-26T05:00:26.960656+0000","last_unstale":"2024-03-26T05:24:54.306297+0000","last_undegraded":"2024-03-26T05:24:54.306297+0000","last_fullsized":"2024-03-26T05:24:54.306297+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:49.385874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:52.271117+0000","last_clean_scrub_stamp":"2024-03-26T05:24:49.385874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:05.350862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"443'43","reported_seq":113,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866147+0000","last_change":"2024-03-26T05:23:56.461342+0000","last_active":"2024-03-26T05:24:53.866147+0000","last_peered":"2024-03-26T05:24:53.866147+0000","last_clean":"2024-03-26T05:24:53.866147+0000","last_became_active":"2024-03-26T05:23:23.921513+0000","last_became_peered":"2024-03-26T05:23:23.921513+0000","last_unstale":"2024-03-26T05:24:53.866147+0000","last_undegraded":"2024-03-26T05:24:53.866147+0000","last_fullsized":"2024-03-26T05:24:53.866147+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'43","last_scrub_stamp":"2024-03-26T05:23:56.461189+0000","last_deep_scrub":"443'43","last_deep_scrub_stamp":"2024-03-26T05:23:56.461189+0000","last_clean_scrub_stamp":"2024-03-26T05:23:56.461189+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:14.572658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":975,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628378+0000","last_change":"2024-03-26T05:24:27.651958+0000","last_active":"2024-03-26T05:24:54.628378+0000","last_peered":"2024-03-26T05:24:54.628378+0000","last_clean":"2024-03-26T05:24:54.628378+0000","last_became_active":"2024-03-26T05:23:23.923551+0000","last_became_peered":"2024-03-26T05:23:23.923551+0000","last_unstale":"2024-03-26T05:24:54.628378+0000","last_undegraded":"2024-03-26T05:24:54.628378+0000","last_fullsized":"2024-03-26T05:24:54.628378+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:27.651844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:24:27.651844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:38.146689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1035,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.072770+0000","last_change":"2024-03-26T05:24:55.072793+0000","last_active":"2024-03-26T05:24:55.072770+0000","last_peered":"2024-03-26T05:24:55.072770+0000","last_clean":"2024-03-26T05:24:55.072770+0000","last_became_active":"2024-03-26T05:20:15.376729+0000","last_became_peered":"2024-03-26T05:20:15.376729+0000","last_unstale":"2024-03-26T05:24:55.072770+0000","last_undegraded":"2024-03-26T05:24:55.072770+0000","last_fullsized":"2024-03-26T05:24:55.072770+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:55.072657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:55.072657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:58.783140+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"504'13","reported_seq":1227,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704603+0000","last_change":"2024-03-26T05:24:14.563653+0000","last_active":"2024-03-26T05:24:58.704603+0000","last_peered":"2024-03-26T05:24:58.704603+0000","last_clean":"2024-03-26T05:24:58.704603+0000","last_became_active":"2024-03-26T05:23:23.927485+0000","last_became_peered":"2024-03-26T05:23:23.927485+0000","last_unstale":"2024-03-26T05:24:58.704603+0000","last_undegraded":"2024-03-26T05:24:58.704603+0000","last_fullsized":"2024-03-26T05:24:58.704603+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'13","last_scrub_stamp":"2024-03-26T05:24:14.563317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:14.563317+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:30.281483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"443'60","reported_seq":120,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114001+0000","last_change":"2024-03-26T05:24:50.472093+0000","last_active":"2024-03-26T05:24:55.114001+0000","last_peered":"2024-03-26T05:24:55.114001+0000","last_clean":"2024-03-26T05:24:55.114001+0000","last_became_active":"2024-03-26T05:23:26.807667+0000","last_became_peered":"2024-03-26T05:23:26.807667+0000","last_unstale":"2024-03-26T05:24:55.114001+0000","last_undegraded":"2024-03-26T05:24:55.114001+0000","last_fullsized":"2024-03-26T05:24:55.114001+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'60","last_scrub_stamp":"2024-03-26T05:24:50.471950+0000","last_deep_scrub":"243'36","last_deep_scrub_stamp":"2024-03-26T05:09:59.155894+0000","last_clean_scrub_stamp":"2024-03-26T05:24:50.471950+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:19.488139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"153'6","reported_seq":974,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628451+0000","last_change":"2024-03-26T05:24:07.574811+0000","last_active":"2024-03-26T05:24:54.628451+0000","last_peered":"2024-03-26T05:24:54.628451+0000","last_clean":"2024-03-26T05:24:54.628451+0000","last_became_active":"2024-03-26T05:23:23.920284+0000","last_became_peered":"2024-03-26T05:23:23.920284+0000","last_unstale":"2024-03-26T05:24:54.628451+0000","last_undegraded":"2024-03-26T05:24:54.628451+0000","last_fullsized":"2024-03-26T05:24:54.628451+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'6","last_scrub_stamp":"2024-03-26T05:24:07.574701+0000","last_deep_scrub":"153'6","last_deep_scrub_stamp":"2024-03-26T05:11:29.959120+0000","last_clean_scrub_stamp":"2024-03-26T05:24:07.574701+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:31.102020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"94'2","reported_seq":96,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114802+0000","last_change":"2024-03-26T05:24:06.352202+0000","last_active":"2024-03-26T05:24:55.114802+0000","last_peered":"2024-03-26T05:24:55.114802+0000","last_clean":"2024-03-26T05:24:55.114802+0000","last_became_active":"2024-03-26T05:23:26.809224+0000","last_became_peered":"2024-03-26T05:23:26.809224+0000","last_unstale":"2024-03-26T05:24:55.114802+0000","last_undegraded":"2024-03-26T05:24:55.114802+0000","last_fullsized":"2024-03-26T05:24:55.114802+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-03-26T05:24:06.352098+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-03-26T05:01:36.945708+0000","last_clean_scrub_stamp":"2024-03-26T05:24:06.352098+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:23.245143+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"542'28","reported_seq":1080,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.629130+0000","last_change":"2024-03-26T05:24:36.788404+0000","last_active":"2024-03-26T05:24:54.629130+0000","last_peered":"2024-03-26T05:24:54.629130+0000","last_clean":"2024-03-26T05:24:54.629130+0000","last_became_active":"2024-03-26T05:15:32.640871+0000","last_became_peered":"2024-03-26T05:15:32.640871+0000","last_unstale":"2024-03-26T05:24:54.629130+0000","last_undegraded":"2024-03-26T05:24:54.629130+0000","last_fullsized":"2024-03-26T05:24:54.629130+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"542'28","last_scrub_stamp":"2024-03-26T05:24:36.788275+0000","last_deep_scrub":"542'28","last_deep_scrub_stamp":"2024-03-26T05:24:36.788275+0000","last_clean_scrub_stamp":"2024-03-26T05:24:36.788275+0000","objects_scrubbed":4,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:44.656721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":32,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"188'2","reported_seq":1128,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.945050+0000","last_change":"2024-03-26T05:24:51.876501+0000","last_active":"2024-03-26T05:24:54.945050+0000","last_peered":"2024-03-26T05:24:54.945050+0000","last_clean":"2024-03-26T05:24:54.945050+0000","last_became_active":"2024-03-26T05:23:26.814453+0000","last_became_peered":"2024-03-26T05:23:26.814453+0000","last_unstale":"2024-03-26T05:24:54.945050+0000","last_undegraded":"2024-03-26T05:24:54.945050+0000","last_fullsized":"2024-03-26T05:24:54.945050+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'2","last_scrub_stamp":"2024-03-26T05:24:51.876276+0000","last_deep_scrub":"188'2","last_deep_scrub_stamp":"2024-03-26T05:17:50.147911+0000","last_clean_scrub_stamp":"2024-03-26T05:24:51.876276+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:11.333124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114426+0000","last_change":"2024-03-26T05:24:51.503660+0000","last_active":"2024-03-26T05:24:55.114426+0000","last_peered":"2024-03-26T05:24:55.114426+0000","last_clean":"2024-03-26T05:24:55.114426+0000","last_became_active":"2024-03-26T05:23:26.800666+0000","last_became_peered":"2024-03-26T05:23:26.800666+0000","last_unstale":"2024-03-26T05:24:55.114426+0000","last_undegraded":"2024-03-26T05:24:55.114426+0000","last_fullsized":"2024-03-26T05:24:55.114426+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:51.503539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:51.503539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.057042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":956,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628204+0000","last_change":"2024-03-26T05:23:58.627233+0000","last_active":"2024-03-26T05:24:54.628204+0000","last_peered":"2024-03-26T05:24:54.628204+0000","last_clean":"2024-03-26T05:24:54.628204+0000","last_became_active":"2024-03-26T05:23:23.925613+0000","last_became_peered":"2024-03-26T05:23:23.925613+0000","last_unstale":"2024-03-26T05:24:54.628204+0000","last_undegraded":"2024-03-26T05:24:54.628204+0000","last_fullsized":"2024-03-26T05:24:54.628204+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:58.627125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:23:58.627125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:10.873968+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"443'39","reported_seq":1120,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863888+0000","last_change":"2024-03-26T05:24:18.886779+0000","last_active":"2024-03-26T05:24:53.863888+0000","last_peered":"2024-03-26T05:24:53.863888+0000","last_clean":"2024-03-26T05:24:53.863888+0000","last_became_active":"2024-03-26T05:15:32.634434+0000","last_became_peered":"2024-03-26T05:15:32.634434+0000","last_unstale":"2024-03-26T05:24:53.863888+0000","last_undegraded":"2024-03-26T05:24:53.863888+0000","last_fullsized":"2024-03-26T05:24:53.863888+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'39","last_scrub_stamp":"2024-03-26T05:24:18.886672+0000","last_deep_scrub":"443'39","last_deep_scrub_stamp":"2024-03-26T05:24:18.886672+0000","last_clean_scrub_stamp":"2024-03-26T05:24:18.886672+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:41.880187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":925,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627523+0000","last_change":"2024-03-26T05:23:57.572827+0000","last_active":"2024-03-26T05:24:54.627523+0000","last_peered":"2024-03-26T05:24:54.627523+0000","last_clean":"2024-03-26T05:24:54.627523+0000","last_became_active":"2024-03-26T05:23:23.925756+0000","last_became_peered":"2024-03-26T05:23:23.925756+0000","last_unstale":"2024-03-26T05:24:54.627523+0000","last_undegraded":"2024-03-26T05:24:54.627523+0000","last_fullsized":"2024-03-26T05:24:54.627523+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:57.572747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:23:57.572747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:01.874256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1025,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.629034+0000","last_change":"2024-03-26T05:23:18.608175+0000","last_active":"2024-03-26T05:24:54.629034+0000","last_peered":"2024-03-26T05:24:54.629034+0000","last_clean":"2024-03-26T05:24:54.629034+0000","last_became_active":"2024-03-26T05:15:32.641031+0000","last_became_peered":"2024-03-26T05:15:32.641031+0000","last_unstale":"2024-03-26T05:24:54.629034+0000","last_undegraded":"2024-03-26T05:24:54.629034+0000","last_fullsized":"2024-03-26T05:24:54.629034+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:18.608076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:18.010011+0000","last_clean_scrub_stamp":"2024-03-26T05:23:18.608076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1063,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306900+0000","last_change":"2024-03-26T05:24:48.425579+0000","last_active":"2024-03-26T05:24:54.306900+0000","last_peered":"2024-03-26T05:24:54.306900+0000","last_clean":"2024-03-26T05:24:54.306900+0000","last_became_active":"2024-03-26T05:23:23.945259+0000","last_became_peered":"2024-03-26T05:23:23.945259+0000","last_unstale":"2024-03-26T05:24:54.306900+0000","last_undegraded":"2024-03-26T05:24:54.306900+0000","last_fullsized":"2024-03-26T05:24:54.306900+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:48.425489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:48.425489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:01.965172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"443'32","reported_seq":112,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:04.536324+0000","last_change":"2024-03-26T05:25:04.536358+0000","last_active":"2024-03-26T05:25:04.536324+0000","last_peered":"2024-03-26T05:25:04.536324+0000","last_clean":"2024-03-26T05:25:04.536324+0000","last_became_active":"2024-03-26T05:23:26.798031+0000","last_became_peered":"2024-03-26T05:23:26.798031+0000","last_unstale":"2024-03-26T05:25:04.536324+0000","last_undegraded":"2024-03-26T05:25:04.536324+0000","last_fullsized":"2024-03-26T05:25:04.536324+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'32","last_scrub_stamp":"2024-03-26T05:25:04.536248+0000","last_deep_scrub":"98'20","last_deep_scrub_stamp":"2024-03-26T05:04:12.659322+0000","last_clean_scrub_stamp":"2024-03-26T05:25:04.536248+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:22.353963+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1060,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863470+0000","last_change":"2024-03-26T05:24:28.910622+0000","last_active":"2024-03-26T05:24:53.863470+0000","last_peered":"2024-03-26T05:24:53.863470+0000","last_clean":"2024-03-26T05:24:53.863470+0000","last_became_active":"2024-03-26T05:15:32.632387+0000","last_became_peered":"2024-03-26T05:15:32.632387+0000","last_unstale":"2024-03-26T05:24:53.863470+0000","last_undegraded":"2024-03-26T05:24:53.863470+0000","last_fullsized":"2024-03-26T05:24:53.863470+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:28.910532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:45.352678+0000","last_clean_scrub_stamp":"2024-03-26T05:24:28.910532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:57.527714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"336'11","reported_seq":1344,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704446+0000","last_change":"2024-03-26T05:24:23.840064+0000","last_active":"2024-03-26T05:24:58.704446+0000","last_peered":"2024-03-26T05:24:58.704446+0000","last_clean":"2024-03-26T05:24:58.704446+0000","last_became_active":"2024-03-26T05:15:32.636392+0000","last_became_peered":"2024-03-26T05:15:32.636392+0000","last_unstale":"2024-03-26T05:24:58.704446+0000","last_undegraded":"2024-03-26T05:24:58.704446+0000","last_fullsized":"2024-03-26T05:24:58.704446+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-03-26T05:24:23.839981+0000","last_deep_scrub":"120'9","last_deep_scrub_stamp":"2024-03-26T05:03:20.186818+0000","last_clean_scrub_stamp":"2024-03-26T05:24:23.839981+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:43.739324+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":111,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:59.794453+0000","last_change":"2024-03-26T05:24:59.794477+0000","last_active":"2024-03-26T05:24:59.794453+0000","last_peered":"2024-03-26T05:24:59.794453+0000","last_clean":"2024-03-26T05:24:59.794453+0000","last_became_active":"2024-03-26T05:23:26.814104+0000","last_became_peered":"2024-03-26T05:23:26.814104+0000","last_unstale":"2024-03-26T05:24:59.794453+0000","last_undegraded":"2024-03-26T05:24:59.794453+0000","last_fullsized":"2024-03-26T05:24:59.794453+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:59.794371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:59.794371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:25.337751+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"443'49","reported_seq":1121,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628941+0000","last_change":"2024-03-26T05:24:00.812655+0000","last_active":"2024-03-26T05:24:54.628941+0000","last_peered":"2024-03-26T05:24:54.628941+0000","last_clean":"2024-03-26T05:24:54.628941+0000","last_became_active":"2024-03-26T05:15:32.626448+0000","last_became_peered":"2024-03-26T05:15:32.626448+0000","last_unstale":"2024-03-26T05:24:54.628941+0000","last_undegraded":"2024-03-26T05:24:54.628941+0000","last_fullsized":"2024-03-26T05:24:54.628941+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'49","last_scrub_stamp":"2024-03-26T05:24:00.812557+0000","last_deep_scrub":"345'39","last_deep_scrub_stamp":"2024-03-26T05:15:56.674423+0000","last_clean_scrub_stamp":"2024-03-26T05:24:00.812557+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:06.645746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":959,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627933+0000","last_change":"2024-03-26T05:24:00.584116+0000","last_active":"2024-03-26T05:24:54.627933+0000","last_peered":"2024-03-26T05:24:54.627933+0000","last_clean":"2024-03-26T05:24:54.627933+0000","last_became_active":"2024-03-26T05:23:23.924133+0000","last_became_peered":"2024-03-26T05:23:23.924133+0000","last_unstale":"2024-03-26T05:24:54.627933+0000","last_undegraded":"2024-03-26T05:24:54.627933+0000","last_fullsized":"2024-03-26T05:24:54.627933+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:00.584025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:24:00.584025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:28.716607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"508'12","reported_seq":116,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.704700+0000","last_change":"2024-03-26T05:24:17.405498+0000","last_active":"2024-03-26T05:24:58.704700+0000","last_peered":"2024-03-26T05:24:58.704700+0000","last_clean":"2024-03-26T05:24:58.704700+0000","last_became_active":"2024-03-26T05:23:26.806234+0000","last_became_peered":"2024-03-26T05:23:26.806234+0000","last_unstale":"2024-03-26T05:24:58.704700+0000","last_undegraded":"2024-03-26T05:24:58.704700+0000","last_fullsized":"2024-03-26T05:24:58.704700+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'12","last_scrub_stamp":"2024-03-26T05:24:17.405360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:17.405360+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:29.961077+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1048,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863661+0000","last_change":"2024-03-26T05:24:46.144452+0000","last_active":"2024-03-26T05:24:53.863661+0000","last_peered":"2024-03-26T05:24:53.863661+0000","last_clean":"2024-03-26T05:24:53.863661+0000","last_became_active":"2024-03-26T05:15:32.631444+0000","last_became_peered":"2024-03-26T05:15:32.631444+0000","last_unstale":"2024-03-26T05:24:53.863661+0000","last_undegraded":"2024-03-26T05:24:53.863661+0000","last_fullsized":"2024-03-26T05:24:53.863661+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:46.144355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:17:59.508745+0000","last_clean_scrub_stamp":"2024-03-26T05:24:46.144355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:46.492028+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"443'32","reported_seq":109,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866578+0000","last_change":"2024-03-26T05:24:08.668366+0000","last_active":"2024-03-26T05:24:53.866578+0000","last_peered":"2024-03-26T05:24:53.866578+0000","last_clean":"2024-03-26T05:24:53.866578+0000","last_became_active":"2024-03-26T05:23:23.931656+0000","last_became_peered":"2024-03-26T05:23:23.931656+0000","last_unstale":"2024-03-26T05:24:53.866578+0000","last_undegraded":"2024-03-26T05:24:53.866578+0000","last_fullsized":"2024-03-26T05:24:53.866578+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'32","last_scrub_stamp":"2024-03-26T05:24:08.668274+0000","last_deep_scrub":"95'16","last_deep_scrub_stamp":"2024-03-26T05:01:40.462271+0000","last_clean_scrub_stamp":"2024-03-26T05:24:08.668274+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:22.481085+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":105,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:01.681192+0000","last_change":"2024-03-26T05:25:01.681239+0000","last_active":"2024-03-26T05:25:01.681192+0000","last_peered":"2024-03-26T05:25:01.681192+0000","last_clean":"2024-03-26T05:25:01.681192+0000","last_became_active":"2024-03-26T05:23:23.931645+0000","last_became_peered":"2024-03-26T05:23:23.931645+0000","last_unstale":"2024-03-26T05:25:01.681192+0000","last_undegraded":"2024-03-26T05:25:01.681192+0000","last_fullsized":"2024-03-26T05:25:01.681192+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:25:01.681095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:25:01.681095+0000","last_clean_scrub_stamp":"2024-03-26T05:25:01.681095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:23.546578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1080,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613567+0000","last_change":"2024-03-26T05:23:39.196051+0000","last_active":"2024-03-26T05:24:54.613567+0000","last_peered":"2024-03-26T05:24:54.613567+0000","last_clean":"2024-03-26T05:24:54.613567+0000","last_became_active":"2024-03-26T05:23:26.813536+0000","last_became_peered":"2024-03-26T05:23:26.813536+0000","last_unstale":"2024-03-26T05:24:54.613567+0000","last_undegraded":"2024-03-26T05:24:54.613567+0000","last_fullsized":"2024-03-26T05:24:54.613567+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:39.195947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:55.496800+0000","last_clean_scrub_stamp":"2024-03-26T05:23:39.195947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:02.584927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1044,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628103+0000","last_change":"2024-03-26T05:23:37.641257+0000","last_active":"2024-03-26T05:24:54.628103+0000","last_peered":"2024-03-26T05:24:54.628103+0000","last_clean":"2024-03-26T05:24:54.628103+0000","last_became_active":"2024-03-26T05:23:26.802281+0000","last_became_peered":"2024-03-26T05:23:26.802281+0000","last_unstale":"2024-03-26T05:24:54.628103+0000","last_undegraded":"2024-03-26T05:24:54.628103+0000","last_fullsized":"2024-03-26T05:24:54.628103+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:37.641159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:02:16.072965+0000","last_clean_scrub_stamp":"2024-03-26T05:23:37.641159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"443'21","reported_seq":1086,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613655+0000","last_change":"2024-03-26T05:23:46.347900+0000","last_active":"2024-03-26T05:24:54.613655+0000","last_peered":"2024-03-26T05:24:54.613655+0000","last_clean":"2024-03-26T05:24:54.613655+0000","last_became_active":"2024-03-26T05:00:24.939288+0000","last_became_peered":"2024-03-26T05:00:24.939288+0000","last_unstale":"2024-03-26T05:24:54.613655+0000","last_undegraded":"2024-03-26T05:24:54.613655+0000","last_fullsized":"2024-03-26T05:24:54.613655+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'21","last_scrub_stamp":"2024-03-26T05:23:46.347776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:23:46.347776+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:03.079113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":930,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628241+0000","last_change":"2024-03-26T05:24:13.545357+0000","last_active":"2024-03-26T05:24:54.628241+0000","last_peered":"2024-03-26T05:24:54.628241+0000","last_clean":"2024-03-26T05:24:54.628241+0000","last_became_active":"2024-03-26T05:23:23.930076+0000","last_became_peered":"2024-03-26T05:23:23.930076+0000","last_unstale":"2024-03-26T05:24:54.628241+0000","last_undegraded":"2024-03-26T05:24:54.628241+0000","last_fullsized":"2024-03-26T05:24:54.628241+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:13.545257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.764742+0000","last_clean_scrub_stamp":"2024-03-26T05:24:13.545257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:42.876092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1032,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628355+0000","last_change":"2024-03-26T05:24:16.861274+0000","last_active":"2024-03-26T05:24:54.628355+0000","last_peered":"2024-03-26T05:24:54.628355+0000","last_clean":"2024-03-26T05:24:54.628355+0000","last_became_active":"2024-03-26T05:00:26.960354+0000","last_became_peered":"2024-03-26T05:00:26.960354+0000","last_unstale":"2024-03-26T05:24:54.628355+0000","last_undegraded":"2024-03-26T05:24:54.628355+0000","last_fullsized":"2024-03-26T05:24:54.628355+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:16.861192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:04.767790+0000","last_clean_scrub_stamp":"2024-03-26T05:24:16.861192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:35.838026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1071,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863121+0000","last_change":"2024-03-26T05:23:46.864136+0000","last_active":"2024-03-26T05:24:53.863121+0000","last_peered":"2024-03-26T05:24:53.863121+0000","last_clean":"2024-03-26T05:24:53.863121+0000","last_became_active":"2024-03-26T05:23:26.809570+0000","last_became_peered":"2024-03-26T05:23:26.809570+0000","last_unstale":"2024-03-26T05:24:53.863121+0000","last_undegraded":"2024-03-26T05:24:53.863121+0000","last_fullsized":"2024-03-26T05:24:53.863121+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:46.864038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:23:46.864038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:01.818874+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"443'33","reported_seq":1092,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628368+0000","last_change":"2024-03-26T05:24:20.888480+0000","last_active":"2024-03-26T05:24:54.628368+0000","last_peered":"2024-03-26T05:24:54.628368+0000","last_clean":"2024-03-26T05:24:54.628368+0000","last_became_active":"2024-03-26T05:15:32.636359+0000","last_became_peered":"2024-03-26T05:15:32.636359+0000","last_unstale":"2024-03-26T05:24:54.628368+0000","last_undegraded":"2024-03-26T05:24:54.628368+0000","last_fullsized":"2024-03-26T05:24:54.628368+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'33","last_scrub_stamp":"2024-03-26T05:24:20.888373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:20.888373+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:48.499006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":100,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:05.509193+0000","last_change":"2024-03-26T05:25:05.509240+0000","last_active":"2024-03-26T05:25:05.509193+0000","last_peered":"2024-03-26T05:25:05.509193+0000","last_clean":"2024-03-26T05:25:05.509193+0000","last_became_active":"2024-03-26T05:23:26.801339+0000","last_became_peered":"2024-03-26T05:23:26.801339+0000","last_unstale":"2024-03-26T05:25:05.509193+0000","last_undegraded":"2024-03-26T05:25:05.509193+0000","last_fullsized":"2024-03-26T05:25:05.509193+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:25:05.509098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:50.406745+0000","last_clean_scrub_stamp":"2024-03-26T05:25:05.509098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:33.584124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":968,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627411+0000","last_change":"2024-03-26T05:24:12.573345+0000","last_active":"2024-03-26T05:24:54.627411+0000","last_peered":"2024-03-26T05:24:54.627411+0000","last_clean":"2024-03-26T05:24:54.627411+0000","last_became_active":"2024-03-26T05:23:23.923115+0000","last_became_peered":"2024-03-26T05:23:23.923115+0000","last_unstale":"2024-03-26T05:24:54.627411+0000","last_undegraded":"2024-03-26T05:24:54.627411+0000","last_fullsized":"2024-03-26T05:24:54.627411+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:12.573245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:03:06.806863+0000","last_clean_scrub_stamp":"2024-03-26T05:24:12.573245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:19.220471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1036,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628606+0000","last_change":"2024-03-26T05:24:37.960766+0000","last_active":"2024-03-26T05:24:54.628606+0000","last_peered":"2024-03-26T05:24:54.628606+0000","last_clean":"2024-03-26T05:24:54.628606+0000","last_became_active":"2024-03-26T05:00:29.461176+0000","last_became_peered":"2024-03-26T05:00:29.461176+0000","last_unstale":"2024-03-26T05:24:54.628606+0000","last_undegraded":"2024-03-26T05:24:54.628606+0000","last_fullsized":"2024-03-26T05:24:54.628606+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:37.960659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:53.017168+0000","last_clean_scrub_stamp":"2024-03-26T05:24:37.960659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:02.519656+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"443'14","reported_seq":1056,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628081+0000","last_change":"2024-03-26T05:23:45.659583+0000","last_active":"2024-03-26T05:24:54.628081+0000","last_peered":"2024-03-26T05:24:54.628081+0000","last_clean":"2024-03-26T05:24:54.628081+0000","last_became_active":"2024-03-26T05:00:24.941845+0000","last_became_peered":"2024-03-26T05:00:24.941845+0000","last_unstale":"2024-03-26T05:24:54.628081+0000","last_undegraded":"2024-03-26T05:24:54.628081+0000","last_fullsized":"2024-03-26T05:24:54.628081+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'14","last_scrub_stamp":"2024-03-26T05:23:45.659465+0000","last_deep_scrub":"100'6","last_deep_scrub_stamp":"2024-03-26T05:01:58.079363+0000","last_clean_scrub_stamp":"2024-03-26T05:23:45.659465+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:58.734800+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1083,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863851+0000","last_change":"2024-03-26T05:23:57.771237+0000","last_active":"2024-03-26T05:24:53.863851+0000","last_peered":"2024-03-26T05:24:53.863851+0000","last_clean":"2024-03-26T05:24:53.863851+0000","last_became_active":"2024-03-26T05:23:26.809979+0000","last_became_peered":"2024-03-26T05:23:26.809979+0000","last_unstale":"2024-03-26T05:24:53.863851+0000","last_undegraded":"2024-03-26T05:24:53.863851+0000","last_fullsized":"2024-03-26T05:24:53.863851+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:57.771156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:11.070341+0000","last_clean_scrub_stamp":"2024-03-26T05:23:57.771156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:04.197280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1057,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.945906+0000","last_change":"2024-03-26T05:23:59.761514+0000","last_active":"2024-03-26T05:24:54.945906+0000","last_peered":"2024-03-26T05:24:54.945906+0000","last_clean":"2024-03-26T05:24:54.945906+0000","last_became_active":"2024-03-26T05:23:23.936128+0000","last_became_peered":"2024-03-26T05:23:23.936128+0000","last_unstale":"2024-03-26T05:24:54.945906+0000","last_undegraded":"2024-03-26T05:24:54.945906+0000","last_fullsized":"2024-03-26T05:24:54.945906+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:59.761427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:23:59.761427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:10.149772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":955,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627283+0000","last_change":"2024-03-26T05:24:15.611940+0000","last_active":"2024-03-26T05:24:54.627283+0000","last_peered":"2024-03-26T05:24:54.627283+0000","last_clean":"2024-03-26T05:24:54.627283+0000","last_became_active":"2024-03-26T05:23:23.923501+0000","last_became_peered":"2024-03-26T05:23:23.923501+0000","last_unstale":"2024-03-26T05:24:54.627283+0000","last_undegraded":"2024-03-26T05:24:54.627283+0000","last_fullsized":"2024-03-26T05:24:54.627283+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:15.611825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:15.611825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:42.657185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"443'39","reported_seq":1239,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.945493+0000","last_change":"2024-03-26T05:23:49.645029+0000","last_active":"2024-03-26T05:24:54.945493+0000","last_peered":"2024-03-26T05:24:54.945493+0000","last_clean":"2024-03-26T05:24:54.945493+0000","last_became_active":"2024-03-26T05:23:26.814859+0000","last_became_peered":"2024-03-26T05:23:26.814859+0000","last_unstale":"2024-03-26T05:24:54.945493+0000","last_undegraded":"2024-03-26T05:24:54.945493+0000","last_fullsized":"2024-03-26T05:24:54.945493+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'39","last_scrub_stamp":"2024-03-26T05:23:49.644913+0000","last_deep_scrub":"413'31","last_deep_scrub_stamp":"2024-03-26T05:18:46.881054+0000","last_clean_scrub_stamp":"2024-03-26T05:23:49.644913+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:09.049245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1040,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628012+0000","last_change":"2024-03-26T05:23:40.634385+0000","last_active":"2024-03-26T05:24:54.628012+0000","last_peered":"2024-03-26T05:24:54.628012+0000","last_clean":"2024-03-26T05:24:54.628012+0000","last_became_active":"2024-03-26T05:00:22.779084+0000","last_became_peered":"2024-03-26T05:00:22.779084+0000","last_unstale":"2024-03-26T05:24:54.628012+0000","last_undegraded":"2024-03-26T05:24:54.628012+0000","last_fullsized":"2024-03-26T05:24:54.628012+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:40.634265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:18:47.776444+0000","last_clean_scrub_stamp":"2024-03-26T05:23:40.634265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1046,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863953+0000","last_change":"2024-03-26T05:24:48.133964+0000","last_active":"2024-03-26T05:24:53.863953+0000","last_peered":"2024-03-26T05:24:53.863953+0000","last_clean":"2024-03-26T05:24:53.863953+0000","last_became_active":"2024-03-26T05:15:32.634716+0000","last_became_peered":"2024-03-26T05:15:32.634716+0000","last_unstale":"2024-03-26T05:24:53.863953+0000","last_undegraded":"2024-03-26T05:24:53.863953+0000","last_fullsized":"2024-03-26T05:24:53.863953+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:48.133880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:23:18.758720+0000","last_clean_scrub_stamp":"2024-03-26T05:24:48.133880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.554815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1074,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:04.412556+0000","last_change":"2024-03-26T05:25:04.412579+0000","last_active":"2024-03-26T05:25:04.412556+0000","last_peered":"2024-03-26T05:25:04.412556+0000","last_clean":"2024-03-26T05:25:04.412556+0000","last_became_active":"2024-03-26T05:23:26.814722+0000","last_became_peered":"2024-03-26T05:23:26.814722+0000","last_unstale":"2024-03-26T05:25:04.412556+0000","last_undegraded":"2024-03-26T05:25:04.412556+0000","last_fullsized":"2024-03-26T05:25:04.412556+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:25:04.412472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:25:04.412472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:11.382373+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"443'42","reported_seq":1121,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863816+0000","last_change":"2024-03-26T05:24:31.996949+0000","last_active":"2024-03-26T05:24:53.863816+0000","last_peered":"2024-03-26T05:24:53.863816+0000","last_clean":"2024-03-26T05:24:53.863816+0000","last_became_active":"2024-03-26T05:15:32.634216+0000","last_became_peered":"2024-03-26T05:15:32.634216+0000","last_unstale":"2024-03-26T05:24:53.863816+0000","last_undegraded":"2024-03-26T05:24:53.863816+0000","last_fullsized":"2024-03-26T05:24:53.863816+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'42","last_scrub_stamp":"2024-03-26T05:24:31.996840+0000","last_deep_scrub":"290'32","last_deep_scrub_stamp":"2024-03-26T05:12:15.970313+0000","last_clean_scrub_stamp":"2024-03-26T05:24:31.996840+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:44.828643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":1064,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613257+0000","last_change":"2024-03-26T05:23:55.228827+0000","last_active":"2024-03-26T05:24:54.613257+0000","last_peered":"2024-03-26T05:24:54.613257+0000","last_clean":"2024-03-26T05:24:54.613257+0000","last_became_active":"2024-03-26T05:23:23.930527+0000","last_became_peered":"2024-03-26T05:23:23.930527+0000","last_unstale":"2024-03-26T05:24:54.613257+0000","last_undegraded":"2024-03-26T05:24:54.613257+0000","last_fullsized":"2024-03-26T05:24:54.613257+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-26T05:23:55.228704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:23:55.228704+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:24.587746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"504'13","reported_seq":1401,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:58.929348+0000","last_change":"2024-03-26T05:24:58.929348+0000","last_active":"2024-03-26T05:24:58.929348+0000","last_peered":"2024-03-26T05:24:58.929348+0000","last_clean":"2024-03-26T05:24:58.929348+0000","last_became_active":"2024-03-26T05:23:23.935698+0000","last_became_peered":"2024-03-26T05:23:23.935698+0000","last_unstale":"2024-03-26T05:24:58.929348+0000","last_undegraded":"2024-03-26T05:24:58.929348+0000","last_fullsized":"2024-03-26T05:24:58.929348+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'13","last_scrub_stamp":"2024-03-26T05:24:58.929226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:58.929226+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:04.689329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"429'2","reported_seq":987,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627530+0000","last_change":"2024-03-26T05:23:31.438151+0000","last_active":"2024-03-26T05:24:54.627530+0000","last_peered":"2024-03-26T05:24:54.627530+0000","last_clean":"2024-03-26T05:24:54.627530+0000","last_became_active":"2024-03-26T05:23:26.797077+0000","last_became_peered":"2024-03-26T05:23:26.797077+0000","last_unstale":"2024-03-26T05:24:54.627530+0000","last_undegraded":"2024-03-26T05:24:54.627530+0000","last_fullsized":"2024-03-26T05:24:54.627530+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'2","last_scrub_stamp":"2024-03-26T05:23:31.438043+0000","last_deep_scrub":"429'2","last_deep_scrub_stamp":"2024-03-26T05:19:54.275264+0000","last_clean_scrub_stamp":"2024-03-26T05:23:31.438043+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:56.650917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"443'38","reported_seq":1131,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:25:00.935067+0000","last_change":"2024-03-26T05:25:00.935129+0000","last_active":"2024-03-26T05:25:00.935067+0000","last_peered":"2024-03-26T05:25:00.935067+0000","last_clean":"2024-03-26T05:25:00.935067+0000","last_became_active":"2024-03-26T05:23:23.935314+0000","last_became_peered":"2024-03-26T05:23:23.935314+0000","last_unstale":"2024-03-26T05:25:00.935067+0000","last_undegraded":"2024-03-26T05:25:00.935067+0000","last_fullsized":"2024-03-26T05:25:00.935067+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'38","last_scrub_stamp":"2024-03-26T05:25:00.934972+0000","last_deep_scrub":"322'32","last_deep_scrub_stamp":"2024-03-26T05:15:10.715786+0000","last_clean_scrub_stamp":"2024-03-26T05:25:00.934972+0000","objects_scrubbed":6,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:13.778155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866780+0000","last_change":"2024-03-26T05:23:53.488591+0000","last_active":"2024-03-26T05:24:53.866780+0000","last_peered":"2024-03-26T05:24:53.866780+0000","last_clean":"2024-03-26T05:24:53.866780+0000","last_became_active":"2024-03-26T05:23:23.940669+0000","last_became_peered":"2024-03-26T05:23:23.940669+0000","last_unstale":"2024-03-26T05:24:53.866780+0000","last_undegraded":"2024-03-26T05:24:53.866780+0000","last_fullsized":"2024-03-26T05:24:53.866780+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:53.488486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:46.410356+0000","last_clean_scrub_stamp":"2024-03-26T05:23:53.488486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:04.275443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.867090+0000","last_change":"2024-03-26T05:24:15.607395+0000","last_active":"2024-03-26T05:24:53.867090+0000","last_peered":"2024-03-26T05:24:53.867090+0000","last_clean":"2024-03-26T05:24:53.867090+0000","last_became_active":"2024-03-26T05:23:23.937195+0000","last_became_peered":"2024-03-26T05:23:23.937195+0000","last_unstale":"2024-03-26T05:24:53.867090+0000","last_undegraded":"2024-03-26T05:24:53.867090+0000","last_fullsized":"2024-03-26T05:24:53.867090+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:15.607260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:15.607260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:19.431547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1062,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.629087+0000","last_change":"2024-03-26T05:24:35.078487+0000","last_active":"2024-03-26T05:24:54.629087+0000","last_peered":"2024-03-26T05:24:54.629087+0000","last_clean":"2024-03-26T05:24:54.629087+0000","last_became_active":"2024-03-26T05:23:26.803457+0000","last_became_peered":"2024-03-26T05:23:26.803457+0000","last_unstale":"2024-03-26T05:24:54.629087+0000","last_undegraded":"2024-03-26T05:24:54.629087+0000","last_fullsized":"2024-03-26T05:24:54.629087+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:35.078380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:20:01.576615+0000","last_clean_scrub_stamp":"2024-03-26T05:24:35.078380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:40.685466+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"443'39","reported_seq":1123,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.306655+0000","last_change":"2024-03-26T05:24:32.574978+0000","last_active":"2024-03-26T05:24:54.306655+0000","last_peered":"2024-03-26T05:24:54.306655+0000","last_clean":"2024-03-26T05:24:54.306655+0000","last_became_active":"2024-03-26T05:00:24.943151+0000","last_became_peered":"2024-03-26T05:00:24.943151+0000","last_unstale":"2024-03-26T05:24:54.306655+0000","last_undegraded":"2024-03-26T05:24:54.306655+0000","last_fullsized":"2024-03-26T05:24:54.306655+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'39","last_scrub_stamp":"2024-03-26T05:24:32.574853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:32.574853+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:54.392821+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1058,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863831+0000","last_change":"2024-03-26T05:23:41.727083+0000","last_active":"2024-03-26T05:24:53.863831+0000","last_peered":"2024-03-26T05:24:53.863831+0000","last_clean":"2024-03-26T05:24:53.863831+0000","last_became_active":"2024-03-26T05:23:23.929621+0000","last_became_peered":"2024-03-26T05:23:23.929621+0000","last_unstale":"2024-03-26T05:24:53.863831+0000","last_undegraded":"2024-03-26T05:24:53.863831+0000","last_fullsized":"2024-03-26T05:24:53.863831+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:41.727001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:07:53.149071+0000","last_clean_scrub_stamp":"2024-03-26T05:23:41.727001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:07.634291+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1052,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613902+0000","last_change":"2024-03-26T05:24:02.227643+0000","last_active":"2024-03-26T05:24:54.613902+0000","last_peered":"2024-03-26T05:24:54.613902+0000","last_clean":"2024-03-26T05:24:54.613902+0000","last_became_active":"2024-03-26T05:23:23.930052+0000","last_became_peered":"2024-03-26T05:23:23.930052+0000","last_unstale":"2024-03-26T05:24:54.613902+0000","last_undegraded":"2024-03-26T05:24:54.613902+0000","last_fullsized":"2024-03-26T05:24:54.613902+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:02.227560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:02.227560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:23.706995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":103,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.867096+0000","last_change":"2024-03-26T05:24:19.669790+0000","last_active":"2024-03-26T05:24:53.867096+0000","last_peered":"2024-03-26T05:24:53.867096+0000","last_clean":"2024-03-26T05:24:53.867096+0000","last_became_active":"2024-03-26T05:23:26.816310+0000","last_became_peered":"2024-03-26T05:23:26.816310+0000","last_unstale":"2024-03-26T05:24:53.867096+0000","last_undegraded":"2024-03-26T05:24:53.867096+0000","last_fullsized":"2024-03-26T05:24:53.867096+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:19.669658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:01:50.502459+0000","last_clean_scrub_stamp":"2024-03-26T05:24:19.669658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:36.243542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"443'18","reported_seq":105,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866419+0000","last_change":"2024-03-26T05:24:25.668061+0000","last_active":"2024-03-26T05:24:53.866419+0000","last_peered":"2024-03-26T05:24:53.866419+0000","last_clean":"2024-03-26T05:24:53.866419+0000","last_became_active":"2024-03-26T05:23:23.927909+0000","last_became_peered":"2024-03-26T05:23:23.927909+0000","last_unstale":"2024-03-26T05:24:53.866419+0000","last_undegraded":"2024-03-26T05:24:53.866419+0000","last_fullsized":"2024-03-26T05:24:53.866419+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-03-26T05:24:25.667897+0000","last_deep_scrub":"321'16","last_deep_scrub_stamp":"2024-03-26T05:15:00.995735+0000","last_clean_scrub_stamp":"2024-03-26T05:24:25.667897+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:26.347631+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114170+0000","last_change":"2024-03-26T05:24:31.419665+0000","last_active":"2024-03-26T05:24:55.114170+0000","last_peered":"2024-03-26T05:24:55.114170+0000","last_clean":"2024-03-26T05:24:55.114170+0000","last_became_active":"2024-03-26T05:23:26.811878+0000","last_became_peered":"2024-03-26T05:23:26.811878+0000","last_unstale":"2024-03-26T05:24:55.114170+0000","last_undegraded":"2024-03-26T05:24:55.114170+0000","last_fullsized":"2024-03-26T05:24:55.114170+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:31.419548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:15:00.635308+0000","last_clean_scrub_stamp":"2024-03-26T05:24:31.419548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:36.294443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":966,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.627532+0000","last_change":"2024-03-26T05:24:16.565833+0000","last_active":"2024-03-26T05:24:54.627532+0000","last_peered":"2024-03-26T05:24:54.627532+0000","last_clean":"2024-03-26T05:24:54.627532+0000","last_became_active":"2024-03-26T05:23:23.924963+0000","last_became_peered":"2024-03-26T05:23:23.924963+0000","last_unstale":"2024-03-26T05:24:54.627532+0000","last_undegraded":"2024-03-26T05:24:54.627532+0000","last_fullsized":"2024-03-26T05:24:54.627532+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:16.565704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:12:18.959687+0000","last_clean_scrub_stamp":"2024-03-26T05:24:16.565704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:17.798547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":100,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:57.570572+0000","last_change":"2024-03-26T05:24:57.570614+0000","last_active":"2024-03-26T05:24:57.570572+0000","last_peered":"2024-03-26T05:24:57.570572+0000","last_clean":"2024-03-26T05:24:57.570572+0000","last_became_active":"2024-03-26T05:23:26.802657+0000","last_became_peered":"2024-03-26T05:23:26.802657+0000","last_unstale":"2024-03-26T05:24:57.570572+0000","last_undegraded":"2024-03-26T05:24:57.570572+0000","last_fullsized":"2024-03-26T05:24:57.570572+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:57.570470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:57.570470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:18.424129+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"443'33","reported_seq":111,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866361+0000","last_change":"2024-03-26T05:24:12.574819+0000","last_active":"2024-03-26T05:24:53.866361+0000","last_peered":"2024-03-26T05:24:53.866361+0000","last_clean":"2024-03-26T05:24:53.866361+0000","last_became_active":"2024-03-26T05:23:23.930895+0000","last_became_peered":"2024-03-26T05:23:23.930895+0000","last_unstale":"2024-03-26T05:24:53.866361+0000","last_undegraded":"2024-03-26T05:24:53.866361+0000","last_fullsized":"2024-03-26T05:24:53.866361+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'33","last_scrub_stamp":"2024-03-26T05:24:12.574705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:12.574705+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:28.337465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.115042+0000","last_change":"2024-03-26T05:24:08.338763+0000","last_active":"2024-03-26T05:24:55.115042+0000","last_peered":"2024-03-26T05:24:55.115042+0000","last_clean":"2024-03-26T05:24:55.115042+0000","last_became_active":"2024-03-26T05:23:26.811430+0000","last_became_peered":"2024-03-26T05:23:26.811430+0000","last_unstale":"2024-03-26T05:24:55.115042+0000","last_undegraded":"2024-03-26T05:24:55.115042+0000","last_fullsized":"2024-03-26T05:24:55.115042+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:08.338680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:24:08.338680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:31.121401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":92,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:55.114102+0000","last_change":"2024-03-26T05:24:42.348161+0000","last_active":"2024-03-26T05:24:55.114102+0000","last_peered":"2024-03-26T05:24:55.114102+0000","last_clean":"2024-03-26T05:24:55.114102+0000","last_became_active":"2024-03-26T05:23:26.793735+0000","last_became_peered":"2024-03-26T05:23:26.793735+0000","last_unstale":"2024-03-26T05:24:55.114102+0000","last_undegraded":"2024-03-26T05:24:55.114102+0000","last_fullsized":"2024-03-26T05:24:55.114102+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:42.348053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:24:42.348053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:57.786404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1048,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863362+0000","last_change":"2024-03-26T05:24:35.038720+0000","last_active":"2024-03-26T05:24:53.863362+0000","last_peered":"2024-03-26T05:24:53.863362+0000","last_clean":"2024-03-26T05:24:53.863362+0000","last_became_active":"2024-03-26T05:15:32.640052+0000","last_became_peered":"2024-03-26T05:15:32.640052+0000","last_unstale":"2024-03-26T05:24:53.863362+0000","last_undegraded":"2024-03-26T05:24:53.863362+0000","last_fullsized":"2024-03-26T05:24:53.863362+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:35.038610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:35.038610+0000","last_clean_scrub_stamp":"2024-03-26T05:24:35.038610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:45.104003+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"520'74","reported_seq":1156,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628355+0000","last_change":"2024-03-26T05:23:49.635195+0000","last_active":"2024-03-26T05:24:54.628355+0000","last_peered":"2024-03-26T05:24:54.628355+0000","last_clean":"2024-03-26T05:24:54.628355+0000","last_became_active":"2024-03-26T05:00:24.932344+0000","last_became_peered":"2024-03-26T05:00:24.932344+0000","last_unstale":"2024-03-26T05:24:54.628355+0000","last_undegraded":"2024-03-26T05:24:54.628355+0000","last_fullsized":"2024-03-26T05:24:54.628355+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'74","last_scrub_stamp":"2024-03-26T05:23:49.635086+0000","last_deep_scrub":"520'74","last_deep_scrub_stamp":"2024-03-26T05:23:49.635086+0000","last_clean_scrub_stamp":"2024-03-26T05:23:49.635086+0000","objects_scrubbed":11,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:14.138112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":98,"num_read_kb":85,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1042,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628605+0000","last_change":"2024-03-26T05:24:18.885706+0000","last_active":"2024-03-26T05:24:54.628605+0000","last_peered":"2024-03-26T05:24:54.628605+0000","last_clean":"2024-03-26T05:24:54.628605+0000","last_became_active":"2024-03-26T05:00:23.346462+0000","last_became_peered":"2024-03-26T05:00:23.346462+0000","last_unstale":"2024-03-26T05:24:54.628605+0000","last_undegraded":"2024-03-26T05:24:54.628605+0000","last_fullsized":"2024-03-26T05:24:54.628605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:18.885629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:05:12.713011+0000","last_clean_scrub_stamp":"2024-03-26T05:24:18.885629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:43.265977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":97,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.865836+0000","last_change":"2024-03-26T05:24:16.642162+0000","last_active":"2024-03-26T05:24:53.865836+0000","last_peered":"2024-03-26T05:24:53.865836+0000","last_clean":"2024-03-26T05:24:53.865836+0000","last_became_active":"2024-03-26T05:23:23.927943+0000","last_became_peered":"2024-03-26T05:23:23.927943+0000","last_unstale":"2024-03-26T05:24:53.865836+0000","last_undegraded":"2024-03-26T05:24:53.865836+0000","last_fullsized":"2024-03-26T05:24:53.865836+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:16.642049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:24:16.642049+0000","last_clean_scrub_stamp":"2024-03-26T05:24:16.642049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:44.028895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1020,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.628379+0000","last_change":"2024-03-26T05:24:10.134983+0000","last_active":"2024-03-26T05:24:54.628379+0000","last_peered":"2024-03-26T05:24:54.628379+0000","last_clean":"2024-03-26T05:24:54.628379+0000","last_became_active":"2024-03-26T05:15:32.637717+0000","last_became_peered":"2024-03-26T05:15:32.637717+0000","last_unstale":"2024-03-26T05:24:54.628379+0000","last_undegraded":"2024-03-26T05:24:54.628379+0000","last_fullsized":"2024-03-26T05:24:54.628379+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:24:10.134860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:24:10.134860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:16.518730+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"443'27","reported_seq":109,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866010+0000","last_change":"2024-03-26T05:24:21.705542+0000","last_active":"2024-03-26T05:24:53.866010+0000","last_peered":"2024-03-26T05:24:53.866010+0000","last_clean":"2024-03-26T05:24:53.866010+0000","last_became_active":"2024-03-26T05:23:23.928598+0000","last_became_peered":"2024-03-26T05:23:23.928598+0000","last_unstale":"2024-03-26T05:24:53.866010+0000","last_undegraded":"2024-03-26T05:24:53.866010+0000","last_fullsized":"2024-03-26T05:24:53.866010+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'27","last_scrub_stamp":"2024-03-26T05:24:21.705431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:23.918327+0000","last_clean_scrub_stamp":"2024-03-26T05:24:21.705431+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:42.216471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1076,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.863734+0000","last_change":"2024-03-26T05:23:58.803459+0000","last_active":"2024-03-26T05:24:53.863734+0000","last_peered":"2024-03-26T05:24:53.863734+0000","last_clean":"2024-03-26T05:24:53.863734+0000","last_became_active":"2024-03-26T05:23:26.810639+0000","last_became_peered":"2024-03-26T05:23:26.810639+0000","last_unstale":"2024-03-26T05:24:53.863734+0000","last_undegraded":"2024-03-26T05:24:53.863734+0000","last_fullsized":"2024-03-26T05:24:53.863734+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:58.803375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:21.748689+0000","last_clean_scrub_stamp":"2024-03-26T05:23:58.803375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:00.549208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1065,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:54.613316+0000","last_change":"2024-03-26T05:23:31.098901+0000","last_active":"2024-03-26T05:24:54.613316+0000","last_peered":"2024-03-26T05:24:54.613316+0000","last_clean":"2024-03-26T05:24:54.613316+0000","last_became_active":"2024-03-26T05:23:26.818750+0000","last_became_peered":"2024-03-26T05:23:26.818750+0000","last_unstale":"2024-03-26T05:24:54.613316+0000","last_undegraded":"2024-03-26T05:24:54.613316+0000","last_fullsized":"2024-03-26T05:24:54.613316+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:31.098786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:25.933152+0000","last_clean_scrub_stamp":"2024-03-26T05:23:31.098786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:59.884114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":103,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-26T05:24:53.866510+0000","last_change":"2024-03-26T05:23:59.562452+0000","last_active":"2024-03-26T05:24:53.866510+0000","last_peered":"2024-03-26T05:24:53.866510+0000","last_clean":"2024-03-26T05:24:53.866510+0000","last_became_active":"2024-03-26T05:23:26.820867+0000","last_became_peered":"2024-03-26T05:23:26.820867+0000","last_unstale":"2024-03-26T05:24:53.866510+0000","last_undegraded":"2024-03-26T05:24:53.866510+0000","last_fullsized":"2024-03-26T05:24:53.866510+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:23:59.562345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:00:27.943471+0000","last_clean_scrub_stamp":"2024-03-26T05:23:59.562345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:25:08.378788+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":638,"num_read_kb":633,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1823,"num_read_kb":1548,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":7018,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3666,"internal_metadata":0},"log_size":1321,"ondisk_log_size":1321,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":335,"seq":1438814044277,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":523564,"kb_used_data":1060,"kb_used_omap":4,"kb_used_meta":522491,"kb_avail":93221588,"statfs":{"total":95995035648,"available":95458906112,"internally_reserved":0,"allocated":1085440,"data_stored":766308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":535031464},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332403,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":731612,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93013540,"statfs":{"total":95995035648,"available":95245864960,"internally_reserved":0,"allocated":741376,"data_stored":437112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":748419526},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575181,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938496,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92806656,"statfs":{"total":95995035648,"available":95034015744,"internally_reserved":0,"allocated":778240,"data_stored":437936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738706,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952376,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92792776,"statfs":{"total":95995035648,"available":95019802624,"internally_reserved":0,"allocated":770048,"data_stored":438781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411656,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":731880,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93013272,"statfs":{"total":95995035648,"available":95245590528,"internally_reserved":0,"allocated":1015808,"data_stored":700054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117314,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907132,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92838020,"statfs":{"total":95995035648,"available":95066132480,"internally_reserved":0,"allocated":1363968,"data_stored":1027960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":927529415},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":501,"seq":2151778615320,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1208176,"kb_used_data":1145384,"kb_used_omap":7,"kb_used_meta":62776,"kb_avail":92536976,"statfs":{"total":95995035648,"available":94757863424,"internally_reserved":0,"allocated":1172873216,"data_stored":1172549216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8112,"internal_metadata":64282704},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659448,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894628,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":92850524,"statfs":{"total":95995035648,"available":95078936576,"internally_reserved":0,"allocated":1339392,"data_stored":1027806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":914749893},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3666,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:25:10.546 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T05:25:10.546 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-26T05:25:10.787 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:25:11.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:10 smithi080 ceph-mon[29691]: 3.11 scrub starts 2024-03-26T05:25:11.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:10 smithi080 ceph-mon[29691]: 3.11 scrub ok 2024-03-26T05:25:11.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:10 smithi080 ceph-mon[29691]: 4.e scrub starts 2024-03-26T05:25:11.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:10 smithi080 ceph-mon[29691]: 4.e scrub ok 2024-03-26T05:25:11.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:10 smithi080 ceph-mon[29691]: from='client.26255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:25:11.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:10 smithi080 ceph-mon[29691]: pgmap v1182: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:25:11.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:10 smithi080 ceph-mon[40404]: 3.11 scrub starts 2024-03-26T05:25:11.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:10 smithi080 ceph-mon[40404]: 3.11 scrub ok 2024-03-26T05:25:11.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:10 smithi080 ceph-mon[40404]: 4.e scrub starts 2024-03-26T05:25:11.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:10 smithi080 ceph-mon[40404]: 4.e scrub ok 2024-03-26T05:25:11.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:10 smithi080 ceph-mon[40404]: from='client.26255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:25:11.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:10 smithi080 ceph-mon[40404]: pgmap v1182: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:25:11.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:10 smithi132 ceph-mon[33500]: 3.11 scrub starts 2024-03-26T05:25:11.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:10 smithi132 ceph-mon[33500]: 3.11 scrub ok 2024-03-26T05:25:11.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:10 smithi132 ceph-mon[33500]: 4.e scrub starts 2024-03-26T05:25:11.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:10 smithi132 ceph-mon[33500]: 4.e scrub ok 2024-03-26T05:25:11.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:10 smithi132 ceph-mon[33500]: from='client.26255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:25:11.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:10 smithi132 ceph-mon[33500]: pgmap v1182: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:25:12.205 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:11 smithi080 ceph-mon[40404]: 1.4 scrub starts 2024-03-26T05:25:12.206 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:11 smithi080 ceph-mon[40404]: 1.4 scrub ok 2024-03-26T05:25:12.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:11 smithi080 ceph-mon[29691]: 1.4 scrub starts 2024-03-26T05:25:12.206 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:11 smithi080 ceph-mon[29691]: 1.4 scrub ok 2024-03-26T05:25:12.291 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:11 smithi132 ceph-mon[33500]: 1.4 scrub starts 2024-03-26T05:25:12.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:11 smithi132 ceph-mon[33500]: 1.4 scrub ok 2024-03-26T05:25:12.927 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:12] ENGINE Bus STOPPING 2024-03-26T05:25:12.927 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:25:12.927 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:12] ENGINE Bus STOPPED 2024-03-26T05:25:12.927 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:12] ENGINE Bus STARTING 2024-03-26T05:25:12.927 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:12] ENGINE Serving on http://:::9283 2024-03-26T05:25:12.927 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:12] ENGINE Bus STARTED 2024-03-26T05:25:13.254 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[29691]: 3.7 scrub starts 2024-03-26T05:25:13.254 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[29691]: 3.7 scrub ok 2024-03-26T05:25:13.254 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[29691]: 5.19 scrub starts 2024-03-26T05:25:13.254 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[29691]: 5.19 scrub ok 2024-03-26T05:25:13.254 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[29691]: pgmap v1183: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:25:13.254 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1312825995' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T05:25:13.254 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[29691]: from='client.? 172.21.15.80:0/1312825995' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-26T05:25:13.255 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:25:13.255 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:25:13.255 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:25:13.255 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[29691]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:25:13.255 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[40404]: 3.7 scrub starts 2024-03-26T05:25:13.255 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[40404]: 3.7 scrub ok 2024-03-26T05:25:13.255 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[40404]: 5.19 scrub starts 2024-03-26T05:25:13.255 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[40404]: 5.19 scrub ok 2024-03-26T05:25:13.255 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[40404]: pgmap v1183: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:25:13.255 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1312825995' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T05:25:13.255 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[40404]: from='client.? 172.21.15.80:0/1312825995' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-26T05:25:13.255 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:25:13.256 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:25:13.256 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:25:13.256 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:12 smithi080 ceph-mon[40404]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:25:13.279 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T05:25:13.289 INFO:tasks.cephadm:Teardown begin 2024-03-26T05:25:13.289 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:25:13.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:12 smithi132 ceph-mon[33500]: 3.7 scrub starts 2024-03-26T05:25:13.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:12 smithi132 ceph-mon[33500]: 3.7 scrub ok 2024-03-26T05:25:13.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:12 smithi132 ceph-mon[33500]: 5.19 scrub starts 2024-03-26T05:25:13.292 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:12 smithi132 ceph-mon[33500]: 5.19 scrub ok 2024-03-26T05:25:13.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:12 smithi132 ceph-mon[33500]: pgmap v1183: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:25:13.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:12 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1312825995' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T05:25:13.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:12 smithi132 ceph-mon[33500]: from='client.? 172.21.15.80:0/1312825995' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-26T05:25:13.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:12 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:25:13.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:12 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:25:13.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:12 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:25:13.293 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:12 smithi132 ceph-mon[33500]: from='mgr.14580 172.21.15.80:0/86307406' entity='mgr.y' 2024-03-26T05:25:13.347 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:25:13.402 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T05:25:13.403 DEBUG:teuthology.orchestra.run.smithi080:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T05:25:13.432 DEBUG:teuthology.orchestra.run.smithi132:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T05:25:13.452 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T05:25:13.453 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T05:25:13.453 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.a 2024-03-26T05:25:13.825 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:13 smithi080 systemd[1]: Stopping Ceph mon.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:25:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:13 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a[29687]: 2024-03-26T05:25:13.833+0000 7f70d7d0f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:25:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:13 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a[29687]: 2024-03-26T05:25:13.833+0000 7f70d7d0f700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T05:25:14.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:13 smithi080 podman[222354]: 2024-03-26 05:25:13.987175536 +0000 UTC m=+0.305967841 container died d7b1d029f188e01e68fd186f3f6072fc520eb5b4a9bd58256503a363d33db90b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124) 2024-03-26T05:25:14.116 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:13 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:13] ENGINE Bus STOPPING 2024-03-26T05:25:14.116 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:13 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:25:14.116 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:13 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:13] ENGINE Bus STOPPED 2024-03-26T05:25:14.117 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:13 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:13] ENGINE Bus STARTING 2024-03-26T05:25:14.117 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:13 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:13] ENGINE Serving on http://:::9283 2024-03-26T05:25:14.117 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:13 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:13] ENGINE Bus STARTED 2024-03-26T05:25:14.117 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:14 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: ::ffff:172.21.15.132 - - [26/Mar/2024:05:25:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:25:14.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:14 smithi080 podman[222354]: 2024-03-26 05:25:14.442974736 +0000 UTC m=+0.761767074 container cleanup d7b1d029f188e01e68fd186f3f6072fc520eb5b4a9bd58256503a363d33db90b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-26T05:25:14.616 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 26 05:25:14 smithi080 bash[222354]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-a 2024-03-26T05:25:14.817 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T05:25:14.889 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:25:14.889 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T05:25:14.890 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-26T05:25:14.890 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.c 2024-03-26T05:25:15.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:14 smithi080 systemd[1]: Stopping Ceph mon.c for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:25:15.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:15 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-c[40400]: 2024-03-26T05:25:15.213+0000 7ff2de60c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:25:15.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:15 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-c[40400]: 2024-03-26T05:25:15.213+0000 7ff2de60c700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T05:25:15.616 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:15 smithi080 podman[222490]: 2024-03-26 05:25:15.36836828 +0000 UTC m=+0.290639204 container died 2ccfda2e5cae31b065fb0d39518431ba9e80454ec55ba02bdad97200f38ad5de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:25:15.616 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:15 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:15] ENGINE Bus STOPPING 2024-03-26T05:25:15.616 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:15 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:15] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:25:15.617 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:15 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:15] ENGINE Bus STOPPED 2024-03-26T05:25:15.617 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:15 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:15] ENGINE Bus STARTING 2024-03-26T05:25:15.617 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:15 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:15] ENGINE Serving on http://:::9283 2024-03-26T05:25:15.617 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:15 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: [26/Mar/2024:05:25:15] ENGINE Bus STARTED 2024-03-26T05:25:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:15 smithi080 podman[222490]: 2024-03-26 05:25:15.848576776 +0000 UTC m=+0.770847707 container cleanup 2ccfda2e5cae31b065fb0d39518431ba9e80454ec55ba02bdad97200f38ad5de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=squid-5e70124) 2024-03-26T05:25:16.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 26 05:25:15 smithi080 bash[222490]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-c 2024-03-26T05:25:16.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T05:25:16.287 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:25:16.287 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-26T05:25:16.288 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-26T05:25:16.288 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.b 2024-03-26T05:25:16.541 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:16 smithi132 systemd[1]: Stopping Ceph mon.b for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:25:16.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:25:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug there is no tcmu-runner data available 2024-03-26T05:25:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-b[33496]: 2024-03-26T05:25:16.654+0000 7f608f422700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:25:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-b[33496]: 2024-03-26T05:25:16.654+0000 7f608f422700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T05:25:17.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:16 smithi132 podman[157225]: 2024-03-26 05:25:16.808936554 +0000 UTC m=+0.244416484 container died 7747b853140c851ae358488216623b90ae62dcd5e46bc5429f501eaccf9b0600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T05:25:17.430 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:17 smithi132 podman[157225]: 2024-03-26 05:25:17.238831685 +0000 UTC m=+0.674311604 container cleanup 7747b853140c851ae358488216623b90ae62dcd5e46bc5429f501eaccf9b0600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-b, GIT_CLEAN=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T05:25:17.430 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:17 smithi132 bash[157225]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-b 2024-03-26T05:25:17.654 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T05:25:17.680 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:17 smithi132 podman[157240]: 2024-03-26 05:25:17.440099372 +0000 UTC m=+0.633545973 container remove 7747b853140c851ae358488216623b90ae62dcd5e46bc5429f501eaccf9b0600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mon-b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-26T05:25:17.680 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:17 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-26T05:25:17.680 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:17 smithi132 systemd[1]: Stopped Ceph mon.b for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:25:17.680 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 26 05:25:17 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mon.b.service: Consumed 22.245s CPU time. 2024-03-26T05:25:17.712 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:25:17.712 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-26T05:25:17.712 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T05:25:17.712 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mgr.y 2024-03-26T05:25:18.026 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:17 smithi080 systemd[1]: Stopping Ceph mgr.y for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:25:18.335 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:18 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y[29924]: 2024-03-26T05:25:18.035+0000 7f304cad7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:25:18.335 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:18 smithi080 podman[222623]: 2024-03-26 05:25:18.066944882 +0000 UTC m=+0.150861123 container died fbbbaa9143e7b70f9bb80add4374aad1520ba304e797fd655fc7ddb0afe858d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-26T05:25:18.616 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:18 smithi080 podman[222623]: 2024-03-26 05:25:18.528341521 +0000 UTC m=+0.612257750 container cleanup fbbbaa9143e7b70f9bb80add4374aad1520ba304e797fd655fc7ddb0afe858d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T05:25:18.616 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 26 05:25:18 smithi080 bash[222623]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-y 2024-03-26T05:25:18.936 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T05:25:18.972 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:25:18.972 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T05:25:18.972 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T05:25:18.972 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mgr.x 2024-03-26T05:25:19.292 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:25:19 smithi132 systemd[1]: Stopping Ceph mgr.x for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:25:19.708 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:25:19 smithi132 podman[157346]: 2024-03-26 05:25:19.30752989 +0000 UTC m=+0.137858194 container died 09998ae5d2ac79cc4768befc5ed43a49f9958cfa05e4480dd2daa6224a9c3e3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-5e70124) 2024-03-26T05:25:20.041 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:25:19 smithi132 podman[157346]: 2024-03-26 05:25:19.718558093 +0000 UTC m=+0.548886455 container cleanup 09998ae5d2ac79cc4768befc5ed43a49f9958cfa05e4480dd2daa6224a9c3e3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T05:25:20.042 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:25:19 smithi132 bash[157346]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x 2024-03-26T05:25:20.042 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:25:19 smithi132 podman[157360]: 2024-03-26 05:25:19.962251468 +0000 UTC m=+0.658542453 container remove 09998ae5d2ac79cc4768befc5ed43a49f9958cfa05e4480dd2daa6224a9c3e3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-mgr-x, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T05:25:20.042 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Mar 26 05:25:19 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-26T05:25:20.119 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T05:25:20.156 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:25:20.156 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T05:25:20.156 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T05:25:20.157 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.0 2024-03-26T05:25:20.456 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:20 smithi080 systemd[1]: Stopping Ceph osd.0 for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:25:20.866 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0[44401]: 2024-03-26T05:25:20.465+0000 7f1085401700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:25:20.866 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0[44401]: 2024-03-26T05:25:20.465+0000 7f1085401700 -1 osd.0 544 *** Got signal Terminated *** 2024-03-26T05:25:20.866 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:20 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0[44401]: 2024-03-26T05:25:20.465+0000 7f1085401700 -1 osd.0 544 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:25:26.332 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:26 smithi080 podman[222743]: 2024-03-26 05:25:26.028979559 +0000 UTC m=+5.671719680 container died 417fb4798c44477eebbb6cfb5346035420d32114a37a9e44c73dff4204d59435 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:25:26.616 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:26 smithi080 podman[222743]: 2024-03-26 05:25:26.498890151 +0000 UTC m=+6.141630280 container cleanup 417fb4798c44477eebbb6cfb5346035420d32114a37a9e44c73dff4204d59435 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:25:26.616 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:26 smithi080 bash[222743]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0 2024-03-26T05:25:27.025 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:26 smithi080 podman[222760]: 2024-03-26 05:25:26.741915643 +0000 UTC m=+0.708160683 container remove 417fb4798c44477eebbb6cfb5346035420d32114a37a9e44c73dff4204d59435 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:25:27.294 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:27 smithi080 podman[222853]: 2024-03-26 05:25:26.93773285 +0000 UTC m=+0.018174097 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:25:27.294 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:27 smithi080 podman[222853]: 2024-03-26 05:25:27.042970487 +0000 UTC m=+0.123411735 container create 9486bc2393ed8b58e62ce3e16026a573a7a577dc7e14273a5a015fd70636efbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS) 2024-03-26T05:25:27.294 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:27 smithi080 podman[222853]: 2024-03-26 05:25:27.30374511 +0000 UTC m=+0.384186357 container init 9486bc2393ed8b58e62ce3e16026a573a7a577dc7e14273a5a015fd70636efbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124) 2024-03-26T05:25:27.542 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:25:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:25:27.177Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-26T05:25:27.542 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:25:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:25:27.177Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-26T05:25:27.542 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:25:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:25:27.182Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-26T05:25:27.542 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:25:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:25:27.182Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-26T05:25:27.616 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:27 smithi080 podman[222853]: 2024-03-26 05:25:27.307043508 +0000 UTC m=+0.387484767 container start 9486bc2393ed8b58e62ce3e16026a573a7a577dc7e14273a5a015fd70636efbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T05:25:27.616 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:27 smithi080 podman[222853]: 2024-03-26 05:25:27.412148747 +0000 UTC m=+0.492590005 container attach 9486bc2393ed8b58e62ce3e16026a573a7a577dc7e14273a5a015fd70636efbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-26T05:25:28.116 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 26 05:25:27 smithi080 podman[222853]: 2024-03-26 05:25:27.847263137 +0000 UTC m=+0.927704387 container died 9486bc2393ed8b58e62ce3e16026a573a7a577dc7e14273a5a015fd70636efbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-0-deactivate, RELEASE=squid-5e70124, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T05:25:28.474 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T05:25:28.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:25:28.516 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T05:25:28.517 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T05:25:28.517 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.1 2024-03-26T05:25:28.851 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:28 smithi080 systemd[1]: Stopping Ceph osd.1 for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:25:29.116 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:28 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1[50406]: 2024-03-26T05:25:28.860+0000 7f3d1ac11700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:25:29.116 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:28 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1[50406]: 2024-03-26T05:25:28.860+0000 7f3d1ac11700 -1 osd.1 544 *** Got signal Terminated *** 2024-03-26T05:25:29.116 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:28 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1[50406]: 2024-03-26T05:25:28.860+0000 7f3d1ac11700 -1 osd.1 544 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:25:34.768 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:34 smithi080 podman[222963]: 2024-03-26 05:25:34.510147253 +0000 UTC m=+5.759486321 container died 6e4d9c61dab97e91860d005ccf96f8793fd2c71ebc3f8770e2435152e193a8b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0) 2024-03-26T05:25:35.093 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:34 smithi080 podman[222963]: 2024-03-26 05:25:34.918406312 +0000 UTC m=+6.167745378 container cleanup 6e4d9c61dab97e91860d005ccf96f8793fd2c71ebc3f8770e2435152e193a8b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T05:25:35.094 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:34 smithi080 bash[222963]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1 2024-03-26T05:25:35.366 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:35 smithi080 podman[222980]: 2024-03-26 05:25:35.103021598 +0000 UTC m=+0.595967444 container remove 6e4d9c61dab97e91860d005ccf96f8793fd2c71ebc3f8770e2435152e193a8b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, ceph=True) 2024-03-26T05:25:35.667 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:35 smithi080 podman[223073]: 2024-03-26 05:25:35.327056523 +0000 UTC m=+0.017497246 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:25:35.667 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:35 smithi080 podman[223073]: 2024-03-26 05:25:35.43750472 +0000 UTC m=+0.127945463 container create e37f4a352a739bdc12c8266fc739eaff1c3c374551d7784e900e27061efc915d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-26T05:25:35.667 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:35 smithi080 podman[223073]: 2024-03-26 05:25:35.673960346 +0000 UTC m=+0.364401047 container init e37f4a352a739bdc12c8266fc739eaff1c3c374551d7784e900e27061efc915d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T05:25:35.667 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:35 smithi080 podman[223073]: 2024-03-26 05:25:35.677145824 +0000 UTC m=+0.367586534 container start e37f4a352a739bdc12c8266fc739eaff1c3c374551d7784e900e27061efc915d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-5e70124) 2024-03-26T05:25:35.952 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:35 smithi080 podman[223073]: 2024-03-26 05:25:35.773007085 +0000 UTC m=+0.463447801 container attach e37f4a352a739bdc12c8266fc739eaff1c3c374551d7784e900e27061efc915d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-26T05:25:36.291 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:35 smithi080 podman[223073]: 2024-03-26 05:25:35.961436589 +0000 UTC m=+0.651877281 container died e37f4a352a739bdc12c8266fc739eaff1c3c374551d7784e900e27061efc915d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:25:36.556 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 26 05:25:36 smithi080 podman[223099]: 2024-03-26 05:25:36.543366827 +0000 UTC m=+0.572461107 container remove e37f4a352a739bdc12c8266fc739eaff1c3c374551d7784e900e27061efc915d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T05:25:36.561 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T05:25:36.606 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:25:36.606 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T05:25:36.606 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T05:25:36.606 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.2 2024-03-26T05:25:36.866 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:36 smithi080 systemd[1]: Stopping Ceph osd.2 for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:25:37.366 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:36 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2[56429]: 2024-03-26T05:25:36.920+0000 7fd549586700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:25:37.366 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:36 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2[56429]: 2024-03-26T05:25:36.920+0000 7fd549586700 -1 osd.2 544 *** Got signal Terminated *** 2024-03-26T05:25:37.366 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:36 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2[56429]: 2024-03-26T05:25:36.920+0000 7fd549586700 -1 osd.2 544 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:25:42.804 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:42 smithi080 podman[223179]: 2024-03-26 05:25:42.416686595 +0000 UTC m=+5.583474822 container died 374f81256ba305015d2c35f364249379a8098130a5984d075ed8e47ba1c3a788 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-03-26T05:25:42.804 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:42 smithi080 podman[223179]: 2024-03-26 05:25:42.81347901 +0000 UTC m=+5.980267268 container cleanup 374f81256ba305015d2c35f364249379a8098130a5984d075ed8e47ba1c3a788 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T05:25:43.116 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:42 smithi080 bash[223179]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2 2024-03-26T05:25:43.116 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:43 smithi080 podman[223195]: 2024-03-26 05:25:43.04693434 +0000 UTC m=+0.624849626 container remove 374f81256ba305015d2c35f364249379a8098130a5984d075ed8e47ba1c3a788 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-26T05:25:43.616 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:43 smithi080 podman[223290]: 2024-03-26 05:25:43.347698444 +0000 UTC m=+0.105482270 container create 19632ab1a490300de5952fce043b53f1786c0587d2b59586894625c1edf936a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T05:25:43.616 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:43 smithi080 podman[223290]: 2024-03-26 05:25:43.260322336 +0000 UTC m=+0.018106170 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:25:43.616 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:43 smithi080 podman[223290]: 2024-03-26 05:25:43.575028101 +0000 UTC m=+0.332811928 container init 19632ab1a490300de5952fce043b53f1786c0587d2b59586894625c1edf936a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T05:25:43.616 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:43 smithi080 podman[223290]: 2024-03-26 05:25:43.578289344 +0000 UTC m=+0.336073174 container start 19632ab1a490300de5952fce043b53f1786c0587d2b59586894625c1edf936a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD) 2024-03-26T05:25:44.111 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:43 smithi080 podman[223290]: 2024-03-26 05:25:43.641821425 +0000 UTC m=+0.399605274 container attach 19632ab1a490300de5952fce043b53f1786c0587d2b59586894625c1edf936a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:25:44.111 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:43 smithi080 podman[223290]: 2024-03-26 05:25:43.867484364 +0000 UTC m=+0.625268196 container died 19632ab1a490300de5952fce043b53f1786c0587d2b59586894625c1edf936a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2-deactivate, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-26T05:25:44.362 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 26 05:25:44 smithi080 podman[223316]: 2024-03-26 05:25:44.370219317 +0000 UTC m=+0.494633292 container remove 19632ab1a490300de5952fce043b53f1786c0587d2b59586894625c1edf936a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-26T05:25:44.383 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T05:25:44.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:25:44.427 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T05:25:44.427 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T05:25:44.428 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.3 2024-03-26T05:25:44.755 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:44 smithi080 systemd[1]: Stopping Ceph osd.3 for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:25:45.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3[62541]: 2024-03-26T05:25:44.764+0000 7fc5d367a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:25:45.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3[62541]: 2024-03-26T05:25:44.764+0000 7fc5d367a700 -1 osd.3 544 *** Got signal Terminated *** 2024-03-26T05:25:45.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:44 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3[62541]: 2024-03-26T05:25:44.764+0000 7fc5d367a700 -1 osd.3 544 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:25:47.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:47 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:47.148+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:47.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:47 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:47.238+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:48.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:48 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:48.173+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:48.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:48 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:48.199+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:49.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:49 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:49.201+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:49.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:49 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:49.139+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:50.128 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:49 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:49.875+0000 7fd9745c5700 -1 osd.4 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.766158+0000 front 2024-03-26T05:25:21.766124+0000 (oldest deadline 2024-03-26T05:25:49.366344+0000) 2024-03-26T05:25:50.128 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:49 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:49.888+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:25:50.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:50 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:50.186+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:50.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:50 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:50.137+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:50.616 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:50 smithi080 podman[223395]: 2024-03-26 05:25:50.32079541 +0000 UTC m=+5.654864803 container died 6e4be109e5b6bee44e9968d3f17ad9084156dd2c84639f5960cbb9e299b4f4b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T05:25:51.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:50 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:50.846+0000 7fd9745c5700 -1 osd.4 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.766158+0000 front 2024-03-26T05:25:21.766124+0000 (oldest deadline 2024-03-26T05:25:49.366344+0000) 2024-03-26T05:25:51.098 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:50 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:50.860+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:25:51.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:50 smithi080 podman[223395]: 2024-03-26 05:25:50.722263855 +0000 UTC m=+6.056333290 container cleanup 6e4be109e5b6bee44e9968d3f17ad9084156dd2c84639f5960cbb9e299b4f4b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-26T05:25:51.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:50 smithi080 bash[223395]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3 2024-03-26T05:25:51.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:50 smithi080 podman[223412]: 2024-03-26 05:25:50.956794305 +0000 UTC m=+0.632117827 container remove 6e4be109e5b6bee44e9968d3f17ad9084156dd2c84639f5960cbb9e299b4f4b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-03-26T05:25:51.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:51 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:51.106+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:51.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:51 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:51.195+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:51.566 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:51 smithi080 podman[223502]: 2024-03-26 05:25:51.163483962 +0000 UTC m=+0.017593215 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:25:51.566 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:51 smithi080 podman[223502]: 2024-03-26 05:25:51.265755907 +0000 UTC m=+0.119865151 container create 765ae254b832ed8f3e25966fa7e8c8e41eb1522a48b36d7c430619abab7ace94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T05:25:51.567 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:51 smithi080 podman[223502]: 2024-03-26 05:25:51.492941872 +0000 UTC m=+0.347051118 container init 765ae254b832ed8f3e25966fa7e8c8e41eb1522a48b36d7c430619abab7ace94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:25:51.567 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:51 smithi080 podman[223502]: 2024-03-26 05:25:51.496060092 +0000 UTC m=+0.350169346 container start 765ae254b832ed8f3e25966fa7e8c8e41eb1522a48b36d7c430619abab7ace94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:25:51.567 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:51 smithi080 podman[223502]: 2024-03-26 05:25:51.576205001 +0000 UTC m=+0.430314247 container attach 765ae254b832ed8f3e25966fa7e8c8e41eb1522a48b36d7c430619abab7ace94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T05:25:51.866 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 26 05:25:51 smithi080 podman[223502]: 2024-03-26 05:25:51.785818492 +0000 UTC m=+0.639927737 container died 765ae254b832ed8f3e25966fa7e8c8e41eb1522a48b36d7c430619abab7ace94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True) 2024-03-26T05:25:52.203 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:51 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:51.824+0000 7fd9745c5700 -1 osd.4 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.766158+0000 front 2024-03-26T05:25:21.766124+0000 (oldest deadline 2024-03-26T05:25:49.366344+0000) 2024-03-26T05:25:52.203 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:52.062+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:52.204 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:51 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:51.851+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:25:52.438 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T05:25:52.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:25:52.481 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T05:25:52.481 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T05:25:52.481 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.4 2024-03-26T05:25:52.510 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:52.212+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:52.792 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:52 smithi132 systemd[1]: Stopping Ceph osd.4 for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:25:53.212 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:53 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:53.046+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:53.213 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:52.846+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:25:53.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:52.814+0000 7fd9745c5700 -1 osd.4 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.766158+0000 front 2024-03-26T05:25:21.766124+0000 (oldest deadline 2024-03-26T05:25:49.366344+0000) 2024-03-26T05:25:53.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:52.920+0000 7fd977fd1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:25:53.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:52.920+0000 7fd977fd1700 -1 osd.4 544 *** Got signal Terminated *** 2024-03-26T05:25:53.213 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:52 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:52.920+0000 7fd977fd1700 -1 osd.4 544 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:25:53.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:53 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:53.221+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:54.262 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:54.045+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:54.262 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:53 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:53.813+0000 7fd9745c5700 -1 osd.4 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.766158+0000 front 2024-03-26T05:25:21.766124+0000 (oldest deadline 2024-03-26T05:25:49.366344+0000) 2024-03-26T05:25:54.262 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:53 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:53.833+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:25:54.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:54.271+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:55.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:54.838+0000 7fd9745c5700 -1 osd.4 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.766158+0000 front 2024-03-26T05:25:21.766124+0000 (oldest deadline 2024-03-26T05:25:49.366344+0000) 2024-03-26T05:25:55.280 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:55.046+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:55.281 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:54 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:54.861+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:25:55.541 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:55.289+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:56.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:55.796+0000 7fd9745c5700 -1 osd.4 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.766158+0000 front 2024-03-26T05:25:21.766124+0000 (oldest deadline 2024-03-26T05:25:49.366344+0000) 2024-03-26T05:25:56.042 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:56.001+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:56.042 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:56.001+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:25:56.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:55 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:55.844+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:25:56.763 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:56.316+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:57.042 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:56.773+0000 7fd9745c5700 -1 osd.4 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.766158+0000 front 2024-03-26T05:25:21.766124+0000 (oldest deadline 2024-03-26T05:25:49.366344+0000) 2024-03-26T05:25:57.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:56 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:56.838+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:25:57.042 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:57.020+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:57.042 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:57.020+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:25:57.792 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:57.307+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:57.792 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:57.307+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:30.464929+0000 front 2024-03-26T05:25:30.464834+0000 (oldest deadline 2024-03-26T05:25:56.364725+0000) 2024-03-26T05:25:58.056 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:57.826+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:25:58.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:57.808+0000 7fd9745c5700 -1 osd.4 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.766158+0000 front 2024-03-26T05:25:21.766124+0000 (oldest deadline 2024-03-26T05:25:49.366344+0000) 2024-03-26T05:25:58.056 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:57 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4[154395]: 2024-03-26T05:25:57.808+0000 7fd9745c5700 -1 osd.4 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.367716+0000 front 2024-03-26T05:25:33.367545+0000 (oldest deadline 2024-03-26T05:25:57.467330+0000) 2024-03-26T05:25:58.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:58 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:58.296+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:58.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:58 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:58.296+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:30.464929+0000 front 2024-03-26T05:25:30.464834+0000 (oldest deadline 2024-03-26T05:25:56.364725+0000) 2024-03-26T05:25:58.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:58 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:58.064+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:58.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:58 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:58.064+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:25:58.543 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:58 smithi132 podman[157479]: 2024-03-26 05:25:58.31072961 +0000 UTC m=+5.620714025 container died c3d49c2f4beb095c3be1ee80514079d073f7a68ae16b60dab4ca2bb9079769a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-26T05:25:58.860 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:58 smithi132 podman[157479]: 2024-03-26 05:25:58.754644635 +0000 UTC m=+6.064629094 container cleanup c3d49c2f4beb095c3be1ee80514079d073f7a68ae16b60dab4ca2bb9079769a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-26T05:25:58.860 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:58 smithi132 bash[157479]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4 2024-03-26T05:25:59.292 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:59 smithi132 podman[157496]: 2024-03-26 05:25:59.073290467 +0000 UTC m=+0.766083574 container remove c3d49c2f4beb095c3be1ee80514079d073f7a68ae16b60dab4ca2bb9079769a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS) 2024-03-26T05:25:59.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:59 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:59.029+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:25:59.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:25:59 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:25:59.029+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:25:59.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:58 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:58.869+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:25:59.293 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:58 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:58.869+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:25:59.569 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:59 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:59.318+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:25:59.569 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:25:59 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:25:59.318+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:30.464929+0000 front 2024-03-26T05:25:30.464834+0000 (oldest deadline 2024-03-26T05:25:56.364725+0000) 2024-03-26T05:25:59.569 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:59 smithi132 podman[157588]: 2024-03-26 05:25:59.290895085 +0000 UTC m=+0.017987278 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:25:59.569 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:59 smithi132 podman[157588]: 2024-03-26 05:25:59.407636717 +0000 UTC m=+0.134728945 container create 7861d0b5596bc20112596858fdde85beb526b6ad54ef3a798ac2da650cd2ef09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:25:59.867 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:59 smithi132 podman[157588]: 2024-03-26 05:25:59.685682954 +0000 UTC m=+0.412775136 container init 7861d0b5596bc20112596858fdde85beb526b6ad54ef3a798ac2da650cd2ef09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T05:25:59.868 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:59 smithi132 podman[157588]: 2024-03-26 05:25:59.688976873 +0000 UTC m=+0.416069059 container start 7861d0b5596bc20112596858fdde85beb526b6ad54ef3a798ac2da650cd2ef09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-26T05:25:59.868 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:25:59 smithi132 podman[157588]: 2024-03-26 05:25:59.769511232 +0000 UTC m=+0.496603432 container attach 7861d0b5596bc20112596858fdde85beb526b6ad54ef3a798ac2da650cd2ef09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-26T05:26:00.280 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:00 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:00.073+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:00.280 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:00 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:00.073+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:00.280 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 26 05:26:00 smithi132 podman[157588]: 2024-03-26 05:26:00.033286186 +0000 UTC m=+0.760378368 container died 7861d0b5596bc20112596858fdde85beb526b6ad54ef3a798ac2da650cd2ef09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T05:26:00.280 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:59 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:59.876+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:00.280 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:25:59 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:25:59.876+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:00.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:00 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:00.289+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:26:00.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:00 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:00.289+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:30.464929+0000 front 2024-03-26T05:25:30.464834+0000 (oldest deadline 2024-03-26T05:25:56.364725+0000) 2024-03-26T05:26:00.692 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T05:26:00.774 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:26:00.775 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T05:26:00.775 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T05:26:00.775 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.5 2024-03-26T05:26:01.042 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:00 smithi132 systemd[1]: Stopping Ceph osd.5 for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:26:01.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:00 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:00.866+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:01.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:00 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:00.866+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:01.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:01 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:01.159+0000 7f63ce218700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:26:01.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:01 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:01.159+0000 7f63ce218700 -1 osd.5 544 *** Got signal Terminated *** 2024-03-26T05:26:01.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:01 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:01.159+0000 7f63ce218700 -1 osd.5 544 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:26:01.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:01 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:01.253+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:26:01.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:01 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:01.253+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:30.464929+0000 front 2024-03-26T05:25:30.464834+0000 (oldest deadline 2024-03-26T05:25:56.364725+0000) 2024-03-26T05:26:01.543 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:01 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:01.117+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:01.543 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:01 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:01.117+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:02.130 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:01 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:01.847+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:02.130 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:01 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:01.847+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:02.541 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:02.140+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:02.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:02.140+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:02.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:02.277+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:26:02.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:02.277+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:30.464929+0000 front 2024-03-26T05:25:30.464834+0000 (oldest deadline 2024-03-26T05:25:56.364725+0000) 2024-03-26T05:26:03.082 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:02.828+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:03.082 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:02 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:02.828+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:03.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:03.091+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:03.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:03.091+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:03.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:03.305+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:26:03.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:03.305+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:30.464929+0000 front 2024-03-26T05:25:30.464834+0000 (oldest deadline 2024-03-26T05:25:56.364725+0000) 2024-03-26T05:26:03.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:03.305+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:37.466083+0000 front 2024-03-26T05:25:37.466140+0000 (oldest deadline 2024-03-26T05:26:02.765948+0000) 2024-03-26T05:26:04.121 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:03.811+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:04.121 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:03 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:03.812+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:04.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:04.130+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:04.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:04.130+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:04.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:04.312+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:26:04.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:04.312+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:30.464929+0000 front 2024-03-26T05:25:30.464834+0000 (oldest deadline 2024-03-26T05:25:56.364725+0000) 2024-03-26T05:26:04.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:04.312+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:37.466083+0000 front 2024-03-26T05:25:37.466140+0000 (oldest deadline 2024-03-26T05:26:02.765948+0000) 2024-03-26T05:26:05.041 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:04.796+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:05.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:04.796+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:05.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:04 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:04.796+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:05.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:05.304+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:21.863044+0000 front 2024-03-26T05:25:21.863071+0000 (oldest deadline 2024-03-26T05:25:46.562899+0000) 2024-03-26T05:26:05.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:05.304+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:30.464929+0000 front 2024-03-26T05:25:30.464834+0000 (oldest deadline 2024-03-26T05:25:56.364725+0000) 2024-03-26T05:26:05.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5[44210]: 2024-03-26T05:26:05.304+0000 7f63ca80c700 -1 osd.5 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:37.466083+0000 front 2024-03-26T05:25:37.466140+0000 (oldest deadline 2024-03-26T05:26:02.765948+0000) 2024-03-26T05:26:05.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:05.165+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:05.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:05.165+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:05.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:05.165+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.904742+0000 front 2024-03-26T05:25:40.904787+0000 (oldest deadline 2024-03-26T05:26:05.004680+0000) 2024-03-26T05:26:06.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:05.797+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:06.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:05.797+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:06.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:05 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:05.797+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:06.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:06 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:06.155+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:06.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:06 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:06.155+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:06.542 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:06 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:06.155+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.904742+0000 front 2024-03-26T05:25:40.904787+0000 (oldest deadline 2024-03-26T05:26:05.004680+0000) 2024-03-26T05:26:07.042 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:06 smithi132 podman[157704]: 2024-03-26 05:26:06.789879854 +0000 UTC m=+5.770999018 container died c9953e21114a903b5d35a14e16333b552d1dda2656770e96c33e067fc7e24d98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-26T05:26:07.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:06 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:06.841+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:07.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:06 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:06.841+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:07.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:06 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:06.841+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:07.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:07 smithi132 podman[157704]: 2024-03-26 05:26:07.139693893 +0000 UTC m=+6.120813035 container cleanup c9953e21114a903b5d35a14e16333b552d1dda2656770e96c33e067fc7e24d98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-26T05:26:07.465 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:07 smithi132 bash[157704]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5 2024-03-26T05:26:07.465 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:07.125+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:07.465 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:07.125+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:07.465 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:07.125+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.904742+0000 front 2024-03-26T05:25:40.904787+0000 (oldest deadline 2024-03-26T05:26:05.004680+0000) 2024-03-26T05:26:07.772 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:07 smithi132 podman[157723]: 2024-03-26 05:26:07.47453739 +0000 UTC m=+0.679464386 container remove c9953e21114a903b5d35a14e16333b552d1dda2656770e96c33e067fc7e24d98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-26T05:26:08.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:07.863+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:08.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:07.863+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:08.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:07 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:07.863+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:08.042 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:07 smithi132 podman[157816]: 2024-03-26 05:26:07.685893532 +0000 UTC m=+0.018169466 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:26:08.042 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:07 smithi132 podman[157816]: 2024-03-26 05:26:07.783815144 +0000 UTC m=+0.116091102 container create 6c9a97fcbcf148e5d4eef173b17589632410df7b1ecbae78476d1ca84554ec2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:26:08.042 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:08 smithi132 podman[157816]: 2024-03-26 05:26:08.011277959 +0000 UTC m=+0.343553875 container init 6c9a97fcbcf148e5d4eef173b17589632410df7b1ecbae78476d1ca84554ec2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-26T05:26:08.042 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:08 smithi132 podman[157816]: 2024-03-26 05:26:08.014439592 +0000 UTC m=+0.346715508 container start 6c9a97fcbcf148e5d4eef173b17589632410df7b1ecbae78476d1ca84554ec2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T05:26:08.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:08 smithi132 podman[157816]: 2024-03-26 05:26:08.094666133 +0000 UTC m=+0.426942059 container attach 6c9a97fcbcf148e5d4eef173b17589632410df7b1ecbae78476d1ca84554ec2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-26T05:26:08.542 INFO:journalctl@ceph.osd.5.smithi132.stdout:Mar 26 05:26:08 smithi132 podman[157816]: 2024-03-26 05:26:08.306238693 +0000 UTC m=+0.638514620 container died 6c9a97fcbcf148e5d4eef173b17589632410df7b1ecbae78476d1ca84554ec2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_CLEAN=True) 2024-03-26T05:26:08.543 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:08 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:08.157+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:08.543 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:08 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:08.157+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:08.543 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:08 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:08.157+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.904742+0000 front 2024-03-26T05:25:40.904787+0000 (oldest deadline 2024-03-26T05:26:05.004680+0000) 2024-03-26T05:26:08.904 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T05:26:08.948 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:26:08.948 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T05:26:08.948 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T05:26:08.948 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.6 2024-03-26T05:26:09.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:08 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:08.889+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:09.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:08 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:08.889+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:09.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:08 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:08.889+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:09.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:09 smithi132 systemd[1]: Stopping Ceph osd.6 for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:26:09.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:09 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:09.133+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:09.293 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:09 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:09.133+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:09.293 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:09 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:09.133+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.904742+0000 front 2024-03-26T05:25:40.904787+0000 (oldest deadline 2024-03-26T05:26:05.004680+0000) 2024-03-26T05:26:09.792 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:09 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:09.335+0000 7f27901e2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:26:09.792 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:09 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:09.335+0000 7f27901e2700 -1 osd.6 544 *** Got signal Terminated *** 2024-03-26T05:26:09.792 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:09 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:09.335+0000 7f27901e2700 -1 osd.6 544 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:26:10.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:10 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:10.106+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:10.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:10 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:10.106+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:10.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:10 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:10.106+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.904742+0000 front 2024-03-26T05:25:40.904787+0000 (oldest deadline 2024-03-26T05:26:05.004680+0000) 2024-03-26T05:26:10.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:09 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:09.886+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:10.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:09 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:09.886+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:10.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:09 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:09.886+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:11.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:10 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:10.931+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:11.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:10 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:10.931+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:11.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:10 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:10.931+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:11.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:10 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:10.931+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-26T05:25:49.025482+0000 front 2024-03-26T05:25:49.025470+0000 (oldest deadline 2024-03-26T05:26:10.725459+0000) 2024-03-26T05:26:11.042 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:10 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:10.931+0000 7fb149da4700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24385.0:1503 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e544) 2024-03-26T05:26:11.336 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:11 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:11.084+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:11.336 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:11 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:11.084+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:11.336 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:11 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:11.084+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.904742+0000 front 2024-03-26T05:25:40.904787+0000 (oldest deadline 2024-03-26T05:26:05.004680+0000) 2024-03-26T05:26:12.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:12.048+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:12.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:12.048+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:12.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:12.048+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.904742+0000 front 2024-03-26T05:25:40.904787+0000 (oldest deadline 2024-03-26T05:26:05.004680+0000) 2024-03-26T05:26:12.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:11 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:11.930+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:12.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:11 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:11.930+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:12.293 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:11 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:11.930+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:12.293 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:11 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:11.930+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-26T05:25:49.025482+0000 front 2024-03-26T05:25:49.025470+0000 (oldest deadline 2024-03-26T05:26:10.725459+0000) 2024-03-26T05:26:12.293 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:11 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:11.930+0000 7fb149da4700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24385.0:1503 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e544) 2024-03-26T05:26:13.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:13.043+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:13.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:13.043+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:13.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:13.043+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.904742+0000 front 2024-03-26T05:25:40.904787+0000 (oldest deadline 2024-03-26T05:26:05.004680+0000) 2024-03-26T05:26:13.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:12.921+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:13.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:12.921+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:13.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:12.921+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:13.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:12.921+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-26T05:25:49.025482+0000 front 2024-03-26T05:25:49.025470+0000 (oldest deadline 2024-03-26T05:26:10.725459+0000) 2024-03-26T05:26:13.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:12 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:12.921+0000 7fb149da4700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24385.0:1503 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e544) 2024-03-26T05:26:14.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:13.995+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:22.802728+0000 front 2024-03-26T05:25:22.802787+0000 (oldest deadline 2024-03-26T05:25:46.902636+0000) 2024-03-26T05:26:14.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:13.995+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:32.803634+0000 front 2024-03-26T05:25:32.803689+0000 (oldest deadline 2024-03-26T05:25:55.103501+0000) 2024-03-26T05:26:14.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6[50060]: 2024-03-26T05:26:13.995+0000 7f27887ce700 -1 osd.6 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.904742+0000 front 2024-03-26T05:25:40.904787+0000 (oldest deadline 2024-03-26T05:26:05.004680+0000) 2024-03-26T05:26:14.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:13.933+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:14.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:13.933+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:14.293 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:13.933+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:14.293 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:13.933+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-26T05:25:49.025482+0000 front 2024-03-26T05:25:49.025470+0000 (oldest deadline 2024-03-26T05:26:10.725459+0000) 2024-03-26T05:26:14.293 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:13 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:13.933+0000 7fb149da4700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24385.0:1503 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e544) 2024-03-26T05:26:15.141 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:14 smithi132 podman[157922]: 2024-03-26 05:26:14.861454723 +0000 UTC m=+5.672379118 container died 1a7cf4ef9e5e0589170be97501148468da42206b41ba0276b5aa1c60289f9c99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:26:15.141 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:14.890+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:15.141 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:14.890+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:15.141 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:14.890+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:15.141 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:14.890+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-26T05:25:49.025482+0000 front 2024-03-26T05:25:49.025470+0000 (oldest deadline 2024-03-26T05:26:10.725459+0000) 2024-03-26T05:26:15.141 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:14 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:14.890+0000 7fb149da4700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24385.0:1503 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e544) 2024-03-26T05:26:15.468 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:15 smithi132 podman[157922]: 2024-03-26 05:26:15.251474743 +0000 UTC m=+6.062399149 container cleanup 1a7cf4ef9e5e0589170be97501148468da42206b41ba0276b5aa1c60289f9c99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T05:26:15.468 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:15 smithi132 bash[157922]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6 2024-03-26T05:26:15.766 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:15 smithi132 podman[157941]: 2024-03-26 05:26:15.477780353 +0000 UTC m=+0.606509762 container remove 1a7cf4ef9e5e0589170be97501148468da42206b41ba0276b5aa1c60289f9c99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T05:26:16.021 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:15 smithi132 podman[158035]: 2024-03-26 05:26:15.6787082 +0000 UTC m=+0.018410297 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:26:16.021 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:15 smithi132 podman[158035]: 2024-03-26 05:26:15.812041619 +0000 UTC m=+0.151743702 container create beb1696737a6d812c342ec09e7bbea0dde386c16fa8f109cfee8edc39d63bdd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:26:16.022 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:15.861+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:16.022 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:15.861+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:16.022 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:15.861+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:16.022 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:15.861+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-26T05:25:49.025482+0000 front 2024-03-26T05:25:49.025470+0000 (oldest deadline 2024-03-26T05:26:10.725459+0000) 2024-03-26T05:26:16.022 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:15 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:15.861+0000 7fb149da4700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24385.0:1503 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e544) 2024-03-26T05:26:16.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:16 smithi132 podman[158035]: 2024-03-26 05:26:16.031437577 +0000 UTC m=+0.371139671 container init beb1696737a6d812c342ec09e7bbea0dde386c16fa8f109cfee8edc39d63bdd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6-deactivate, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:26:16.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:16 smithi132 podman[158035]: 2024-03-26 05:26:16.034549255 +0000 UTC m=+0.374251340 container start beb1696737a6d812c342ec09e7bbea0dde386c16fa8f109cfee8edc39d63bdd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-26T05:26:16.292 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:16 smithi132 podman[158035]: 2024-03-26 05:26:16.123059759 +0000 UTC m=+0.462761859 container attach beb1696737a6d812c342ec09e7bbea0dde386c16fa8f109cfee8edc39d63bdd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-26T05:26:16.600 INFO:journalctl@ceph.osd.6.smithi132.stdout:Mar 26 05:26:16 smithi132 podman[158035]: 2024-03-26 05:26:16.320524113 +0000 UTC m=+0.660226206 container died beb1696737a6d812c342ec09e7bbea0dde386c16fa8f109cfee8edc39d63bdd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T05:26:16.875 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:16.851+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:16.875 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:16.851+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:16.875 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:16.851+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:16.875 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:16.851+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-26T05:25:49.025482+0000 front 2024-03-26T05:25:49.025470+0000 (oldest deadline 2024-03-26T05:26:10.725459+0000) 2024-03-26T05:26:16.875 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:16 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:16.851+0000 7fb149da4700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24385.0:1503 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e544) 2024-03-26T05:26:16.898 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T05:26:16.941 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:26:16.941 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T05:26:16.941 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T05:26:16.941 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.7 2024-03-26T05:26:17.287 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:17 smithi132 systemd[1]: Stopping Ceph osd.7 for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:26:17.542 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:17 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:17.295+0000 7fb1517b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:26:17.542 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:17 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:17.295+0000 7fb1517b8700 -1 osd.7 544 *** Got signal Terminated *** 2024-03-26T05:26:17.542 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:17 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:17.295+0000 7fb1517b8700 -1 osd.7 544 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:26:18.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:17 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:17.842+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:18.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:17 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:17.842+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:18.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:17 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:17.842+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:18.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:17 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:17.842+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-26T05:25:49.025482+0000 front 2024-03-26T05:25:49.025470+0000 (oldest deadline 2024-03-26T05:26:10.725459+0000) 2024-03-26T05:26:18.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:17 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:17.842+0000 7fb149da4700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24385.0:1503 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e544) 2024-03-26T05:26:19.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:18 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:18.859+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:19.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:18 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:18.859+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:19.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:18 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:18.859+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:19.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:18 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:18.859+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-26T05:25:49.025482+0000 front 2024-03-26T05:25:49.025470+0000 (oldest deadline 2024-03-26T05:26:10.725459+0000) 2024-03-26T05:26:19.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:18 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:18.859+0000 7fb149da4700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24385.0:1503 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e544) 2024-03-26T05:26:20.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:19 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:19.872+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:20.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:19 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:19.872+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:20.293 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:19 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:19.872+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:20.293 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:19 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:19.872+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-26T05:25:49.025482+0000 front 2024-03-26T05:25:49.025470+0000 (oldest deadline 2024-03-26T05:26:10.725459+0000) 2024-03-26T05:26:20.293 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:19 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:19.872+0000 7fb149da4700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24385.0:1503 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e544) 2024-03-26T05:26:21.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:20 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:20.880+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:21.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:20 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:20.880+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:21.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:20 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:20.880+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:21.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:20 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:20.880+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-26T05:25:49.025482+0000 front 2024-03-26T05:25:49.025470+0000 (oldest deadline 2024-03-26T05:26:10.725459+0000) 2024-03-26T05:26:21.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:20 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:20.880+0000 7fb149da4700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24385.0:1503 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e544) 2024-03-26T05:26:22.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:21 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:21.904+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-26T05:25:24.522321+0000 front 2024-03-26T05:25:24.522396+0000 (oldest deadline 2024-03-26T05:25:49.822135+0000) 2024-03-26T05:26:22.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:21 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:21.904+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-26T05:25:33.323324+0000 front 2024-03-26T05:25:33.323219+0000 (oldest deadline 2024-03-26T05:25:58.623140+0000) 2024-03-26T05:26:22.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:21 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:21.904+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-26T05:25:40.924246+0000 front 2024-03-26T05:25:40.924140+0000 (oldest deadline 2024-03-26T05:26:03.824169+0000) 2024-03-26T05:26:22.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:21 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:21.904+0000 7fb149da4700 -1 osd.7 544 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-26T05:25:49.025482+0000 front 2024-03-26T05:25:49.025470+0000 (oldest deadline 2024-03-26T05:26:10.725459+0000) 2024-03-26T05:26:22.292 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:21 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7[123111]: 2024-03-26T05:26:21.904+0000 7fb149da4700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24385.0:1503 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e544) 2024-03-26T05:26:23.153 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:22 smithi132 podman[158140]: 2024-03-26 05:26:22.89135846 +0000 UTC m=+5.708793774 container died ae9ffa1b045527d180f2c6dd5ce3b13115027dad875931631dae6ad18f4ea3a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:26:23.542 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:23 smithi132 podman[158140]: 2024-03-26 05:26:23.262695914 +0000 UTC m=+6.080131223 container cleanup ae9ffa1b045527d180f2c6dd5ce3b13115027dad875931631dae6ad18f4ea3a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T05:26:23.542 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:23 smithi132 bash[158140]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7 2024-03-26T05:26:23.861 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:23 smithi132 podman[158170]: 2024-03-26 05:26:23.555621508 +0000 UTC m=+0.660327821 container remove ae9ffa1b045527d180f2c6dd5ce3b13115027dad875931631dae6ad18f4ea3a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-26T05:26:24.149 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:23 smithi132 podman[158259]: 2024-03-26 05:26:23.774439517 +0000 UTC m=+0.017923220 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:26:24.149 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:23 smithi132 podman[158259]: 2024-03-26 05:26:23.889717019 +0000 UTC m=+0.133200697 container create 913cf61a160b9c06fb31e15f12ff725e7df4d7fcb7d070502338ed0b11155817 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-26T05:26:24.149 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:24 smithi132 podman[158259]: 2024-03-26 05:26:24.092066019 +0000 UTC m=+0.335549701 container init 913cf61a160b9c06fb31e15f12ff725e7df4d7fcb7d070502338ed0b11155817 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:26:24.149 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:24 smithi132 podman[158259]: 2024-03-26 05:26:24.095128465 +0000 UTC m=+0.338612146 container start 913cf61a160b9c06fb31e15f12ff725e7df4d7fcb7d070502338ed0b11155817 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:26:24.149 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:24 smithi132 podman[158259]: 2024-03-26 05:26:24.15859986 +0000 UTC m=+0.402083545 container attach 913cf61a160b9c06fb31e15f12ff725e7df4d7fcb7d070502338ed0b11155817 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS) 2024-03-26T05:26:24.542 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:24 smithi132 podman[158259]: 2024-03-26 05:26:24.383469567 +0000 UTC m=+0.626953258 container died 913cf61a160b9c06fb31e15f12ff725e7df4d7fcb7d070502338ed0b11155817 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T05:26:24.892 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T05:26:24.919 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:24 smithi132 podman[158284]: 2024-03-26 05:26:24.887060604 +0000 UTC m=+0.495236989 container remove 913cf61a160b9c06fb31e15f12ff725e7df4d7fcb7d070502338ed0b11155817 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T05:26:24.919 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:24 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-26T05:26:24.920 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:24 smithi132 systemd[1]: Stopped Ceph osd.7 for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:26:24.920 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 26 05:26:24 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@osd.7.service: Consumed 7min 48.044s CPU time. 2024-03-26T05:26:24.936 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:26:24.936 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T05:26:24.936 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T05:26:24.936 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T05:26:25.366 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:26:24 smithi080 systemd[1]: Stopping Ceph rgw.foo.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:26:25.366 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:26:25 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-rgw-foo-a[67232]: 2024-03-26T05:26:25.217+0000 7f3d9aeae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:26:25.366 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:26:25 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-rgw-foo-a[67232]: 2024-03-26T05:26:25.217+0000 7f3da972bac0 -1 shutting down 2024-03-26T05:26:27.542 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:27.177Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-26T05:26:27.542 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:27.179Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-26T05:26:27.542 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:27.182Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-26T05:26:27.542 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:27 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:27.182Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-26T05:26:35.492 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:26:35 smithi080 bash[223606]: time="2024-03-26T05:26:35Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T05:26:35.492 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:26:35 smithi080 podman[223606]: 2024-03-26 05:26:35.237268054 +0000 UTC m=+10.100566591 container died e30d26b8491614153ed9bc8a081f5bf47eb914d62ed44f73f16469adbda3f81d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5) 2024-03-26T05:26:35.866 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:26:35 smithi080 podman[223606]: 2024-03-26 05:26:35.666039191 +0000 UTC m=+10.529337770 container cleanup e30d26b8491614153ed9bc8a081f5bf47eb914d62ed44f73f16469adbda3f81d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-rgw-foo-a, ceph=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:26:35.866 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:26:35 smithi080 bash[223606]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T05:26:36.120 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T05:26:36.144 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:26:35 smithi080 podman[223621]: 2024-03-26 05:26:35.883891974 +0000 UTC m=+0.634941626 container remove e30d26b8491614153ed9bc8a081f5bf47eb914d62ed44f73f16469adbda3f81d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-26T05:26:36.144 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:26:35 smithi080 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T05:26:36.144 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:26:36 smithi080 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-26T05:26:36.144 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:26:36 smithi080 systemd[1]: Stopped Ceph rgw.foo.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:26:36.145 INFO:journalctl@ceph.rgw.foo.a.smithi080.stdout:Mar 26 05:26:36 smithi080 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@rgw.foo.a.service: Consumed 3.582s CPU time. 2024-03-26T05:26:36.156 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:26:36.156 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T05:26:36.156 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T05:26:36.156 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T05:26:36.479 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 systemd[1]: Stopping Ceph prometheus.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:26:36.771 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:36.489Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T05:26:36.771 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:36.489Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T05:26:36.771 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:36.489Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T05:26:36.771 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:36.489Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T05:26:36.771 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:36.489Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T05:26:36.771 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:36.489Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T05:26:36.771 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:36.489Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T05:26:36.771 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:36.489Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T05:26:36.771 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:36.498Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T05:26:36.771 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:36.498Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T05:26:36.772 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:36.498Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T05:26:36.772 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a[69564]: ts=2024-03-26T05:26:36.498Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T05:26:36.772 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 podman[158364]: 2024-03-26 05:26:36.509700277 +0000 UTC m=+0.127682860 container died bcfb97148560743163156adcfac48e9ae3c4df8a3db0cfd86d2acd8cef5109c0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:26:37.042 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 podman[158364]: 2024-03-26 05:26:36.880313184 +0000 UTC m=+0.498295766 container cleanup bcfb97148560743163156adcfac48e9ae3c4df8a3db0cfd86d2acd8cef5109c0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:26:37.042 INFO:journalctl@ceph.prometheus.a.smithi132.stdout:Mar 26 05:26:36 smithi132 bash[158364]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T05:26:37.251 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T05:26:37.288 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:26:37.288 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T05:26:37.288 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T05:26:37.476 INFO:teuthology.orchestra.run.smithi080.stdout:Deleting cluster with fsid: dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T05:26:40.044 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:26:39 smithi080 systemd[1]: Stopping Ceph alertmanager.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:26:40.361 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:26:40 smithi080 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a[85141]: ts=2024-03-26T05:26:40.054Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T05:26:40.361 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:26:40 smithi080 podman[224101]: 2024-03-26 05:26:40.095143981 +0000 UTC m=+0.127143491 container died ee2c30ab9fb52eba29be1c968fe603e27bf98a5ef95e41d82323317d48fd263a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:26:40.616 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:26:40 smithi080 podman[224101]: 2024-03-26 05:26:40.538179564 +0000 UTC m=+0.570179128 container cleanup ee2c30ab9fb52eba29be1c968fe603e27bf98a5ef95e41d82323317d48fd263a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:26:40.616 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:26:40 smithi080 bash[224101]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T05:26:41.116 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:26:40 smithi080 podman[224113]: 2024-03-26 05:26:40.805935191 +0000 UTC m=+0.711130166 container remove ee2c30ab9fb52eba29be1c968fe603e27bf98a5ef95e41d82323317d48fd263a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:26:41.116 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:26:40 smithi080 podman[224113]: 2024-03-26 05:26:40.990518301 +0000 UTC m=+0.895713276 volume remove c0da353ee31bccddb1cefeadd5794e84b349c21397cc765455fb48b37749935c 2024-03-26T05:26:41.616 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:26:41 smithi080 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T05:26:41.616 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:26:41 smithi080 systemd[1]: Stopped Ceph alertmanager.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:26:41.616 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 26 05:26:41 smithi080 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 1.701s CPU time. 2024-03-26T05:26:41.616 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:26:41 smithi080 systemd[1]: Stopping Ceph node-exporter.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:26:41.925 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:26:41 smithi080 podman[224227]: 2024-03-26 05:26:41.670359953 +0000 UTC m=+0.107720039 container died cb2f32aef494cc3676fbb30e776b930dc07143735b1f50c585ca2391196b1590 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:26:42.300 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:26:42 smithi080 podman[224227]: 2024-03-26 05:26:42.067888942 +0000 UTC m=+0.505249082 container cleanup cb2f32aef494cc3676fbb30e776b930dc07143735b1f50c585ca2391196b1590 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:26:42.301 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:26:42 smithi080 bash[224227]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T05:26:42.567 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:26:42 smithi080 podman[224241]: 2024-03-26 05:26:42.310541334 +0000 UTC m=+0.644653613 container remove cb2f32aef494cc3676fbb30e776b930dc07143735b1f50c585ca2391196b1590 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:26:42.567 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:26:42 smithi080 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T05:26:42.567 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:26:42 smithi080 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T05:26:42.567 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:26:42 smithi080 systemd[1]: Stopped Ceph node-exporter.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:26:42.568 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 26 05:26:42 smithi080 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 4.791s CPU time. 2024-03-26T05:26:43.087 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T05:26:43.279 INFO:teuthology.orchestra.run.smithi132.stdout:Deleting cluster with fsid: dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T05:26:45.450 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:26:45 smithi132 systemd[1]: Stopping Ceph iscsi.iscsi.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:26:45.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:26:45 smithi132 ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a[60340]: debug Shutdown received 2024-03-26T05:26:55.703 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:26:55 smithi132 bash[158777]: time="2024-03-26T05:26:55Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-26T05:26:55.704 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:26:55 smithi132 podman[158777]: 2024-03-26 05:26:55.474757263 +0000 UTC m=+10.098437177 container died 179562eae4a9422a12d89b5428ebc402ca399b8d757d1980e24a8a522caf98d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T05:26:56.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:26:55 smithi132 podman[158777]: 2024-03-26 05:26:55.79120522 +0000 UTC m=+10.414885133 container cleanup 179562eae4a9422a12d89b5428ebc402ca399b8d757d1980e24a8a522caf98d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-26T05:26:56.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:26:55 smithi132 bash[158777]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a 2024-03-26T05:26:56.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:26:56 smithi132 podman[158790]: 2024-03-26 05:26:56.033993215 +0000 UTC m=+0.554740460 container remove 179562eae4a9422a12d89b5428ebc402ca399b8d757d1980e24a8a522caf98d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-iscsi-iscsi-a, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-26T05:26:56.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:26:56 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T05:26:56.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:26:56 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-26T05:26:56.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:26:56 smithi132 systemd[1]: Stopped Ceph iscsi.iscsi.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:26:56.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi132.stdout:Mar 26 05:26:56 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Consumed 3.503s CPU time. 2024-03-26T05:26:57.968 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:26:57 smithi132 systemd[1]: Stopping Ceph grafana.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:26:58.249 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:26:57 smithi132 podman[159064]: 2024-03-26 05:26:57.97680297 +0000 UTC m=+0.160007830 container died 3b5402cd4a1ba9f216040ca62203b2bd505af7e4e1937624e7411c2a338aeebe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, architecture=x86_64, version=8.5, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, vcs-type=git) 2024-03-26T05:26:58.516 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:26:58 smithi132 podman[159064]: 2024-03-26 05:26:58.333043212 +0000 UTC m=+0.516248072 container cleanup 3b5402cd4a1ba9f216040ca62203b2bd505af7e4e1937624e7411c2a338aeebe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, description=Ceph Grafana Container, com.redhat.component=ubi8-container, version=8.5, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., distribution-scope=public) 2024-03-26T05:26:58.517 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:26:58 smithi132 bash[159064]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a 2024-03-26T05:26:58.775 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:26:58 smithi132 podman[159080]: 2024-03-26 05:26:58.634434402 +0000 UTC m=+0.660315816 container remove 3b5402cd4a1ba9f216040ca62203b2bd505af7e4e1937624e7411c2a338aeebe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-grafana-a, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, release=239.1651231664, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, maintainer=Paul Cuzner , version=8.5, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, distribution-scope=public, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-26T05:26:58.775 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:26:58 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T05:26:58.775 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:26:58 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T05:26:58.775 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:26:58 smithi132 systemd[1]: Stopped Ceph grafana.a for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:26:58.776 INFO:journalctl@ceph.grafana.a.smithi132.stdout:Mar 26 05:26:58 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@grafana.a.service: Consumed 13.171s CPU time. 2024-03-26T05:26:59.042 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:26:58 smithi132 systemd[1]: Stopping Ceph node-exporter.b for dd09e9cc-eb2c-11ee-95d0-87774f69a715... 2024-03-26T05:26:59.527 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:26:59 smithi132 podman[159190]: 2024-03-26 05:26:59.214512437 +0000 UTC m=+0.104787763 container died f91e694fff90be92cce0c8b796b51c27e84372e1b80ef2a16c70fb8a5dff3326 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:26:59.527 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:26:59 smithi132 podman[159190]: 2024-03-26 05:26:59.537405402 +0000 UTC m=+0.427680733 container cleanup f91e694fff90be92cce0c8b796b51c27e84372e1b80ef2a16c70fb8a5dff3326 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:26:59.791 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:26:59 smithi132 bash[159190]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T05:26:59.792 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:26:59 smithi132 podman[159204]: 2024-03-26 05:26:59.721786556 +0000 UTC m=+0.507903509 container remove f91e694fff90be92cce0c8b796b51c27e84372e1b80ef2a16c70fb8a5dff3326 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:26:59.792 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:26:59 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T05:27:00.292 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:26:59 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T05:27:00.292 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:26:59 smithi132 systemd[1]: Stopped Ceph node-exporter.b for dd09e9cc-eb2c-11ee-95d0-87774f69a715. 2024-03-26T05:27:00.292 INFO:journalctl@ceph.node-exporter.b.smithi132.stdout:Mar 26 05:26:59 smithi132 systemd[1]: ceph-dd09e9cc-eb2c-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 4.308s CPU time. 2024-03-26T05:27:00.601 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:27:00.636 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:27:00.666 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T05:27:00.668 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053/remote/smithi080/crash 2024-03-26T05:27:00.669 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/crash -- . 2024-03-26T05:27:00.725 INFO:teuthology.orchestra.run.smithi080.stderr:tar: /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T05:27:00.725 INFO:teuthology.orchestra.run.smithi080.stderr:tar: Error is not recoverable: exiting now 2024-03-26T05:27:00.728 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053/remote/smithi132/crash 2024-03-26T05:27:00.729 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/crash -- . 2024-03-26T05:27:00.769 INFO:teuthology.orchestra.run.smithi132.stderr:tar: /var/lib/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T05:27:00.769 INFO:teuthology.orchestra.run.smithi132.stderr:tar: Error is not recoverable: exiting now 2024-03-26T05:27:00.770 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T05:27:00.770 DEBUG:teuthology.orchestra.run.smithi080:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T05:27:00.868 INFO:teuthology.orchestra.run.smithi080.stdout:2024-03-26T05:10:00.000257+0000 mon.a (mon.0) 1299 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-26T05:27:00.868 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T05:27:00.868 DEBUG:teuthology.orchestra.run.smithi080:> sudo egrep '\[SEC\]' /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T05:27:00.898 DEBUG:teuthology.orchestra.run.smithi080:> sudo egrep '\[ERR\]' /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T05:27:00.969 DEBUG:teuthology.orchestra.run.smithi080:> sudo egrep '\[WRN\]' /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T05:27:01.038 INFO:teuthology.orchestra.run.smithi080.stdout:2024-03-26T05:10:00.000257+0000 mon.a (mon.0) 1299 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-26T05:27:01.038 INFO:tasks.cephadm:Compressing logs... 2024-03-26T05:27:01.038 DEBUG:teuthology.orchestra.run.smithi080:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T05:27:01.081 DEBUG:teuthology.orchestra.run.smithi132:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T05:27:01.134 INFO:teuthology.orchestra.run.smithi080.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T05:27:01.134 INFO:teuthology.orchestra.run.smithi080.stderr:: No such file or directory 2024-03-26T05:27:01.138 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T05:27:01.141 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T05:27:01.142 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T05:27:01.142 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T05:27:01.142 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T05:27:01.142 INFO:teuthology.orchestra.run.smithi080.stderr: 57.7% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T05:27:01.142 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T05:27:01.142 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T05:27:01.142 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T05:27:01.142 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.log 2024-03-26T05:27:01.142 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T05:27:01.142 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T05:27:01.142 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T05:27:01.152 INFO:teuthology.orchestra.run.smithi132.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T05:27:01.152 INFO:teuthology.orchestra.run.smithi132.stderr:: No such file or directory 2024-03-26T05:27:01.152 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T05:27:01.152 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T05:27:01.152 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T05:27:01.153 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mon.c.log: 91.8% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T05:27:01.154 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T05:27:01.154 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T05:27:01.154 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T05:27:01.154 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/tcmu-runner.log 2024-03-26T05:27:01.156 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T05:27:01.156 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.log 2024-03-26T05:27:01.156 INFO:teuthology.orchestra.run.smithi132.stderr: 67.6% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/tcmu-runner.log.gz 2024-03-26T05:27:01.156 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T05:27:01.156 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T05:27:01.156 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.log: /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/objectstore_tool.2.log 2024-03-26T05:27:01.162 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mon.b.log: /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/objectstore_tool.2.log: 92.4% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T05:27:01.256 INFO:teuthology.orchestra.run.smithi132.stderr: 91.1% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T05:27:01.296 INFO:teuthology.orchestra.run.smithi132.stderr: 90.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T05:27:01.296 INFO:teuthology.orchestra.run.smithi132.stderr: 81.5% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T05:27:01.297 INFO:teuthology.orchestra.run.smithi080.stderr: 91.2% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T05:27:01.304 INFO:teuthology.orchestra.run.smithi132.stderr: 90.0% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T05:27:01.306 INFO:teuthology.orchestra.run.smithi132.stderr: 85.4% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T05:27:01.446 INFO:teuthology.orchestra.run.smithi132.stderr: 94.5% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/objectstore_tool.2.log.gz 2024-03-26T05:27:01.503 INFO:teuthology.orchestra.run.smithi080.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T05:27:01.652 INFO:teuthology.orchestra.run.smithi080.stderr: 89.0% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T05:27:02.113 INFO:teuthology.orchestra.run.smithi080.stderr: 93.9% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T05:27:03.093 INFO:teuthology.orchestra.run.smithi132.stderr: 91.0% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T05:27:04.541 INFO:teuthology.orchestra.run.smithi080.stderr: 89.6% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T05:27:06.044 INFO:teuthology.orchestra.run.smithi080.stderr: 91.1% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T05:27:34.170 INFO:teuthology.orchestra.run.smithi080.stderr: 90.4% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T05:29:07.839 INFO:teuthology.orchestra.run.smithi132.stderr: 93.2% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T05:31:17.163 INFO:teuthology.orchestra.run.smithi132.stderr: 93.5% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T05:31:33.928 INFO:teuthology.orchestra.run.smithi132.stderr: 93.8% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T05:31:54.239 INFO:teuthology.orchestra.run.smithi132.stderr: 93.7% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T05:31:54.241 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-26T05:31:54.241 INFO:teuthology.orchestra.run.smithi132.stderr:real 4m53.145s 2024-03-26T05:31:54.241 INFO:teuthology.orchestra.run.smithi132.stderr:user 4m16.193s 2024-03-26T05:31:54.241 INFO:teuthology.orchestra.run.smithi132.stderr:sys 0m11.071s 2024-03-26T05:39:36.840 INFO:teuthology.orchestra.run.smithi080.stderr: 93.3% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T05:41:50.263 INFO:teuthology.orchestra.run.smithi080.stderr: 93.2% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T05:41:59.985 INFO:teuthology.orchestra.run.smithi080.stderr: 92.9% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T05:42:28.060 INFO:teuthology.orchestra.run.smithi080.stderr: 92.8% -- replaced with /var/log/ceph/dd09e9cc-eb2c-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T05:42:28.063 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-26T05:42:28.063 INFO:teuthology.orchestra.run.smithi080.stderr:real 15m26.969s 2024-03-26T05:42:28.063 INFO:teuthology.orchestra.run.smithi080.stderr:user 6m55.657s 2024-03-26T05:42:28.063 INFO:teuthology.orchestra.run.smithi080.stderr:sys 0m19.703s 2024-03-26T05:42:28.063 INFO:tasks.cephadm:Archiving logs... 2024-03-26T05:42:28.064 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053/remote/smithi080/log 2024-03-26T05:42:28.065 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T05:43:31.729 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053/remote/smithi132/log 2024-03-26T05:43:31.730 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T05:44:09.784 INFO:tasks.cephadm:Removing cluster... 2024-03-26T05:44:09.785 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 --force 2024-03-26T05:44:10.039 INFO:teuthology.orchestra.run.smithi080.stdout:Deleting cluster with fsid: dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T05:44:10.827 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dd09e9cc-eb2c-11ee-95d0-87774f69a715 --force 2024-03-26T05:44:11.048 INFO:teuthology.orchestra.run.smithi132.stdout:Deleting cluster with fsid: dd09e9cc-eb2c-11ee-95d0-87774f69a715 2024-03-26T05:44:11.617 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T05:44:11.617 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T05:44:11.648 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T05:44:11.668 INFO:tasks.cephadm:Teardown complete 2024-03-26T05:44:11.668 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T05:44:11.678 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T05:44:11.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T05:44:11.691 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T05:44:11.794 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T05:44:11.803 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T05:44:11.803 DEBUG:teuthology.orchestra.run.smithi080:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T05:44:11.806 DEBUG:teuthology.orchestra.run.smithi132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T05:44:11.818 INFO:teuthology.orchestra.run.smithi080.stderr:bash: line 1: ntpq: command not found 2024-03-26T05:44:11.819 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-03-26T05:44:11.839 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T05:44:11.839 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-03-26T05:44:11.839 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 +944us[ +955us] +/- 74ms 2024-03-26T05:44:11.839 INFO:teuthology.orchestra.run.smithi132.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -714us[ -704us] +/- 58ms 2024-03-26T05:44:11.839 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +26us[ +37us] +/- 60ms 2024-03-26T05:44:11.840 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T05:44:11.843 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T05:44:11.843 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2024-03-26T05:44:11.843 INFO:teuthology.orchestra.run.smithi080.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +935us[ +934us] +/- 74ms 2024-03-26T05:44:11.843 INFO:teuthology.orchestra.run.smithi080.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -726us[ -727us] +/- 58ms 2024-03-26T05:44:11.843 INFO:teuthology.orchestra.run.smithi080.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +23us[ +22us] +/- 60ms 2024-03-26T05:44:11.844 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T05:44:11.844 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T05:44:11.853 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T05:44:11.853 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T05:44:11.866 DEBUG:teuthology.orchestra.run.smithi080:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T05:44:11.984 DEBUG:teuthology.orchestra.run.smithi132:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T05:44:12.046 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T05:44:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:44:12.075 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T05:44:12.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:44:12.103 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T05:44:12.113 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T05:44:12.123 INFO:teuthology.task.internal:Duration was 3410.556827 seconds 2024-03-26T05:44:12.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T05:44:12.132 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T05:44:12.132 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T05:44:12.135 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T05:44:12.204 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T05:44:12.238 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T05:44:12.445 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T05:44:12.445 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi080.front.sepia.ceph.com 2024-03-26T05:44:12.445 DEBUG:teuthology.orchestra.run.smithi080:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T05:44:12.465 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com 2024-03-26T05:44:12.465 DEBUG:teuthology.orchestra.run.smithi132:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T05:44:12.493 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T05:44:12.493 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T05:44:12.508 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T05:44:12.561 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T05:44:12.561 DEBUG:teuthology.orchestra.run.smithi080:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T05:44:12.578 DEBUG:teuthology.orchestra.run.smithi132:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T05:44:14.217 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T05:44:14.227 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T05:44:14.227 DEBUG:teuthology.orchestra.run.smithi080:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T05:44:14.262 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T05:44:14.315 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T05:44:14.324 DEBUG:teuthology.orchestra.run.smithi080:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T05:44:14.327 DEBUG:teuthology.orchestra.run.smithi132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T05:44:14.358 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = core 2024-03-26T05:44:14.360 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = core 2024-03-26T05:44:14.403 DEBUG:teuthology.orchestra.run.smithi080:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T05:44:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:44:14.439 DEBUG:teuthology.orchestra.run.smithi132:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T05:44:14.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:44:14.458 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T05:44:14.469 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T05:44:14.470 DEBUG:teuthology.misc:Transferring archived files from smithi080:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053/remote/smithi080 2024-03-26T05:44:14.470 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T05:44:14.545 DEBUG:teuthology.misc:Transferring archived files from smithi132:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053/remote/smithi132 2024-03-26T05:44:14.545 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T05:44:14.586 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T05:44:14.587 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T05:44:14.589 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T05:44:14.636 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T05:44:14.648 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T05:44:14.648 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T05:44:14.657 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T05:44:14.657 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T05:44:14.659 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T05:44:14.671 INFO:teuthology.orchestra.run.smithi080.stdout: 265955 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 05:44 /home/ubuntu/cephtest 2024-03-26T05:44:14.691 INFO:teuthology.orchestra.run.smithi132.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 05:44 /home/ubuntu/cephtest 2024-03-26T05:44:14.692 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T05:44:14.703 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T05:44:14.722 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T05:44:14.748 DEBUG:teuthology.nuke:shortname: smithi080 2024-03-26T05:44:14.748 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T05:44:14.770 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:38:21.003983', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJgeH1Q56uVFk/VLjC3/lV252QgRNpcwV2BCZGbDiqXsPobNpYIrINifR5iz5gSVNoaxtwdTgRDg7KoNE3clG2o='} 2024-03-26T05:44:14.772 DEBUG:teuthology.nuke:shortname: smithi132 2024-03-26T05:44:14.773 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T05:44:14.796 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622053', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:38:21.002591', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCbBxpL2wQKtIofp6kPNucKbb+MVQDHk2s0F3nlHmaXReniGPSLyYC5gVqXo8+TVbSHRsXz8FQoddUqGvmYwT9Y='} 2024-03-26T05:44:14.828 INFO:teuthology.orchestra.console.smithi080:Power off 2024-03-26T05:44:14.829 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T05:44:14.853 DEBUG:teuthology.orchestra.console.smithi080:power off output: Chassis Power Control: Down/Off 2024-03-26T05:44:14.854 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:44:14.866 INFO:teuthology.orchestra.console.smithi132:Power off 2024-03-26T05:44:14.867 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T05:44:14.878 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is on 2024-03-26T05:44:14.887 DEBUG:teuthology.orchestra.console.smithi132:power off output: Chassis Power Control: Down/Off 2024-03-26T05:44:14.887 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:44:14.921 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-03-26T05:44:18.879 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:44:18.923 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:44:18.990 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is on 2024-03-26T05:44:19.032 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-03-26T05:44:22.995 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:44:23.033 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:44:23.107 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is on 2024-03-26T05:44:23.143 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-03-26T05:44:27.110 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:44:27.143 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:44:27.221 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is off 2024-03-26T05:44:27.253 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off 2024-03-26T05:44:27.323 INFO:teuthology.orchestra.console.smithi080:Power off completed 2024-03-26T05:44:27.354 INFO:teuthology.orchestra.console.smithi132:Power off completed 2024-03-26T05:44:27.459 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} duration: 3410.556827068329 failure_reason: '"2024-03-26T05:10:00.000257+0000 mon.a (mon.0) 1299 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-26T05:44:27.459 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T05:44:27.552 INFO:teuthology.run:FAIL